Technology

A token is basically just a word. Know how your phone’s auto suggest tries to anticipate the words you want to use as you type? In this case, your phone is using an extremely small token amount (typically only the previous two or three words you have typed) to try and predict your next word, which would also be a token. Your phone only uses a few tokens at a time, because as token count rises, processing requirements also quickly balloon.

And AI chat is basically the same concept, but with a massively inflated token limit. Instead of looking at your previous two or three words, it looks at entire conversations. And it also uses tokens to generate responses, the same way your phone is using one token at a time to predict your next word.

So when you pay for tokens, you’re essentially paying for a word count. As you continue a conversation, the token requirement for each subsequent request will increase, because it is attempting to look at the entire context of the conversation you have had.

Models have built-in token limits, to put a cap on how much memory is required to run the model. As conversations stretch on and you reach the model’s token limits, it will begin losing context for things that happened earlier. It will try to summarize earlier parts of the conversation to shorten them but keep relevant pieces in memory, or it will just outright drop old parts of the conversation and “forget” that context, the same way my phone has already forgotten the start of this sentence.

It’s a little more complicated that “each word is a token”, because the chatbot will combine your prompts with its own internal systems. Especially as conversations stretch on, and it begins to summarize old parts to keep them in memory. But that’s the most straightforward way to explain it.