Understand tokens and context windows in Claude Code

Link copied

Attach video time

When you send a message to Claude Code, it breaks your sentences and words down into tokens, chunking out the text into a format the LLM will understand. Sometimes it's a word, sometimes it's just a letter.

LLMs do this to make processing and responding to queries more efficient. This is how it can store and process millions or billions of pieces of text, make sense of it, and make connections between different characters and words.

Tokens

At its core, LLMs are just prediction machines, and breaking words down into tokens is what makes it all work. Without this breakdown, LLM processing would be significantly slowed down and not as intelligent, making them far less practical.

OpenAI has a pretty cool token visualizer that you can use to see how text gets broken into tokens.

Example:

We're no strangers to love
You know the rules and so do I
A full commitment's what I'm thinkin' of
You wouldn't get this from any other guy

I just wanna tell you how I'm feeling
Gotta make you understand

Never gonna give you up
Never gonna let you down
Never gonna run around and desert you
Never gonna make you cry
Never gonna say goodbye
Never gonna tell a lie and hurt you

Context windows

Every LLM has a context window, and Claude Code is no exception. The context window is the maximum number of tokens an AI can process at once, and includes both input and output tokens combined.

Context limits used to be very small, but they have grown dramatically over the years. The models available within Claude Code come with their own limits. Opus, Sonnet and Fable each have a 1 million token window, and Haiku has a 200k token window. Expect these to continue rising in the future though.

Needle in haystack problem

But the problem with large context windows though is that the larger it is, the harder it is for the LLM to process all of the data in the context window, leading to a "needle in the haystack" problem. That said, models have even gotten better with this problem, especially when using more the more intelligent models such as Opus and Fable. Users can now continue their conversations, and even if they get past the limits, conversations can auto-compact the context window, keep the chat going, and still be aware of everything that is going on.

But still, if you are either on a lesser-intelligent model or see the responses start to drifting, it may be a good idea to clear the conversation with /clear to reset the context window. This is the equivalent of waking up after a good night's sleep and with a reset mind, so you can start the day with a fresh head. LLMs work very similarly.