Understand tokens and context windows in Claude Code

When you send a message to Claude Code, it breaks your sentences and words down into tokens, chunking out the text into a format the LLM will understand. Sometimes it's a word, sometimes it's just a letter.

LLMs do this to make processing and responding to queries more efficient. This is how it can store and process millions or billions of pieces of text, make sense of it, and make connections between different characters and words.

At its core, LLMs are just prediction machines, and breaking words down into tokens is what makes it all work. Without this breakdown, LLM processing would b...