Every large language model, from GPT to Claude to Gemini, reduces to one loop: look at what came before, predict what comes next. That is it. No reasoning engine, no knowledge database, no goals. A very large function that says "given this sequence, the next piece is probably this."
That simplicity is not a shortcoming. It is the trick. This guide walks you through the mechanics so you can reason clearly about what these systems do, where they fail, and why.
Do LLMs read words or tokens?
The model does not read words. It reads tokens. A tokenizer splits your input into chunks that the model has learned to work with. In practice:
- "prediction" might be one token.
- "tokenization" might be two: "token" + "ization".
console.logmight be three: "console", ".", "log".- A space is often merged into the following word.
The exact split depends on the tokenizer vocabulary the model was trained with. The key point: the model never sees letters or words directly. It sees integer IDs. "Hello" might be token 9906.
Tokens matter for you because they determine cost (you pay per token), context window usage, and sometimes surprising behaviour with unusual spellings or rare languages.Common gotchaDo not ask an LLM to count letters, reverse a string, or judge spelling character by character.It sees tokens, not individual letters, so "how many r's in strawberry" trips it up. Hand that work to plain code instead.
How do token IDs become vectors?
An integer ID is not meaningful on its own. The model converts each token ID into a embedding vector. Tokens that appear in similar contexts end up near each other in that space. "cat" and "kitten" land close together. "cat" and "SQL" do not. For a slower walkthrough of how meaning becomes numbers, see embeddings explained for beginners.
The model layers on positional information too, so "the dog chased the cat" and "the cat chased the dog" produce different representations even though they share the same tokens.
You do not need to think about the math. The important mental model: by the time the model processes your input, every token has become a rich numerical representation that encodes something about what that token means and where it sits in the sentence.
How does an LLM generate text?
Once the input is represented as a sequence of vectors, the model runs it through many transformer layers, each one refining the representation by attending to every other token in the context. At the end, it produces a probability distribution over every token in its vocabulary. This is the output: not a word, but a list of probabilities for what word should come next.
The model runs this loop once per output token. To write a 200-word reply, it loops roughly 250 times. Each loop reads the entire context, including everything it has already written.
What does an LLM's probability distribution look like?
At any given step, the model assigns a probability to every token in its vocabulary, which is typically 50,000 to 100,000+ tokens. Most probabilities are near zero. A small cluster at the top carries most of the weight.
Here is a toy example. Suppose the context ends with: "The sky is".
| Token | Probability |
|---|---|
| blue | 0.38 |
| clear | 0.21 |
| dark | 0.14 |
| getting | 0.08 |
| overcast | 0.06 |
| the | 0.02 |
| ... (all other tokens) | ~0.11 combined |
The model does not always pick "blue". It samples from this distribution. Usually "blue" wins because it has the highest probability, but on some runs "dark" or "getting" will be chosen instead.
What is temperature in an LLM?
Temperature is the dial that controls how the model samples from that distribution.
At temperature 0, the model always picks the highest-probability token. The output is fully deterministic. Ask the same question twice, you get the same answer.Common gotchaDo not assume temperature 0 guarantees identical output across real API calls. Floating-point rounding on different hardware, batched requests, and shifting system fingerprints all break a near-tie at the top of the distribution, so the same prompt can still diverge run to run.If you need true reproducibility for a test, pin the response and assert against it rather than trusting temperature 0 alone, and treat low temperature as "much more stable," not "byte-for-byte repeatable."
At temperature 1, the model samples from the raw distribution. "blue" is 3.8x more likely than "getting", but "getting" can still be chosen.
At temperature 2, the distribution is flattened further. The gap between "blue" and "the" narrows. You get more surprising, sometimes incoherent output.
For factual questions and code, lower temperatures produce more reliable results. For creative writing or brainstorming, higher temperatures produce more varied output. Claude Code and most coding tools pin temperature near zero. Chat interfaces often default to 1.
Why does next-word prediction produce capable behaviour?
It is reasonable to ask: how does "predict the next token" produce a model that can write a test suite, explain a concept, or debug your code?
The training data is the answer. The model was trained on a vast corpus of text where capable humans demonstrated reasoning in writing. Math proofs, technical documentation, argument and counter-argument, step-by-step tutorials, code with comments explaining the intent. The model learned that the token sequence "Step 1:" tends to precede a clear explanation, that "def " tends to precede a function with a coherent body, that "because" tends to precede a justification.
Prediction is compression. To predict text well, the model must encode something about the patterns in that text. Those patterns include grammar, facts, logical structure, and domain conventions. The prediction objective forces the model to learn them all.
This is why the same model can write Python, explain photosynthesis, and draft a legal summary. It learned the token patterns that competent humans produce in each of those domains.
What are the limits of LLMs?
Understanding the mechanism makes the limits obvious.
No understanding, just patterns. The model has no internal world model that it is consulting. It has learned which tokens tend to follow which other tokens, across an enormous range of contexts. This produces behaviour that looks like understanding and sometimes is not.
Hallucination is the prediction mechanism working correctly. When the model generates a plausible-sounding but wrong fact, it is doing exactly what it was trained to do: produce the next token that fits the pattern. If the training data contained confident assertions about a topic, the model learned to produce confident assertions about that topic, whether or not the specific claim is accurate.
The context window is the model's entire world. The model has no memory between conversations. It cannot look things up. Everything it knows for your session is either in the weights (learned during training) or in the context window (what you sent). See the context engineering guide for what this means in practice.
Token-by-token generation means no backtracking. The model cannot revise a token it already wrote without being prompted to. Each token is committed. This is why asking the model to think step-by-step before answering often improves results: you are pushing the reasoning into the visible token stream, where the model can attend to it as it continues.
Longer is not always better. Adding more text to the context helps up to a point. Beyond that, relevant information gets diluted. The model reads everything in the context window, but its attention is finite.
What this means for working with LLMs
Once you have the loop in your head, a few practical things become clearer.
Prompting is not magic. You are shaping the probability distribution by providing a context that makes the tokens you want more likely. A good system prompt does not instruct the model to be smart. It creates a context where smart-looking tokens are the natural continuation.
That prompt works because "senior TypeScript engineer reviewing a PR" is a context that, in the training data, preceded high-quality, specific technical feedback. The model learned that pattern.
Temperature matters for your use case. If you are generating test cases where correctness is everything, lean toward lower temperature. If you are generating five alternative names for a feature, a higher temperature gives you more spread.
Longer output is more likely to drift. Each token is predicted from everything before it, including the tokens the model already wrote. Errors early in a long output compound. Shorter, more focused prompts produce more reliable results than "write me a full implementation of X".
The model is not lying when it hallucinates. It is doing prediction. The fix is not to scold it. The fix is to provide context that makes correct tokens more likely: cite the relevant documentation, provide the actual data, structure the prompt so the correct answer is the natural continuation. The stop Claude making things up guide turns this into a practical routine.
Where to go next
If you want to see how this loop is used inside agents, the multi-agent architectures guide shows how multiple models can hand work to each other. If you want to control what the model knows at generation time, the context engineering guide covers what to put in the window and what to leave out.
The predict-sample-append loop runs billions of times a day across every LLM product. It is a remarkably simple idea that rewards you for understanding it precisely.