How encoding works in LLMs

A language model cannot see the letter "a". It sees a number. Before any prediction happens, your text has to be turned into a sequence of integers, and the model's reply has to be turned back into text. That round trip is called encoding, and it is the quiet first step under every chat you have ever had with an LLM.

This guide builds the idea up the way Andrej Karpathy does in his "Let's build GPT" walkthrough. We start with the smallest encoder that works, a character-level dictionary you can type out by hand. Then we hit the wall that makes it impractical, and we reach for the real tools that production models use: Google's SentencePiece and OpenAI's tiktoken. Every snippet here runs as plain Python. None of it calls a model. Encoding is pure bookkeeping, and that is exactly why it is a good place to start.

Why does text need encoding at all?

A neural network is a stack of math. Matrix multiplies, additions, a few curves. It operates on numbers, full stop. "Hello" means nothing to it until "Hello" becomes something like [20, 43, 50, 50, 53].

So every model carries a fixed map between pieces of text and integer IDs. Encoding walks one way: text in, integers out. Decoding walks the other: integers in, text out. The two must agree perfectly, because if encode and decode disagree by even one entry, the model reads gibberish and you read gibberish back.

The set of distinct pieces a tokenizer knows is its vocabularyThe full list of distinct tokens an encoder can produce. Every piece of text must be expressed using only entries from this list.. Pick characters as your pieces and the vocabulary is tiny. Pick whole words and it balloons. Most real systems land in between, on chunks called subwords. That one choice, and what it costs you, is what the rest of this guide covers.

The simplest encoder: a character dictionary

Start with the most literal idea possible. List every unique character in your text, sort it, and number each one. That numbering is your whole vocabulary.

Say your training text is a stretch of Shakespeare. First you find the distinct characters in it.

text = open("input.txt").read()
chars = sorted(set(text))
vocab_size = len(chars)
 
print("".join(chars))
print(vocab_size)
# !$&',-.3:;?ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
# 65

Sixty-five characters. That is the entire alphabet the model will ever deal with: the letters in both cases, a handful of punctuation marks, the space, the newline. Nothing else exists in this little universe.

Now build two lookup tables. One maps a string to an integer (stoiString to integer. A dict that turns each character into its numeric ID.), the other maps an integer back to a string (itosInteger to string. The reverse dict that turns a numeric ID back into its character.). They are mirror images of each other.

stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)}
 
def encode(s):
    return [stoi[c] for c in s]      # string -> list of ints
 
def decode(ids):
    return "".join(itos[i] for i in ids)  # list of ints -> string

That is a complete, working tokenizer. Two dicts and two one-line functions. Watch it go both ways.

encode("hii there")
# [46, 47, 47, 1, 58, 46, 43, 56, 43]
 
decode(encode("hii there"))
# 'hii there'

Encode the string, decode it straight back, and you land on the exact text you started with. That round-trip property is the contract every tokenizer has to honor. Break it and the model is reading one thing while you think it is reading another.

A few things are worth pausing on here. The space character got ID 1, not some special treatment. Both i characters in "hii" map to the same number, because the encoder is deterministic: the same input always yields the same output. And there is no notion of a word at all. The encoder has never heard of "there". It only knows t, h, e, r, e.

Why character encoding does not scale

The character approach is honest and easy to reason about. It is also a poor fit for a real model, and the reason is worth understanding because it explains why everyone moved on.

Sequences get long. A token is one prediction step for the model. With characters, the word "tokenization" is twelve steps. A short paragraph might be six hundred tokens. Since a model can only attend to a fixed window of recent tokens, a character encoder burns that window fast. You spend your whole budget spelling words instead of holding ideas. Tokens are also what you pay for, which token usage and cost breaks down.

The model relearns the obvious. "the" appears constantly in English. A character model has to learn, from scratch and over and over, that t is often followed by h is often followed by e. That pattern is real signal the model could be spending on harder things. Encoding "the" as one piece hands it that fact for free.

Wider vocabularies pack more per step. Sixty-five characters is a very small vocabulary, which sounds efficient, but it means each token carries almost no information. There is a tradeoff hiding here. A bigger vocabulary means each ID stands for more text, so sequences shrink and the model sees more content per step.

Words look tempting in that table, until you meet a word the vocabulary never saw. A pure word encoder either crashes or replaces the unknown with a generic placeholder, which throws away meaning. Names, typos, code, new slang, other languages: all of it falls off the edge. You need something that never gets stuck and still keeps common words whole. That something is subword tokenization.

Subword tokenization, the practical middle

The fix is to learn the pieces from data rather than fixing them in advance. Feed an algorithm a big pile of text and let it find the chunks that pay their way. Common words like "the" and "model" earn a slot of their own. A rare word gets broken into smaller parts it already knows.

The most common method is byte-pair encodingByte-pair encoding. It starts from single characters and repeatedly merges the most frequent adjacent pair into a new token, learning a vocabulary of useful chunks. (BPE). The idea is short. Start with characters. Find the pair of adjacent symbols that occurs most often, merge it into a single new symbol, and record that merge. Repeat a few thousand times. Each round stitches together a slightly bigger chunk, so the vocabulary grows from letters toward syllables, then toward whole words.

The payoff is that nothing is ever unknown. Worst case, a strange string falls all the way back to individual characters or bytes, which always exist. Best case, a familiar word is a single token. You get the completeness of characters and most of the efficiency of words, in one scheme. Two libraries dominate in practice, and they are worth meeting directly.

tiktoken: OpenAI's fast BPE

tiktoken is the tokenizer behind OpenAI's models. It is a BPE implementation written for speed, and you can run it on its own to see exactly how a model would chop your text. Install it and load the encoding a given model uses.

import tiktoken
 
enc = tiktoken.get_encoding("cl100k_base")
 
ids = enc.encode("How encoding works in LLMs")
print(ids)
# [4438, 11058, 4375, 304, 445, 11237, 82]
 
print(enc.decode(ids))
# 'How encoding works in LLMs'

Seven tokens for six words. Notice the count is close to the word count but not equal to it. "LLMs" did not survive as one piece, because it is not a common run of characters, so BPE split it. To see the chunks, decode each ID on its own.

for i in ids:
    print(i, repr(enc.decode([i])))
# 4438  'How'
# 11058 ' encoding'
# 4375  ' works'
# 304   ' in'
# 445   ' L'
# 11237 'LM'
# 82    's'

Two details jump out. The leading space is part of the token: it is ' encoding', not 'encoding'. BPE learned that words usually arrive with a space in front, so it folds the space in rather than spending a separate token on it. And "LLMs" shattered into ' L', 'LM', 's', because the model has never seen that exact capitalized run often enough to keep it whole. Common text stays whole, odd text fragments. That is the deal.

The vocabulary here is large. cl100k_base holds just over 100,000 tokens, which is why most ordinary English words map to a single ID.Common gotchaToken count is not character count and not word count. A rule of thumb is roughly four characters per token for English prose, but code, rare words, emoji, and non-Latin scripts all run higher, sometimes far higher.If you are budgeting context or estimating cost, encode the real text and count the result rather than guessing from length.

SentencePiece: Google's language-agnostic tokenizer

sentencepiece is Google's tokenizer, used by models such as T5 and Llama. It tackles a problem tiktoken sidesteps: many languages do not put spaces between words at all. So SentencePiece treats the raw text as a stream and trains directly on it, marking spaces with a visible symbol (▁, a lower one-eighth block) instead of relying on them to find word edges. That makes it work the same way whether the text is English, Japanese, or a column of code.

You train it on your own corpus, which is part of the appeal. You choose the vocabulary size up front.

import sentencepiece as spm
 
spm.SentencePieceTrainer.train(
    input="input.txt",
    model_prefix="demo",
    vocab_size=8000,
)
 
sp = spm.SentencePieceProcessor(model_file="demo.model")
 
print(sp.encode("How encoding works", out_type=str))
# ['▁How', '▁encod', 'ing', '▁works']
 
print(sp.encode("How encoding works"))
# [305, 1429, 88, 962]

Same text, a different split. With an 8,000-token vocabulary trained on a small corpus, "encoding" came apart into ▁encod and ing, where the larger tiktoken vocabulary kept it whole. The ▁ marks where a space stood, so decoding can rebuild the original spacing exactly.

print(sp.decode([305, 1429, 88, 962]))
# 'How encoding works'

The lesson is that vocabulary size is a dial you control. A smaller vocabulary means more, shorter tokens and a lighter model footprint. A larger one keeps more words whole and shortens sequences, at the cost of a bigger embedding table. Neither is correct in the abstract. It depends on your text, your languages, and your budget.

tiktoken vs SentencePiece on the same text

Both are BPE-family subword tokenizers, and both honor the round trip. The differences are practical, and they show up the moment you run the same sentence through each.

Run "How encoding works in LLMs" through each and you get different ID lists, different counts, and different splits, all of them valid. There is no universally right tokenization. There is only the tokenization the target model was trained with, and you must use that exact one. Encode with cl100k_base and decode with a SentencePiece model and you get nonsense, the same way encode and decode disagreeing in the character example gave nonsense. The contract holds across the whole family: the encoder and decoder must be the matched pair the model expects.

What this means in practice

You will rarely write a tokenizer yourself. The point of building one by hand is that it makes the rest of the stack legible. A handful of habits follow directly from how encoding works.

Count tokens, do not estimate them. When you care about context limits or cost, run the actual text through the actual tokenizer and read the number off. Length in characters is a rough proxy that gets less rough as text gets stranger.

Match the tokenizer to the model. Each model ships with the exact encoder it was trained on. Use a different one and every ID points at the wrong piece. This is the single most common way people produce garbled output without realizing why.

Expect rare text to cost more. Code, long identifiers, unusual names, and non-English scripts fragment into more tokens than plain prose of the same length. If a prompt feels surprisingly expensive, the tokenizer is often the reason.

Spelling tasks are hard for a reason. Because a model reads tokens, not letters, asking it to count the letters in a word or reverse a string fights the way it perceives text. That limitation is a direct, visible consequence of encoding. For the full picture of what the model does once your tokens are in hand, read how LLMs predict the next word. To see how those token IDs turn into meaning the model can compute over, embeddings explained for beginners picks up exactly where this guide stops.

The takeaway

Encoding is the bridge between text and math. A character dictionary, two dicts and two functions, shows the whole shape of it: map text to integers, run the model, map the integers back. It is a fine teaching tool and a bad production tool, because it makes sequences long and forces the model to relearn every common word. Subword tokenizers fixed that by learning their pieces from data. tiktoken and SentencePiece are the two you will meet most, one fixed and fast, one trainable and language-agnostic, both built so that decode always undoes encode. Get that round trip right and the rest of the model has clean numbers to work with.