What Is Tokenization in LLMs Explained Byte Pair Encoding?

Introduction

Every chat with a language model begins with a step that most users never see, and that step shapes cost, quality, and fairness. Developers, students, and curious readers keep typing one compound question into search boxes, namely what is tokenization in LLMs explained byte pair encoding? The short version is that a tokenizer chops text into reusable pieces and maps each piece to an integer the neural network can process. Meta reported that the Llama 3 tokenizer carries a 128K-token vocabulary and emits up to 15 percent fewer tokens than Llama 2 for identical text. That single design choice helped keep inference speed roughly level even though the new model had about one billion more parameters. This guide walks through the full story, from the compression trick behind byte pair encoding to the glitches, costs, and fairness questions that tokenizers create today. By the end you will know how merges are learned, why vocabularies range from 32K to 200K entries, and where tokenization is heading next.

Quick Answers on Tokenization in LLMs and Byte Pair Encoding

What is tokenization in LLMs explained byte pair encoding?

Tokenization converts text into integer token IDs a model can process. Byte pair encoding builds the vocabulary by repeatedly merging the most frequent adjacent symbol pairs until a target vocabulary size is reached.

How many tokens does a typical English word use in LLM tokenization?

OpenAI’s rule of thumb for tokenization says one token is roughly four characters, or about three quarters of a word, so 100 tokens cover roughly 75 English words.

Why does byte-level BPE never produce unknown tokens?

Byte-level BPE starts from the 256 possible byte values instead of Unicode characters, so any text in any language can be encoded without an unknown token appearing in LLMs.

Key Takeaways

• Tokenization turns text into integer IDs, and the token, not the word, is the unit that LLMs read, price, and limit.
• Byte pair encoding learns merges by counting adjacent pairs, which gives a compact vocabulary that still covers rare words through smaller pieces.
• Vocabulary size is a tradeoff: larger tables shorten sequences but add embedding parameters, and teams can offset the bill with prompt caching explained for cost control.
• Tokenizers also create real problems, including multilingual cost gaps, glitch tokens, and spelling errors, which newer byte-level designs try to fix.

What Is Tokenization in LLMs and Where Does BPE Fit?

Tokenization in LLMs converts raw text into integer token IDs. Byte pair encoding builds the vocabulary by repeatedly merging frequent symbol pairs, which is the core answer to what is tokenization in LLMs explained byte pair encoding?

How Raw Text Becomes Token IDs Inside a Language Model

Building on that definition, it helps to follow one sentence through the full pipeline from characters to integers. The tokenizer first normalizes the text, which may mean cleaning up Unicode, and then splits it into chunks using a fixed pattern. Each chunk is broken into the smallest base symbols, and learned merge rules combine those symbols into longer pieces. Every final piece is looked up in a vocabulary table that assigns it one integer ID. The neural network never sees letters at all, only the list of integers that comes out of this lookup. Those integers index rows of an embedding matrix, and the rows become the vectors that flow through the transformer layers. For a refresher on that last step, see this explanation of how word embeddings represent meaning. Following that path from text to integers is the practical answer to what is tokenization in LLMs explained byte pair encoding?

Decoding runs the same process in reverse, which is why the mapping must be lossless for most practical systems. The model predicts one token ID at a time, and the tokenizer converts each predicted ID back into its text fragment. A fragment may be a whole word, a word piece, a single punctuation mark, or a lone byte of a multi-byte character. When the model streams an answer, you are watching these fragments appear one after another. Some fragments look odd on their own, such as half a word, because the boundaries follow statistics instead of grammar.

Everything the model can do is therefore limited to sequences of tokens that its vocabulary can express. The context window, the pricing meter, and the training budget are all measured in tokens rather than words or characters. Two models can read the same paragraph and count very different token totals if their vocabularies differ. That is why a tokenizer is not a minor preprocessing detail but a core part of the model’s identity. Swapping it after pretraining usually means retraining or at least heavy surgery on the embedding layers.

From Gage’s Compression Trick to Modern Subword Vocabularies

Looking back, byte pair encoding began as a data compression method rather than a language technique. Philip Gage described it in 1994 as a way to replace the most frequent pair of bytes in a file with a single unused byte. A lookup table recorded each replacement so the original data could be rebuilt exactly. The idea sat mostly unnoticed in machine learning for two decades because speech and translation researchers worked with whole words. The turning point came when Sennrich, Haddow and Birch adapted subword units for neural machine translation in 2015. Their goal was to let a model translate rare and unseen words by assembling them from smaller pieces. Subword models improved over a back-off dictionary baseline by 1.1 BLEU on English to German and 1.3 BLEU on English to Russian.

From there the technique spread quickly through the transformer era. OpenAI’s GPT-2 pushed it down to bytes, which kept the base vocabulary at 256 symbols instead of a table covering every Unicode character. SentencePiece, released by Kudo and Richardson in 2018, let teams train subword models directly from raw sentences without language-specific pre-splitting. Today nearly every major model family ships a tokenizer descended from one of these two lines. The lineage explains why tokenizers often feel inherited rather than designed for any one model. A vocabulary frozen years ago can still shape how a brand new model prices and parses text.

Why Whole Words and Single Characters Both Fall Short

Stepping back from the algorithm, consider the two obvious ways to split text and why each one breaks. A word-level vocabulary needs an entry for every spelling, so names, typos, and compounds quickly overflow any table you can afford. Unknown words get replaced by a placeholder token, which erases exactly the rare information users often care about. Languages with rich morphology, such as Turkish or Finnish, generate thousands of surface forms from a single root. Even English produces endless variants through prefixes, suffixes, and fresh coinages like brand names. A fixed word list simply cannot cover an open-ended language. The practical result is a model that stumbles on any word it has not seen.

Character-level modeling avoids the unknown-word problem but creates a different one, because sequences become very long. Attention cost grows with the square of sequence length, so a paragraph that takes 150 word tokens may take around 800 characters. Long sequences also fill the context window quickly and leave less room for the instructions and documents users actually care about. The GPT-2 authors observed that byte-level language models still lagged word-level models on large benchmarks such as the One Billion Word Benchmark. Training a model to spell out every word one letter at a time wastes capacity on patterns that frequency statistics could handle for free. The field therefore looked for a middle path between whole words and single characters.

Subword tokenization sits between the two extremes and keeps most of the benefits of each. Frequent words such as “the” earn their own single token, while rare words split into familiar fragments. A long word like “unhappiness” might become three pieces, each of which the model has met thousands of times. This gives a bounded vocabulary, short sequences for common text, and graceful degradation for rare text. The compromise is not perfect, since splits follow frequency statistics rather than linguistic meaning. A deeper look at the basics appears in this guide on what tokenization means in NLP.

Walking Through Byte Pair Encoding One Merge at a Time

Moving on to the mechanics, BPE training is a simple loop that counts, merges, and repeats. The Hugging Face LLM course uses a toy corpus with the words hug (10 times), pug (5), pun (12), bun (4), and hugs (5). Each word is first split into single characters, so the starting vocabulary is just b, g, h, n, p, s, and u. The algorithm then counts every adjacent pair, weighted by how often each word occurs. The pair u and g appears 20 times, more than any other pair, so it becomes the first merge. That merge creates a new token, ug, and adds it to the vocabulary alongside the original characters.

After the first merge the counts change, because ug now behaves as one symbol inside hug, pug, and hugs. The pair u and n appears 16 times across pun and bun, which beats the 15 occurrences of h followed by ug. So the second rule merges u and n into un. On the third round the pair h and ug leads with 15 occurrences, and it becomes the token hug. Each learned rule is stored in order, and that order is the entire trained model of a BPE tokenizer. Nothing else is learned besides the ordered merge list and the resulting vocabulary table.

Encoding new text simply replays those rules, so a new word is split into base characters and the stored merges are applied in the order they were learned. The word bug becomes b and ug, because the rule for u and g fires and nothing else matches. The word mug, in contrast, produces an unknown marker plus ug, because the letter m never appeared in the training corpus. That unknown-character failure is the weakness that byte-level BPE was invented to remove, as the next section explains. The same logic scales from this tiny corpus to billions of words, only with many thousands of merges.

Training stops when the vocabulary reaches a chosen size, which equals the base symbols plus the number of merges. The original GPT model used 478 base symbols and 40,000 merges, which is how it reached a vocabulary of 40,478. Andrej Karpathy’s minbpe repository shows the loop in about a page of Python, starting from the 256 single-byte tokens. Its README example turns the string aaabdaaabac into the IDs 258, 100, 258, 97, and 99 using three merges. Those merges are aa to 256, ab to 257, and 256 plus 257 to 258. Reading that example once makes every production tokenizer feel far less mysterious and answers what is tokenization in LLMs explained byte pair encoding?

Byte-Level BPE and the 256-Symbol Safety Net

Turning to the unknown-character problem, byte-level BPE swaps Unicode characters for raw UTF-8 bytes as the base alphabet. Unicode defines roughly 150,000 characters across 161 scripts, and each character is stored as one to four bytes. Starting from the 256 possible byte values means every string in every language can be represented, with no unknown token ever needed. GPT-2 uses this design with a vocabulary of 50,257 entries, made of 256 byte tokens, 50,000 merges, and one end-of-text token. The guarantee that nothing can fail to encode is the main reason byte-level BPE became the default for large language models. Merges then learn common multi-byte sequences, so ordinary English words still collapse into single tokens.

The safety net has a price, which shows up in languages and symbols that were rare in the tokenizer’s training data. A rare script may never earn merges, so each character falls back to two, three, or four separate byte tokens. Emoji and unusual symbols behave the same way, which is why a single emoji can cost several tokens. SentencePiece-based models such as Llama and Mistral work on code points with a byte fallback for rare characters, a middle design that Karpathy’s tokenizer lecture compares with the byte-first approach. Either way the model still learns embeddings for the byte tokens that appear. Those embeddings are learned by the same neural network training as everything else.

Pre-Tokenization Patterns That Keep Merges Honest

Beyond the merge loop, production tokenizers split text with a regular expression before any merging begins. The GPT-2 team noticed that plain BPE produces wasteful merges, such as separate tokens for “dog.” and “dog!” that waste vocabulary slots. Their input representation section describes preventing merges across character categories, with an exception for spaces. That rule keeps letters, digits, and punctuation in separate lanes, so the vocabulary avoids glued combinations. Pre-tokenization decides which boundaries a merge is never allowed to cross. It also makes training faster, since pairs are counted only inside each chunk.

GPT-4 introduced a revised pattern that handles contractions in a case-insensitive way and changes how digit runs and whitespace are chunked. The minbpe library ships it as the default for its RegexTokenizer, so no merge can cross a category boundary. The pattern is a design decision, not a law of nature, and different model families choose different ones. Small regex differences explain why two byte-level BPE tokenizers can disagree on the same sentence. They also explain why swapping a tokenizer file between models silently breaks the pairing with learned weights.

Pre-tokenization has visible consequences in everyday prompts that many users never think about. The lecture write-up from fast.ai notes that the number 677 can be split into two tokens. Digits of equal value are therefore chunked inconsistently across different numbers, which makes arithmetic harder to learn. Leading and trailing spaces change the tokens produced, which is why a stray space at the end of a prompt can alter model behavior. Code indentation, repeated newlines, and mixed languages all follow the same chunking rules. Understanding the pattern turns many baffling prompt quirks into predictable engineering facts.

What Vocabulary Size Is Right? Tradeoffs From 32K to 200K Tokens

Weighing the options, vocabulary size is the main dial a tokenizer designer turns, and the industry has pushed it steadily upward. The original GPT model used 40,478 entries, GPT-2 settled on 50,257, and the cl100k_base encoding behind GPT-4 holds about 100,000. Meta’s Llama 3 moved to a vocabulary of 128K tokens, and the o200k_base encoding name used by newer OpenAI models signals roughly 200,000 entries. Each jump trades a larger lookup table for shorter token sequences on the same text. Bigger vocabularies compress text better, but every extra entry must be learned from data. The trend line shows designers consistently choosing compression and multilingual coverage over table size.

The tradeoffs of a larger vocabulary are concrete rather than philosophical, and they show up in memory and speed. Longer token sequences cost more to process because attention compares every token with every other token in the window. A larger vocabulary shortens sequences, so Meta could report up to 15 percent fewer tokens than Llama 2 for the same text. The price is a bigger embedding matrix at the input and a bigger output layer that must produce a probability for every entry. That final layer ends with the softmax function in neural networks, whose cost grows with the number of choices. Rare entries also receive few training examples, and undertrained rows are where odd behavior later appears.

Teams usually choose a size empirically by running experiments instead of relying on a rule of thumb. They train candidate tokenizers on a representative corpus and measure fertility, the average number of tokens per word, on every language they care about. They then pick the smallest vocabulary at which further gains flatten out. Karpathy’s minbpe examples train at 4,096 and 32,768 entries, which shows how modest a research tokenizer can be. Multilingual products tend to need far more entries, because each additional script wants its own merges. Whatever size is chosen, it is frozen at pretraining time and travels with the model for life.

Choosing Among WordPiece, Unigram and SentencePiece

Choosing among tokenizer families starts with the realization that they differ in how they build a vocabulary, not in their end goal. BPE builds bottom-up by merging the most frequent pair at each step. WordPiece, used by BERT-family models such as DistilBERT and Electra, builds bottom-up too but picks merges by a score instead of raw frequency. The Hugging Face tokenizer summary gives that score as the frequency of the pair divided by the product of the frequencies of its two parts. WordPiece therefore prefers pairs that co-occur far more often than chance would predict. The algorithm differs by only a few lines of code, yet it can pick different merges from BPE on the same corpus.

The Hugging Face example makes the difference between the two scoring rules tangible with just a handful of pairs. Using its toy counts, the pair g and s scores 0.050, which is higher than the 0.028 scored by u and g, u and n, or h and u. A frequency-only BPE would instead have started with u and g, which appears 20 times. WordPiece reaches a different first merge because g and s are individually rare but almost always appear together. Neither choice is universally better, since downstream quality depends on the task, the corpus, and the vocabulary size. In practice the family is usually dictated by whichever model you are fine-tuning.

Unigram flips the construction order by starting with a very large candidate vocabulary and pruning it. At each round it measures how much the training loss would rise if each token were removed. It then drops the tokens whose removal hurts least, usually the bottom 10 to 20 percent, while always keeping the base characters. Because Unigram assigns probabilities, one word can have several valid segmentations, and training can sample among them as a form of regularization. At inference time the model simply returns the single highest-probability split of each word. This makes Unigram attractive for noisy or morphologically rich text.

SentencePiece is best understood as a toolkit rather than a competing algorithm. It treats the space as an ordinary symbol, usually displayed as ▁, so it needs no language-specific pre-splitting and handles Chinese or Japanese naturally. It can train either BPE or Unigram models, and decoding is lossless because tokens simply concatenate. OpenAI’s tiktoken library takes the opposite route with a fast byte-level BPE implementation that it says is three to six times faster than a comparable open source tokeniser. Llama and Mistral use SentencePiece, while GPT models use tiktoken encodings such as cl100k_base and o200k_base. Picking between the two usually means picking an ecosystem of models, libraries, and conventions.

Special Tokens, Chat Templates and Control Sequences

Beyond the learned merges, every vocabulary reserves a few special tokens that mark structure instead of language. The end-of-text token is the classic example, with ID 50256 in GPT-2 and ID 100257 in GPT-4’s encoding. These entries sit outside normal BPE processing, so the merge rules never create or split them. They tell the model where a document ends, where a prompt begins, or which role is speaking. A special token is a control signal, and a tokenizer must never let ordinary user text impersonate one. Karpathy’s minbpe project handles this by requiring an explicit allowed-special setting before special strings are encoded as control tokens.

Chat models add a layer on top, wrapping each message in template tokens that label the role and delimit turns. The tiktoken README demonstrates the idea by extending cl100k_base into a custom encoding that adds start and end markers for chat messages. If a fine-tuning script uses a different template than the one the model was trained with, quality quietly degrades without any error message. Tool calls and structured outputs add still more reserved sequences, which is part of why function calling in LLMs depends on strict formats. Developers should therefore treat the template as part of the tokenizer and version it alongside the weights. When something looks wrong in a chat deployment, the template is often the first place to check.

Putting a Tokenizer to Work in Real Projects

In practice, most teams never train a tokenizer from scratch because they load the one that ships with their model. OpenAI’s help center points developers to its web tokenizer and to tiktoken for counting tokens before a request. The OpenAI guidance on counting tokens also reminds readers that one token is about four characters and that other languages have different ratios. A typical workflow calls the encoding_for_model helper, encodes the prompt, and compares the length with the model’s context limit. Budgeting this way prevents truncated inputs, silent context overflow, and surprise invoices at the end of the month. Counting tokens before sending a request is the cheapest quality control step in any LLM pipeline. Teams that master this habit have effectively learned what is tokenization in LLMs explained byte pair encoding?

Training your own tokenizer makes sense when the domain differs sharply from web text, such as source code, legal contracts, or biomedical notes. The process is short: sample a representative corpus, choose an algorithm, pick a vocabulary size, train, and measure tokens per word on held-out text. The minbpe repository trains a basic tokenizer at 4,096 entries and a regex tokenizer at 32,768 in a few lines. SentencePiece offers a command-line trainer that handles the same job for teams working with raw multilingual text. One constraint matters above all, because a tokenizer trained after a model exists cannot simply replace the original. Changing it means retraining embeddings, so new tokenizers belong at the start of a project.

Testing is where many deployments cut corners, even though the checks are quick to run. Always verify that encoding followed by decoding returns the original string, including emoji, accented characters, and unusual whitespace. Compare token counts across the languages your users actually write, since fertility can differ by several times. Log token counts in production so cost spikes can be traced to specific prompts or customers. Teams new to the discipline can start with the natural language processing fundamentals before moving on to tokenizer internals. These habits cost almost nothing and prevent most of the tokenization incidents that reach production.

Where Tokenization Falls Short in Spelling, Arithmetic and Glitch Tokens

Looking at failure modes, many famous LLM quirks trace back to the tokenizer rather than the neural network. The fast.ai write-up of Karpathy’s lecture puts it bluntly, saying that much of the odd behavior in large language models can be traced back to tokenization. A model that sees a word as two or three chunks cannot easily count its letters or reverse its spelling. Tasks that depend on characters, such as anagrams or counting repeated letters, fight against the chunking. The model reasons over tokens, so anything defined at the character level is indirect for it. Larger models may partly compensate by memorizing common cases, but the underlying blind spot remains.

Arithmetic suffers for a related reason that involves digit chunking. The number 677 splits into two tokens while another number of similar size may split differently. The model must therefore learn irregular patterns for what humans see as a uniform digit system. Carrying and aligning columns becomes harder when digit groupings shift unpredictably from one number to the next. Formatting choices also matter, because some data formats consume more tokens than others for the same information. The lecture write-up even recommends preferring YAML over JSON in some cases for this reason. These effects are small individually but they add up across long calculations.

Glitch tokens are the most dramatic failure of the tokenization step. In an analysis of GPT-2 and GPT-3 vocabularies, researchers found that of 50,257 tokens, 374 were problematic after a second filtering pass. They split those into 133 truly weird entries and 241 merely confused ones. Strings such as SolidGoldMagikarp made models evade, hallucinate, or answer with unrelated words. When asked to repeat that token, ChatGPT returned “distribute”, which shows how little the model understood the entry. The authors suggest, without confirming, that the tokenizer was built from scraped web text rich in e-commerce backends and gaming logs. The model’s own training data was more curated, so those tokens barely appeared during learning.

The fixes are practical rather than exotic, and most of them are cheap. Build the tokenizer on data that resembles the model’s training corpus, so no entry exists that the model never saw. Audit vocabularies for undertrained tokens, for example by looking at embeddings that sit unusually close to the centroid. Keep byte-level fallback so rare strings decompose into familiar pieces instead of becoming opaque units. Probe new models with token-level tests before release, including strange strings, leading spaces, and long repeated characters. None of this removes the limitation entirely, but it turns a mysterious bug class into a checklist.

Token Costs, Latency and Context Windows in Production

Moving from theory to invoices, APIs bill per token, so tokenizer efficiency turns directly into money. OpenAI’s rule of thumb of four characters per English token means a 1,000-word document costs roughly 1,300 tokens. Languages with weaker tokenization can cost several times more for equivalent content, which the section on the multilingual token tax quantifies below. Output tokens are generated one at a time, so more tokens also mean more latency for the user. For a broader look at budgeting, the analysis of generative AI cost per token tracks how prices have moved. Because billing, speed, and memory all scale with token count, the tokenizer is a hidden cost lever.

Context windows are also measured in tokens, which makes tokenizer efficiency a capacity issue. A window of 128,000 tokens holds about 96,000 English words at the 0.75 words-per-token rule of thumb. The same window holds noticeably fewer words in a language that fragments into many tokens. Even inside the window, quality can slip, as the article on context rot in large language models explains. Efficient tokenization therefore buys more usable context without changing the model at all. It is one of the few optimizations that helps cost, latency, and capability at the same time.

Engineers can pull several levers without ever touching the tokenizer or the model weights. Shorten prompts, reuse stable prefixes, compress retrieved documents, and prefer compact output formats. Cache static instructions so repeated tokens are not reprocessed on every request. Measure tokens per request as a first-class metric next to latency and error rate. Review the highest-volume prompts monthly, because small wording changes can shift totals by double digits. These steps often deliver savings that rival a model downgrade with none of the quality risk.

Ethical Questions Around the Multilingual Token Tax

Setting cost aside, tokenization raises a fairness question because the same meaning can cost very different amounts in different languages. OpenAI’s GPT-4o announcement lists token counts for the same greeting sentence in 20 languages before and after its new tokenizer. Gujarati fell from 145 tokens to 33, a 4.4 times reduction, and Hindi fell from 90 to 31. English moved only from 27 to 24 tokens, a 1.1 times change. The comparison implies that the older tokenizer charged Gujarati speakers more than five times as many tokens as English speakers for one sentence. When pricing, context limits, and speed all scale with tokens, a tokenizer quietly decides who pays more for the same idea.

The ethical response to the token tax has several parts that different actors can own. Tokenizer builders can train on balanced multilingual data, publish fertility numbers for each supported language, and report them alongside accuracy benchmarks. Vendors can consider pricing models that reflect content rather than raw token counts. Researchers can extend vocabularies for underrepresented languages instead of leaving them to byte fallback. A caution applies to these headline figures, because one short sentence per language is a convenient sample rather than a population average. The direction of the gap is still clear and the improvement from a better tokenizer is large.

Tokenizer Quirks as a Security Risk

Given the way tokenizers split text, they also create an attack surface that classic security reviews miss. The write-up on the TokenBreak exploit against AI defenses describes manipulating tokenization to evade LLM content moderation systems. The core idea is that a guardrail model and a target model may split the same string differently. A harmful instruction that one tokenizer recognizes can look benign after a small spelling change under another. Whenever two components tokenize the same text differently, an attacker can try to exploit the disagreement. Defenders therefore need to test filters against perturbed spellings, not just clean strings.

Special-token injection is a related risk that deserves its own line in a threat model. If user text containing a literal control string is encoded as a real control token, a user could break out of the intended structure. This is why libraries such as minbpe require explicit permission before special strings are interpreted. Broader defenses against hostile inputs are covered in this guide to prompt injection attacks and defenses. Teams should sanitize inputs, keep guardrail and model tokenizers aligned where possible, and log suspicious token patterns. Security review of an LLM application should include the tokenizer, not just the prompt.

The Future of Tokenization Beyond Fixed Subword Vocabularies

Looking ahead, the most radical proposal is to remove the fixed vocabulary entirely. Meta researchers introduced the Byte Latent Transformer, which reads raw bytes and groups them into variable-length patches. Patch boundaries depend on how predictable the next byte is, so the model spends more compute on hard data and less on easy data. The authors report that it matches tokenization-based LLM performance at scale, with a study reaching 8 billion parameters and 4 trillion training bytes. If patching holds up, the tokenizer becomes a learned part of the network instead of a frozen preprocessing step. That would erase glitch tokens and unfair per-language fragmentation at the source.

Other changes are already visible in shipping systems and open research projects. Vocabularies keep growing, from 50K to 128K and beyond, and multilingual coverage is becoming a headline feature. Researchers are exploring tokenizers that adapt to a domain or extend an existing vocabulary without full retraining. Cheaper long-context attention may eventually reduce the pressure to compress text aggressively. Each of these paths trades engineering complexity against the simplicity of a lookup table. Progress is likely to be gradual because the ecosystem of weights, tools, and benchmarks is built around subword tokens.

For practitioners, the near-term advice is simple and refreshingly practical. Subword tokenizers will remain the default for years, so learn how your model’s tokenizer behaves and measure it on your own data. Plan budgets in tokens, and track fertility for every language you serve. Stay alert to byte-level and patch-based models, since they may change cost structures quickly if they mature. The ideas in reducing LLM inference costs will carry over regardless of which tokenization scheme wins. Understanding byte pair encoding today is still the best foundation for whatever replaces it, which is why people keep asking what is tokenization in LLMs explained byte pair encoding?

Key Insights

• The Llama 3 tokenizer carries a 128K-token vocabulary and produces up to 15 percent fewer tokens than Llama 2, which helped keep inference speed roughly level.
• GPT-2's byte-level BPE vocabulary has 50,257 entries made of 256 byte tokens, 50,000 merges, and one end-of-text token, so no input becomes unknown.
• In OpenAI's launch examples for GPT-4o, Gujarati fell from 145 to 33 tokens for one sentence, a 4.4 times reduction that shows how tokenizer design drives multilingual cost.
• A study of ten LLMs on AfriMMLU found that each extra token per word lowered accuracy by roughly 8 to 18 percentage points across 16 African languages.
• Of the 50,257 tokens in GPT-2 and GPT-3 vocabularies, researchers flagged 374 as problematic after filtering, showing how undertrained entries create glitch behavior.
• OpenAI says one token is roughly four characters of English text, or about 0.75 words, so 100 tokens cover roughly 75 words.
• Subword models beat a back-off dictionary baseline by 1.1 BLEU on English to German and 1.3 BLEU on English to Russian in the original WMT 15 experiments.
• OpenAI reports that its tiktoken library runs three to six times faster than a comparable open source tokeniser in a one gigabyte GPT-2 benchmark.

Taken together, these figures show that tokenization is a design decision with consequences far beyond preprocessing. Byte pair encoding offers a robust compromise, with a vocabulary that is compact, learnable, and free of unknown inputs when it works on bytes. Larger vocabularies such as the 128K entries in Llama 3 compress text better, but they enlarge the embedding tables and leave rare entries undertrained. The same fragmentation that inflates costs for Gujarati or Hindi also lowers accuracy for African languages in controlled studies. Glitch tokens and tokenizer-based bypasses show that the vocabulary is part of the security and reliability surface. Practitioners who measure tokens per word on their own data, and who watch newer byte-level research, will make better choices than those who treat the tokenizer as a black box.

Side-by-Side Comparison of Tokenization Approaches

Stepping back from individual algorithms, a side-by-side view shows how the main approaches differ on the dimensions that matter in production. The table below compares word-level, character-level, byte-level BPE, WordPiece, and Unigram models across eight dimensions. No single approach wins everywhere, because each one trades vocabulary size, sequence length, and robustness differently. Teams that already manage large context windows should read it alongside context engineering for LLM agents. Many of the rows also connect to the broader natural language processing challenges and fixes that practitioners face. Use the table as a starting point for your own measurements, since the right answer depends on your languages and data.

Dimension	Word-level	Character-level	Byte-level BPE	WordPiece	Unigram (SentencePiece)
Unit of vocabulary	Whole words	Single characters	Byte sequences built by merging	Subword pieces chosen by score	Subword pieces kept after pruning
Unknown input handling	Replaced by an unknown token	Rare characters can be unknown	Never unknown, falls back to bytes	Unknown token for unseen characters	Unknown token unless byte fallback is on
Typical sequence length	Short	Very long	Moderate	Moderate	Moderate
Typical vocabulary size	Hundreds of thousands	Tens to thousands	Roughly 50K to 200K	Tens of thousands	Tens of thousands
How entries are chosen	Word frequency counts	Fixed alphabet	Most frequent adjacent pair	Pair frequency relative to its parts	Tokens whose removal hurts loss least
Segmentation at inference	Dictionary lookup	One symbol per character	Ordered merge rules replayed	Learned subword rules	Highest-probability split
Multilingual behavior	Weak for rich morphology	Works but costs many tokens	Uneven, rare scripts cost more tokens	Depends on training data	Language independent via raw-text training
Example models or tools	Early NLP systems	Small research models	GPT-2, GPT-4, tiktoken	BERT, DistilBERT, Electra	Llama and Mistral via SentencePiece

Tokenization in Practice: Three Production Tokenizers

GPT-2's Byte-Level BPE Tokenizer

Among the most influential production tokenizers, GPT-2's byte-level BPE started from a base vocabulary of 256 bytes instead of the more than 130,000 symbols needed for Unicode code points. OpenAI trained the vocabulary on its WebText corpus and ended with 50,257 entries, made of 256 byte tokens, 50,000 merges, and one end-of-text token. Merges were blocked across character categories, so the GPT-2 paper explains that fragments such as "dog." and "dog!" no longer waste separate slots. The measurable result was a reduction of more than 99 percent in the base alphabet compared with a code-point alphabet, with no unknown tokens at all. The design handled arbitrary input, which let the model be evaluated on any dataset regardless of preprocessing. Its limit appeared later, when researchers found hundreds of undertrained glitch tokens and noted that rare scripts still cost many byte tokens per character.

Meta's Llama 3 128K-Token Tokenizer

Meta rolled out a tokenizer with a 128K-token vocabulary for Llama 3, which it describes as more efficient than the one used by Llama 2. The company reports that the new tokenizer produces up to 15 percent fewer tokens for the same text. Meta also says this efficiency helped keep inference speed roughly on par with Llama 2 7B, even though Llama 3 8B has about one billion more parameters. The model was pretrained on over 15 trillion tokens drawn from publicly available sources, so every saved token is multiplied across an enormous corpus. Meta's announcement presents these figures as vendor measurements rather than independent benchmarks. A larger table is still a trade-off, since it enlarges the embedding and output layers and gives rare entries fewer training examples.

OpenAI's GPT-4o Multilingual Tokenizer

OpenAI rolled out a new tokenizer with GPT-4o and published token counts for one greeting sentence in 20 languages. Gujarati dropped from 145 tokens to 33, a 4.4 times reduction, while Telugu fell from 159 to 45 and Hindi from 90 to 31. Western European languages gained far less, with Spanish, Portuguese, and French each improving by only about 1.1 times. English moved from 27 tokens to 24, so the largest savings went to scripts that were previously fragmented. For a user writing in Gujarati, that reduction means cheaper prompts, faster replies, and more room in the context window. The comparison has a clear limit, because it uses a single sentence per language rather than a large corpus. The announcement also does not detail how the new vocabulary was built, so outsiders cannot reproduce the result.

Lessons From Case Studies in Tokenization Failures and Fixes

Case Study: SolidGoldMagikarp and the Glitch Token Mystery

Despite the polish of GPT-3 era chat models, researchers faced a problem when they noticed bizarre behavior on a handful of strange strings. Asked to repeat a token such as SolidGoldMagikarp, the models evaded the request, hallucinated, insulted the user, or produced unrelated words. ChatGPT, for instance, answered with the word "distribute" instead of echoing the string. The team faced a puzzle because the tokens existed in the vocabulary, yet the models seemed never to have learned them. To investigate, they developed prompt-based probes and analyzed embeddings, looking for tokens that sit unusually close to the centroid of the embedding space. The LessWrong investigation filtered the 50,257-token vocabulary in two passes.

The impact was a list of 374 problematic tokens, about 0.74 percent of the vocabulary, divided into 133 truly weird and 241 merely confused entries. The authors hypothesized, without confirming it, that the tokenizer had been built from scraped web text containing e-commerce backends, Reddit threads, and gaming logs. The model's training data was more heavily curated, so those strings appeared rarely or never during learning. As a result their embeddings stayed near the center of the space, and the model did not know what to do with them. The explanation is still a hypothesis, and the effect was documented mainly in specific GPT-2 and GPT-3 family models. The episode left a lasting lesson that vocabularies and training data should be audited together.

Case Study: The Token Tax on African Languages

A research team studying multilingual models faced a simple but unsettling problem, which was that some languages need far more tokens per word than others. Their paper, The Token Tax, evaluated 10 LLMs on AfriMMLU, a benchmark of 9,000 multiple-choice questions across 5 subjects and 16 African languages. The researchers measured fertility, meaning tokens per word, and used mixed-effects regressions to relate it to accuracy. Each additional token per word was associated with a drop of roughly 8 to 18 percentage points in accuracy. Fertility explained between 20 and 50 percent of the variance in accuracy in their regressions. African languages trailed English by about 25 accuracy points on average.

The authors then built a cost argument on top of these measurements. Because transformer cost scales quadratically with sequence length, doubling tokens could quadruple training cost and time, which they illustrate with a Llama-3.1-405B example rising from $105 million to $420 million. They also found that reasoning models such as DeepSeek and o1 outperformed non-reasoning peers by 8 to 12 points on African languages. These cost figures come from what the authors call a thought exercise with assumed token counts, not from measured training runs. The study is correlational, so fertility may stand in for other factors such as limited training data in those languages. The central concern, that identical content costs and performs differently by language, is still well supported.

Case Study: Subword Units for Rare Words in Machine Translation

In 2015, neural machine translation systems faced a bottleneck because they relied on a fixed vocabulary even though translation is open-ended. Rare and unknown words were handled by falling back to a dictionary, which broke the end-to-end learning loop. Sennrich, Haddow and Birch proposed a simpler solution, splitting rare words into subword units that the network could translate piece by piece. They compared character n-gram segmentation with a method based on the BPE compression algorithm on the WMT 15 English-German and English-Russian tasks. Subword models beat the back-off dictionary baseline by a lift of 1.1 BLEU for English to German. The original arXiv paper reports a similar gain of 1.3 BLEU for English to Russian. The abstract reports these gains for subword models in general, so it does not isolate the contribution of BPE alone. Frequency-driven splits can also cut across meaningful word parts, which remains a drawback of the approach today.

Common Questions About Tokenization in LLMs and Byte Pair Encoding