How tokenization works in LLMs

If you follow AI topics, you’ve probably come across tokenization. Depending on the context, it can mean data security, blockchain assets or text analysis. Here, we’ll focus on tokenization in Large Language Models (LLMs).

What is tokenization?

Tokenization breaks human language into smaller, more manageable units called tokens.

Every input sent to an LLM passes through a tokenizer. It turns the input into a sequence of tokens, representing words with numbers because LLMs can only process numerical data.

The tokenization process

There are three core steps:

Input data: The raw text is handled before it reaches the model. The nature of that text determines how it should be divided into tokens.

Transformers: This is the model architecture that processes tokenized input. Tokenization puts the input into the right format so the model can process it effectively.

Prediction output: The model generates predictions, but the quality of its output is influenced by how well the input was tokenized. Good tokenization preserves the original text’s meaning and structure, helping the model produce more accurate, coherent responses.

Types of tokenization

Text can be tokenized in three main ways:

• Word based: Each word becomes an individual token.
• Character based: Each character becomes a token.
• Subword based: Words are divided into subword units rather than kept whole or split into individual characters.

What is a subword?

Subwords are smaller, meaningful units within a word.

For example: unstoppable → un + stoppable

This helps a model understand and generate variations of words it may never have encountered before. That ability is crucial during training, particularly with large, diverse datasets.

LLMs primarily use subword based tokenization because it balances vocabulary size with contextual understanding.

Disadvantages of word based tokenization

Repetition of similar words: LLMs struggle to connect words with the same root. For example, “run,” “running,” and “runner” are treated as separate tokens without recognizing their relationship.

Filler words: Common words such as “and,” “or,” “the,” and “is” occupy token space without adding significant meaning to a sentence.

Out-of-vocabulary issues: Models struggle with words they did not encounter during training, including spelling mistakes and new terminology.

Disadvantages of character based tokenization

Small vocabulary, long sequences: A small vocabulary comes with much longer token sequences, making processing inefficient.

Lack of meaning: Individual characters carry no semantic meaning on their own, making context and relationships harder for models to understand.

Loss of language structure: This approach misses important patterns and word structures needed for language understanding.

How do we decide where to split words?

Byte Pair Encoding (BPE) is a popular approach. It merges frequently occurring character pairs or subwords to reduce the overall token count while retaining meaning. Its efficiency makes it widely used in natural language processing systems.

Tokens and pricing

Using AI comes at a cost, at least when you need advanced features.

Tokenization directly affects LLM pricing. Rather than paying a fixed amount for every interaction, you pay according to the number of tokens you use.

API customers typically encounter two types of pricing:

• Input pricing: The cost of tokens in your prompts and queries.
• Output pricing: The cost of tokens in the model’s response.

Providers often list prices per 1,000 or 1 million tokens, helping you estimate the cost.

Token efficiency: How many tokens for the same sentence?

Consider the sentence: “Artificial intelligence is transforming the world.”

• GPT-4: 8 tokens
• Claude: 8 tokens (estimated)
• Mistral: 9 tokens
• LLaMA 3: 10 tokens

Pro tip: Tools such as ChatGPT’s tokenizer page show how many tokens your text contains and their corresponding token IDs.

Why tokenization matters

Anyone working with LLMs benefits from understanding tokenization because it affects:

• Cost management: Token counts directly influence API costs.
• Performance optimization: Efficient tokenization supports better model performance.
• Prompt engineering: Understanding how text is tokenized helps you write more effective prompts.

Tokenization may seem technical, but it is fundamental to understanding how LLMs interpret and process language. At DataNorth, we empower businesses to use and implement artificial intelligence in their operations.

Curious how AI can optimize and facilitate your processes? Get in touch with us to explore the possibilities!

Originally published by Nick Moesker, DataNorth*, on* 5 June 2025*:* Tokenization for LLMs.