Tokenisation — the unsung hero of NLP

📅 Day 50. Halfway through the sprint. Welcome to NLP and transformers week.

📝 Let's start with the layer no one talks about but everyone uses — tokenisation.

A tokenizer turns text into a list of integer IDs that the model can process. Modern tokenizers (BPE, WordPiece, SentencePiece) split text into subword units, NOT whole words.

🔤 Examples:
→ 'tokenization' might be ['token', 'ization'] — two tokens.
→ 'unbelievable' might be ['un', 'bel', 'ievable'] — three tokens.
→ 'hello world' is usually ['hello', ' world'] (the leading space is part of the token).
→ Numbers and punctuation each become tokens.
→ Emojis often become 2-4 tokens because they're encoded in multi-byte Unicode.

💰 Why this matters in practice — APIs charge by token, not by word.

A 1000-word English blog post is roughly 1300 tokens. A 1000-character code snippet is roughly 250-400 tokens (code is denser). A page of dense math symbols is hundreds of tokens for a few lines.

GPT-4-class models cost a few dollars per million tokens. For high-volume apps, token efficiency matters.

🌐 Tokenization is also language-dependent. English averages ~1.3 tokens per word. Chinese, Japanese, Korean often hit 1-2 tokens per character (much higher per equivalent meaning). Multi-lingual apps need to budget for this.

🎯 The takeaway:
→ Tokens ≠ words. The mental model 'each word is one token' is wrong.
→ Token counts are language-specific and content-specific.
→ Always count tokens before sending to APIs to estimate cost and check context limits.

💡 We'll write code to count tokens later in the day. For now — internalise that 'token' is the model's unit, not the human's.

🧮 Byte-Pair Encoding (BPE) is the most common subword tokenisation algorithm in 2026. The whole idea fits in four sentences.

1️⃣ Start with the corpus split into characters. Each character is a token.

2️⃣ Find the most-frequent adjacent pair of tokens. Merge them into a new combined token.

3️⃣ Repeat the merge step until the vocabulary reaches the desired size (typically 32k-100k tokens for modern LLMs).

4️⃣ The result — a vocabulary that covers common words as single tokens and rare words as combinations of subwords.

🎯 Why BPE wins:

→ No 'unknown word' problem. Any string is decomposable, even names you've never seen, even random gibberish, even other languages. The worst case is per-character tokenisation; never an outright failure.

→ Frequent words stay efficient. 'the', 'and', 'is' are single tokens. Common bigrams like 'New York' might also be a single token if frequent enough in training.

→ Rare words decompose meaningfully. 'unhappiness' might split as ['un', 'happiness'], retaining morphological information.

→ Cross-language. A multilingual BPE trained on many languages handles all of them. The model implicitly learns shared subwords (Latin alphabet shows up in many languages).

📦 Variants:
→ WordPiece (BERT) — similar to BPE, slightly different merge criterion (likelihood-based, not frequency-based).
→ SentencePiece (Llama, T5) — operates on raw bytes/Unicode, no whitespace pre-tokenisation. More language-agnostic.
→ Byte-level BPE (GPT-2 onwards) — tokens are byte sequences. Handles any string at byte level. The standard in modern LLMs.

🏆 GPT, Llama, Claude — all variants of BPE.

💡 You don't usually train your own tokenizer. You use the pretrained one that came with your model. But understanding BPE makes you a better LLM user.

💻 transformers ships with the tokenizer for every modern LLM. Three lines from import to encoded text.

Look at the snippet.

📦 Line 1 — import AutoTokenizer.

🔧 Line 2 — load by model name. AutoTokenizer.from_pretrained('gpt2'). Downloads the tokenizer (small, fast). Same API for every model — gpt2, llama-3, mistral, claude (via Anthropic SDK), gemini (via Google SDK).

🔢 Line 3 — encode text into integer IDs. tok.encode('Saurav Danej · 90 days of AI/ML in public.') returns a list like [27817, 526, 65, ...].

📝 Line 4 — decode IDs back to text. tok.decode(ids) returns the original string.

For production use, the typical call is:

result = tok(text, padding=True, truncation=True, return_tensors='pt')

This returns a dict with input_ids (the integer IDs), attention_mask (1 for real tokens, 0 for padding), and possibly token_type_ids for some models. Pad and truncate to a fixed length; convert to PyTorch tensors. Ready to feed into the model.

🛠 Common kwargs:
→ max_length=512 — truncate to this many tokens.
→ padding='max_length' — pad to max_length.
→ truncation=True — truncate longer inputs.
→ return_tensors='pt' for PyTorch, 'tf' for TensorFlow.

⚠️ Important — when fine-tuning a model, always save the tokenizer alongside it. The tokenizer must match the model exactly. Different tokenizers produce different IDs for the same text; the model only understands its own.

💾 Save with tok.save_pretrained('my-model/'). Load alongside the model.

💡 The HuggingFace tokenizer API is one of the cleanest in NLP. Three lines to encode anything. Use it.

💡 Pro tip — count your tokens BEFORE sending to an LLM API. Two reasons: cost and context limits.

💰 Cost. Major LLM APIs charge per token. GPT-4 class models are roughly $5-15 per million input tokens. For high-volume applications, token estimation is the difference between $50/month and $5000/month in production.

🚧 Context limits. Every model has a maximum context (input + output) it can handle. GPT-4o is 128k tokens. Claude 3.5 Sonnet is 200k. Gemini Pro 2 is up to 2M. If you exceed the limit, the API returns an error — you won't get a partial response, you'll get rejected.

Accidentally sending a 200k-token document to a 128k-token-limit model fails immediately. You discover this in production at 2am.

🛠 The tools:

→ tiktoken (OpenAI). pip install tiktoken. Then tokenizer = tiktoken.encoding_for_model('gpt-4o') and len(tokenizer.encode(text)) gives you the count. Fast — pure Rust under the hood.

→ AutoTokenizer (HuggingFace). For open models. len(tok.encode(text)).

→ anthropic.beta.messages.count_tokens — Anthropic's official token counter for Claude.

→ Google's count_tokens method on the GenerativeModel object for Gemini.

📋 Pre-flight check pattern:

token_count = count_tokens(prompt)
if token_count > MAX_TOKENS - response_buffer:
    truncate_or_chunk(prompt)
else:
    send_to_api(prompt)

🎯 For RAG specifically — pre-flight checks save you from the 'I retrieved 20 chunks and the prompt is too big' bug. Truncate to the most-relevant N chunks before sending.

💡 Tokens > characters > words for sizing prompts. Get the unit right. Estimate before you spend.

📅 End of Day 50.

✅ Recap:

📝 Tokens are not words. BPE/WordPiece subword tokenisation gives ~1.3 tokens per English word. Languages and content type vary.

🧮 BPE in 4 sentences. Start from characters; merge frequent pairs; repeat. Vocabulary size around 32k-100k for modern LLMs. No unknown words.

💻 HuggingFace AutoTokenizer in 3 lines. Same API across all transformer models. Save the tokenizer alongside the model when fine-tuning.

💰 Always count tokens before API calls. tiktoken for OpenAI, AutoTokenizer for open models, official counters for Anthropic and Google. Budget by token, not by character or word.

🧠 Reflection — tokenisation is the most under-discussed part of LLM engineering. Token efficiency directly affects cost and latency. Understanding the model's actual unit (token, not word) makes everything downstream — context limits, API pricing, prompt design — clearer.

🚀 Tomorrow, Day 51 — embeddings. Coordinates of meaning. The bridge from token IDs to dense vectors that capture semantic similarity. The MTEB leaderboard and how to pick an embedding model.

💼 Halfway through the sprint. The next four weeks are the modern AI stack — NLP, transformers, RAG, agents.

👋 See you in the morning.