Word embeddings — meaning as a vector

📅 Day 51. Embeddings might be the most important concept in modern NLP. They turn discrete things (words, sentences, images, even users) into dense vectors where similar things have nearby vectors.

🧭 An embedding is a function — input X (text, image, anything) → vector in some d-dimensional space. The function is learned from data. The space is shaped so that semantically similar inputs end up close together.

🎯 Word2Vec (2013) famously demonstrated this with vector arithmetic:

vector('king') - vector('man') + vector('woman') ≈ vector('queen')

The vector difference between 'king' and 'man' captures the concept of 'royalty without gender'. Add it to 'woman' → you get 'queen'. The embedding space encodes meaning structurally.

🌍 Modern embeddings (OpenAI text-embedding-3, BAAI/bge, jina-embeddings, voyage-3) are trained on web-scale text with contrastive learning. They don't just map words; they map sentences, paragraphs, documents — entire chunks of meaning into single vectors.

💡 Where embeddings power 2026 AI:

→ Semantic search. Embed query and documents; return the most similar by cosine. Better than keyword search for natural-language queries.

→ RAG retrieval. The R in RAG. Find relevant chunks for a question by embedding both and matching.

→ Clustering and topic modelling. Embed documents; cluster the vectors; each cluster is a topic.

→ Recommendation systems. Embed users and items; recommend items with vectors close to the user's.

→ Anomaly detection. Embed events; outliers in embedding space are anomalies.

📊 Cross-modal embeddings. CLIP embeds both images and text into the same space. Search for images using text queries. The bridge between modalities.

🧠 If you understand embeddings, you understand half of LLM applications. Tomorrow we go into how to pick a model. Wednesday we put it into a RAG system.

🤔 Choosing the right embedding model matters more than people realise. The same downstream RAG/search system can have wildly different quality depending on the embedding model. Three questions guide the choice.

💰 Question 1 — Cost vs control.

→ API embeddings (OpenAI text-embedding-3-small, Voyage AI, Cohere). Pay per token. Fast. No infrastructure. Quality is consistently strong. Best for prototyping and low-volume production.

→ Self-hosted embeddings (BAAI/bge, sentence-transformers). Free per inference (after compute cost). Requires GPU/CPU resources. Quality competitive with APIs for top models. Best for high-volume production or privacy-sensitive data.

My default — start with OpenAI text-embedding-3-small ($0.02/M tokens) for prototyping. Switch to bge-small or bge-m3 self-hosted when scale or privacy demands.

📐 Question 2 — Dimension.

→ 384 dim — bge-small. Fast retrieval, lower memory.
→ 768 dim — bge-base, sentence-transformers. Good balance.
→ 1024 dim — bge-large, voyage-3.
→ 1536 dim — OpenAI text-embedding-3-small (default).
→ 3072 dim — OpenAI text-embedding-3-large, optional dimensions.

Higher dim = more nuance + more memory + slower retrieval. For most retrieval tasks, 768-1024 is the sweet spot. Higher only helps when the data is genuinely complex.

Note: text-embedding-3 supports 'matryoshka' embeddings — train at high dim, truncate to lower dim at inference. Best of both worlds.

🎯 Question 3 — Domain.

→ General-purpose models for most cases.
→ Code-specific models (voyage-code-3) for codebases.
→ Multilingual models (bge-m3, jina-embeddings-v3) for non-English data.
→ Domain-specific (biomedical, legal) for niche content.

📋 The rule — always evaluate on YOUR data, not benchmark scores. A 'top MTEB' model might lose to a smaller model on your specific corpus. Build a 30-pair eval set; test 3 candidates; pick the winner.

💻 Free, runs on CPU or GPU, 5 lines from import to embedding. The fastest path to a working embedding setup for prototyping or self-hosted production.

Look at the snippet.

📦 Import sentence_transformers. The library is the easiest way to use BAAI/bge, sentence-transformers' own models, mxbai, and many others through a single API.

🔧 Load model. SentenceTransformer('BAAI/bge-small-en-v1.5'). Downloads from HuggingFace on first use. Subsequently cached locally.

The 'small' in the name means 384 dim. The 'large' variants (bge-large-en-v1.5) are 1024 dim. The trade-off is quality vs speed/memory.

📝 Define documents. A list of strings.

🔢 Encode. model.encode(docs, normalize_embeddings=True) returns a NumPy array of shape (n, dim). normalize_embeddings=True scales each vector to unit length, which makes cosine similarity equivalent to dot product downstream — saves a normalisation step at retrieval time.

📊 print(vecs.shape) confirms the result. (2, 384) means 2 documents, 384-dimensional each.

🚀 Now you can:
→ Compute similarity — np.dot(vec_a, vec_b) gives cosine similarity (because vectors are normalised).
→ Find nearest — sort by similarity, pick top-k.
→ Cluster — pass vectors to KMeans.
→ Persist — save to a vector database like Qdrant.

📋 For batch processing — pass a list of strings. The library batches internally for GPU efficiency. For 100k documents, this typically takes 30-60 seconds on a modest GPU.

💡 Performance tips:
→ Use a GPU if available. CPU is fine for prototyping (a few documents per second); GPU is needed for production scale.
→ Set show_progress_bar=False to suppress the progress bar in scripts.
→ For very long documents, the model truncates at its max sequence length (typically 512 tokens). Chunk before embedding for long docs.

💡 Pro tip — embeddings should be unit-normalised before you do cosine similarity or store in a vector DB. Otherwise you have subtle bugs.

🧮 The math. Cosine similarity between vectors a and b is (a · b) / (||a|| * ||b||). The dot product divided by the product of magnitudes.

If both vectors are unit-normalised (||a|| = ||b|| = 1), the formula simplifies to just (a · b). Cosine = dot product. Saves a normalisation step at every comparison.

📊 If they're NOT normalised:
→ The cosine formula computes the right answer, but slowly (every comparison divides by the product of norms).
→ ANN libraries (Faiss, ScaNN) often assume unit norms. Without normalisation, they return the wrong neighbours.
→ Some vector DBs (Qdrant with COSINE distance) handle non-normalised inputs correctly. Others assume normalisation. Read your DB's docs.
→ Comparing similarity scores across different documents is harder when norms vary.

✅ The fix — normalise once, at encode time, before storage:

v = v / np.linalg.norm(v, axis=-1, keepdims=True)

NumPy does this in one line. After this, every comparison is just dot product (or matrix multiplication for batches), which is much faster.

📦 Most modern embedding APIs return normalised vectors by default — OpenAI text-embedding-3, voyage, jina. Some don't (some sentence-transformers models). Always check.

For sentence-transformers — pass normalize_embeddings=True to model.encode(). The flag is right there in the API.

For custom models — add the normalisation as a post-processing step in your encoding pipeline.

🎯 The cost — one line of code per encode call. The savings — correct retrieval, faster comparisons, compatible with all vector DBs.

💡 Always normalise. The bug from skipping is silent and ranking-distorting.

📅 End of Day 51.

✅ Recap:

🧭 Embeddings = learned coordinates of meaning. Vector arithmetic captures relationships. Modern embeddings power semantic search, RAG, recommendation, anomaly detection.

🤔 Pick by 3 questions — cost (API vs self-hosted), dim (768-1024 sweet spot), domain (general vs specific). Always evaluate on YOUR data.

💻 5-line local embedding with sentence-transformers + bge-small. Free, runs on CPU/GPU.

📐 Always normalise before cosine. Saves a step at every comparison; required by most ANN libraries; eliminates a class of subtle ranking bugs.

🧠 Reflection — embeddings are the underrated workhorse of modern AI. Every search bar that 'just understands what you meant' is using embeddings. Every recommendation system that 'gets your taste' is using embeddings. The technique itself is decades old (latent semantic analysis from the 90s); the modern realisation (contrastive learning + transformers) is what made them dominant.

🚀 Tomorrow, Day 52 — attention. Q, K, V — the three letters behind every transformer. The mechanism that ate ML.

💼 We're building toward transformers (Day 53), then BERT/GPT (54-55), then RAG (week 9). Each day builds on the last; you'll see the pattern.

👋 See you in the morning.