Embeddings Explained for Builders
Embeddings are the backbone of RAG - they turn text into numbers that capture meaning. Understanding how they work helps you make better decisions about chunking, model selection, and when retrieval fails. This guide covers what you need to know as a builder, without the mathematics.
What Is an Embedding?
An embedding is a list of numbers (a vector) that represents the meaning of a piece of text. An embedding model maps text to a point in high-dimensional space - similar texts map to nearby points; unrelated texts map to distant points.
"The dog chased the ball" and "A puppy ran after a sphere" will have similar embeddings because they mean similar things. "Quarterly earnings report" will have a very different embedding from either.
How RAG Retrieval Works
At index time: you split your documents into chunks, embed each chunk, and store (chunk text, embedding vector) in a vector database.
At query time: you embed the user's question and find the chunks whose embeddings are most similar to the query embedding. Those chunks are inserted into the LLM's context as relevant source material.
Similarity: Cosine Distance
The most common similarity measure is cosine similarity - the angle between two vectors. A score of 1.0 means identical direction (same meaning); 0 means perpendicular (unrelated); -1 means opposite meaning. Most retrieval systems return results sorted by cosine similarity descending.
Threshold caution
Cosine similarity scores vary by model - a "good match" might be 0.85 with one model and 0.72 with another. Always calibrate thresholds on your own data rather than using defaults from documentation.
Embedding Models (2025–2026)
| Model | Dimensions | Context | Notes |
|---|---|---|---|
| text-embedding-3-large (OpenAI) | 3072 | 8K tokens | Strong general-purpose; supports dimension reduction |
| text-embedding-3-small (OpenAI) | 1536 | 8K tokens | Lower cost, slightly lower quality - good starting point |
| embed-english-v3 (Cohere) | 1024 | 512 tokens | Excellent for hybrid RAG; native int8 compression |
| nomic-embed-text (local) | 768 | 8K tokens | Best local model; runs on CPU; 8K context is rare for open models |
| mxbai-embed-large (local) | 1024 | 512 tokens | Strong MTEB scores; good for Ollama local deployment |
Chunking Strategy
Chunking is how you split documents before embedding. It is the biggest variable in RAG quality that builders control.
Fixed-size chunking
Split every N tokens with M tokens of overlap. Simple. Works well for dense text. Overlap prevents facts from being cut mid-sentence. 256–512 tokens with 50–100 token overlap is a common starting point.
Semantic chunking
Split at semantic boundaries - paragraphs, section headers, sentences. More expensive but better-aligned chunks improve retrieval precision. LangChain and LlamaIndex both support semantic chunking.
Parent-child chunking
Embed small chunks (128 tokens) for precise retrieval, but return larger parent chunks (512 tokens) to the LLM for context. Small chunks match queries better; large chunks give the model enough context to answer correctly.
When to Fine-Tune Embeddings
Off-the-shelf embedding models work well for general text but struggle with domain- specific terminology (legal, medical, code, proprietary product names). Fine-tune embeddings when:
- Your domain vocabulary is not in standard training data
- Retrieval quality on your evaluation set is below threshold despite good chunking
- You have 500+ labelled (query, relevant passage) pairs available for training
Fine-tuning uses contrastive learning - making the model pull similar pairs together and push dissimilar pairs apart. OpenAI, Cohere, and local models via sentence-transformers all support embedding fine-tuning.