Embedding

Definition

A numerical representation of text, images, or other data that AI models can process and compare.

Why It Matters

Comparing text by exact match misses anything that paraphrases. Embeddings turn each chunk of text into a fixed-length vector where semantic similarity becomes geometric distance. That's the foundation for search, recommendation, RAG, deduplication and clustering of any unstructured data.

Key Points

• Leading embedding models (E5-large, BGE-M3, text-embedding-3-large) output 768–3072-dimensional float vectors.
• Cosine similarity is the standard distance metric; dot product is equivalent when vectors are L2-normalised to unit length.
• Chunking strategy matters: 256–512 tokens with ~20 % overlap balances recall coverage and precision in retrieval.
• MTEB (Massive Text Embedding Benchmark) is the standard comparison for retrieval tasks, always check the retrieval subtask, not just the average score.
• Bi-encoders (e.g. Sentence-BERT) are fast but less accurate; cross-encoders are accurate but slow, reranking combines both by using a cross-encoder on the top-K bi-encoder results.

Example

The sentences "I bought a car" and "I purchased an automobile" get embeddings that point in nearly the same direction in 768-dimensional space (cosine similarity ~0.95) even though they share only one word. A keyword search would miss the match.

Related Terms

• RAG (Retrieval-Augmented Generation)A technique where AI retrieves relevant documents before generating a response, improving accuracy.
• NLP (Natural Language Processing)The field of AI focused on understanding and generating human language.
• TransformerThe neural network architecture behind modern AI models. Introduced in the 2017 paper "Attention Is All You Need."

Quick Facts