What is embedding?

Embeddings are numerical vector representations of data—such as words, sentences, images, users, or products—designed so that items with similar meaning or characteristics are located closer together in a vector space.

For example, a text embedding model might represent:

"dog"  → [0.21, -0.48, 0.73, ...]
"puppy" → [0.24, -0.44, 0.70, ...]
"database" → [-0.61, 0.18, -0.27, ...]

The real vectors usually contain hundreds or thousands of dimensions. The important property is not an individual number; it is the relative position of vectors.

How embeddings are created

An embedding model learns a function:

input → embedding model → vector

During training, the model learns representations that capture useful patterns in its training objective. In language models, token embeddings are learned parameters that map token IDs to vectors. Separate embedding models can also convert complete sentences or documents into vectors intended for semantic comparison.

How similarity is measured

A common metric is cosine similarity, which compares the direction of two vectors:

cosine_similarity(A, B) = (A · B) / (||A|| ||B||)

A higher cosine similarity generally indicates that two vectors are more similar according to the embedding model.

A simple Python example:

PYTHON
import numpy as np

def cosine_similarity(a, b):
    a = np.array(a)
    b = np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

query = [0.2, 0.7, -0.1]
document = [0.18, 0.73, -0.08]

print(cosine_similarity(query, document))

In a production system, an embedding model generates these vectors rather than developers manually defining them.

Why embeddings are useful

Embeddings make unstructured information searchable and comparable mathematically. Common applications include:

• Semantic search: retrieve documents based on meaning rather than exact keyword matches.
• RAG: embed documents and queries, retrieve relevant chunks, then provide them to an LLM.
• Recommendation systems: represent users and items so similar preferences can be matched.
• Clustering: group semantically related documents or products.
• Classification: use embeddings as features for downstream models.
• Duplicate detection: identify text or other content with very similar representations.

Embeddings in RAG

A typical retrieval workflow is:

Documents
   ↓
Split into chunks
   ↓
Embedding model
   ↓
Vectors + metadata
   ↓
Vector database

User question
   ↓
Embedding model
   ↓
Query vector
   ↓
Similarity search
   ↓
Relevant chunks
   ↓
LLM
   ↓
Answer

The query and stored documents should normally be embedded using compatible representations from the same embedding model or model family intended for that retrieval task.

Embeddings vs. tokens

Tokens and embeddings are related but different.

A token is a discrete unit produced by tokenization. An embedding is a continuous numerical vector representing a token or larger piece of information.

For example:

"Transformers are powerful"
        ↓ tokenization
["Transform", "ers", " are", " powerful"]
        ↓ embedding lookup / model
vectors representing those tokens

In short, embeddings turn information into vectors that machine-learning systems can compare and process mathematically while preserving useful learned relationships.