Building a Retrieval-Augmented Generation (RAG) system allows Large Language Models (LLMs) to answer questions based on your private enterprise documents without re-training the model.

At the core of every RAG system is a **Vector Database** (or extension like PostgreSQL pgvector) that performs semantic similarity searches over high-dimensional embeddings. Here is how vector distance calculations and indexing work.

1. What is a Vector Embedding?

An embedding model converts text chunks into floating-point numerical arrays (e.g. 1,536 dimensions). In this multidimensional space, words or passages with similar meanings end up close together.

Unlike relational SQL databases that use exact string equality (WHERE title = 'Docker'), vector databases measure geometric distance between vector coordinates.

2. Similarity Distance Metrics

When comparing a user query vector against millions of document vectors, three primary metrics are used:

  • Cosine Distance: Measures the angle between two vectors, ignoring vector length. Perfect for text semantics where document chunk length varies.
  • Dot Product: Multiplies corresponding vector elements. Fast when vectors are normalized to unit length.
  • Euclidean Distance (L2): Measures straight-line distance between two point coordinates.

3. HNSW Indexing for Scale

Calculating exact distance against 1 million 1536-dimensional vectors takes several seconds using brute-force \(O(N)\) linear scans.

To achieve sub-50ms queries, vector databases construct a Hierarchical Navigable Small World (HNSW) graph index. HNSW links vectors into multi-layer graph structures: upper layers quickly skip across wide vector regions, while lower layers refine search to nearest neighbors, executing in \(O(\log N)\) time.

4. PostgreSQL pgvector Implementation

You don't always need a separate vector database. If you already run PostgreSQL, the pgvector extension adds native vector columns and HNSW indexes:

-- 1. Enable pgvector extension CREATE EXTENSION IF NOT EXISTS vector; -- 2. Create table storing 1536-dim embeddings CREATE TABLE tech_documents ( id BIGSERIAL PRIMARY KEY, title TEXT NOT NULL, content TEXT NOT NULL, embedding vector(1536) ); -- 3. Create HNSW Cosine Index CREATE INDEX ON tech_documents USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64); -- 4. Retrieve Top 3 Relevant Chunks for a Query SELECT title, content, 1 - (embedding <=> '[0.012, -0.045, ...]') AS similarity FROM tech_documents ORDER BY embedding <=> '[0.012, -0.045, ...]' LIMIT 3;

5. Key RAG Production Advice

  1. Chunk Size: Use 500-token chunks with 50-token overlap to maintain contextual continuity.
  2. Hybrid Search: Combine vector cosine search with PostgreSQL full-text keyword search (BM25) for best retrieval accuracy.