Standard relational B-Tree indexes excel at exact key lookups (e.g. WHERE user_id = 42), but scale poorly for full-text search over millions of unstructured text documents. Executing SQL wildcard queries like LIKE '%kubernetes%' forces full table scans that take seconds or minutes.

**Elasticsearch** is a distributed real-time search engine built on top of **Apache Lucene**. It provides near-instantaneous full-text search by organizing data into immutable **Inverted Indexes**, compressed using **Finite State Transducers (FST)**, and ranked via the **Okapi BM25** relevance scoring algorithm.

1. The Anatomy of an Inverted Index

Instead of mapping Document -> Words like a traditional database table, an Inverted Index maps Word -> List of Document IDs (Posting List):

Term (Tokenized) Posting List (Doc IDs + Positions + Frequencies)
kubernetes Doc 1 (tf: 3, pos: [4, 18, 92]), Doc 4 (tf: 1, pos: [12])
docker Doc 1 (tf: 1, pos: [5]), Doc 2 (tf: 4, pos: [2, 10, 44, 88])
ebpf Doc 3 (tf: 2, pos: [1, 15])

Finite State Transducers (FST) Memory Compression

Because term dictionaries contain millions of unique words, storing them in RAM is expensive. Lucene compresses term dictionaries into a **Finite State Transducer (FST)** in memory. An FST is a directed acyclic graph that shares common prefixes and suffixes across dictionary words (such as cat, cats, catering), requiring minimal RAM while supporting fast prefix and fuzzy searches.

2. Mathematical Mechanics of Okapi BM25 Scoring

When a search query matches multiple documents, Elasticsearch ranks results using the **Okapi BM25** algorithm:

$$\text{Score}(D, Q) = \sum_{i=1}^{n} \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}$$

BM25 builds upon traditional TF-IDF by adding two crucial parameters:

  • Term Frequency Saturation ($k_1 \approx 1.2$): Prevents a document that repeats a term 100 times from scoring 100x higher than a document that mentions it 3 times. As term frequency $f(q_i, D)$ increases, score contribution quickly asymptotically approaches a plateau limit.
  • Document Length Normalization ($b \approx 0.75$): Penalizes long documents (such as 50-page PDFs) that mention a search term incidentally compared to concise, focused articles where the term occupies a large percentage of total text depth.

3. Elasticsearch Cluster & Shard Architecture

An Elasticsearch **Index** is a logical namespace composed of one or more physical Lucene **Shards**:

  • Primary Shards: Each document written to an index is assigned to a specific primary shard using consistent hashing: shard = hash(routing_id) % number_of_primary_shards.
  • Replica Shards: Synchronously or asynchronously copy primary shard data to provide read throughput and failover redundancy if a node dies.
  • Immutable Segments: A Lucene shard consists of immutable log segments. New documents buffer in memory (RAM buffer) and periodically flush to disk as new immutable segments every second (providing near real-time searchability).

4. Production Elasticsearch Query DSL Example

Below is a production JSON Query DSL request combining full-text BM25 matching, term filtering, and aggregations:

POST /engineering_articles/_search { "query": { "bool": { "must": [ { "match": { "content": { "query": "kubernetes architecture etcd", "fuzziness": "AUTO", "operator": "and" } } } ], "filter": [ { "term": { "category": "Cloud Infrastructure" } }, { "range": { "reading_time_mins": { "gte": 10 } } } ] } }, "aggs": { "topics_breakdown": { "terms": { "field": "tags.keyword", "size": 10 } } }, "_source": ["title", "slug", "author"], "highlight": { "fields": { "content": { "number_of_fragments": 2 } } } }

5. Operational Performance Tuning Guidelines

  • Set JVM Heap Size to Max 32GB: Never allocate more than 31-32GB of RAM to Elasticsearch JVM heap. Staying below 32GB allows JVM to use 32-bit Compressed Object Pointers (Compressed OOPs), doubling pointer density in CPU cache.
  • Leave 50% RAM for OS Page Cache: If a node has 64GB RAM, allocate 30GB to JVM heap and reserve the remaining 34GB for Linux OS Page Cache so Lucene segment files stay cached in memory.