Elasticsearch Optimization: Lucene Architecture, Index Design, and Performance Tuning

Introduction: The data platform has evolved through three versions, encountering many common challenges; this article shares finalized documentation focusing on Elasticsearch (ES) optimization.

Requirement: The system must support cross‑month queries, retain over a year of historical data, and return query results within seconds despite daily tables containing billions of rows.

ES Retrieval Principles: An overview of ES and Lucene’s basic structure, introducing concepts such as Cluster, Node, Index, Type, Document, Shards, Replicas, and how Lucene underpins indexing and searching.

Lucene Index Implementation: Description of Lucene’s file structures—dictionary, posting lists, stored fields, DocValues—and their impact on storage size and random‑read performance.

Shard Allocation: Explanation of routing logic (shard = hash(routing) % number_of_primary_shards) and how explicit _routing can concentrate related data on the same shard to reduce search load.

Optimization Cases: Practical measures including bulk writes, multi‑threaded ingestion, increasing refresh_interval (e.g., "-1" during bulk load), allocating ~50% of node memory for Lucene cache, using SSDs, custom ID strategies, segment merge throttling, and configuring merge thread counts.

Search Performance Tuning: Recommendations to disable unnecessary doc values, prefer keyword over numeric range fields, turn off _source storage for unused fields, use filter queries instead of scoring, and adopt pagination strategies (from+size, search_after, scroll) to avoid costly deep pagination.

Performance Testing: Benchmark plans covering single‑node (50M‑100M docs) and cluster (1B‑3B docs) scenarios, measuring disk I/O, memory, CPU, network usage, and comparing SSD versus HDD performance.

Production Results: After applying the optimizations, the platform reliably serves billions of records with 100‑result queries completing in under 3 seconds, and pagination remaining fast.

Configuration Example:

{ "mappings": { "data": { "dynamic": "false", "_source": { "includes": ["XXX"] }, "properties": { "state": { "type": "keyword", "doc_values": false }, "b": { "type": "long" } } } }, "settings": { ... } }