Elasticsearch Optimization and Performance Tuning for Billion‑Scale Data

The data platform has undergone three major versions, encountering many common challenges; this article consolidates the refined documentation, focusing on Elasticsearch (ES) optimization to help readers avoid pitfalls.

Background: a business system generates over a hundred million rows per day per table, partitioned by day, but only three months of data can be retained in the DB; the goal is to enable cross‑month queries, retain over a year of history, and achieve second‑level query latency.

ES retrieval principles are introduced, describing core concepts such as Cluster, Node, Index, Type, Document, Shards, Replicas, and the underlying Lucene architecture that powers ES indexing and search.

Lucene’s index structure consists of dictionaries, posting lists, stored fields, and DocValues; the article highlights the cost of random disk reads for .fdt, .tim, and .doc files and the impact of scoring on performance.

Optimization cases are divided into index‑side and search‑side improvements. Index‑side tips include bulk and multi‑threaded writes, increasing the refresh interval (e.g., "refresh_interval": "-1" and manual refresh), allocating ~50% of system memory to Lucene cache, using SSDs, custom ID generation aligned with HBase row keys, and tuning segment‑merge parameters (e.g., "indices.store.throttle.max_bytes_per_sec": "200mb"). Search‑side tips cover disabling unnecessary doc values, preferring keyword fields over numeric ranges, disabling _source storage for unused fields, using filter queries or constantScoreQuery to avoid scoring, and employing pagination strategies such as from+size, search_after, and scroll.

{
    "mappings": {
        "data": {
            "dynamic": "false",
            "_source": {
                "includes": ["XXX"]
            },
            "properties": {
                "state": {
                    "type": "keyword",
                    "doc_values": false
                },
                "b": {
                    "type": "long"
                }
            }
        }
    },
    "settings": {......}
}

Performance testing includes single‑node tests with 50‑100 million records, cluster tests up to 3 billion records, random query combinations, and SSD versus HDD comparisons, emphasizing the need for baseline benchmarks.

In production, the platform now handles billions of records with 100‑row queries returning within 3 seconds, and further scaling can be achieved by adding nodes to distribute load.