Elasticsearch Indexing and Search Performance Tuning Guide

First Part: Tuning Indexing Speed

Use bulk requests instead of single‑document indexing to achieve much higher performance; determine the optimal bulk size by benchmarking on a single‑node, single‑shard cluster, gradually increasing the batch size until indexing speed stabilizes, while avoiding requests that are too large to prevent memory pressure.

Send data from multiple workers or threads, as a single thread cannot fully utilize the cluster's indexing capacity; this also helps reduce the cost of each fsync.

Watch for TOO_MANY_REQUESTS (429) responses (EsRejectedExecutionException) and apply random exponential back‑off when the cluster cannot keep up.

Increase the index.refresh_interval (e.g., to 30s) to reduce segment creation frequency, and for large bulk loads temporarily set index.refresh_interval=-1 and index.number_of_replicas=0 to speed up indexing, restoring the original values after the load.

Prevent the OS from swapping the Elasticsearch JVM process, allocate roughly half of the physical memory to the filesystem cache, use auto‑generated document IDs to skip existence checks, and consider better hardware (SSD, local storage, more cache memory, faster CPUs).

Increase index.memory.index_buffer_size if the node is dedicated to heavy indexing, ensuring the buffer is large enough (up to 512 MB per shard) without exceeding a few tens of megabytes per request.

Disable the _field_names field if you do not need existence queries, and review the "tune for disk usage" guide for additional disk‑related optimizations.

Second Part: Tuning Search Speed

Allocate as much memory as possible to the filesystem cache (at least half of available RAM) because Elasticsearch heavily relies on it for fast search.

Use appropriate hardware: SSDs, local storage, and sufficient CPU resources; avoid remote filesystems like NFS or SMB.

Model documents efficiently: avoid joins (nested objects, parent‑child relationships) and use keyword fields for identifiers; prefer exact matches over costly joins.

Pre‑index frequently queried data (e.g., range‑based price fields) and use keyword mappings where possible; avoid unnecessary scripts, and prefer painless or expression scripts when needed.

Use rounded dates with now for better query caching, force‑merge read‑only indices, and enable eager global ordinals for fields that are heavily aggregated.

Pre‑warm the filesystem cache by setting index.store.preload so that hot index segments are loaded into memory after a restart.

Leverage index sorting to speed up joins, and use the preference parameter to improve cache utilization by routing similar queries to the same shard.

Configure an appropriate number of replicas to balance throughput and fault tolerance; more replicas can increase throughput but also increase cache pressure.

Enable adaptive replica selection to let Elasticsearch choose the fastest replica based on response time, service time, and queue size.

Third Part: General Recommendations

Do not request large result sets; Elasticsearch excels at returning the top‑N matching documents. Use the Scroll API if you need to retrieve all documents.

Avoid large documents; keep individual docs small to stay within default size limits and reduce memory, network, and disk overhead.

Minimize sparsity by normalizing document structures, avoiding unnecessary fields, and disabling norms and doc_values on sparse fields when they are not needed.

Mix exact queries with stemming using multi‑fields to support both precise and fuzzy matching.

Ensure consistent scoring by using the preference parameter to route repeated queries to the same shard; be aware that deleted documents and differing shard statistics can cause score variations.

If score inconsistencies persist, consider using dfs_query_then_fetch to gather global index statistics before executing the query, or reduce the number of primary shards for small datasets.