JD ElasticSearch Supports ZSTD Compression: Implementation, Performance Evaluation, and Usage Guide

JD ElasticSearch now supports the ZSTD compression algorithm, offering higher compression ratios and faster speeds, which reduces storage and bandwidth costs while maintaining performance.

Motivation for adoption includes lack of official ZSTD support in Elastic, competitive pressure, the need for better control over open‑source components, integration of JD retail ES with cloud ES, and cost‑efficiency gains.

Performance test results (4c8g cluster, 3 primary shards, 1 replica) show that the jd_zstd codec at compression level 3 improves write performance by 38.46% over best_compression and 5.88% over LZ4, while saving about 24% storage compared to LZ4. Detailed file‑size comparisons for different compression levels (3, 6, 9) are provided.

Usage instructions :

Create an index with the ZSTD codec by setting index.codec: jd_zstd (level 3) or index.codec: jd_zstd_6 (level 6) in the index settings.

Note that the codec cannot be changed after index creation.

Example API calls:

# Create index with ZSTD level 3
PUT zstdtest
{
    "settings": {
      "index": {
        "codec": "jd_zstd"
      }
    }
}

# Create index with ZSTD level 6
PUT zstdtest_6
{
    "settings": {
      "index": {
        "codec": "jd_zstd_6"
      }
    }
}

Technical implementation :

The ZSTD algorithm is implemented via the zstd-jni library (version 1.5.5‑1). A custom ZstdCodec class extends Elasticsearch’s CompressionMode , providing ZstdCompressor and ZstdDecompressor with configurable compression levels and block sizes. The codec is registered in server/src/main/resources/META-INF/services/org.apache.lucene.codecs.Codec and required security permissions are added to server/src/main/resources/org/elasticsearch/bootstrap/security.policy .

grant codeBase "${codebase.zstd-jni}" {
  permission java.lang.RuntimePermission "loadLibrary.*";
  permission java.lang.RuntimePermission "libzstd.*";
};

The article also outlines the relevant Lucene file types affected by compression (row‑store files .fdm, .fdt, .fdx , column‑store .dvd , etc.) and provides reference tables comparing file sizes under LZ4, best_compression, and ZSTD at various levels.

References include the ZSTD GitHub repository, Lucene codec documentation, and papers on approximate nearest‑neighbor search.