Deep Dive into Apache Druid V1 Storage Format: Index Structures and Disk Layout

Apache Druid is a high‑performance OLAP engine whose storage format is a key component for achieving sub‑second query latency on massive data. The article examines the V1 version storage format, focusing on index structures and how data is persisted on disk.

Druid stores data column‑wise; each column is a logical file. Columns are split into dimensions and metrics. Dimensions are indexed and consist of three parts: a dictionary, encoded dimension values, and an inverted index, while metrics simply store row values.

The dictionary de‑duplicates all column values, sorts them, and uses the array index as the encoded integer. Physically, the dictionary is a linear array with an index section storing offsets for each variable‑length value and a data section storing the actual bytes.

Encoded dimension values use a variable‑length integer scheme. The encoding rules are:

1 – 2^8‑1   => 1 byte
2^8 – 2^16‑1 => 2 bytes
2^16 – 2^24‑1 => 3 bytes
2^24 – 2^32‑1 => 4 bytes
2^32 – 2^40‑1 => 5 bytes

For example, the "city" dimension in the sample data has three unique values, so each value is stored in a single byte.

The inverted index is implemented as a bitmap for each dictionary entry; a bit set to 1 indicates that the corresponding row contains that dictionary value. Bitmaps are compressed to save space, resulting in variable‑length storage.

Array‑type dimensions follow the same three‑part storage model, but the encoded values form a two‑level structure: an outer variable‑length list whose elements are inner fixed‑length lists.

Physical storage metadata includes fields such as:

version: 1 byte
allowReverseLookup: 1 byte (whether reverse lookup is enabled)
numBytesUsed: 4 bytes (total bytes occupied)
numElements: 4 bytes (total number of elements)

A sample query (SELECT city, SUM(click_cnt) FROM table_t WHERE category=0 OR category=1 GROUP BY city) demonstrates how the dictionary, encoded values, and bitmap inverted index are used during query execution.

The article concludes with a summary of fixed‑length versus variable‑length storage patterns and their impact on Druid's performance.