How Uber Cuts Storage Costs with ZSTD Compression in Apache Parquet

Background

Uber’s analytics platform runs on Apache Hadoop, storing hundreds of petabytes in an HDFS‑based data lake. The platform uses Apache Hudi for table management, Apache Parquet as the underlying file format, and Apache Hive, Presto, and Spark for interactive and long‑running queries.

Apache Parquet in Uber

Parquet is a columnar storage format that groups rows into row groups, each containing column chunks that are further divided into pages. This layout enables efficient encoding and compression at the page level.

Encoding and Compression

Parquet supports several encodings (dictionary, RLE, delta, byte‑split) and compression algorithms (SNAPPY, GZIP, LZO, BROTLI, LZ4, ZSTD). Choosing a compressor involves balancing compression ratio against read/write speed.

ZSTD Introduction

ZSTD offers higher compression ratios with configurable levels 1‑22, allowing users to trade speed for size.

ZSTD vs. GZIP and SNAPPY

Benchmarking the top 10 Hive tables showed that converting from GZIP (level 3) to ZSTD reduced storage by ~7%, while switching from SNAPPY to ZSTD cut size by ~39%. Uncompressed tables saw up to a 79% reduction. The tests also measured vCore usage for representative Hive queries.

Compression‑Level Benchmark

Using 65 ≈ 1 GB Parquet files originally compressed with GZIP, Uber recompressed them with ZSTD levels 1‑22. Results indicated diminishing storage gains after level 8, a steady increase in write time with higher levels, and unchanged read time. Level 19 was selected as the optimal point where further write‑time cost did not yield additional size benefits.

Enabling ZSTD in Parquet

Parquet 1.11.1 and earlier rely on Hadoop 2.9.1+ for ZSTD, which Uber’s older Hadoop version did not support. Uber replaced Hadoop’s ZSTD with a ZSTD‑JNI wrapper, merged the change into Parquet 1.12.0+, and contributed patches to Presto (issue #15939) to add native ZSTD support.

Column Deletion

Many tables contain unused columns that waste space. Uber identified such columns via table lineage and audit logs, then built a Parquet column‑prune tool (merged as PARQUET‑1800) that removes columns during file copy, achieving a 20× speedup over a Spark‑based approach.

Column Reordering

Inspired by the paper “Reordering Rows for Better Compression,” Uber reordered columns (UUID, time_ms, latitude, longitude), which reduced file size by 32% for a test table.

Advanced Encodings

Future work includes delta encoding (based on Lemire & Boytsov’s binary packing) and byte‑stream split encoding introduced in Parquet 1.12.0, both of which improve compression efficiency.

Key Takeaways

ZSTD dramatically reduces storage size and vCore consumption; open‑source contributions eliminate the need for Hadoop‑level ZSTD support; column pruning and reordering further cut space; and large‑scale data conversion can be accelerated tenfold with the new tools.