Reading Encrypted ORC Files in StarRocks: Architecture and Implementation Details

This article explains how StarRocks reads encrypted ORC files, covering background, ORC file structure, encryption concepts, and the concrete implementation in StarRocks.

Background : To protect sensitive Hive table data, encryption is applied at the column level. Spark can read/write encrypted ORC files using the Java ORC library, but the C++ ORC library used by StarRocks lacks encryption support, requiring a modification to enable decryption.

Problem Statement : StarRocks must be able to query Hive tables stored as encrypted ORC files. The solution involves extending the Apache ORC C++ module to handle encryption.

ORC File Overview : ORC (Optimized Row Columnar) is a column‑oriented storage format designed for high I/O efficiency. It consists of a four‑layer hierarchy – File, Stripe, Stream, and Group – and provides three levels of index (FileStat, StripeStat, IndexData) to enable predicate push‑down and selective reading.

File Structure Details : The file tail contains PostScript, Footer, and MetaData. The Footer stores column statistics, stripe metadata, and encryption information. The body holds multiple Stripes, each composed of Streams (index‑Stream and data‑Stream). Reading starts from the tail, extracts PostScript, then Footer and MetaData, and finally the required Stripes.

Encryption Basics : Symmetric encryption (AES‑128‑CTR/NoPadding) is used. Keys include a masterKey (stored in Hive table properties), statKey (decrypts FileStat/StripeStat), and dataKey (decrypts IndexData and row data). The IV is 16 bytes, encoding column ID, stream type, stripe ID, and a counter (min_count).

StarRocks Reading Flow :

1) User submits an SQL query to the FE node.

2) StarRocks retrieves the encrypted masterKey from Hive table metadata and decrypts it via KMS.

3) The plaintext masterKey is passed to the BE node.

4) BE uses the masterKey to decrypt statKey and dataKey stored in the ORC file.

5) BE decrypts statistics and actual data, then applies predicate push‑down using the three‑level index.

Implementation Details : The reading pipeline follows the Decorator pattern – a raw file stream is wrapped by a DecryptionInputStream (if encrypted) and then by a DecompressionStream. Each layer reads data from the underlying stream, processes it, and passes it upward.

Encrypted Block Skip‑Read Mechanism : Data is divided into encrypted blocks within compression blocks. To read a specific group, the system calculates the compression block index and offset, derives the corresponding min_count for the IV, reads the entire compression block, decrypts it, and finally extracts the target group.

Example formulas (shown in code style): block_index = group_offset / zipBlockSize and zip_head_offset = block_index * zipBlockSize . The min_count is computed as min_count = zip_head_offset / encrypted_block_size .

Q&A Highlights :

Decompression can be performed on individual compression blocks, not the whole file.

ORC’s three‑level index enables precise data retrieval without full scans.

ORC’s columnar indexing does not rely on the left‑most prefix rule used in row‑store databases.

Compression precedes encryption to improve encryption efficiency.

Conclusion : The article provides a comprehensive view of how StarRocks integrates encryption handling into ORC file reading, covering key concepts, file architecture, encryption key management, and the layered I/O implementation. Readers interested in the actual code can refer to the open‑source PR linked at the end.