Boosting Ozone Block Reads with gRPC Streaming: Up to 30% Faster

Background

Large‑scale read scenarios such as Docker image repositories (often >20 GB) and model file distribution (hundreds of MB to several GB) involve many clients pulling the same file simultaneously.

1.2 Limitations of the existing Ozone read method

Ozone stores objects as blocks, which are composed of chunks. The original read process first obtains the chunk list, then requests each chunk sequentially. A gap exists between each chunk request due to message exchange, which becomes significant for large files.

The community proposed a gRPC bidirectional streaming read to eliminate the repeated readChunk calls and reduce request overhead.

2 grpc streaming read improvement

2.1 Overview

The improvement uses gRPC’s bidirectional stream. The client sends the required read length once; the DataNode (DN) streams the entire block data back, which the client caches and reads locally without further chunk requests.

A new readBlock service was added on the server side, and the client side implements a StreamBlockInput that uses this service.

2.2 Client improvements

2.2.1 New StreamBlockInput

The extended Ozone read stream currently supports only three‑replica mode (no EC) and caches the whole block data.

Prefer cached data when available.

If not cached, read from the container via readDataFromContainer .

readBlock can request the entire length in a single call.

Buffer is released automatically after reading.

2.2.2 readBlock RPC flow

Client sends a readBlock request.

DN opens the corresponding container and obtains the chunk list.

Depending on client version, DN reads either one chunk or a checksum length at a time.

DN streams each read result back to the client.

After sending all data, DN notifies the client of completion.

Client merges received data into StreamData .

Client returns StreamData to the container protocol.

The data is assembled into a StreamBlockInput buffer.

Once the block data is cached, subsequent reads retrieve chunks from the cache without additional requests.

3 Performance testing

3.1 Single‑thread read test

Reading a 1 GB file with and without streaming on a HDD‑based Ozone cluster showed streaming times between 4.8–5.2 s versus 7.3–7.4 s for the old method, a reduction of roughly 30%.

3.2 Multi‑thread read test

Using Ozone’s Freon tool with 4 concurrent threads for 1 GB objects, streaming completed 61 reads (≈586 MB/s) while the non‑streaming approach completed 48 reads (≈427 MB/s), an improvement of about 27% in bandwidth.

4 Future plans

The streaming read feature is not yet in the community edition; plans include upstreaming it and adding pre‑fetch logic on the DataNode side using posix_fadvise to load upcoming chunks into the system cache, further boosting throughput for hot files.