Optimizing Ozone S3 Gateway with Fast/Slow Dual‑Queue Isolation and Rate‑Limiting

Problem Overview

In the Ozone S3 gateway, client requests are divided into fast requests (high‑frequency, short‑latency operations such as GetObject, PutObject) and slow requests (low‑frequency, long‑running operations such as ListObjects, ListParts, DeleteObjects). Both types share a single Jetty QueuedThreadPool, causing slow requests to occupy threads for seconds, filling the pool and forcing fast requests to wait, with no traffic‑control or back‑pressure mechanism.

Root Cause

The lack of isolation, load‑shedding, and tiered rate‑limiting leads to long‑tail resource contention under mixed load, making core read/write services unstable.

Design Goals

Vertical isolation: separate thread pools for fast and slow requests.

Fast‑fail: reject new requests when a queue approaches capacity.

Elastic protection: pre‑emptively reject slow requests when the fast‑queue reaches a water‑mark.

Observability: distinct metrics for success, timeout, queue‑full rejection, and water‑mark rejection.

Core Implementation

Fast/Slow Dispatch

All S3 requests enter Jetty’s ServerConnector, are handed to a FastSlowQueueHandler, which parses the bucket and operation type to classify the request. Slow requests are routed to a dedicated slowExecutor with few threads and a small queue that overflows quickly; fast requests go to a larger fastExecutor with ample threads and a big queue.

Rate‑Limiting Mechanism

When either queue nears full, subsequent requests receive HTTP 429. For fast‑queue download scenarios, a bucket‑level water‑mark is evaluated; if the bucket is configured for throttling, its download requests are rejected with 429 while other requests continue.

Trigger condition: pending requests would fill the thread‑pool queue.

Water‑mark rule: reject only the throttled bucket’s download requests.

Statistics: separate counters for queue‑full rejections and water‑mark rejections.

Asynchronous Context and Timeout Control

After dispatch, an AsyncContext is created. The built‑in global timeout is disabled. IdleTimeout is set per request type—longer for slow requests, shorter for fast ones. To avoid a race where Jetty clears _dispatcherType before the async handler uses it, the code forces REQUEST type before the handler chain:

// Ensure _dispatcherType is set before entering handler chain.
// Connector thread's HttpChannel.dispatch() finally block may race
// to clear _dispatcherType to null. We use a pre-check + retry loop
// to handle this race condition robustly.
baseRequest.setDispatcherType(DispatcherType.REQUEST);

All requests then pass through a fixed filter chain (RootPageDisplayFilter → EmptyContentTypeFilter → S3GAccessLogFilter → S3GAuthFilter) before reaching the Jersey container for S3 REST routing.

Request Completion and Metrics

Successful processing increments a completion counter and ends the async context.

Idle‑timeout aborts increment a timeout counter and mark the request as failed.

Metrics cover normal completion, timeout, and rate‑limit rejections for full‑stack observability.

Advantages

Zero‑intrusion: only the dispatch layer is modified; existing auth, logging, and S3 logic remain untouched.

Complete isolation: fast and slow thread pools prevent slow requests from blocking core I/O.

Precise throttling: only configured buckets’ download traffic is limited.

Negligible overhead: static request‑type detection avoids heavy computation.

Reasonable timeout strategy: per‑connection idle timeout replaces a single global timeout.

Operations‑friendly: full‑stack metrics enable quick diagnosis of queue saturation, timeouts, and throttling events.

Conclusion

The FastSlowQueueHandler‑based solution adds dual‑queue isolation, dual water‑level throttling, async timeout tuning, and comprehensive monitoring to the Ozone S3 gateway without third‑party middleware, delivering a lightweight, high‑availability improvement for mixed‑load object‑storage workloads.