Optimizing Kafka Producer for High Throughput: Batching, Asynchronous Sending, Compression, and Concurrency

Kafka is a core component of large‑scale data architectures; this article explains how to achieve high throughput with Kafka producers.

Batch Sending Optimization – The producer collects multiple records into a batch before sending, reducing network overhead. Adjusting linger.ms allows the producer to wait a few milliseconds to form larger batches, but setting it too high increases end‑to‑end latency.

Asynchronous Sending Mechanism – The send() method returns a Future<RecordMetadata> immediately. Results are handled via callbacks, allowing the producer to continue sending without waiting for broker acknowledgments.

producer.send(record, (metadata, exception) -> {
    if (exception == null) {
        System.out.println("Success: " + metadata.offset());
    } else {
        exception.printStackTrace();
    }
});

Compression Mechanism – Compressing messages on the producer side reduces network traffic. Common compressors include gzip (high compression, higher CPU), snappy (fast, moderate compression), lz4 (very fast, low CPU, lower compression), and zstd (high compression with good speed). For high‑throughput scenarios, lz4 or zstd are recommended; for CPU‑sensitive systems, snappy is a good choice.

Concurrent Sending Capability – Kafka brokers use the operating system’s page cache for sequential writes, turning most disk I/O into fast memory operations. This design enables the broker to handle extremely high write throughput.