Key Kafka Producer Configuration Parameters and Tuning Recommendations

Building on the previous article about Kafka producer message sending mechanisms, this piece focuses on the producer side, emphasizing the need to balance high throughput with zero message loss and introducing the essential configuration parameters.

acks : Determines how many replica acknowledgments are required before the producer considers a message committed; three options are 0 (no wait), 1 (leader only), and all/-1 (all in‑sync replicas). Higher durability reduces throughput.

max.request.size : Sets the maximum size of a single request (default 1 MiB). Increasing it (e.g., to 10 MiB) helps avoid failures when sending large messages.

retries : Number of retry attempts when a send fails (default 0). Setting a positive value (e.g., 3) mitigates transient failures such as network glitches or leader elections; retry.backoff.ms controls the pause between attempts.

compression.type : Controls message compression (none, gzip, snappy, lz4, zstd). Compression reduces network and disk I/O at the cost of CPU; lz4 is recommended for a good balance, zstd for maximum compression.

buffer.memory : Size of the producer’s memory buffer (default 32 MiB). Larger values allow more in‑flight data but consume more heap; when the buffer fills, the producer blocks for max.block.ms or throws a TimeoutException.

batch.size : Target batch size for records sent together (default 16 KB). Larger batches improve throughput, while smaller batches lower latency.

linger.ms : Maximum time to wait for additional records before sending a batch (default 0). Setting a small positive value (e.g., 100 ms) can increase throughput by reducing request frequency.

request.timeout.ms : Maximum time the producer waits for a broker response (default 30 s). Increase it for heavy loads to avoid premature time‑outs.

max.in.fight.requests.per.connection : Maximum number of unacknowledged requests per connection (default 5). Reducing it to 1 helps prevent out‑of‑order messages.

For a no‑loss setup, the article recommends: acks=all (or -1), retries≥3, max.in.fight.requests.per.connection=1, using send(record, callback) with error handling, unclean.leader.election.enable=false, replication.factor=3, min.insync.replicas=2, and enable.auto.commit=false on the consumer side.

In summary, the article reviews each producer parameter, explains its impact on reliability and performance, and provides a concise configuration checklist; further details can be found in the official Kafka documentation.