Kafka Production Optimization: Reducing Load and Improving Compression via Filebeat Tuning

Background

The Kafka logging cluster handles the entire company's application logs, reaching peak traffic of billions of messages per minute and daily data volumes in the petabyte range.

Key Terminology

Broker : a node in the Kafka cluster.

Network Idle Rate : average idle proportion of network thread pool threads, indicating cluster busyness.

Request Queue : queue where client requests are stored before the server processes them.

Kubernetes : container orchestration platform used for deploying Filebeat sidecars.

Kafka Production Pain Points and Optimization Goals

Pain Points

During traffic spikes, IO, storage, and idle rates surge, causing slower server responses and Kafka consumer backlog; some services were degraded, affecting data completeness.

Goals

Fundamentally resolve the issue by optimizing the transmission path, increasing batch compression, and reducing server request volume, network traffic, and CPU consumption without impacting business read/write operations.

Optimization Process

Root Cause Analysis

The main cause of high pressure was low compression ratio in production.

Parameter Adjustments

Investigated Filebeat parameters bulk_flush_frequency (default 0) and bulk_max_size (default 2048). Tested increasing bulk_flush_frequency from 0.1 to 0.2.

bulk_flush_frequency: 0.2
bulk_max_size: 2048

Adjusted Filebeat memory queue settings:

queue.mem:
  events: 4096
flush.min_events: 2048
flush.timeout: 1s

Found that increasing flush.timeout to 5s improved compression when partition count was low, but not at production scale.

Discovered round_robin.group_events (default 1) which groups events per partition before selection, reducing partition switches and increasing batch size.

Testing Results

Increasing batch size raised compression ratio, reducing network and disk usage. SNAPPY compression tests showed size reductions up to 35% for batches over 50 messages.

Count

Compressed Size

1

1.1K

5

1.3K

20

2.2K

50

4K

100

8K

200

16K

Verification and Production Rollout

Online gray‑scale tests with tuned Filebeat parameters ( queue.mem.events: 4096 , flush.min_events: 2048 , flush.timeout: 5s , round_robin.group_events: 10 ) showed:

Production request count reduced by 30%.

Traffic reduced by 30‑40%.

Post‑deployment metrics for three representative topics:

App

Before Traffic

Before Requests

After Traffic

After Requests

test_1***

40G

16M

23G

5M

test_2***

33G

15M

21G

2M

test_3***

280G

220M

220G

120M

Overall impact after full rollout:

CPU usage dropped from 36% to 22%.

Topic production requests decreased by 42%.

Cluster traffic fell by 20%.

Optimization Summary

Reduced per‑minute client request volume by over a billion, traffic down 35%.

Increased per‑minute message capacity from 2.6B to 3.3B after the May stress test (35% improvement).

Improved idle rate to keep the cluster stable, lowered IO and network load.

Added comprehensive monitoring metrics for client read/write counts, idle rates, compression ratios, and resource usage.

Future Planning

As business grows, Kafka data volume will continue to expand, increasing pressure on IO, network, storage, and CPU. Future work will focus on balancing compression settings to maintain latency while further expanding cluster capacity.