Understanding SFMQ: A Kafka Proxy Layer that Solves Common Message‑Queue Challenges

Kafka is widely used but many teams face issues such as inconsistent client configurations, multiple SDK versions, painful upgrades, rebalance storms during rolling restarts, tight coupling between partitions and consumer instances, and the lack of built‑in delay or transactional messaging.

Through a narrative about a developer named Bo and his mentor Li, the article introduces SFMQ, an internal proxy layer built on top of Kafka that hides language‑specific client details, makes version upgrades transparent, isolates rebalance effects to the proxy, reduces Kafka broker load, and centralises common capabilities like delayed messages, transaction handling, retry, dead‑letter queues, filtering, and load‑balancing.

SFMQ implements delayed messaging using Redis sorted sets (zset), where the scheduled delivery time is stored as the score; a dedicated thread polls for scores less than the current time and pushes ready messages to Kafka. For high‑volume scenarios, multiple zsets are sharded across proxy instances, each with its own consumer thread, and a shard‑service coordinates assignment and failover.

Transactional messaging in SFMQ follows a three‑step protocol: a half‑message is first stored in Redis, then an ACK (commit or rollback) is sent based on the local DB transaction outcome, and finally a back‑check ensures the half‑message is resolved, guaranteeing exactly‑once delivery despite failures.

On the consumer side, SFMQ adopts a push model: the proxy pulls from Kafka and pushes HTTP requests to downstream services, preserving order per key while allowing concurrent processing via a window component. It also provides built‑in dead‑letter handling and offset management to avoid blocking other messages.

The article compares native Kafka consumption problems—limited partition scaling, head‑of‑line blocking, and message latency—with SFMQ’s solutions, showing how SFMQ’s windowed push, ordered key handling, and dead‑letter retries eliminate these bottlenecks.

In summary, SFMQ abstracts Kafka behind a simple HTTP API, offering unified access, upgrade safety, resource efficiency, and a rich set of middleware features that let developers focus on business logic rather than messaging infrastructure.