Design Principles and Architecture of Message Queues

Message queues are core components for decoupling and traffic control in distributed systems.

The overall design consists of three main roles: Producer (sends messages), Broker (stores and forwards messages), and Consumer (receives and acknowledges messages).

Broker design focuses on message dumping, establishing a common pattern for decoupling, eventual consistency, and peak‑shaving, effectively performing two RPC calls plus optional storage, or three RPC calls when consumption confirmation is required.

Key functional requirements are message transmission, storage, and consumption, which lead to three design aspects: communication protocol, storage choice, and consumer relationship maintenance.

Communication protocols include JMS (Java Message Service) with point‑to‑point and publish‑subscribe models, AMQP (Advanced Message Queuing Protocol) as a language‑agnostic binary wire‑level protocol, and Kafka’s custom TCP‑based protocol tailored to its own business needs.

Storage options range from in‑memory, local file systems, distributed file systems, NoSQL stores, to relational databases; the choice depends on performance, reliability, and business scenario requirements.

Consumer relationship handling distinguishes between unicast (point‑to‑point) and broadcast (one‑to‑many), often leveraging external systems like Zookeeper to maintain subscription information.

Advanced features that a robust message queue should support include message ordering, delivery reliability, persistence, multiple messaging models, clustering, and transactional capabilities.

The article concludes with a note that further details on advanced features will be covered later.