This article introduces Apache Kafka’s core components and key features, then details seven practical use cases—including log processing, recommendation streams, monitoring, CDC, system migration, event sourcing, and message queuing—illustrated with diagrams and step‑by‑step workflows for distributed systems.
This article explores eight practical retry mechanisms for handling unreliable third‑party APIs in backend systems, covering simple loops, recursion, built‑in client retries, Spring Retry, Resilience4j, custom utilities, asynchronous thread‑pool retries, and message‑queue based approaches, plus best‑practice tips.
This article recounts the step‑by‑step evolution of a ride‑hailing order service’s data layer—from a single SQLServer database through SQL optimization, read‑write splitting, business‑domain sharding, caching and MQ integration, to a custom sharding component and smooth migration to MySQL—highlighting challenges, solutions, and lessons learned.
This article explains Kafka's fundamentals, including topics, partitions, brokers, replication, producer‑consumer workflow, consumer groups, offset management, and common exception handling, while providing code examples and diagrams to help developers understand and effectively use this distributed messaging system.
This article introduces the LMAX Disruptor library, explains its core concepts such as Ring Buffer, Sequencer and Wait Strategy, and provides a step‑by‑step Java demo with full source code to build a fast, lock‑free in‑memory message queue.
This article explains the core components of RocketMQ—including NameServer, Broker, Producer, and Consumer—detailing their functions, deployment diagrams, and how the middleware supports high‑concurrency scenarios such as peak shaving, asynchronous messaging, fault tolerance, and system decoupling.
This article explains why batch messaging in RocketMQ is crucial for high-throughput systems, outlines its characteristics and ideal scenarios, and provides complete Java code examples for sending and splitting large batches to improve performance and reduce downstream API calls.
This article introduces asynchronous messaging patterns, then compares RabbitMQ and Apache Kafka by examining their internal architectures, message models, and trade‑offs, helping architects choose the appropriate solution based on scenario requirements in modern.
This article explains how to apply gray release techniques in RocketMQ by upgrading consumers, upgrading producers, and using consumer‑side filtering, tag filtering, and SQL92 filtering, providing concrete code examples, configuration steps, and trade‑off analysis for each approach.
This article explains seven typical message‑queue use cases—including ordinary, ordered, delayed, transactional, trace, dead‑letter, and priority messages—detailing their business motivations, implementation challenges, and concrete code examples for Kafka, RocketMQ, Pulsar, and RabbitMQ.
This article presents a complete Netty TCP client demo built with Spring Boot, detailing the project architecture, module layout, business flow, key code implementations—including a local BlockingQueue, message processing with Redis locks, client initialization, handler logic, and testing endpoints—providing a practical reference for backend developers.
This article compares RabbitMQ and Apache Kafka across dimensions such as message ordering, routing, timing, retention, fault handling, scalability, and consumer complexity, and provides practical guidance on when to choose each platform for backend architectures.
This article explains the fundamental asynchronous messaging patterns, compares RabbitMQ and Apache Kafka in terms of architecture, queue and pub/sub models, and provides guidance on selecting the appropriate technology for different backend scenarios.
This article explains Kafka’s fundamental role as a message system, detailing topics, partitions, producers, consumers, replica management, consumer groups, the controller, Zookeeper coordination, and performance optimizations such as sequential writes, zero‑copy, log segmentation, and network design, providing a comprehensive overview for big‑data practitioners.
The article recounts a series of real‑world anecdotes exposing how excessive architectural ambitions—such as needless microservice fragmentation, ubiquitous message‑queue usage, multi‑level caching, over‑engineered design patterns, and configuration‑driven code—lead to wasted resources, maintenance headaches, and fragile systems, urging a return to simplicity.
This course provides an in‑depth study of RocketMQ, covering its fundamentals, source‑code analysis, advanced features such as transactional and ordered messages, performance optimization, operational monitoring, and hands‑on projects that guide learners through installation, configuration, clustering, and integration with Spring Boot.
This article explains Kafka's role as a messaging system, covering core concepts such as topics, partitions, producers, consumers, messages, cluster architecture, replicas, consumer groups, controller coordination with Zookeeper, and performance optimizations like sequential writes and zero‑copy networking.
This article details the architecture, technical requirements, and implementation techniques—including hybrid push‑pull, heterogeneous storage, distributed scheduling, task chunking, and at‑least‑once delivery—used to build a high‑concurrency, reliable reminder notification service for QQ.
This guide explains how to implement delayed and priority queues in RabbitMQ 3.8 using Spring Boot 2.6, covering dead‑letter exchange configuration, message expiration, priority settings, code examples for sending and consuming messages, and practical considerations such as priority limits and ordering behavior.
This tutorial walks through RabbitMQ fundamentals, installation on macOS, core AMQP concepts, common exchange types, and step‑by‑step SpringBoot integration with code samples, while highlighting practical issues like connection leaks and the need for a connection pool.
This article provides an in‑depth technical comparison of RabbitMQ and Apache Kafka, covering their core architectural differences, message ordering, routing, timing, retention, fault handling, scalability, consumer complexity, and offers guidance on selecting the appropriate platform for various backend scenarios.
This article explains how to detect, diagnose, and prevent message loss in Kafka-based message queue systems by covering system decoupling, traffic control, data consistency challenges, producer, broker, and consumer issues, and offering configuration, monitoring, and operational best‑practice solutions.
This article explains RocketMQ's internal workflow for concurrent and ordered message consumption, covering thread‑pool configuration, pull‑message handling, request submission, execution steps, offset management, expired‑message cleanup, and answers common questions about batch retries and offset ordering.
This article explains why using Redis List for a message queue is limited, introduces Redis Stream as a purpose‑built MQ with consumer groups, ACKs, and persistence, and dives into its internal Radix‑Tree and listpack structures, command usage, and memory‑saving techniques.
This article introduces Apache Pulsar, an open‑source cloud‑native distributed messaging platform, detailing its storage‑compute separation architecture, multi‑tenant support, load balancing, fault tolerance, schema handling, functions, IO connectors, tiered storage, cross‑region replication, and operational commands for managing brokers, bookies, and namespaces.
This article breaks down real‑world high‑concurrency strategies—horizontal scaling, caching, Elasticsearch, sharding, message‑queue smoothing, and cellization—explaining when each applies, their trade‑offs, and practical tips for building scalable, reliable backend services.
This article explains how ordering issues in message queues like RabbitMQ and Kafka can break MySQL binlog synchronization, illustrates common pitfalls, and offers practical solutions to guarantee correct processing order in high‑throughput backend systems.
This article provides a detailed overview of popular message queue systems, explains their core concepts such as decoupling and eventual consistency, and compares the advantages and disadvantages of Kafka, RocketMQ, RabbitMQ, and other notable MQ solutions for high‑concurrency scenarios.
Redis is a high-performance, open-source in-memory key-value database offering rich data structures, persistence options, and advanced features like replication, clustering, and pub/sub, making it ideal for caching, counters, message queues, leaderboards, and session storage in demanding, high-concurrency applications.
The article analyzes the root causes and system‑wide consequences of oversized messages in JD Logistics, explains middleware limits of JMQ and JSF, and presents design principles, code‑level checks, and practical mitigation techniques such as pagination, claim‑check pattern, batch sizing, and monitoring to prevent service outages.
This article compares Kafka and RabbitMQ across scalability, durability, latency, data flow, ordering, reliability, persistence, extensibility, and complexity, then outlines ideal use cases for each and offers practical guidance on selecting the right message queue for a project.
This article explains the three core benefits of message queues, compares major MQ products, details RocketMQ's advantages, internal models, reliability mechanisms, load‑balancing, storage design, and practical code examples, guiding engineers to make informed architectural decisions.
This guide explains what a process is, how it differs from a thread, various ways to create and terminate processes in Linux, and provides detailed examples of inter‑process communication mechanisms—including message queues, shared memory, UNIX domain sockets, pipes, and semaphores—complete with C code samples.
This article provides a thorough overview of message queues, covering their definition, core components, types (point‑to‑point and publish/subscribe), key characteristics such as asynchrony, loose coupling, distribution and reliability, selection criteria among popular MQ products, and typical application scenarios in high‑concurrency systems.
This article details the architectural redesign of QQ Music's comment platform—migrating to MongoDB, introducing threaded comments, and employing caching and message‑queue decoupling—to handle massive read/write spikes during celebrity live‑drop events while maintaining high availability and performance.
QQ Music’s comment system was re‑engineered with a MongoDB backend, cache layer, asynchronous writes, split read/write services, priority queues and rate‑limiting, enabling it to endure celebrity‑driven traffic spikes, maintain data consistency, and deliver high‑availability, low‑latency user experiences.
This tutorial explains how to create delayed and priority queues in RabbitMQ using Spring Boot, covering dead‑letter queue configuration, TTL settings, priority parameters, and provides complete Java code examples for sending and consuming prioritized messages.
This article explains Kafka's core architecture—including topics, producers, brokers, and consumers—its underlying mechanisms, the role of Zookeeper, key characteristics such as high throughput and fault tolerance, and common use cases like log collection, activity tracking, and stream processing.
Learn a step‑by‑step strategy to synchronize millions of records from dozens of third‑party systems across all provinces, using SFTP for bulk transfers, standardized file formats, scheduled jobs, Redis‑based rate limiting, MQ‑driven ingestion, and data‑consistency checks, while addressing security and performance challenges.
This article explains RocketMQ transaction messages by covering their use cases, underlying mechanisms, and a step‑by‑step implementation that demonstrates how to ensure eventual consistency between message production and local database transactions in distributed Java back‑end systems.
This article provides a comprehensive walkthrough of RocketMQ 4.9.x consumption architecture, covering the four core roles, publish‑subscribe model, storage structures, load‑balancing, long‑polling, concurrent and ordered consumption flows, progress persistence, and retry mechanisms, with illustrative diagrams and code snippets.
This article traces the development of message queues from early decoupling solutions like ActiveMQ and RabbitMQ, through high‑throughput designs such as Kafka and RocketMQ, to modern platform‑centric systems like Pulsar, while detailing core concepts, architecture diagrams, storage mechanisms and trade‑offs.
This article explains how to connect to Redis from PHP, demonstrates core Redis commands such as set, get, list, hash, and sorted‑set operations, and describes typical use cases like caching, counters, distributed locks, and message queues.
This article explores RocketMQ's high availability mechanism, focusing on DLedger's implementation of Raft-based leader election for distributed message middleware, covering both traditional and Controller mode architectures.
This guide walks through building a SpringBoot architecture that updates MySQL first, deletes Redis cache asynchronously via Canal and RabbitMQ, resolves common Canal meta‑data mismatches, and configures manual RabbitMQ acknowledgments to guarantee reliable cache invalidation.
This article provides a comprehensive overview of RocketMQ, covering its origin, core concepts, component roles, cluster deployment architecture, workflow, feature details, persistence mechanisms, and cleanup policies, offering developers a solid foundation for using this high‑performance messaging middleware.
Kafka is an open‑source, distributed streaming platform that uses a publish/subscribe message queue architecture to provide high‑throughput, fault‑tolerant real‑time data processing, featuring topics, partitions, replicas, consumer groups, and multiple APIs for producers, consumers, streams, connectors, and administration.
This article explains Redis Pub/Sub fundamentals, demonstrates channel‑based and pattern‑based subscription with command‑line examples, dives into the internal data structures that power them, and shows how to integrate Redis Pub/Sub in Spring Boot using Redisson for lightweight messaging.
This article walks through RocketMQ 4.9.x’s consumption architecture, explaining the roles of NameServer, Broker, Producer and Consumer, the publish‑subscribe model, storage structures, load‑balancing algorithms, long‑polling, concurrent and ordered consumption, progress persistence, and the built‑in retry mechanism.
This article provides a comprehensive overview of Kafka, covering its core value for decoupling and asynchronous processing, fundamental concepts such as producers, consumers, topics, partitions and replication, high‑performance mechanisms like zero‑copy and OS cache, detailed resource evaluation for CPU, memory, disk and network, operational tools, consumer‑group rebalance strategies, LEO/HW offsets, controller management, and delayed‑task scheduling.
This article comprehensively explores Kafka’s core concepts, architecture, and the techniques it employs—such as ack strategies, replica synchronization, high‑watermark, leader‑epoch, zero‑copy, batch sending, compression, and reactor‑based networking—to achieve both strong reliability and high throughput in distributed messaging systems.
This article explains how Redis List can be used as a FIFO message queue, covering core concepts, command usage, reliability techniques, and Java integration with Redisson for building a practical backend messaging solution.
RocketMQ 5.1.0 introduces a tiered storage module that offloads messages from local disks to cheaper storage, enabling longer retention without impacting hot data performance, and the article explains its design, implementation details, configuration steps, and current challenges.
This course covers an introduction to message queues and their use cases, detailed RabbitMQ concepts and installation, various PHP RabbitMQ patterns including simple, worker, fanout, direct, and topic modes, as well as dead‑letter and delayed queues, culminating in a TP6 practical implementation.
This article explains how to set up a PHP development environment, install RabbitMQ and its dependencies, and provides complete producer and consumer code examples to create a simple "Hello World" message queue using RabbitMQ's AMQP protocol.
This article details how WeChat's Gemini‑2.0 big‑data platform leverages Apache Pulsar, outlining cloud‑native advantages, load‑balancing refinements, cache and SSD tuning, high‑availability safeguards, and cost‑saving strategies that together enable large‑scale, real‑time, deep‑learning recommendation workloads.
This article provides an in-depth overview of Kafka, covering its core value as a message queue, fundamental concepts, cluster architecture, producer and consumer configurations, scaling strategies, monitoring tools, and practical operational commands for building and maintaining high‑throughput, highly available streaming systems.
An in‑depth tutorial demonstrates building a Netty‑based TCP client with Spring Boot, covering project architecture, module layout, message‑queue simulation, detailed Java code for client creation, connection handling, and testing endpoints, providing a complete, production‑ready demo for IoT socket communication.
This article demonstrates how to achieve cache‑database consistency by updating MySQL first, deleting Redis cache asynchronously via Canal, publishing change events to RabbitMQ, and handling acknowledgments manually in a Spring Boot application, complete with configuration steps and troubleshooting tips.
This article compares four common approaches for synchronizing MySQL data to Elasticsearch—synchronous dual write, asynchronous dual write via message queues, scheduled tasks, and binlog‑based data subscription—detailing their advantages, drawbacks, implementation steps, and tool choices such as Canal, Maxwell, and Python‑MySQL‑Replication.
This article provides a comprehensive overview of Kafka, covering its core value as a message queue, fundamental concepts, cluster architecture, log storage mechanisms, zero‑copy data transfer, high‑throughput and high‑availability design, consumer group behavior, rebalance strategies, and practical operational commands for managing topics, partitions, and offsets.
This article provides a comprehensive technical overview of Kafka, covering its core message‑queue value, architecture components such as producers, consumers, topics, partitions and replication, high‑performance mechanisms like zero‑copy and OS cache, resource planning for disks, memory, CPU and network, operational tools and commands, consumer‑group management, rebalance strategies, and internal scheduling mechanisms such as the time‑wheel.
This article comprehensively explores Kafka’s core architecture, explaining how asynchronous decoupling and traffic shaping are achieved, detailing the roles of producers, brokers, consumers, and ZooKeeper, and analyzing the reliability and performance techniques such as ACK policies, replication, idempotent and transactional producers, page‑cache flushing, zero‑copy, compression, batching, and load‑balancing strategies.
The article explains how RocketMQ 5.1.0 introduces a tiered storage module that offloads messages to cheaper media, describes its design, architecture layers, quick‑start configuration, upload and read mechanisms, prefetch cache, fault recovery, current development plans, and remaining challenges.
This article provides a comprehensive overview of major message‑queue middleware—including RocketMQ, Kafka, Pulsar, and RabbitMQ—covering fundamental concepts such as tags, groups, offsets, architectural components, storage mechanisms, transaction workflows, rebalance strategies, and recent developments, while comparing their features and performance characteristics.
This article provides a detailed side‑by‑side comparison of five popular message‑queue systems—Kafka, RabbitMQ, ZeroMQ, RocketMQ, and ActiveMQ—covering documentation, programming languages, supported protocols, storage, transactions, load balancing, clustering, management UI, availability, duplication handling, throughput, subscription models, ordering, acknowledgments, replay, retry, concurrency, and includes promotional information from the author.
This article outlines a comprehensive flash‑sale system architecture covering front‑end separation, Nginx static handling, traffic shaping, Redis caching and locking, message‑queue throttling, database sharding, and captcha mechanisms to achieve high concurrency while ensuring security and reliability.
This article explains the architecture of a flash‑sale system from seven dimensions, covering characteristics, overall design, peak‑limiting, security, page optimization, asynchronous processing, hotspot isolation, Nginx configuration, Redis clustering, message‑queue throttling, database design, and captcha mechanisms to handle massive concurrent requests.
This guide introduces RabbitMQ, explains how to install it with Docker, covers basic and advanced producer/consumer usage including acknowledgments, durability, QoS, bindings, and exchange types, and discusses a typical Python reconnection problem, providing a practical overview for backend developers.
This article explains why MQ middleware like Kafka is introduced for system decoupling and traffic control, outlines the three key challenges of message loss detection, loss points, and prevention, and provides detailed configurations, monitoring tips, and code examples to ensure reliable, loss‑free message delivery.
This article breaks down RocketMQ’s storage architecture, explaining how its commitlog, consumequeue, and index files work together to ensure reliable persistence, quick offset-based lookups, and efficient message consumption in a publish‑subscribe model.
This article explains the common interview question of guaranteeing 100% message reliability in MQ middleware such as Kafka or RabbitMQ, outlines the three lifecycle stages of a message, discusses detection mechanisms, id generation, idempotent consumption, and handling message backlog, providing concrete design patterns and practical examples.
This article explains the key characteristics of flash‑sale (秒杀) business and presents a comprehensive backend architecture—including Nginx, CDN, gateway, Redis, MQ, asynchronous processing, hotspot isolation, and database strategies—to handle massive simultaneous requests while ensuring performance, reliability, and security.
This article examines the various reasons why RocketMQ can deliver the same message to consumers multiple times, covering send‑retry mechanisms, consumer‑side exceptions, offset commit and persistence failures, master‑slave synchronization issues, rebalance events, and long‑processing cleanup, and offers guidance on designing idempotent solutions.
This article details the architectural evolution of JD's VOP message warehouse, describing the challenges of massive data volumes, the transition from V1.0 to V3.0+ through database sharding, MongoDB adoption, traffic governance, stability improvements, and cost reduction strategies, while presenting performance metrics and future outlook.
This article surveys eleven practical techniques for implementing delayed tasks in Java, covering JDK utilities like DelayQueue, Timer, and ScheduledThreadPoolExecutor, as well as message‑queue solutions such as RocketMQ, RabbitMQ, Redis key‑expiration, Redisson, Netty, Hutool, Quartz, and simple polling, with code demos and implementation principles.
Facing chaotic usage, maintenance challenges, partition tolerance issues, and performance bottlenecks with RabbitMQ, our middleware team decided to fully migrate to Kafka, outlining reasons, comparative models, migration strategies, and verification steps to ensure a smooth, high‑availability, high‑performance transition.
This article explores eight Java asynchronous programming techniques—from raw threads and Future to CompletableFuture, Spring @Async, event-driven approaches, message queues, Hutool ThreadUtil, and Guava ListenableFuture—detailing their concepts, advantages, limitations, and providing complete code examples for each.
This article explains why asynchronous execution improves latency in common scenarios like sending SMS or emails, and presents eight Java techniques—including threads, Future, CompletableFuture, Spring @Async, ApplicationEvent, message queues, third‑party utilities, and Guava—to implement robust asynchronous programming.
This article details Tencent Cloud's year‑long production experience with Apache Pulsar, covering why Pulsar was chosen over Kafka, deep dives into Ack hole handling, TTL/Backlog/Retention strategies, zk‑node and ledger leaks, cache optimizations, and concrete code snippets that illustrate the stability and performance improvements.
A detailed investigation reveals that RocketMQ can report successful writes to the broker while consumers see minutes‑long delays because the transient store pool and commit‑log forwarding mechanisms delay the transfer of messages from the broker’s memory to the consumer queue.
This article explains RocketMQ's consumer startup process in pull mode, detailing the underlying push‑like implementation, configuration validation, instance naming, client initialization, load‑balancing, wrapper handling, offset storage, scheduled tasks, and trace dispatching, accompanied by Java code examples.
This article explores four common approaches for synchronizing product data from MySQL to Elasticsearch in e‑commerce systems—synchronous dual write, asynchronous dual write with message queues, scheduled tasks, and binlog‑based data subscription—detailing their advantages, drawbacks, and implementation considerations.
This article explores four common approaches for synchronizing product data from MySQL to Elasticsearch—including synchronous dual writes, asynchronous messaging, scheduled jobs, and binlog‑based data subscription—detailing their advantages, drawbacks, and implementation considerations for e‑commerce search systems.
This article examines four common approaches for synchronizing product data from MySQL to Elasticsearch—synchronous dual write, asynchronous dual write with message queues, scheduled batch jobs, and binlog‑based data subscription—detailing their advantages, drawbacks, and implementation considerations.
This article explains how Redis implements the Pub/Sub messaging pattern by storing subscription relationships in hash‑based dictionaries, detailing the processing flow of SUBSCRIBE and PUBLISH commands, cluster propagation, and key limitations such as lack of persistence and client buffer constraints.
This article introduces eight practical techniques for achieving asynchronous processing in Java—including raw threads, Future, CompletableFuture, Spring @Async, ApplicationEvent, message queues, Hutool ThreadUtil, and Guava ListenableFuture—complete with code examples and discussion of their advantages and limitations.
This guide explains how to use Alibaba Cloud EventBridge to create and manage RocketMQ 4.0, 5.0, and self‑hosted Apache RocketMQ instances, covering architecture, API parameters, VPC and security settings, and step‑by‑step procedures for event‑driven integration and message routing.
This article introduces the Disruptor library, explains its core concepts such as Ring Buffer, Sequencer, and Wait Strategy, and provides a complete eight‑step Spring Boot example with code to create, publish, and consume high‑throughput messages in Java.
The article analyzes three data‑level classifications, examines four cache‑database write strategies, discusses their pros and cons, and presents practical solutions such as delayed double deletion, message‑queue compensation, and binlog‑driven cache updates to maintain consistency.
This article explains the core distinctions between Remote Procedure Call (RPC) and Message Queue (MQ) technologies, covering their architectures, communication patterns, functional features, and performance considerations, and outlines typical use cases such as synchronous calls, decoupling, traffic shaping, and asynchronous processing in distributed systems.
This article walks through every stage of a RocketMQ message—how producers create and route messages, the storage mechanisms and zero‑copy techniques used by brokers, high‑availability modes, consumption models, ordering guarantees, and the automatic cleanup policies that finally retire messages.
This article explains how to use RabbitMQ's delayed message plugin, covering its features, installation on multiple platforms, configuration steps, and complete Java Spring Boot code examples for producing and consuming delayed messages in a backend system.
This article analyzes the functional decomposition, data structures, system architecture, push‑pull mode selection, and message flow of a one‑to‑one instant messaging service, providing practical design guidelines and example Redis/MySQL schemas for building a scalable chat backend.
This article breaks down the functional requirements, data models, layered architecture, push‑pull messaging choices, and message flow for building a robust one‑to‑one instant‑messaging service using Redis, MySQL and asynchronous queues.
This article explains why message queues are used to decouple services, enable asynchronous processing, smooth traffic spikes, and improve performance, while also detailing the new challenges they introduce such as reduced availability, increased complexity, duplicate consumption, data consistency, message loss, ordering, and backlog, along with practical solutions for each issue.
This article explains the concept of idempotency, illustrates why repeated operations must yield identical results, and details how RocketMQ’s components—NameServer, Broker, Producer, and Consumer—can be designed with unique message IDs and acknowledgment mechanisms to achieve reliable, duplicate‑free messaging in backend systems.
This article reviews the evolution of message queues, explains why Pulsar was adopted at Tencent, details the challenges of open‑source solutions, and presents the architectural design, performance and observability optimizations, plus a real‑world case study and future roadmap.
Apache RocketMQ 5.0 introduces a stateless proxy, gRPC‑based multi‑language SDK with immutable APIs, enhanced error handling, SimpleConsumer, and pop consumption, and the article outlines the technical improvements, roadmap, and step‑by‑step guide for developers to set up the environment, understand the new protocols, and contribute language client implementations.
This article introduces the fundamentals of message middleware, outlines its core components, and compares RocketMQ and Kafka architectures to show how they achieve high‑availability through replication, leader election, and coordinated producer‑consumer interactions.
