1 views collected around this technical thread.
This article explains why Kafka consumers often become bottlenecks—due to complex processing, resource constraints, or sub‑optimal configuration—and provides concrete steps such as profiling Java code, simplifying logic, using background threads, scaling consumer instances, and tuning key consumer parameters.
This article explains the three message delivery semantics, the common causes of duplicate messages in queue systems, and presents concrete producer‑side, broker‑side, and consumer‑side deduplication techniques for Kafka and Pulsar, including code samples and best‑practice recommendations.
This article explains how Kafka achieves high concurrency through its distributed broker cluster, partitioned topics, sequential log writes, message compression, asynchronous producer mechanisms, and OS page‑cache techniques, illustrating the combined architectural and performance optimizations that enable massive throughput.
This article explains Kafka's event streaming concept, detailing events and streams, core components such as producers, topics, partitions, consumers, persistence, and typical real‑time data pipeline, event‑driven architecture, stream processing, and log aggregation use cases, highlighting its role as a foundational big‑data infrastructure.
This tutorial walks through downloading, extracting, configuring, starting, creating topics, producing and consuming messages, and finally stopping Apache Kafka 3.8.1 on a Linux system, including all necessary command‑line instructions.
This article examines the typical reasons Kafka messages are lost across producers, brokers, and consumers, and provides detailed configuration recommendations and best‑practice solutions to significantly reduce the risk of data loss in distributed streaming systems.
This article details practical techniques for boosting Kafka producer throughput, covering batch sending optimization, linger.ms tuning, asynchronous send handling, compression options, and concurrent sending strategies, while providing code examples and performance considerations for high‑throughput streaming applications.
This article explains why Kafka is widely adopted by top companies, outlines its high throughput, scalability, and durability, and describes key real‑time data pipeline, stream processing, and big‑data integration scenarios, concluding that mastering Kafka is essential for modern backend and data engineering roles.
This article reviews six mainstream solutions for keeping MySQL and Elasticsearch in sync—including synchronous double‑write, asynchronous MQ‑based double‑write, Logstash polling, Canal binlog listening, DataX batch migration, and Flink stream processing—detailing their scenarios, advantages, drawbacks, and practical code examples to guide optimal technical selection.
Kafka 4.0 replaces ZooKeeper with the native KRaft (Kafka Raft) consensus layer, simplifying deployment, boosting scalability, cutting latency, and introducing a new incremental consumer rebalance protocol and share‑group model that together reshape how distributed streaming applications are built and operated.
This technical case study details how a high‑traffic Kafka logging cluster was optimized by adjusting Filebeat and Kafka parameters, increasing compression batch size, and tuning Kubernetes settings, resulting in significant reductions in request volume, network traffic, CPU usage, and overall resource consumption.
This article explains how to import 1 billion 1 KB log records stored in HDFS or S3 into MySQL by analyzing single‑table limits, using batch inserts, choosing storage engines, sharding, optimizing file‑reading methods, and coordinating distributed tasks with Redis, Redisson, and Zookeeper to ensure ordered, reliable, and high‑throughput data loading.
This guide reviews common backend interview topics, explaining TCP TIME_WAIT behavior, multi‑port listening, full page load steps, Redis data types, Kafka consumer sizing and at‑most‑once semantics, the CAP theorem, Singleton usage, C++ std::map complexity, and an O(n) doubly‑linked list reversal algorithm.
This article explains how to leverage Spring's TransactionSynchronizationManager to detect active transactions and register synchronization callbacks that asynchronously send Kafka messages after transaction commit, illustrated with a practical payment‑system example and complete Java code snippets.
This guide walks through initializing a Spring Boot project, handling IDE setup challenges, managing version compatibility between Spring Cloud, Spring Boot, and Kafka, and provides practical code examples for global exception handling, logging, CORS configuration, Swagger integration, and useful development tools.
This article visually breaks down Kafka’s fundamental concepts—including producers, consumers, topics, consumer groups, partitions, and cluster architecture—explaining how messages flow, are stored, and achieve load balancing and ordering within a distributed streaming system.
The article explains how to guarantee that messages are processed only once in high‑concurrency environments by combining production‑side idempotent publishing, broker‑level deduplication with unique IDs, and consumption‑side business idempotency such as database constraints or distributed locks, while also recommending monitoring, metrics, and reconciliation as safety nets.
This article explains common causes of duplicate message consumption in high‑traffic systems and presents a three‑layer defense—producer idempotence, broker de‑duplication, and consumer idempotent design—plus monitoring and reconciliation strategies to achieve reliable exactly‑once processing.
This article explains how RocketMQ, a Chinese‑origin message queue, simplifies Kafka’s architecture while adding powerful features such as tag‑based filtering, transactional messaging, delayed and dead‑letter queues, and a unified commit‑log storage model, making delayed processing and high‑throughput scenarios easier to implement.
This article introduces a reusable asynchronous processing SDK built on Spring, Kafka, and MySQL that leverages @AsyncExec annotations, transactional event listeners, and configurable thread pools to ensure reliable, non‑blocking execution, data consistency, and fault‑tolerant handling of business logic in backend applications.