Introduction to Apache Flume: Architecture, Core Concepts, Configuration and Usage

Flume is a distributed, reliable, and highly available system for massive log aggregation that allows users to define custom data sources, perform simple processing, and write events to various configurable sinks.

Design goals: (1) Reliability – three levels of guarantee (end‑to‑end, store‑on‑failure, best‑effort); (2) Scalability – three‑tier architecture (agent, collector, storage) with horizontal scaling and ZooKeeper‑managed masters; (3) Manageability – unified master control, web and shell management interfaces; (4) Extensibility – pluggable agents, collectors, and storages with many built‑in components.

Core concepts: Agent (runs on a JVM, may contain multiple sources and sinks), Client (produces data), Source (collects data from clients), Sink (consumes data from channels), Channel (queues events between sources and sinks), and Event (basic data payload).

Typical deployment involves downloading the latest Flume package (e.g., 1.6.0) from flume.apache.org , extracting it, and preparing configuration files.

Common configuration patterns include:

Pattern 1 – scanning a specific file.

Pattern 2 – (image omitted for brevity).

Pattern 3 – scanning a directory for new files.

For this tutorial the first pattern is used to integrate Flume with Kafka.

Before starting Flume, ensure ZooKeeper and Kafka are running:

./zkServer.sh start

Start Kafka (example command):

.fka-server-start.sh -daemon ../configrver.properties

Consume messages with the default Kafka console consumer:

.fka-console-consumer.sh -zookeeper localhost:2181 --from-beginning --topic testKJ1

Generate test log data with a simple shell script (output.sh):

for((i=0;i<=50000;i++));
 do echo "test-"$i>>abc.log;
 done

Run the script, then start Flume with the following command:

./bin/flume-ng agent -n agent -c conf -f conf/hw.conf -Dflume.root.logger=INFO,console

The final log line "Component type:SINK,name:k1 started" indicates successful startup.

The complete workflow is illustrated in the following diagram:

In summary, this guide introduces the basics of Flume, its architecture, configuration, and integration with Kafka, laying the groundwork for deeper source‑code analysis and advanced tuning.