Design and Architecture of JD.com’s Buffalo Distributed DAG Scheduling System

1. Introduction Buffalo is a JD.com‑developed distributed DAG scheduling system that provides data engineers, algorithm engineers, and analysts with offline job orchestration, debugging, monitoring, and resource‑containerized execution, aiming for a stable, efficient, and user‑friendly ETL platform.

Challenges The system faces (1) complex business‑driven dependency graphs with hundreds to thousands of upstream/downstream tasks spanning dozens of layers, (2) massive scale with hundreds of thousands of tasks, millions of dependencies, and millions of daily schedule executions demanding high stability and performance, and (3) diverse data processing scenarios requiring rich scheduling capabilities.

2. Core Technical Solutions

2.1 Entity and Orchestration Model A two‑layer entity model is used: Action (the smallest executable unit containing script, parameters, environment) and Task (a DAG composed of one or more Actions plus trigger rules). This double‑layer model enhances flexibility and manageability compared with a single‑layer model.

2.2 Instance‑Based Scheduling Tasks are defined statelessly; at each run cycle a task instance is created (instantiation) based on the task’s configuration (actions, upstream dependencies, data dependencies, schedule). Instances are the executable, stateful objects. Benefits include stable periodic execution and explicit, traceable dependencies.

2.3 Classification‑Based Scheduling Tasks are classified and prioritized; higher‑priority tasks receive resource guarantees during contention, and classification information propagates to the underlying compute clusters for differentiated handling.

3. High‑Availability Architecture Buffalo consists of three layers, each built with high‑availability designs: Manager (stateless management services for task creation and operation, horizontally scalable), Scheduler (NameNode) (core engine handling instance generation, DAG scheduling, and resource allocation, using active‑active multi‑active architecture with master‑backup for state handling), and Execution Layer (supports both long‑running workers on physical machines and short‑lived pods on Kubernetes, both with failover mechanisms).

4. High Performance

4.1 Horizontal Scaling The multi‑active scheduler distributes load via hash‑based sharding, allowing the system to scale out horizontally while ensuring idempotent processing of task batches.

4.2 Event‑Driven Execution Instead of periodic polling, Buffalo reacts to state‑change events, performing condition checks only when relevant dependencies change, reducing unnecessary scans and enabling parallel asynchronous handling.

4.3 In‑Memory Scheduling Resource scheduling is performed in‑memory on a primary node selected from the NameNode cluster, avoiding distributed locks and external storage bottlenecks, thus greatly improving throughput.

4.4 Hot‑Cold Data Separation Completed task instances (cold data) are periodically migrated to separate storage, while active instances (hot data) remain in the primary store. An index table records partition ranges to enable fast location of cold data, and cold data can be restored to hot tables for operations such as re‑run or forced success.

5. Open Capabilities Buffalo exposes HTTP APIs for task configuration, instance manipulation, status and log queries, and provides asynchronous event streams via JDQ for real‑time status propagation to downstream business systems.

6. Future Plans Ongoing enhancements target better user experience, extreme performance, containerization, plug‑in extensibility, finer‑grained resource management, and broader open capabilities, inviting community feedback for continuous improvement.