Expert Interview: Architecture, Components, and Future Trends of Big Data Platforms

DataFun interviewed several big‑data platform experts to discuss the current architecture, key components, challenges and future trends of big data platforms.

01 Big Data Platform Architecture

The platform consists of core modules such as data integration, storage & computation, distributed scheduling and query analysis. The article follows this diagram to explore each area.

02 Data Integration

Key technologies include log synchronization (Flume, Vector), data extraction tools (DataX, BitSail) and transmission queues (Kafka, RabbitMQ, Pulsar). Experts emphasize the importance of reliable, high‑throughput log sync and robust data pipelines.

Log synchronization must handle large volumes and guarantee continuity with buffering mechanisms to avoid data loss.

Data integration is the first touchpoint for business; slow or lossy pipelines erode trust in the platform.

Kafka is widely known but less user‑friendly; Pulsar offers a more advanced architecture.

03 Data Processing: Storage & Computation

Storage relies on HDFS, which offers horizontal scalability and high fault tolerance. Experts note optimization of HDFS architecture, read/write separation and emerging meta‑data‑separated systems like JuiceFS.

Computation engines include batch (MapReduce, Hive, Spark) and streaming (Storm, Spark Streaming, Flink). Flink is highlighted for its unified batch‑and‑stream capabilities, though it still lags in massive offline workloads.

Flink excels at real‑time processing but needs better stability and offline performance.

04 Data Scheduling

Common schedulers mentioned are Crontab, Apache Airflow, Oozie, Azkaban, Kettle, XXL‑JOB, Apache DolphinScheduler and SeaTunnel. DolphinScheduler is praised for its Chinese UI and big‑data focus, while Airflow remains popular internationally.

Resource schedulers such as YARN and Azkaban are also discussed, with YARN being widely adopted.

05 Big Data Query

OLAP engines compared include Presto, StarRocks and Impala. StarRocks delivers the highest performance but at higher resource cost; Impala can approach StarRocks after tuning; Presto is easy to use but slower.

Query‑optimization tools like Alluxio, JuiceFS and JindoFS provide data orchestration and caching, each with different strengths and deployment scenarios.

06 Future Trends

Experts foresee OLAP as the focal point, emphasizing faster computation, elastic storage, and cloud‑native designs. Object storage adoption is growing for cost‑effective, low‑maintenance data lakes. Cloud‑native architectures will gain importance, though stability and hardware performance remain challenges. Real‑time computing, especially Flink, still has room for improvement.

The interview featured Zhang Yaodong (Xiaomi), Zhu Jianghua (NetEase) and Fan Yuchen (NetEase), who shared their practical experiences and insights.