Comprehensive Overview of Big Data Architecture, Lambda/Kappa Models, and End-to-End Data Platform Design

In recent years, rapid advances in IT, big data, machine learning, and algorithms have led many enterprises to treat data as a valuable asset. Without a coherent overall data architecture, businesses face gaps between data and applications, resulting in unknown data sources, unmet requirements, and poor data sharing.

1. Big Data Technology Stack

The article first outlines the basic components of big data and introduces the overall technology stack, which includes data acquisition, transmission, real‑time processing, batch processing, and storage. (Image omitted for brevity.)

2. Lambda and Kappa Architectures

Most modern big‑data systems are built on either the Lambda or Kappa model. Lambda provides a dual‑pipeline (batch + speed) architecture with high flexibility, scalability, and fault tolerance. Kappa simplifies the design by using a single stream processing pipeline, eliminating the cost of maintaining two separate data‑processing paths.

3. Typical Big Data Architecture under Lambda/Kappa

Typical implementations combine components such as Kafka, Flink/Spark, HBase, Elasticsearch, and data warehouses to form an integrated pipeline (illustrated in the original diagram).

4. End‑to‑End Pain Points

Lack of an integrated data‑development IDE for managing the whole lifecycle.

No standardized data‑modeling system, leading to inconsistent metric definitions.

High skill barrier for business users to directly use components like HBase or ES.

Complex team structures make issue tracing difficult.

Data silos hinder cross‑team data sharing.

Separate batch and stream computation models increase development effort.

Missing enterprise‑level metadata governance.

These issues make data platform governance and open‑capability provision challenging.

5. Exemplary Big Data Architecture Design

A well‑designed platform should provide:

Multi‑source data acquisition.

One‑click data synchronization.

Data quality and modeling tools.

Metadata management.

Unified data access.

Real‑time and batch computation engines.

Resource scheduling.

One‑stop development IDE.

(Diagram of the integrated platform omitted.)

6. Metadata – The Foundation of Big Data Systems

Metadata records the complete lineage from data generation to consumption, covering static schema information, dynamic task dependencies, data‑warehouse models, lifecycle, and ETL scheduling. It enables data graphs, DAG orchestration, quality governance, and resource‑usage overview. Without a comprehensive metadata layer, organizations face traceability, permission, resource, and sharing problems.

7. Stream‑Batch Unified Computing

Maintaining separate engines (e.g., Spark for batch, Flink for streaming) burdens users. A custom DSL can abstract engine‑specific syntax, allowing developers to write a single language that targets multiple back‑ends.

8. Real‑Time & Batch ETL Platform

ETL platforms should support multiple data sources, a rich set of operators (filter, split, transform, output), and dynamic logic updates via hot‑swap JARs.

9. Intelligent Unified Query Platform

Traditional point‑to‑point APIs lead to coarse granularity, low reusability, and high maintenance. An intelligent query layer abstracts underlying stores (e.g., HBase) and provides unified access, simplifying permission management and reducing duplicated development.

10. Data‑Warehouse Modeling Standards

Inconsistent naming (e.g., good_id vs. spu_id) and ambiguous metric definitions cause confusion and high development cost. A unified modeling framework (e.g., Alibaba’s OneData) enforces naming conventions, granularity standards, and reuse policies.

11. One‑Click Integration Platform

Data from various sources (binlog, logs, front‑end events, Kafka, etc.) can be ingested with a single click, routed through the transmission layer to ETL, linked with metadata for schema governance, and finally delivered to real‑time or batch compute engines.

12. Data Development IDE – End‑to‑End Tool

An integrated IDE offers data integration, development, management, quality, and service capabilities, enabling developers to work with data as easily as writing SQL. References include Alibaba Cloud DataWorks.

13. Additional Considerations

Complete data‑system engineering also involves alerting, monitoring, resource isolation, quality detection, and a one‑stop data processing suite.