Overview of the Qirin Big Data Platform Architecture and Core Modules

The Qirin big data platform, developed by the 360 System Department since 2010, provides a comprehensive one‑stop solution for the entire big data development lifecycle, serving over 30 departments, 1,000+ users, with more than 25,000 servers and exabyte‑scale storage.

1. Platform Architecture – The platform consists of eight functional modules from bottom to top: Resource Management, Metadata Management, Data Ingestion, Task Development, Interactive Query, Data Services, Permission Center, and System Management, all designed to lower usage barriers and accelerate business data value.

2. Resource Accounts – Multi‑tenant support is achieved through departments, resource groups, and project accounts, which manage storage, compute, and permission isolation; resource groups define business topics, while project accounts are used for accessing storage and submitting tasks.

3. Resource Management – Users can request various resources such as file, table, and object storage (HDFS/Hive/HBase/Cassandra), compute resources (YARN, containers), message buses (Kafka, QBus, NSQ), analytical engines (Druid, Doris), search indexes (Poseidon, ElasticSearch), and service resources (ScribeQ), with budgeting and cost tracking.

4. Metadata Management – Provides a unified view of all data assets, supporting APIs for data governance; it manages data sources like MySQL, Hive, Kafka, HDFS and metadata objects such as tables, topics, and paths.

5. Data Ingestion – The xCollector module enables real‑time collection of logs and other data, supporting file and Kafka sources, and can push data to Kafka, HDFS, or ElasticSearch after parsing and transformation at the edge.

6. Task Development – Offers visual workflow composition and code editing for batch and streaming jobs, supporting Shell, TransX, MapReduce, Spark, Flink, SQL, and provides scheduling, monitoring, and alerting capabilities, as well as open APIs for integration.

7. Interactive Query – Built on the BigSQL engine, it allows fast, cross‑source SQL queries over Hive, MySQL, Druid, ElasticSearch, HBase, etc., with result download and simple visualization, and supports saved query templates.

8. Self‑Service Analysis – A lightweight visual analytics tool based on open‑source components, enabling users to create dimensional models, dashboards, and share reports via links.

9. Future Roadmap – Plans include continuous enhancement of metadata, task development, and ingestion capabilities; opening more APIs for data governance; expanding resource services; delivering generic big data solutions for external projects; and integrating container‑cloud management for elastic scaling.

10. Conclusion – Qirin 1.0 was built in six months, encapsulating distributed big data components into a unified kernel‑library‑interface platform, and will continue to evolve with a focus on productization, service orientation, and supporting enterprise data governance.