Douyin Group's Data Management: Strategies for Metric Construction, Management, Production, and Consumption

Douyin Group, a core unit of ByteDance, manages data volumes exceeding the exabyte level with over 600 PB of metric assets, presenting significant challenges for data quality and efficiency.

The Volcano Engine data platform is organized into four layers—data engine, data construction management, data analysis application, and solution/consulting services—supporting agile, real‑time processing and seamless integration with tools like Feishu and calendars.

Three major pain points are identified: inconsistent metric definitions, unclear metric scopes, and fragmented metric consumption, which lead to redundant tables and wasted resources.

A three‑layer technical solution is proposed: (1) metric production, ensuring model design and data quality; (2) metric management, focusing on efficiency and consistency; (3) metric consumption, delivering metrics as service‑oriented topics for multi‑point reuse.

Metric management practice emphasizes consistency (avoiding "same name, different meaning"), continuous freshness through iterative mechanisms, and efficiency via streamlined definition and decomposition processes.

Organizational design assigns clear responsibilities to business owners, data application teams, and public‑layer data teams to maintain data integrity and accelerate decision‑making.

Consistency is enforced through strict decomposition standards, unique metric validation, and similarity checks of atomic metrics and modifiers.

Efficiency improvements include a focus on core metrics, process optimization, batch scripts for automated decomposition, and exploration of large‑model‑driven automatic metric splitting.

Metric production follows a hierarchical model design—from detailed to aggregated layers—balancing dimensional coverage, performance, and extensibility.

A comprehensive quality assurance system covers product modules, responsibility allocation, and standards for accuracy and timeliness, with full‑link lineage tracking and daily governance via alerting tools.

Stability is achieved through standardized data layer outputs, upstream link optimization, and daily operational practices such as on‑call rotations, SLA agreements, and fault‑response mechanisms.

Metric consumption is realized through metric topics that act as virtual tables, offering low‑cost setup, fast discovery, cross‑cluster/data‑source analysis, and intelligent routing.

Topic management provides flexible directory structures, fine‑grained permission control, and easy import processes, while topic lists and detail pages give clear visibility of business impact, technical definitions, and consumption lineage.

Future work focuses on standardizing, configuring, and automating metric production, leveraging large‑model automation for metric decomposition, and delivering a unified data architecture that enables one‑definition‑multiple‑consumption across the organization.