Design and Implementation of a Scalable User Tag Production Platform

This article introduces the concept of user tags as dimensional descriptors of users that enable product optimization, precise marketing, and other business scenarios. It outlines the need for a flexible, comprehensive tag system and describes the challenges of coverage and practical adoption.

It then details the requirements of a tag production platform, emphasizing application‑driven tag construction, systematic organization of tags according to usage purpose, and three major tag categories: numeric aggregation tags, cohort (group) tags, and state‑transition tags.

The technical design is presented as a batch‑stream integrated architecture. The data flow starts from various SDKs, passes through ingestion, storage, and query layers, and is split into user‑behavior and user‑attribute tables. Offline batch jobs (using Impala + HDFS parquet) read these tables, apply SQL‑based tag rules, and generate daily tag tables.

To improve query performance, individual tag tables are periodically merged into a wide view, which is later materialized as a physical table. Bitmap (RoaringBitmap) techniques are used to accelerate cohort filtering.

Real‑time tags are computed with Flink: a Flink job consumes Kafka event topics, maintains state machines, stores intermediate state in Flink state and KV stores, and outputs tag updates to a Kafka tag topic. Offline tag results are synchronized to KV for fast lookup.

The overall system combines batch efficiency with stream low‑latency, allowing both daily offline tag updates and second‑level real‑time tag changes, while supporting billions of users with scalable storage and computation.

Finally, the article summarizes that a successful tag platform must be application‑oriented, flexible in rule definition, and built on a stable data model, leveraging batch‑stream integration to meet diverse business needs.