Design and Implementation of Snowball's Model Feature Management Platform

Overview Snowball's algorithm engineering team needed a platform to manage the rapid growth of features for ranking and search in its stock‑community scenario, where traditional code‑centric integration limited iteration speed and system stability.

Background and Problem Description The platform must handle massive feature volumes, support fast feature iteration, efficient computation, and maintain engineering performance while serving personalized content to millions of daily active users.

New Platform Architecture In 2021 a new model‑feature platform (ugc‑model‑sled) was built, consisting of a web‑UI management module and an online engine module. The management module stores metadata for models, features, and configurations in a database, while the engine assembles features and serves scoring requests.

Model Management Models are registered with metadata (name, version, type, output definition, feature set) and managed through an automated online suite that handles model deployment, version control, and consistency across prediction clusters.

Feature Management Features are organized into three layers: raw data layer (original user/post/stock attributes), derived feature layer (cross‑features and transformations), and model‑feature layer (pre‑processed inputs for models). Each layer has its own metadata tables and APIs for registration and lifecycle control.

Online Engine Design and Optimizations The engine retrieves raw features, applies derived and model‑feature transformations, and forwards data to the prediction service. Optimizations include aggregated gRPC queries, protobuf+snappy compression, asynchronous feature fetching, and off‑heap caching to reduce I/O and CPU overhead.

Feature Logging Solution To record features for offline training without impacting online latency, logging is decoupled from scoring by performing an asynchronous feature fetch after the recommendation result is returned, ensuring consistency and low overhead.

Benefits and Outcomes The platform provides full feature lifecycle management, reduces feature integration time from weeks to days, enables feature reuse across projects, supports online/offline consistency, and markedly improves system stability and engineering efficiency.

Conclusion and Outlook After a year of evolution, the platform supports over 200 features and 50 models, with future work focusing on further reducing raw‑data acquisition latency and extending the feature‑layer to additional model families such as XGBoost.