Model Serving in Real-Time: Insights from Alibaba’s User Interest Center

In this post, the author introduces the concept of model serving, which addresses how to perform real‑time inference with models that have been trained offline.

Traditional serving faces challenges when QPS reaches thousands per node, requiring a dedicated model server to deliver predictions within tens of milliseconds.

The article reviews four mainstream serving methods—self‑built platforms, pre‑trained embeddings with lightweight models, PMML‑style serialization tools, and native TensorFlow Serving—then focuses on Alibaba’s User Interest Center (UIC) solution.

Two architectural patterns are presented: Architecture A, a classic pipeline where offline training produces a model that consumes user behavior features and ad features in an online prediction server; and Architecture B, which replaces the online user‑behavior feature store with a User Interest Representation store powered by UIC.

UIC generates and updates user interest embeddings (vectors) based on real‑time behavior events, allowing the online prediction server to skip costly sequence‑model inference and directly run a lightweight MLP, thus cutting latency dramatically.

The solution essentially follows an “Embedding + lightweight online model” deployment, with embeddings refreshed asynchronously (near‑real‑time) via offline inference triggered by behavior changes.

Empirical data shows that under 500 QPS, DIEN’s inference time drops from 200 ms to 19 ms using the UIC architecture.

The article concludes that Alibaba’s UIC‑based serving method combines theoretical machine‑learning advances with practical engineering, offering a best‑practice for teams struggling with serving efficiency and latency.

Finally, two discussion questions are posed: the trigger mechanism for embedding updates in UIC, and whether sequence models must remain offline or can be accelerated for online inference.