Evolution of E‑commerce Platform Recommendation System Architecture

The talk, originally presented at DataFunTalk, outlines how recommendation systems for C2C marketplaces such as Xianyu have progressed from a basic two‑stage offline‑online framework (the "Stone Age") to increasingly sophisticated architectures.

In the early stage, offline jobs generated recommendation material using Spark matrix factorization, while online services performed simple table look‑ups, resulting in low personalization, limited recall dimensions, and poor cache efficiency.

The subsequent "Bronze Age" introduced finer granularity by shifting from category‑level to item‑level targeting, adding multiple recall dimensions through collaborative filtering combined with user and item profiles, and improving data reuse across the pipeline.

The "Industrial Revolution I" phase focused on real‑time recommendation, making both offline mining and user‑interest inference live, which boosted conversion rates by 80‑90% and established continuous improvement as a core principle.

Finally, the "Machine‑Learning Era" emphasizes building a robust ML pipeline for feature generation, model training, and real‑time serving, enabling both ranking and recall models, better utilization of negative feedback, and alignment of optimization goals with business metrics.

The article also stresses the importance of effective user profiling: creating useful, well‑connected, and detailed profiles, avoiding over‑ or under‑granular attributes, and ensuring consistent, real‑time availability across teams and storage systems.

Key challenges discussed include synchronization of profile updates, heterogeneous downstream consumption (e.g., Redis vs. Elasticsearch), and coordinated upgrades across multiple services.

Overall, the evolution follows four main directions—refined relevance, multi‑dimensional recall/sort, real‑time feedback, and model‑driven computation—each contributing to incremental improvements in recommendation performance.