Common Pitfalls in Recommendation Systems: Metrics, Exploration‑Exploitation, and Offline‑Online Discrepancies

Recommendation systems often face unclear evaluation metrics; optimizing for CTR can conflict with user satisfaction, while high‑stay‑time or read‑U metrics push different content types, leading to trade‑offs that are not well‑defined.

Metrics such as CTR, stay‑time, and read‑U are interdependent, and over‑optimizing one can degrade others, as seen in platforms like 今日头条 and Medium.

The exploration‑exploitation (E&E) dilemma highlights the need to balance precise recommendations with user interest discovery, acknowledging that overly narrow feeds can reduce long‑term engagement.

Offline‑online gaps arise from three main issues: (1) data leakage where features strongly correlated with labels leak information; (2) inconsistency between offline and online feature pipelines, often due to different codebases or timing delays; (3) distribution shifts where offline training data (the "iceberg tip") differs from the full online data distribution.

Solutions include ensuring identical feature extraction code for training and serving, aligning data timestamps, up‑sampling unbiased samples, and blending online and offline model scores with a linear combination.

Additional practical pitfalls discussed involve magic‑number parameters in similarity calculations, limited adoption of advanced algorithms like SVD, and the impact of business constraints (rules, popularity weighting) on model performance.

The article concludes that while technical issues can be mitigated, business‑driven constraints often present the toughest challenges for recommendation system engineers.