Calibration Techniques for User Behavior Prediction in Online Advertising: Background, Algorithm Evolution, and Engineering Practice

Introduction – Calibration is a research branch of trustworthy machine learning that improves the uncertainty of model predictions in domains like medical diagnosis, weather forecasting, autonomous driving, and especially online advertising.

Background – In advertising, click‑through rate (CTR) and other user‑behavior probabilities are estimated but cannot be observed directly; models therefore suffer from bias, and traditional metrics like AUC only assess ranking quality, not absolute value accuracy.

Need for Calibration – Accurate probability estimates affect bidding fairness, revenue, and plan stability, especially under mixed bidding types, diverse ad formats, and ad‑recommendation mixing.

Calibration Objectives & Related Work – The goal is to make predicted values as close as possible to true probabilities. Early work includes Platt Scaling, Histogram Binning, Isotonic Regression, and scaling methods; recent research focuses on post‑processing approaches for flexibility.

Algorithm Evolution Smoothed Isotonic Regression (SIR): combines binning, isotonic regression, and linear scaling to handle data sparsity while preserving monotonicity. Bayes‑SIR: adds Bayesian smoothing to address cold‑start problems by incorporating prior CTR distributions. RTW‑Bayes‑SIR: mitigates temporal performance fluctuations by correcting distribution drift in real time. Post‑Click Conversion Estimation Model (PCCEM): extends calibration to downstream metrics (conversion, add‑to‑cart) using short‑term user signals to predict long‑term outcomes.

Engineering Practice – The calibration module is placed between the prediction and ranking stages, allowing independent deployment. Data flow diagrams illustrate how shallow metric calibration uses tracked data directly, while downstream metrics require an intermediate click‑quality estimation before calibration.

Summary & Outlook – Calibration has been widely applied in Alibaba’s display advertising since 2018, improving system stability, ROI, and bidding fairness. Future work includes exploring more sophisticated, fine‑grained methods, deeper theoretical analysis, and broader applications beyond advertising.

Q&A Highlights – Discussed differences between prior and observation data, causes of data drift, update frequency of calibration models, and applicability of calibration to recommendation systems.