Calibration Techniques for User Response Prediction in Online Advertising

Abstract: Calibration is a research branch of trustworthy machine learning that improves the uncertainty of model predictions in terms of accuracy and confidence. This talk presents the evolution and engineering practice of calibration algorithms used in Alibaba Mama's display advertising system.

Background: In computational advertising, user‑behavior probability estimation (e.g., click‑through rate, conversion rate) is core to the system. Although deep learning has dramatically improved model capacity, the true click probability cannot be observed directly, leading to systematic bias in predictions. Traditional metrics such as AUC only measure ranking quality and ignore the absolute magnitude of the predicted values, which is critical for bidding, budget allocation, and fairness.

Need for Calibration: Accurate magnitude of predicted values (size‑accuracy) is essential for:

Precise bidding (CPC, CPM, Auto‑Bid, OCPX)

Stable auction outcomes

Fairness across mixed‑bidding and mixed‑media scenarios

Examples illustrate how over‑ or under‑estimation of pCTR can cause revenue loss for the platform or advertisers.

Calibration Objective: Make the predicted probability f(x) as close as possible to the true probability Y . The goal is to improve absolute accuracy while preserving ranking performance.

Related Work: Early calibration methods (Platt Scaling, Histogram Binning, Isotonic Regression, Scaling) are post‑processing techniques that adjust the output of a base model. Recent research combines these methods and provides theoretical guarantees.

Post‑Processing Approach at Alibaba: The calibration module is decoupled from the base prediction and ranking modules, enabling flexible plug‑in deployment and rapid response to distribution shifts.

Key Technical Issues: Selecting appropriate clustering (PV grouping) for calibration, balancing granularity against data sparsity, and handling data drift.

Calibration Evaluation Metrics:

Predict Click Over Click (PCOC): ratio of calibrated CTR to posterior CTR; closer to 1 indicates better size‑accuracy.

Calibration‑N (Cal‑N): computes PCOC on multiple clusters and aggregates deviation from 1.

Grouped Calibration‑N (GC‑N): extends Cal‑N with dimension‑wise weighting.

Algorithm Evolution:

Smoothed Isotonic Regression (SIR): Combines binning, isotonic regression, and linear scaling. Uses monotonic smoothing to alleviate data sparsity while remaining lightweight and interpretable.

Bayesian SIR (Bayes‑SIR): Addresses cold‑start problems by incorporating prior click‑rate distributions from historical data and applying Bayesian updating to sparse new observations.

Real‑Time Wave‑Adjustment Bayes‑SIR (RTW‑BSIR): Mitigates temporal performance fluctuations caused by distribution drift by estimating and correcting the bias between training and online data in real time.

Post‑Click Conversion Estimation Model (PCCEM): Extends calibration to downstream metrics (conversion, add‑to‑cart, etc.) that suffer from delayed feedback and extreme sparsity. It predicts long‑term conversion using short‑term post‑click signals and then applies the calibrated CTR methods.

Engineering Practice: The calibration module sits between the prediction and ranking stages. Data flow diagrams show separate paths for shallow metric calibration (direct SIR/PCCEM) and deep metric calibration (click‑quality estimation followed by calibration).

Summary & Outlook: Calibration has been widely applied in weather forecasting, medical diagnosis, autonomous driving, and advertising. Alibaba Mama has deployed lightweight, interpretable calibration methods since 2018, achieving significant business gains. Future work includes exploring more complex, fine‑grained calibration techniques and strengthening theoretical research.

Q&A Highlights:

Prior data are historical, possibly coarse, while observation data are short‑term, real‑time samples.

Data drift arises from user behavior changes, advertiser actions, and system latency.

Calibration models benefit from frequent (hourly) updates to track rapid distribution changes.

Window length for training data balances responsiveness and statistical confidence.

Calibration is applicable to recommendation systems when absolute metric accuracy is required.

Bucket count in SIR trades off granularity against confidence; optimal values are chosen empirically.

For more details, see the referenced GitHub repository and related conference papers (ICML, NIPS, KDD, etc.).