Practical Guide to Modeling Stability: Feature PSI, Model PSI, and Monitoring Techniques

Background : Modeling is a core task for algorithm and data‑mining engineers, yet the stability of features and models is often ignored.

The author shares practical experience so readers can immediately apply the methods to avoid hidden problems in production.

Stability in modeling consists of two parts: feature stability and model stability.

Feature Stability : It refers to whether a feature’s values change dramatically over time. This should be evaluated before model building to prevent unstable features from entering the model. The common metric is the Population Stability Index (PSI). The PSI formula is shown below:

PSI compares a base dataset with a test dataset (or expected vs. actual). The calculation steps are:

Divide the feature values in the base set into equal‑frequency bins (typically 10).

For each bin, compute the proportion of records (or users, stores, etc.) in both base and test sets.

Apply the PSI formula to obtain the stability score.

If a PSI value is below 0.1 for a six‑month span, the feature is generally considered stable, though business context may allow a higher threshold.

In Ant Financial’s PAI platform, a PSI component can batch‑process hundreds of features automatically, as illustrated in the workflow diagram:

The author describes a concrete example: using data from 20171130 as the base set, PSI is computed for each feature against the previous six months, and features with PSI < 0.1 are kept.

Model Stability : Similar to feature stability, model stability focuses on the output probability (prediction_prob) of a binary classifier. Treat this probability as a feature and compute its PSI using the same method.

Additional practices include mapping the continuous probability to an integer rank (e.g., 1‑10, 1‑100) to suppress volatility. The mapping process involves:

Calculating quantiles of the probability distribution (10 quantiles for a 1‑10 scale, 100 for a 1‑100 scale).

Assigning each probability to the corresponding integer bucket.

After ranking, two monitoring tasks are recommended: monthly drift of quantile boundaries and month‑to‑month variation of the integer scores.

Monitoring : Implement daily monitoring of PSI and rank drift, either via the internal monitoring platform or by scheduling SQL jobs that generate reports.

The article concludes by inviting readers to share their own experiences and suggestions on modeling stability.