PSI for Model Monitoring: Useful Signal, Not a Verdict

A model can perform well in one period and degrade after its input population changes. PSI is one diagnostic that can surface distribution shift before or alongside business-metric movement; it does not explain every performance change by itself.

What PSI Measures

Population Stability Index measures how much a variable's distribution has shifted between two time periods — typically your training data (baseline) versus current production data.

If your model was trained on data where 60% of users were mobile and now 80% are mobile, the model is serving a population mix that differs from the baseline. PSI summarizes that distribution difference without identifying its cause or performance impact.

The Formula

PSI = SUM((Actual% - Expected%) * ln(Actual% / Expected%))

Where:

• Expected% = distribution in training data (baseline)
• Actual% = distribution in current production data
• Calculated across N bins — typically 10 or 20

Interpreting PSI Values

• PSI below 0.10 — often treated as a low-shift signal, subject to feature and sample context
• PSI 0.10 to 0.20 — commonly used as an investigation range, not an automatic verdict
• PSI above 0.20 — a strong review signal; validate sample size, binning, model performance and business impact

Python Implementation

import numpy as np
import pandas as pd

def calculate_psi(expected, actual, buckets=10):
    breakpoints = np.percentile(expected,
                                np.linspace(0, 100, buckets + 1))
    breakpoints = np.unique(breakpoints)

    expected_counts = np.histogram(expected, bins=breakpoints)[0]
    actual_counts = np.histogram(actual, bins=breakpoints)[0]

    expected_pct = expected_counts / len(expected) + 1e-6
    actual_pct = actual_counts / len(actual) + 1e-6

    psi_values = (actual_pct - expected_pct) * np.log(actual_pct / expected_pct)
    return np.sum(psi_values)

# Usage
train_scores = model.predict_proba(X_train)[:, 1]
prod_scores = model.predict_proba(X_production)[:, 1]

psi = calculate_psi(train_scores, prod_scores)
print(f"Score PSI: {psi:.4f}")

if psi > 0.20:
    print("REVIEW: distribution shift exceeds the configured threshold")
elif psi > 0.10:
    print("REVIEW: distribution shift is above the investigation threshold")

Score PSI vs Feature PSI — Critical Distinction

Monitoring only score-level PSI is incomplete. Score PSI can look stable even when individual features have shifted because feature-level changes can offset one another at the aggregate score.

Score-level PSI alone may be incomplete. Monitor risk-relevant input features where lineage, volume and actionability justify it. In the synthetic CreditScoreV4 example—not a real customer or production incident—overall score PSI is 0.12 (below the example's critical threshold) while device_risk_score reaches 0.31. The example shows why score-only monitoring can miss a feature-level shift.

feature_psi_report = {}

for feature in X_train.columns:
    psi = calculate_psi(
        X_train[feature].values,
        X_production[feature].values
    )
    feature_psi_report[feature] = psi

psi_df = pd.DataFrame.from_dict(
    feature_psi_report, orient='index', columns=['psi']
).sort_values('psi', ascending=False)

print(psi_df[psi_df['psi'] > 0.10])

Common Mistakes

• Monitoring only the score can conceal offsetting feature-level shifts
• Bin choice affects sensitivity, especially in tails and sparse regions
• Thresholds should reflect feature type, baseline volume and the cost of false alerts
• PSI is more useful as contextual trend evidence than as a universal pass/fail rule

PSI in a Production Pipeline

Teams can choose a monitoring cadence and feature scope based on data volume, model risk, latency and operational cost. PSI results may feed dashboards or alerts, but each threshold needs a documented response and should be interpreted alongside data-quality, performance and business evidence.