What Feature Drift Is and Why It's Silent
Feature drift is when the statistical distribution of one or more input features changes between the time a model was trained and when it runs in production. The model doesn't know this happened. It keeps producing outputs — just increasingly wrong ones.
The danger is not the drift itself. It's that nothing breaks loudly. No error thrown, no alert fired. AUC slowly degrades. Business metrics move weeks later. By the time someone notices, the model has been making bad decisions for months.
Scroll horizontally to inspect the complete figure.
PSI = 0.31 for this feature. A shift this large means the model is scoring a fundamentally different population than it was trained on.
Two Types of Drift — Know the Difference
- Covariate drift (feature drift): the input distribution P(X) changes, but the relationship between X and Y stays the same. The model's logic is still valid — it just hasn't seen this population. Retraining on fresh data usually fixes it.
- Concept drift: the relationship between X and Y changes — what used to predict Y no longer does. This is harder. Feature engineering and model architecture may need to change, not just retraining.
PSI detects covariate drift. Detecting concept drift requires tracking prediction error rates over time, which is a separate monitoring layer.
PSI Implementation for Production Monitoring
import numpy as np
import pandas as pd
def calculate_psi(reference, production, buckets=10):
"""
reference: array from training/baseline period
production: array from current production window
"""
# Use percentile-based bins from reference distribution
breakpoints = np.percentile(reference, np.linspace(0, 100, buckets + 1))
breakpoints = np.unique(breakpoints)
ref_counts = np.histogram(reference, bins=breakpoints)[0]
prod_counts = np.histogram(production, bins=breakpoints)[0]
# Percentages with small epsilon to avoid log(0)
ref_pct = ref_counts / len(reference) + 1e-8
prod_pct = prod_counts / len(production) + 1e-8
psi = np.sum((prod_pct - ref_pct) * np.log(prod_pct / ref_pct))
return psi
def monitor_all_features(X_reference, X_production, threshold_warn=0.10, threshold_crit=0.20):
results = []
for col in X_reference.columns:
psi = calculate_psi(X_reference[col].values, X_production[col].values)
status = "OK" if psi < threshold_warn else ("WARN" if psi < threshold_crit else "CRITICAL")
results.append({"feature": col, "psi": round(psi, 4), "status": status})
df = pd.DataFrame(results).sort_values("psi", ascending=False)
return df
# Run daily
drift_report = monitor_all_features(X_train, X_today)
critical = drift_report[drift_report["status"] == "CRITICAL"]
if len(critical) > 0:
print("CRITICAL DRIFT DETECTED:")
print(critical.to_string())Scroll horizontally to inspect the complete figure.
In this synthetic example, PSI crosses the warning threshold (0.10) at Week 4, the simulated week of vendor migration. It crosses critical (0.20) by Week 6. Daily monitoring would flag the example two weeks earlier.
Data Contracts — Preventing Drift at Ingestion
PSI catches drift after it enters production. Data contracts prevent bad data from entering the pipeline at all. They define what valid data looks like at the schema level — before the model ever sees it.
import great_expectations as ge
# Define contract for credit feature table
suite = ge.core.ExpectationSuite(expectation_suite_name="credit_features")
# Schema checks
suite.add_expectation(ge.core.ExpectationConfiguration(
expectation_type="expect_column_to_exist",
kwargs={"column": "credit_utilization_ratio"}
))
# NULL rate checks
suite.add_expectation(ge.core.ExpectationConfiguration(
expectation_type="expect_column_values_to_not_be_null",
kwargs={"column": "credit_utilization_ratio", "mostly": 0.95}
))
# Range validation
suite.add_expectation(ge.core.ExpectationConfiguration(
expectation_type="expect_column_values_to_be_between",
kwargs={"column": "credit_utilization_ratio", "min_value": 0, "max_value": 1}
))
# Distribution check — mean within expected range
suite.add_expectation(ge.core.ExpectationConfiguration(
expectation_type="expect_column_mean_to_be_between",
kwargs={"column": "device_risk_score", "min_value": 0.20, "max_value": 0.45}
))
# Run validation
validator = ge.dataset.PandasDataset(df)
results = validator.validate(expectation_suite=suite)
if not results["success"]:
raise ValueError("Data contract failed — halt ingestion pipeline")Operational decision
Retraining Trigger Logic
Retraining should be triggered by evidence of degradation, not by a fixed schedule. Schedule-based retraining is wasteful when data is stable and dangerously slow when it isn't.
def should_retrain(drift_report, performance_metrics):
"""
drift_report: DataFrame with feature PSI values
performance_metrics: dict with current AUC, KS statistic
"""
critical_features = drift_report[drift_report["status"] == "CRITICAL"]
auc_degraded = performance_metrics["auc"] < 0.78
ks_degraded = performance_metrics["ks"] < 0.35
trigger_reasons = []
if len(critical_features) > 0:
trigger_reasons.append(f"{len(critical_features)} features in critical drift")
if auc_degraded:
trigger_reasons.append(f"AUC {performance_metrics['auc']:.3f} below threshold 0.78")
if ks_degraded:
trigger_reasons.append(f"KS {performance_metrics['ks']:.3f} below threshold 0.35")
if trigger_reasons:
print("RETRAINING TRIGGERED:")
for r in trigger_reasons:
print(f" → {r}")
return True
print("No retraining needed")
return FalseScroll horizontally to inspect the complete figure.
Summary
Key Takeaways
- Feature drift is silent — it doesn't break code, it degrades predictions over time
- Score-level PSI can hide offsetting feature shifts; select feature-level monitoring based on risk, lineage and operational value
- Data contracts stop bad data at ingestion — PSI catches what slips through
- Retrain on evidence (PSI + AUC thresholds), not on fixed schedules
- In the synthetic CreditScoreV4 drift scenario, feature-level monitoring flags the simulated shift at Week 4 instead of Week 6