Fairness Monitoring in ML: From Metrics to Investigation

Illustrative boundary

Synthetic Scenario: CreditScoreV4

This is a synthetic illustrative scenario created to explain monitoring controls; it is not a real lender, deployment, customer event, or regulatory complaint. In the scenario, a model passes its initial accuracy thresholds. Six simulated weeks later, AUC falls from 0.81 to 0.74, 30-day delinquency rises from 3.1% to 5.4% and minority-group approval rates fall 11 percentage points.

Scenario setup: an upstream vendor schema change introduces 22% NULL values in a credit feature. Silent median imputation filled the gaps — but that median was stale, inflating approvals for high-risk applicants. A second feature,device_risk_score, drifted from mean 0.31 to 0.44, acting as a geographic proxy that disproportionately penalized minority applicants.

Figure 1 — Synthetic Approval-Rate Example: Pre vs Post Vendor Migration

Scroll horizontally to inspect the complete figure.

Majority group approval rate fell 2.2pp. Minority group fell 11pp. Disparate impact: 0.76 — below the 4/5ths rule threshold of 0.80.

The Three Fairness Metrics You Must Track

1. Demographic Parity

Are approval rates equal across groups regardless of actual creditworthiness?

SELECT
  protected_group,
  COUNT(*) AS total_applicants,
  SUM(CASE WHEN approved = 1 THEN 1 ELSE 0 END) AS approved,
  ROUND(AVG(approved), 4) AS approval_rate
FROM loan_applications
GROUP BY protected_group;

-- Demographic Parity Difference = |rate_A - rate_B|
-- Threshold: < 0.05 (5 percentage points)

2. Disparate Impact Ratio (4/5ths Rule)

A screening ratio often used to flag outcomes for closer review. It compares minority and majority approval rates. A value below 0.80 is not, by itself, a legal finding or proof of discrimination; context and qualified legal review are still required.

# In Python
minority_rate = 0.521
majority_rate = 0.684

disparate_impact = minority_rate / majority_rate
print(f"Disparate Impact: {disparate_impact:.3f}")
# Output: 0.762 — FLAG FOR CONTEXTUAL REVIEW (example threshold: >= 0.80)

four_fifths_compliant = disparate_impact >= 0.80
print(f"4/5ths screening threshold met: {four_fifths_compliant}")
# Output: False — flag for review, not a compliance determination

3. Equal Opportunity Difference

Among applicants who would actually repay the loan (true positives), are approval rates equal across groups? This is more nuanced than demographic parity because it conditions on actual creditworthiness.

# True Positive Rate (recall) per group
tpr_majority = tp_majority / (tp_majority + fn_majority)
tpr_minority = tp_minority / (tp_minority + fn_minority)

eod = abs(tpr_majority - tpr_minority)
print(f"Equal Opportunity Difference: {eod:.3f}")
# Output: 0.18 — FLAG FOR CONTEXTUAL REVIEW (example threshold: < 0.05)

Figure 2 — Synthetic Fairness Dashboard Example

Scroll horizontally to inspect the complete figure.

16.3pp

Demographic Parity Diff

Threshold: < 5pp

REVIEW

0.762

Disparate Impact Ratio

Threshold: ≥ 0.80

REVIEW

0.18

Equal Opportunity Diff

Threshold: < 0.05

REVIEW

Using SHAP to Diagnose Bias Sources

Fairness-screening differences identify outcomes that need investigation. Grouped SHAP analysis can show which features are associated with different model contributions, but it does not establish causation or prove that a feature is the source of bias. Proxy risk still requires domain, data and legal review.

import shap
import pandas as pd

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Compare SHAP importance by group
shap_df = pd.DataFrame(shap_values, columns=X_test.columns)
shap_df['group'] = X_test['protected_group'].values

# Mean absolute SHAP per feature per group
group_shap = shap_df.groupby('group').apply(
    lambda g: g.drop('group', axis=1).abs().mean()
)
print(group_shap.T.sort_values('minority', ascending=False).head(10))

# device_risk_score has 3x higher SHAP for minority group
# This confirms it is acting as a proxy discriminator

Figure 3 — Mean |SHAP| by Feature and Group (Top 5 Features)

Scroll horizontally to inspect the complete figure.

device_risk_score has 3x higher SHAP importance for minority applicants — clear proxy discrimination signal.

Action plan

Remediation Steps

Immediate: pause promotion and investigate device_risk_score, its lineage and subgroup impact before choosing rollback or removal
Short-term: compare mitigations on reviewed data and add documented fairness checks to the release process
Long-term: choose a risk-based review cadence and route threshold alerts to a documented human investigation
Audit: preserve evidence and involve qualified domain, compliance and legal reviewers when impact may require remediation

Deployment decision

Pre-Deployment Fairness Gate

def fairness_gate(model, X_test, y_test, protected_col):
    """Returns True only if model passes all fairness checks."""
    groups = X_test[protected_col].unique()
    rates = {}
    tprs = {}

    for g in groups:
        mask = X_test[protected_col] == g
        preds = model.predict(X_test[mask])
        rates[g] = preds.mean()

        # True positive rate
        tp = ((preds == 1) & (y_test[mask] == 1)).sum()
        fn = ((preds == 0) & (y_test[mask] == 1)).sum()
        tprs[g] = tp / (tp + fn) if (tp + fn) > 0 else 0

    majority = max(rates, key=rates.get)
    minority = min(rates, key=rates.get)

    di = rates[minority] / rates[majority]
    dp_diff = abs(rates[majority] - rates[minority])
    eod = abs(tprs[majority] - tprs[minority])

    passed = di >= 0.80 and dp_diff <= 0.05 and eod <= 0.05

    print(f"Disparate Impact: {di:.3f} ({'PASS' if di >= 0.80 else 'FAIL'})")
    print(f"Dem. Parity Diff: {dp_diff:.3f} ({'PASS' if dp_diff <= 0.05 else 'FAIL'})")
    print(f"Equal Opp. Diff:  {eod:.3f} ({'PASS' if eod <= 0.05 else 'FAIL'})")
    print(f"Gate result: {'APPROVED FOR DEPLOYMENT' if passed else 'BLOCKED'}")
    return passed

Summary

Key Takeaways

Accuracy metrics alone are not enough — a high-AUC model can still discriminate
Track all three: Demographic Parity, Disparate Impact, Equal Opportunity Difference
SHAP by group reveals which features are driving bias — it's the diagnostic, not just a visualization
Automated checks can shorten detection time, while scheduled human review remains necessary for context and remediation
Upstream data changes are the most common source of fairness failures — data contracts prevent them