Applied AI
Insurance Fraud Detection: A Leakage-Safe ML Pipeline
145 behavioural features and a leakage-safe SMOTE + XGBoost pipeline that raised fraud recall from 18.4% to 65.0% at 0.840 ROC-AUC.
Key results
- fraud recall
- 18.4% → 65.0%
- held-out data, before vs after the leakage-safe resampling pipeline
- ROC-AUC
- 0.840
- held-out data
- F1 (fraud class)
- 0.63
- at the chosen decision threshold
- engineered features
- 145
- behavioural and claim-level
Problem#
Fraudulent claims are a small minority of all claims. A model can reach high accuracy by predicting "not fraud" every time — and catch almost nothing.
Approach#
- Engineered 145 behavioural features.
- Applied SMOTE and scaling only inside the training folds of stratified K-fold, so no information from validation data leaks into training.
- Trained XGBoost inside a single scikit-learn pipeline, so training and serving apply identical transformations.
- Explained individual predictions with SHAP, and exposed a cost-based decision threshold that investigators can tune.
- Served through FastAPI and a Streamlit dashboard with Docker Compose, covered by automated tests and schema contracts.
Results#
Recall on the fraud class rose from 18.4% to 65.0% at 0.840 ROC-AUC, with an F1 of 0.63.
Limitations — what this does not prove#
- 65% recall still means roughly one in three fraud cases is missed.
- The model was trained and tested on a public dataset, not an insurer's production data.
- The best threshold depends on cost assumptions that would need to come from the business.