I built a full ML pipeline on a Kaggle dataset and proved it has zero predictive signal — and shipped the null result instead of faking accuracy
A failure mode I see constantly — in portfolios and in vendor models at work —
is reporting a great ROC-AUC without ever asking whether the dataset contains
any signal at all. So I built the opposite: a pipeline designed to falsify its
own results before trusting them.
I took a public BMW sales dataset (50k rows, 2010–2024) and ran the full stack:
econometrics, gradient boosting (XGB/LGBM/CatBoost), a tabular MLP, SHAP. Every
model landed at no-skill — regression R² ≈ 0, classification AUC ≈ 0.51.
Instead of torturing the data, I ran two checks I now apply by default:
- Permutation / label-shuffle test: refit on shuffled labels. If your "real"
score sits inside the shuffled distribution (here p ≈ 0.90), you have nothing.
- Positive control: push a synthetic target with known structure through the
exact same pipeline. It hit R² ≈ 0.86 — proving the pipeline is sound and the
data is the problem, not the code.
I also found the classification target was a deterministic threshold on the
volume column — textbook target leakage that gives a fake 1.00 AUC. Remove it
and AUC collapses to chance.
Since the data can't forecast, the actual deliverable is an explicit what-if
simulator (constant-elasticity demand, literature-grounded priors, Monte-Carlo
intervals) — clearly labelled as a model of assumptions, never a fit to history.
The whole thing is reproducible (Docker, CI, tests) with a live demo so you can
click through the leakage proof yourself. Genuinely curious where this breaks:
what would you put on a "does this dataset have any signal?" checklist?
[live demo] · [repo]