ML Experiment¶
Scaffold a new model-training script or audit an existing one for the problems that make ML results unreproducible or quietly wrong — missing seeds, data leakage between train/val/test, and no baseline comparison. Framework-agnostic.
Trigger: /ml-experiment
How It Works¶
- Mode and data shape first — new script or audit an existing one, and is the data time-ordered or i.i.d.? This decides the split strategy and is the biggest source of silent leakage.
- Split strategy by data shape — random split with a fixed seed for i.i.d. data; a time-based split or walk-forward for time-ordered data (reuse MC Analysis if the project already has one).
- Leakage checks — scalers/encoders fit on the full dataset before splitting, target-derived features, duplicate rows across splits, rolling windows that reach past the prediction point.
- Seeds fixed and logged — numpy, framework, and split
random_state, alongside hyperparameters and a data version/hash, so a run can be reproduced from its own logged settings. - Baseline required — every metric is logged next to a naive baseline (majority-class, mean, previous-value) so an improvement is a real comparison, not an isolated number.
- Audit mode greps for concrete bugs —
fit_transformbefore a split,train_test_splitwith norandom_state,shuffle=Trueon dated data, target-derived features, metrics printed with no baseline. - Report or generate accordingly — scaffold mode produces a training script following the examples below; audit mode reports file:line findings in the same severity-style format as Secrets Audit.
Leakage From Fitting on the Full Dataset¶
Bad: scaler sees the test set before the split \"happens\"
Good: split first, fit only on train
Time-Ordered Data Split Randomly¶
Bad: shuffles away the thing you're trying to predict
Good: train on the past, test on the future
Reporting a Result With No Baseline¶
Good: the number that actually tells you something
Notes¶
- A model that beats its baseline by less than its run-to-run seed variance hasn't beaten anything — check under 2-3 seeds before claiming a real improvement.
- Suspiciously high accuracy/R² is a prompt to re-check the split for leakage, not a reason to celebrate.
- Framework-agnostic by design — a checklist, not a scaffold tied to any specific library. Existing project training conventions win over anything suggested here.
- Comparing a production model against its original backtest is Trade Review's job, not this skill's.
- Natural habit points: before trusting a metric improvement enough to act on it, before merging a new model into a pipeline, and whenever a training script reports a number that seems unusually good.