I found target leakage in my own model

The situation

Nine thousand lines of Formula 1 prediction: telemetry ingestion, per-circuit characteristics, weather, engineered features, gradient-boosted models, a front end. It ran. It produced a predicted finishing order. I had never checked whether the number attached to it meant anything.

It did not. Two independent defects, both structural, both found by reading the evaluation rather than running it, which is the only way to find this class of problem, because the code does not fail. It succeeds, confidently, at the wrong thing.

The feature contained the label

One feature was a driver’s historical affinity for a circuit: how they have gone at this track before. It is built by walking race results over a window of past seasons and storing a per-driver, per-track record.

The training rows were collected over the same window. Not a similar one, the identical expression, computed from the same clock. So when the model was handed the 2024 Monaco Grand Prix as a training example, the affinity feature for each driver already included that driver’s 2024 Monaco result. The label was inside the feature.

This is the failure mode that produces impressive numbers and no predictive power. The model is not learning which drivers do well at Monaco; it is being told the answer in a slightly encoded form and rewarded for decoding it.

The split let it train on the future

The second defect stands on its own. The train/test split was a random shuffle across every race in every season, stratified on the target, seeded for repeatability, and completely wrong for a temporal problem.

A random split assigns races to train and test with no regard for when they happened, so the model is routinely fitted on races that took place after the ones it is scored against. For a task whose entire purpose is predicting an outcome before it occurs, an evaluation that reads the future is not measuring the thing anyone cares about.

I searched the whole repository for a time-ordered split, a walk-forward loop, a year-based holdout, anything. There was nothing. It had never been done a different way.

It could not have been reproduced anyway

Two smaller problems compound the first two. The training window is derived from the current date, so the same command trains on different data depending on the day it runs. Nobody, including me, could re-run it and expect the same model.

And there was no saved model, no recorded metric, and no evaluation output committed anywhere. The code to compare predictions against results existed and had clearly been run at some point, but nothing was kept. There was no number to defend, which in hindsight is the only reason the defects survived: nobody had anything to be suspicious of.

The missing baseline, which decides it

Even a clean pipeline would not have been enough, because there was nothing to compare against. In modern Formula 1, qualifying explains most of the finishing order. The honest baseline is “predict the grid, unchanged”, and it is a hard one to beat.

A top-three classifier that cannot beat the starting grid has added nothing, however good its accuracy looks in isolation. That comparison had never been run, so the project had no way of knowing whether it was useful even in principle.

What a defensible version would require

Order the data by time and split on it: train on everything before a cutoff, test on what comes after, walk the cutoff forward. Recompute the affinity feature strictly from races before the one being predicted, per prediction, rather than once from a global window. Pin the training window to explicit dates instead of the system clock. Commit the model and the metric so a claim can be checked. Then run it against the grid-order baseline and publish the difference, including if the difference is negative.

I have not done that work, and the model stays unpresented until I do. The finding is worth more than the predictor was: a model nobody has audited is a story about optimism, and an audit that killed a model is a story about how you work.

Update, October 2026: the work, done

I rebuilt it. Features are now computed by walking races in order and reading each race's inputs before its result is folded in, and a test deletes every later race and checks that nothing about the earlier one changes. I planted a leak to make sure the test fails when it should. It does.

Every race since 2018 is predicted by a model trained only on the races before it. Settings were chosen on 2018 to 2021 and the headline numbers are 2022 onward, which the tuning never touched. Over those 108 races the model's favourite won 56% of the time; the pole-sitter won 57%. So on picking a name, it does not beat the grid. On the probabilities it does, clearly: it put 44% on the eventual winner on average against the grid's 36%, and its stated odds are calibrated.

That is the result, including the half of it that is not flattering. It is live at tommylu.com/f1, with the race-by-race record, before every race, scored after it.