mundial-2026-ml

This repository’s README said the features are built “with no data leakage”. That is true for nine of the fourteen and false for five, and the five are not five copies of one mistake: they are two mistakes of different shapes. I discovered them auditing the file after writing it.

The first shape is a snapshot. fifa_diff is the difference in FIFA ranking points between the two teams. The code loads a single ranking file, dated 2026-06-11, and queries it for every row of history:

home_fifa = fifa_latest.get(home, home_elo)
away_fifa = fifa_latest.get(away, away_elo)

For a match in 2004 that means the feature carries information from twenty-two years later. It is the hard kind of leak to see: it does not come from the row’s own outcome — the elo_diff sitting beside it is correctly built, written after the row is emitted — it comes from a table that looked static and is not.

And it does not travel alone. Further down the same record, match_features.py:163-164 writes fifa_diff_x_home_form and fifa_diff_x_away_form, which are that same contaminated difference multiplied by each team’s recent form. The snapshot enters the model three times, not once, and I did not notice the other two until I counted.

The second shape is an aggregate computed before the split, and I had not declared it anywhere until now. home_penalty_win_rate and away_penalty_win_rate are each team’s historical shootout win rate. They are built at match_features.py:97-106 and read at :161-162, and the build happens once, over the whole of shootouts.csv, before the walk-forward loop starts. So a match played in 2001 is handed a rate computed from shootouts as recent as 2026-07-07. Of the 682 shootouts in that file, 168 — 24.6% — are dated 2018 or later, which is to say at or beyond the start of the test window: 150 land inside 2018–2025 and 18 are from 2026, past the end of it. Every test row carries a feature that already knows how the test window went.

That second shape is also the hole in the hard wall from decision 2. The wall is real, but it filters rows: _load_features drops everything dated after 10 June 2026 when the model loads the table. The penalty rates were already baked into the table before that filter ran, and the aggregate includes the four 2026 World Cup shootouts played after the cut — Germany–Paraguay and Netherlands–Morocco on 29 June, Australia–Egypt on 3 July, Switzerland–Colombia on 7 July. The tournament the model predicts does reach the model. I built the wall and then walked a feature around it.

Two things I am not going to claim. I am not going to claim the effect is small, because I have not measured it. And I am not going to claim the 59.96% above is free of it, because it is not: that number, and the 0.879, the 0.517 and the 0.172 printed beside it, are all measured with the five leaks inside. They are an optimistic ceiling, not an estimate of how this model would do on matches it had genuinely never met. They are the numbers I have, not the ones I would like to have.

What I can say is what fixing them would cost and how the size of the damage would be known. The snapshot needs the historical FIFA ranking series by publication date, joined to each match on its date with a merge_asof. The penalty rates need to move inside the walk-forward loop and accumulate the way Elo already does, strictly from shootouts that had happened by the date of the row. Then the expanding-window backtest gets re-run. If the weighted accuracy drops a little, the five were nearly redundant with elo_diff and the model never depended on them. If it drops a lot, the published number was optimistic by a margin I would then be able to state, and it has to be republished. Both answers are useful. I have neither yet.

outputs/calibration_report.csv has three rows and says something uncomfortable:

The uncalibrated model wins on all three. Selection settles on sigmoid because it is written like this:

best_method = min(("isotonic", "sigmoid"), key=lambda m: metrics[m]["brier"])

raw is measured, printed and saved to the report, but it never enters the min. That is not a decision, it is an oversight in the selection code, and I would rather call it by its name than defend it.

So should I remove the calibration? I am not sure, and that is the interesting part. What consumes these probabilities is not a metric: it is a Monte Carlo simulator that samples them 5,000 times per tournament. There it is not enough for the probability to be well ordered; its level has to be right, because a model that says 70% where the real frequency is 60% produces too many favourite champions and the champion probabilities come out biased. Brier measures both properties together in one number, which is why it does not answer this question.

The way to answer it is to decompose Brier into reliability and resolution, or to compare directly the champion probabilities the simulator produces with raw against with sigmoid. I have not done it. Until I do, the honest statement is this: the calibration is in place on an argument that sounds good and that I have not verified, and the metrics published above are the backtest’s, not this table’s.

There is not a single test_, no pytest.ini, no workflow. On a project whose central argument is “my evaluation methodology is honest”, the absence of tests is the fairest criticism you can make: the walk-forward feature build is exactly the kind of code a test should pin, because an innocent change in the order of two lines reintroduces leakage without anything complaining. The five leaks above are the proof: nothing in this repository was ever going to catch them except me reading the file.

In the mode that simulates the full tournament, the bracket draw is random rather than following the real group-and-confederation pairing rules. And penalty shootouts are resolved with each team’s historical rate, not with a model — the same rate that leaks into the features above. Both widen the uncertainty on the champion probabilities beyond what the number of simulations suggests.

outputs/backtest_results.csv and outputs/build_metadata.json are dated 5 July; the metadata stamps itself 2026-07-06T02:59:55Z. data/processed/match_features.csv, the two .pkl model files under models/ and outputs/calibration_report.csv are dated 15 July. Ten days apart, and the consequence is not cosmetic: the metrics published on this page do not describe the models currently in the repository. The 59.96%, the 0.879, the 0.517 and the 0.172 come out of the 5 July backtest. The models you would load today are the 15 July ones, fitted on a feature table that was rebuilt after that backtest had already been written.

How far apart the two builds are, I do not know, because I have not re-run the backtest against the current models — and re-running it is one command, python -m scripts.build_all, which is what makes this the least defensible gap on the page. Until I do, read the numbers here as what a previous build reported, not as a measurement of the files you can download.