Twelve years of committee-model backtests

What the committee model is for

MMX-F measures how good teams are. The committee model has a different job: estimate how the College Football Playoff selection committee will rank teams each week. Those are different questions, and the gap between the answers is the most interesting thing on this site every week in November. Before that gap is worth reading, the committee model has to earn some trust. This is its full record on the twelve seasons that already happened, misses included.

What the model looks at

It is trained on every CFP poll since the playoff began in 2014. It works from the kinds of information the committee itself cites: record, resume strength (how hard the schedule was and how good the wins and losses were), recent form, and where teams stood in earlier polls. It honors head-to-head results the way the committee tends to. It does not use anything the committee cannot see.

How it was tested

The test is called leave-one-season-out. Train the model on eleven seasons, grade it on the twelfth, and repeat until every season has been graded by a version of the model that never saw it. Each season gets three scores.

• Top-15 MAE: for the teams the committee actually ranked in its top 15, how far off the model's rank was, on average. Lower is better; zero would be a perfect ordering.
• Spearman: rank correlation across the full top 25. A value of 1.0 means the identical order.
• Final top-12 overlap: of the twelve teams in the committee's final poll, how many the model also had in its final twelve. It measures how consistently the model identifies the teams around the modern playoff cut line.

The targets were set before training: top-15 MAE of 2.0 or better, overlap of at least 10 of 12, and Spearman of at least 0.93.

Reading the table

Across all twelve seasons the model averages a top-15 MAE of 1.44, a Spearman of 0.950, and 11.1 of 12 final-playoff teams. All three targets are met, and every season from 2015 onward clears the MAE target on its own.

2014 is the row that looks like it belongs to a different model. A top-15 MAE of 4.79 and a Spearman of 0.779 are the worst marks in the table by a wide margin. That was the playoff's first season. The committee was working out its own norms in public across seven polls, and the model's most useful input, where the committee had a team the week before, did not exist for the opening poll. Dropping 2014 would produce a prettier average. It stays in the table.

2020 is the other soft spot: five polls, uneven schedules during the pandemic, a top-15 MAE of 1.76, and only 10 of 12 on the final field. Everything from 2015 onward lands between 0.86 and 1.76 on MAE, and the model matched the full final twelve in 2016, 2022, 2023, and 2024.

Where it is strong, and where it is not

The model tracks the committee better than it anticipates the committee's surprises. Most of its accuracy comes from knowing where a team already stood and what happened on Saturday. The resume inputs matter most at the edge of the top 12, which is where the interesting decisions live and where the model is least certain.

It does not predict upsets, and it does not try to. It predicts how the committee reacts to results that have already happened. Playoff odds come from the simulation, which uses MMX-F to play out the games and the committee model to rank the results.

The 2026 ledger

The 2026 ledger has no entries yet. The committee model activates around week 4, once roughly 200 games are final and the resume inputs mean something. The first projection publishes before the first CFP rankings in early November. When the real rankings drop, the projection is graded in public on the same three measures plus the single worst miss, and the results stay up whether they flatter us or not.