Fu Index

Benchmaxx Check

Sort by

By lab

# Model Home advantage Tests Biggest spike Ambition Fu
How the home advantage works

For each launch we take the benchmarks of the headline table of the official launch post or model card (the same tables as the Ambition Score). The Fu Ranking fit gives every benchmark a difficulty and a discrimination. We estimate the model’s ability again, this time only from its scores on benchmarks that are not in the launch table, and predict its score on each launch-table benchmark from that ability.

The residual of a benchmark is the reported score minus the prediction on the logit scale, in units of the fit’s residual SD σ (about how far a typical score sits from its prediction). The raw advantage is the mean residual of the launch-table benchmarks (each capped at ±4 σ, since larger gaps are mostly data errors). Labs show the tests they do well on, so the median launch post has a raw advantage of +0.7 σ; the home advantage shown is the raw advantage minus that median. The ± range is one standard error: the noise of the launch-table scores plus the uncertainty of the ability estimate. A post needs at least 3 launch-table benchmarks and 5 other benchmarks in the fit.

A high home advantage means the model looks better on the benchmarks its lab chose than everywhere else. That fits training on the test sets or their close relatives (benchmaxxing), but also honest specialisation, scaffolds tuned for the launch, or other benchmarks that were run poorly. Most other scores come from the labs too, so the home advantage tends to understate the gap to independent results. It cannot see memorised test cases directly; that takes runs on edited tasks.