Wiki · Reference entry
Benchmark
A shared set of tests everyone reports against.
Benchmarks make models comparable, which is valuable, and they make everyone optimize for the same things, which is not. A model at the top of a leaderboard has been tuned, directly or indirectly, for that leaderboard. None of it tells you how the system behaves inside your operation, with your data and your exceptions.
Why it matters
Leaderboard position and fitness for your workflow are close to unrelated.