78 upvotes
Benchmarks Turned Into Difficulty Scores, Not Just Test Sets
What is Your Benchmark Is an Empirical Measure Over Difficulty?
The paper reframes any benchmark as an empirical measure that quantifies how hard each item is, instead of treating it as a fixed list of questions. This perspective reveals two failure modes, measure mismatch and …, that can skew model comparisons.
Who it is for
Researchers and dataset creators should read it to spot hidden bias in benchmark scores and to design more trustworthy evaluation pipelines.
How to try it
Read the paper on Hugging Face: https://huggingface.co/papers/2610.29927