Epoch AI found 23 false negatives among 131 DeepSWE tasks, implying a ceiling of roughly 79.6%. This suggests the current near-saturated score of about 74% is constrained by artificial benchmark limits rather than model capability.
Super cool work! This also explains why some benches have an "artificial ceiling".
For example, on DeepSWE, Epoch found 23 false negatives out of 131 tasks. This means a ceiling of roughly 79.6%, which actually matches the current (almost) saturated high score of ~74%
Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.
Source: wh · x.comPublished · added here
