Skip to content
Read the original: Elvis Saravia· Published 41/100AI score41/100

Parsewave audit fixes 206 verifier bugs in AutomationBench

Original titleWe need more efforts like this.

AISummary

Parsewave audited all 600 public tasks in Zapier's AutomationBench and human review confirmed 206 real verifier bugs, all of which were fixed in AutomationBench Verified. Replaying 1,235 Kimi K3 runs on the old and fixed verifiers changed 27.9% of grades, with pass rate rising from 18.8% to 43.8% where verifiers were too strict and falling from 60.2% to 49.7% where they were too lenient.

Read the original x.com

Source: Elvis Saravia · x.comPublished · added here