#Eval/Benchmark

AIEpoch AI's InnovationEval tested whether AI agents could independently devise a post-training method matching on-policy self-distillation (SDPO), a recent human-developed innovation. GPT-5.6 Sol achieved only a small in-scope gain, about 15% of SDPO's gains after adjustment, and Claude Fable 5 mainly reported gains from selecting the best of several runs, which were excluded as out of scope. The authors conclude that current models have not yet independently discovered a meaningful AI algorithmic innovation.

Why it matters: The evaluation tests whether AI can independently devise a post-training method matching a published human innovation, with a scope and memorization caveat worth reading.

AIOpenAI is releasing a broad range of new mathematical results produced by an internal frontier model, with release guidance from the Institute for Advanced Study's Advisory Group on Mathematics and Artificial Intelligence. The results are available on GitHub at openai/math. The post itself is brief and emphasizes the results rather than hype.

AIOpenAI says its models solved several important math problems and that the team documented the work in a blog post. The post calls this a remarkable new era of scientific progress, but it does not name the specific problems here.

AITeknium announced Hermes Index, which combines scores from the new HermesBench and three other agent benchmarks. The index aims to help Hermes Agent users find the best model at a given time and at a given price point. It was introduced by Nous Research as a way to inform model choice and show labs their performance in Hermes.

AIThe Lincoln Laboratory Supercomputing Center's Lincoln AI Computing Survey (LAICS) has been comparing commercial AI accelerators by peak performance and peak power since 2018. The latest paper covers more than 120 accelerators, up from 57 in the first, with data drawn from public sources. The team says five to 10 new AI accelerator startups emerge each year, and six have announced their first accelerators in recent months.

AIPerplexity cut its Decision API input pricing by half, to $0.02 per million input tokens. The reduction follows the release of pplx-decider-v1.1-27b, an open-weights multimodal decision model that scores highest on Hugging Face's Decision Index 0.3 benchmark. The model costs half as much as v1.

AIGoogle/Gemini's Paige Bailey thanked attendees for supporting an event featuring Stanford AI Lab, Google DeepMind, Surge AI, and Hands-On Robotics. A companion post from Ugur Yektabasak praised a panel with Bailey, Amaub, and Teresa Nguyen during Tech Week and credited Google DeepMind with organizing it.

AIElvis Saravia argues that building good evals on top of AI systems can put a practitioner at the frontier of their domain or task quickly. He advises readers to learn eval construction, calling it worth the investment. The post responds to Garry Tan's point that agents writing markdown skills on cron jobs can handle most knowledge work.

AIA 9B model trained with SFT and prompt repetition at about $25 of compute reaches roughly 79% of Jev's Decision Index score. It ranks second among all models at 10B parameters or below and 14th of 70 overall.

AIGitHub has released ReviewBench, a new open benchmark for evaluating AI code review agents, shaped by analysis of 103.9M GitHub pull requests. It contains 219 PRs across 19 languages, and developers can bring their own code review agent, evaluate it, and submit their results.

AIResearch by pzeroresearch estimates that frontier models' experimental research taste has doubled roughly every three months since December 2025, with Opus 5.5 now exceeding their expert human baseline. The author of the main post, Boris Power, calls the plot very interesting for recursive self-improvement implications, while noting that the details matter for doing useful work at frontier labs.

AIParsewave audited all 600 public tasks in Zapier's AutomationBench and human review confirmed 206 real verifier bugs, all of which were fixed in AutomationBench Verified. Replaying 1,235 Kimi K3 runs on the old and fixed verifiers changed 27.9% of grades, with pass rate rising from 18.8% to 43.8% where verifiers were too strict and falling from 60.2% to 49.7% where they were too lenient.

AIPerplexity says its open-weights multimodal decision model pplx-decider-v1.1-27b ranks first on the new Hugging Face Decision Index 0.3 benchmark. The updated model is available to all, and it costs $0.02 per million input tokens, half the price of v1.

AIYuchen Jin says Reflection's Beam and Mistral Large 4 both reached roughly GLM-5.2 level within the past two days. He suggests the Western versus Chinese open-source model gap may come down to Chinese labs being able to distill Anthropic and OpenAI models, which Western labs cannot.

AIARC Prize published full Grok 4.7 results from the xAI model on its public leaderboard. The post links to the leaderboard, a GitHub repository for reproducing the results, and the ARC Prize testing policy.

AIGrok 4.7 scored 1.8% on ARC-AGI-3 in the standard harness, which lets models carry notes between turns, slightly below the 2.1% reported for Grok 4.7 in that setting. In a new provider adapter harness that preserves opaque reasoning and enables auto compaction, the score rose to 10.0%.

AIMicrosoft Research podcast host Chad Atalla interviews Jennifer Neville, a partner research manager at Microsoft, about her path into AI and her work on how evaluation exposes surprising failures in models tested beyond traditional benchmarks. The conversation also covers practical guidance for working with current AI systems and why examining underlying data matters when results defy expectations.

AIGoogle DeepMind has released EmbeddingGemma 2, a 740-million-parameter embedding model that maps text, images, audio, and video into a shared space and runs on local hardware under an Apache 2.0 license. Matryoshka Representation Learning lets developers truncate output vectors from 768 dimensions to 512, 256, or 128, and the model supports an 8K-token context window. The model weights are available on Hugging Face and Kaggle, with Gemini Enterprise Agent Platform availability coming soon.

Why it matters: The release shows how a 740M-parameter multimodal embedder runs locally with a 768-to-128 dimension truncation option, useful for judging on-device retrieval designs.

AIARC Prize is hosting its first Frontier AI Benchmarking Ecosystem Dinner with Snorkel AI during SF Tech Week. The event will bring together 40 experts from nonprofit benchmarking organizations, industry, academia, and government to discuss AI benchmark design and building a more scalable, transparent evaluation ecosystem. Attendees named in the post include representatives from METR, Epoch, Vals, Artificial Analysis, Harvard, and Stanford.