#Eval/Benchmark

AIClaude Opus 5.5 leads the benchmark with a score of 63.31 at $4.99 per task, ahead of GPT 6 Astra at 56.25 ($11.61) and Sonnet 5.5 at 53.14 ($2.82). At the low end, DeepSeek V4.1 Flash scores 36.91 at $0.26, and Ling 3.0 Flash scores 21.56 at $0.054.

AINous Research's Hermes Bench runs every model through the same harness, with reasoning set high where offered. The index reports each model's mean score and mean cost per task across four suites.

AINous Research says its evaluation uses an index of four benchmarks, including its newly introduced Hermes Bench. The other three are Terminal-Bench 4.0, Terminal-Bench-Science, and SkillsBench.

AINous Research has introduced Hermes Index, an average of benchmarks that measures model performance within Hermes Agent. The index aims to help users choose between models and show AI labs how their models perform in Hermes.

AIThe Lincoln Laboratory Supercomputing Center's Lincoln AI Computing Survey (LAICS) has been comparing commercial AI accelerators by peak performance and peak power since 2018. The latest paper covers more than 120 accelerators, up from 57 in the first, with data drawn from public sources. The team says five to 10 new AI accelerator startups emerge each year, and six have announced their first accelerators in recent months.

AIPerplexity cut its Decision API input pricing by half, to $0.02 per million input tokens. The reduction follows the release of pplx-decider-v1.1-27b, an open-weights multimodal decision model that scores highest on Hugging Face's Decision Index 0.3 benchmark. The model costs half as much as v1.

AIElvis Saravia argues that building good evals on top of AI systems can put a practitioner at the frontier of their domain or task quickly. He advises readers to learn eval construction, calling it worth the investment. The post responds to Garry Tan's point that agents writing markdown skills on cron jobs can handle most knowledge work.

AICline says Mistral Large 4 beats Opus 5.5 and GPT-6 Astra on cybersecurity benchmarks, largely because it refuses far fewer security tasks. The post claims Opus and Astra had about 40% of tasks blocked by their own safety filters.

AIA 9B model trained with SFT and prompt repetition at about $25 of compute reaches roughly 79% of Jev's Decision Index score. It ranks second among all models at 10B parameters or below and 14th of 70 overall.

AIGitHub has released ReviewBench, a new open benchmark for evaluating AI code review agents, shaped by analysis of 103.9M GitHub pull requests. It contains 219 PRs across 19 languages, and developers can bring their own code review agent, evaluate it, and submit their results.

AIResearch by pzeroresearch estimates that frontier models' experimental research taste has doubled roughly every three months since December 2025, with Opus 5.5 now exceeding their expert human baseline. The author of the main post, Boris Power, calls the plot very interesting for recursive self-improvement implications, while noting that the details matter for doing useful work at frontier labs.

AIParsewave audited all 600 public tasks in Zapier's AutomationBench and human review confirmed 206 real verifier bugs, all of which were fixed in AutomationBench Verified. Replaying 1,235 Kimi K3 runs on the old and fixed verifiers changed 27.9% of grades, with pass rate rising from 18.8% to 43.8% where verifiers were too strict and falling from 60.2% to 49.7% where they were too lenient.

AIMistral Large 4 reverse-engineered an unknown binary end-to-end, concluding it was Cobalt Strike and extracting its IoCs and malware configuration. The model also wrote a report with a YARA rule for detecting future incidents. The task, which could take a day of analyst work, was completed in 12 minutes.

AIMistral Large 4 solved 18 of 19 challenges in a CTF speedrun, with tool calls and solve times drawn from actual runs. The post frames the model as efficient at reasoning over diverse complex challenges compared with other models.

AIMistral AI says its Mistral Large 4 is one of the world's strongest AI models for cybersecurity. The post gives no benchmark scores, comparisons, or availability details.

AIPerplexity says its open-weights multimodal decision model pplx-decider-v1.1-27b ranks first on the new Hugging Face Decision Index 0.3 benchmark. The updated model is available to all, and it costs $0.02 per million input tokens, half the price of v1.

AIA METR researcher discovered a bug in the transcript viewer of Inspect, a popular evaluation framework, that could let an agent show a user a fabricated or edited version of its transcript.

AIYuchen Jin says Reflection's Beam and Mistral Large 4 both reached roughly GLM-5.2 level within the past two days. He suggests the Western versus Chinese open-source model gap may come down to Chinese labs being able to distill Anthropic and OpenAI models, which Western labs cannot.

AIPerplexity says its open-weights multimodal decision model pplx-decider-v1.1-27b is now available and scores highest on Hugging Face's new Decision Index 0.3 benchmark. The model costs $0.02 per million input tokens, half the price of v1.

AIARC Prize reports that Grok 4.7's reasoning-token usage correlates with its ARC-AGI-2 public scores. The low setting averaged about 10k tokens per test-pair attempt and scored 25%, while medium through xhigh used 86k to 120k tokens and scored 57.5% to 60%. ARC Prize suggests the lower token usage may help explain the low setting's lower score.

AIARC Prize published full Grok 4.7 results from the xAI model on its public leaderboard. The post links to the leaderboard, a GitHub repository for reproducing the results, and the ARC Prize testing policy.

AIGrok 4.7 scored 1.8% on ARC-AGI-3 in the standard harness, which lets models carry notes between turns, slightly below the 2.1% reported for Grok 4.7 in that setting. In a new provider adapter harness that preserves opaque reasoning and enables auto compaction, the score rose to 10.0%.

AIOn ARC-AGI-2, Grok 4.7's best score is 61.4% at $2.01 per task, down from Grok 4.6's 67.1% at $0.76 per task. The post notes cost per task rose 166% despite identical input and output token pricing.

AIGrok 4.7 used more reasoning tokens on average than Grok 4.6 on ARC-AGI-2 semi-private tasks at medium, high, and xhigh reasoning levels, raising its cost per task. Per test-pair attempt, medium used 136% more tokens, high 125% more, and xhigh 173% more, while low used 27% fewer. A chart compares the two models at xhigh on the 20 public tasks where Grok 4.7 increased token use the most.

AISpaceXAI's Grok 4.7 scored 90.2% on ARC-AGI-1 at $0.64 per task, higher than Grok 4.6, according to ARC Prize. It reached 61.4% on ARC-AGI-2 at $2.01 per task and 1.8% on ARC-AGI-3 under the standard harness ($2.7k), or 10.0% with the provider adapter harness ($4.8k), both lower than Grok 4.6 on those two benchmarks.

AIMicrosoft Research podcast host Chad Atalla interviews Jennifer Neville, a partner research manager at Microsoft, about her path into AI and her work on how evaluation exposes surprising failures in models tested beyond traditional benchmarks. The conversation also covers practical guidance for working with current AI systems and why examining underlying data matters when results defy expectations.

AIGoogle's EmbeddingGemma 2 is a 740M-parameter embedding model that outperforms some models more than twice its size while using about 191MB to 567MB of active RAM. It offers an 8K context window, 4x larger than the first generation, and can process up to 5.5 minutes of audio, 29 images, or 58 video frames in one pass.

AIGoogle Gemma says EmbeddingGemma 2 delivers best-in-size-class performance on MTEB (Code) and MIEB (Lite). The post highlights local codebase indexing, semantic code search, and coding agent retrieval, with a 9.92-point code performance gain over EmbeddingGemma.

AIGoogle DeepMind has released EmbeddingGemma 2, a 740-million-parameter embedding model that maps text, images, audio, and video into a shared space and runs on local hardware under an Apache 2.0 license. Matryoshka Representation Learning lets developers truncate output vectors from 768 dimensions to 512, 256, or 128, and the model supports an 8K-token context window. The model weights are available on Hugging Face and Kaggle, with Gemini Enterprise Agent Platform availability coming soon.

Why it matters: The release shows how a 740M-parameter multimodal embedder runs locally with a 768-to-128 dimension truncation option, useful for judging on-device retrieval designs.

AIARC Prize has published full DeepSeek V4.1 Flash results on its public leaderboard, alongside the benchmark code and testing policy needed to reproduce them. The post links to the complete results page for the model. No benchmark scores are stated in the post itself.

AIOn ARC-AGI-1, DeepSeek V4.1 Flash scored 88.5% at high reasoning versus 90.5% at low, with high using 35% more output tokens without consistently better answers. On ARC-AGI-2, the reported per-task cost of max reasoning ($0.129) appears slightly lower than high ($0.133), but after excluding incomplete tasks caused by API issues, max is about 4.5% more expensive per task.

AIDeepSeek V4.1 Flash reaches 72.9% on ARC-AGI-2 at $0.13 per task and 94.5% on ARC-AGI-1 at $0.07 per task, according to ARC Prize verification. Compared with V4 Flash's best scores, it gains 11.5 points on ARC-AGI-2 and 5.5 points on ARC-AGI-1, but costs about 250% more per task.

AISimon Willison reports that Mistral can now generate his pelican SVG test, shared via a Markdown SVG renderer. The post links to a rendered result but gives no benchmark or scoring details. Background from Mistral's own announcement describes Mistral Large 4 as a 1T-parameter, natively multimodal model with 49B active parameters, available via API today and with open weights planned for end of October.

AIMistral announced Mistral Large 4, a natively multimodal model with 1T parameters and 49B active, now available via API. Open weights are scheduled for release at the end of October, with a countdown page on Hugging Face showing October 31, 2026.

AIPerplexity's Computer ran two agents playing StarCraft against each other in real time, with the game never paused while each agent thought. Blue, playing with 41 Dragoons, lost the final match 2-5 to Red, which used High Templar and Psionic Storm after Blue failed to scout Red's build. Each agent received only its own fog-of-war-limited game state, and video input was not provided.

AIMistral has announced Mistral Large 4, a natively multimodal model with 1T total parameters and 49B active parameters. The company says it is the best open-weights model from the US or Europe on aggregated benchmarks, and that it is available via API today, with open weights due at the end of October.

AIElevenLabs' Eleven v4 and Eleven v4 Turbo rank #1 and #2 on the Artificial Analysis text-to-speech leaderboard. Per Artificial Analysis, Eleven v4 Turbo leads the Provider Voice arena at an Elo of 1,334, costs $40 per 1M characters, and generates 96 characters per second.

AIMistral Large 4 is a 1T-parameter natively multimodal model with 49B active parameters, which the Mistral account says leads open-weights models from the US or Europe on aggregated benchmarks. The post claims it beats closed frontier models on visual grounding and posts strong results across cybersecurity, legal, and agentic behavior. It is available via API now, with open weights due at the end of October.

AIArthur Mensch says Mistral trained its model on its own compute, and reinforcement learning shows no sign of saturating. The post accompanies Mistral's announcement of Mistral Large 4, a 1T-parameter natively multimodal model with 49B active parameters, available via API today and with open weights planned for end of October.

AIOn human evaluation, the model outperforms GLM 5.3 on STEM, CAD, and finance tasks and performs on par on agentic coding. The post is part 5 of a thread, so the model's name and other details come from earlier posts not included here.