#Eval/Benchmark

AIMax Woolf reports that prompting agents to iteratively optimize code can make it faster than current state-of-the-art libraries. He says the blog post includes his prompts and benchmark results.

AIXiaomi released MiMo-V2.6-Flash-RL, an efficiency-balanced checkpoint in its MiMo-V2.6 series, on Hugging Face. The model is a sparse MoE with 309B total and 15B activated parameters, supports text, image, video, and audio input, and offers a 1M-token context. The technical report says it was trained with a single mixed reinforcement learning run across coding, agent, visual, and cybersecurity tasks.

Why it matters: The report pairs its benchmark tables with the RL training method, which helps readers judge how the checkpoint's scores relate to its training approach.

AIXiaomi MiMo released MiMo-V2.6-Pro-RL on Hugging Face, a sparse MoE model with 1.02T total and 42B activated parameters and a 1M-token context. The technical report says it accepts text, image, video, and audio, and was trained with a single mixed reinforcement learning run across coding, agent, visual, and cybersecurity tasks.

Why it matters: The report pairs a 1.02T-parameter MoE model with an RL-based self-improvement method, useful for judging how reinforcement learning is scaled in frontier open models.

AIResponding to a post by ScarletKc, howie.serious simply agrees with the claim that GPT-6 Astra struggles with open-ended, exploratory work that lacks a fixed correct answer. The main post is a one-word endorsement (), while the quoted post argues GPT models excel at verifiable, goal-defined tasks and that Claude Fable handles open-ended exploration better.

AIMatei Zaharia, a prominent AI researcher, said it is very cool that GEPA works on Jev. The post is a short endorsement, linking to background about a test in which GEPA optimized Jev's prompts for extracting suspected adverse drug effects from medical sentences.

AIFireworks AI reports that an oracle router choosing among 18 models per DeepSWE task reaches 97.6% at $1.88 per task, versus 74.1% at $6.52 for GPT-6 Astra alone. The oracle is hindsight-based, so the authors say a production router must predict the best model before the task starts, which is the harder problem.

AIxAI released Grok 4.7, which it calls its most capable model for coding and knowledge work, built on a larger base model than Grok 4.6 and trained with a longer reinforcement learning run. It is priced from $2 per million input tokens and $6 per million output tokens, the same as Grok 4.6, and is available in Cursor, Grok Build, and the Grok API. xAI reports gains on CursorBench 4.0 (46.3%) and AA Briefcase v1.1 (1,657) over Grok 4.6, and says it posts the strongest safety results it has tested on refusals and jailbreak resistance.

Why it matters: The release pairs a new base model with benchmark tables against named rivals and pricing, letting readers compare its coding and office-work gains against Grok 4.6 and frontier models.

AIvLLM achieved 5000 total token throughput per GPU in high-throughput PD serving of Qwen3.8-2.4T on a GB300 NVL72 cluster under an 8K/1K workload. The low-latency scenario reached 180 generated tokens per user, with both results shown on the Pareto frontier. The post also provides srt-slurm recipes and explains the tuning process used to create them.

AIA developer's Augury plant identification model, built on an OpenBMB model, raised photo top-1 accuracy from 71.8% to 80.2% by merging duplicate species keys and adding PCA whitening. The next steps are reaching 90%+ accuracy and building a phone GUI so farmers can use it on-device.

AIMuon appears to outperform AdamW because it suppresses memorization, according to WeightWatcher experiments on a single-head nanoGPT model across five seeds. At 10,000 steps, teacher-forced recall of planted sequences was about 62% for AdamW versus under 1% for Muon. The author notes that some Muon layers also show α < 2, so α alone does not explain memorization and individual layers and their ESDs should be examined.

AIGoogle has released Android Bench 2.0, an updated benchmark that evaluates AI models on long-horizon tasks such as building apps from scratch, migrating cross-platform codebases to Android, and making complex architectural transitions. The benchmark uses continuous completion scoring to show which tasks each model performs well on.

AISemiAnalysis tested offloading DeepSeek-V4.1-Flash's Engram embedding table from HBM to host DRAM and to local SSD. On B200 configurations, DRAM delivered more total tokens per dollar and higher P90 interactivity than SSD at every measured point. The report concludes SSD offloading is likely not worth the tradeoff for production serving in its unoptimized setup.

AIAnthropic is partnering with Adaptyv Bio on a protein design competition in which over 5,000 designs will be experimentally validated. Anthropic is providing up to $1 million in Claude credits plus funding for experimental validation alongside Adaptyv, while Modal contributes up to $250,000 in compute and Twist Bioscience supplies DNA.

AISierra has become AIUC-1 certified after an independent audit by Schellman and testing by the Artificial Intelligence Underwriting Company (AIUC), a new standard for AI agents that tests resistance to manipulation and unauthorized access. Schellman found that Sierra met all applicable AIUC-1 requirements, and the technical evaluations recur at least quarterly with a full audit each year. The certification complements Sierra's existing SOC 2 Type II, ISO 27001, and ISO 42001 attestations.

AINortheastern University MS student Soham Padia used Ai2's open Olmo 3-32B model to build Steering Arena, a public game in which players submit text prefixes to steer prosocial behavior. About 600 submissions from a few dozen people showed the top 36 entries were unreadable token strings, while the best plain-English entry ranked 37th at about 2.7 times lower score. The results suggest that once an evaluation metric is exposed, it becomes an optimization target.

AIDatabricks rolled out Astra to all of its engineers and reported internal measurements. Engineers given Astra increased overall coding spend by around 60% compared with baseline, and Astra outperformed prior top models on highly complex system design tasks, while the gain on medium or low complexity tasks was unclear.

AIMatei Zaharia says agent harnesses make a large difference in cost, even on open-source coding benchmarks, and Melissa Pan's research examines why. Her quoted evaluation of seven models across Claude Code, Codex, and Pi found harness choice had little effect on task success but significantly affected cost. A simple harness can be competitive, and the native harness is not always the best.

AITencent's WeVisDoc-4B, fine-tuned from Qwen3-VL-4B-Instruct, converts page images into structured Markdown with LaTeX formulas and HTML tables. It scores 95.38 Overall on OmniDocBench v1.6 and a mean Overall of 75.54 across three PureDocBench tracks, ranking first among compared end-to-end parsers in all four reported settings. The model is available on Hugging Face and runs through vLLM, which requires version 0.11.1 or later.

AITencent's WeVisDoc-4B, fine-tuned from Qwen3-VL-4B-Instruct, scores 95.38 Overall on OmniDocBench v1.6 and 75.54 mean Overall across three PureDocBench tracks. The end-to-end parser converts page images into structured Markdown with LaTeX formulas and HTML tables, and the 2B variant is also available. The repository provides vLLM serving scripts with a 32768-token default context and a Python client for batch processing.

AITogether AI says teams can migrate from closed to open-source models in weeks to months, not years, using a managed service. The guide recommends defining the use case, shortlisting candidates with relevant benchmarks, and replaying existing traffic against them to test accuracy, cost, and latency against current targets.

AIHiDream.ai launched vivago R1, a content creation agent, globally, with a domestic version upgrade. The company says R1 can output five-minute high-quality videos through agent planning, with a claimed 85% usable-output rate and support for multi-round extensions. It also released HiDream-O1-Video-1.0, a native omni-modal video model supporting single shots of 5 to 20 seconds at 1080p.

AIJeff Dean congratulated Periodic Labs on results from its materials-science AI work. The post itself provides no specific figures, so the announcement is limited to the congratulations.

AIPeriodic Labs says it used 1,300 H200 GPUs and months of its lab data to mid-train and RL an open-source model called Neon, which it claims surpasses GPT-6 Astra on its analysis benchmark. The company says it is focusing first on hard materials science problems, including superconductors, magnets, and semiconductor materials.

AIJason Wei argues that specialized, often private wet-lab data can let a task-specific model outperform a general frontier model on scientific tasks. He cites Neon, an open-source model that Liam Fedus says was mid-trained and RL-tuned on experimental data using 1,300 H200s to surpass GPT-6 Astra on an analysis benchmark. The post frames this data as a potential moat as work moves toward the frontier of science.

AIGoogle introduces Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking, two live dialogue models for voice agents that reason and speak simultaneously. The Extended Thinking version scores 82.6 on Artificial Analysis' Speech to Speech Quality Index and 97.7% on Big Bench Audio, while 3.8 Live targets scale and cost efficiency. Developers can access both through the Gemini API in Google AI Studio, and enterprise and consumer rollouts vary by product.

Why it matters: The source names the two models, their access paths, and specific benchmark results, showing how the voice agent capabilities differ between the two tiers.

AIPeriodic Labs chose SGLang and Miles to build Neon, an open-source model it says surpasses GPT-6 Astra on its analysis benchmark after mid-training and RL on 1,300 H200s. RadixArk says Periodic extended both frameworks for scientific RL at trillion-parameter scale, delivering more efficient training, lower memory use, and 2.5x faster inference. The work has been contributed back to both projects.

AIShanghai AI Laboratory's Intern Large Models announced Intern-S2-397B, available in BF16 and FP8 under Apache 2.0. The post reports 87.0 on FrontierScience-Olympiad and 84.0 on SWE-bench Multilingual, leading the reported comparison on both, and says it was jointly trained across 20+ scientific domains with long-horizon agent RL.

AIinclusionAI released Step-3.7-Flash-singprobe, an 8.13M-parameter probe that reuses Step-3.7-Flash hidden states to score query intent, response unsafety, and hallucination risk at every generated token. The probe adds less than 0.5% decode-time overhead and reports 0.9858 R-AUC and 0.9295 T-AUC on streaming safety benchmarks. It is supported through SGLang and vLLM integration branches and loads from Hugging Face by checkpoint ID.