#Eval/Benchmark

AIOn ARC-AGI-2, Grok 4.7's best score is 61.4% at $2.01 per task, down from Grok 4.6's 67.1% at $0.76 per task. The post notes cost per task rose 166% despite identical input and output token pricing.

AIGrok 4.7 used more reasoning tokens on average than Grok 4.6 on ARC-AGI-2 semi-private tasks at medium, high, and xhigh reasoning levels, raising its cost per task. Per test-pair attempt, medium used 136% more tokens, high 125% more, and xhigh 173% more, while low used 27% fewer. A chart compares the two models at xhigh on the 20 public tasks where Grok 4.7 increased token use the most.

AISpaceXAI's Grok 4.7 scored 90.2% on ARC-AGI-1 at $0.64 per task, higher than Grok 4.6, according to ARC Prize. It reached 61.4% on ARC-AGI-2 at $2.01 per task and 1.8% on ARC-AGI-3 under the standard harness ($2.7k), or 10.0% with the provider adapter harness ($4.8k), both lower than Grok 4.6 on those two benchmarks.

AIGoogle's EmbeddingGemma 2 is a 740M-parameter embedding model that outperforms some models more than twice its size while using about 191MB to 567MB of active RAM. It offers an 8K context window, 4x larger than the first generation, and can process up to 5.5 minutes of audio, 29 images, or 58 video frames in one pass.

AIGoogle Gemma says EmbeddingGemma 2 delivers best-in-size-class performance on MTEB (Code) and MIEB (Lite). The post highlights local codebase indexing, semantic code search, and coding agent retrieval, with a 9.92-point code performance gain over EmbeddingGemma.

AIOn ARC-AGI-1, DeepSeek V4.1 Flash scored 88.5% at high reasoning versus 90.5% at low, with high using 35% more output tokens without consistently better answers. On ARC-AGI-2, the reported per-task cost of max reasoning ($0.129) appears slightly lower than high ($0.133), but after excluding incomplete tasks caused by API issues, max is about 4.5% more expensive per task.

AIDeepSeek V4.1 Flash reaches 72.9% on ARC-AGI-2 at $0.13 per task and 94.5% on ARC-AGI-1 at $0.07 per task, according to ARC Prize verification. Compared with V4 Flash's best scores, it gains 11.5 points on ARC-AGI-2 and 5.5 points on ARC-AGI-1, but costs about 250% more per task.

AISemiAnalysis rated FarmGPU as ClusterMAX Underperform after its Slurm layer failed to advertise GPU resources and Kubernetes exposed no RDMA devices for scale-out networking. The post credits FarmGPU's Grafana monitoring, provisioning notes, and trustworthy technical team, while noting the team may be stretched thin across small clusters.

AIHamel Husain argues that similar wording does not show whether an LLM answer works for a specific application, so teams should check concrete failure modes. He adds that similarity metrics can still help with retrieval evaluation and measuring output diversity.

AISimon Willison reports that Mistral can now generate his pelican SVG test, shared via a Markdown SVG renderer. The post links to a rendered result but gives no benchmark or scoring details. Background from Mistral's own announcement describes Mistral Large 4 as a 1T-parameter, natively multimodal model with 49B active parameters, available via API today and with open weights planned for end of October.

AIMistral announced Mistral Large 4, a natively multimodal model with 1T parameters and 49B active, now available via API. Open weights are scheduled for release at the end of October, with a countdown page on Hugging Face showing October 31, 2026.

AIPerplexity's Computer ran two agents playing StarCraft against each other in real time, with the game never paused while each agent thought. Blue, playing with 41 Dragoons, lost the final match 2-5 to Red, which used High Templar and Psionic Storm after Blue failed to scout Red's build. Each agent received only its own fog-of-war-limited game state, and video input was not provided.

AIDuke University researcher Alexis Fox has been announced as a speaker at the ARC Prize Research Summit 2026. Fox studies how AI can reason over longer horizons and is lead author of PRO-LONG, which uses programmatic memory to support long-horizon reasoning.

AIMistral says its model outperforms GLM 5.3 in human evaluations across multiple domains. The post gives no benchmark scores, model names, or evaluation details.

AIMistral Large 4 is a 1T-parameter natively multimodal model with 49B active parameters, which the Mistral account says leads open-weights models from the US or Europe on aggregated benchmarks. The post claims it beats closed frontier models on visual grounding and posts strong results across cybersecurity, legal, and agentic behavior. It is available via API now, with open weights due at the end of October.

AIOn human evaluation, the model outperforms GLM 5.3 on STEM, CAD, and finance tasks and performs on par on agentic coding. The post is part 5 of a thread, so the model's name and other details come from earlier posts not included here.

AIMistral says its ML4 model reaches state-of-the-art performance among open models across a wide range of capabilities, and outperforms the best models in visual grounding, legal, and spreadsheet manipulation. The post reports ML4 ranks among the best on the AA Cyber Index, scoring 82% on vulnerability reproduction and patching and 93% on Cybench. It argues that self-hosted, auditable open models are the best defense option for enterprises today, and that they do not refuse to help.

AIMistral's ML4 model matches the best open-weight models on DeepSWE, AutomationBench, and AA-Briefcase, and reaches state-of-the-art results on finance and legal workflows and complex multimodal grounding benchmarks. The post says it can navigate terminal workflows, work across spreadsheets, slides, and PDFs, and reason over scientific and multimodal tasks.

AIMistral has launched a preview of Mistral Large 4 (ML4), a 1T-parameter multimodal model with 49B active parameters. The company says it is the strongest open-weight model from the US or Europe on aggregated benchmarks and is available via API now, with open weights planned for the end of October.

Why it matters: The post gives parameter counts, a preview timeline, and an open-weights release date, which help readers judge how Mistral's model compares with other open-weight options.

AIMistral AI introduced Mistral Large 4, a natively multimodal model with 1T parameters and 49B active parameters. The company says it is the best open-weights model from the US or Europe on aggregated benchmarks and is available via API today, with open weights due at the end of October.

AIAIHOT launched a Pareto frontier view on its leaderboard to compare models by price-performance, accessible via the top-right toggle. The cost calculation varies by scenario: general and office use assumes cache-hit:input:output at 7:2:1, programming at 97:2:1, research at input:output 1:3, and knowledge Q&A at 3:1 without caching.

AIBlack Forest Labs says its FLUX 3 model ranks first on Google DeepMind's Physics-IQ benchmark, which tests whether video models can predict what happens next in real filmed physical experiments. The company says FLUX 3 outperforms Seedance 2.5, MiniMax H3, Gemini Omni 1.1 Flash, Veo 3.1, Sora 2, and Cosmos3 in most cases, and that pairing it with a physics verification layer scores even higher.

AIReflection AI previewed Beam, an MoE open model with 501B total and 23B active parameters, claiming 3–4x better inference efficiency than GLM 5.2. The model was pretrained from scratch on 23.8T tokens in four weeks, and its RL run used 10,500 GB300 GPUs over four weeks, which the post describes as possibly the largest publicly recorded. Reflection positions Beam as a workhorse open model for enterprises, governments, and developers, with full weights due this month.