Updated
Updated
Items with an AI score under 20 are hidden. Show low-relevance items
Perplexity@perplexity_aiAI score20AIPerplexity's embedding preview achieves the highest average nDCG@10 among tested models on ConTEB, though not on every task. It also outperforms voyage-context-4 on chunk retrieval while using 8x less storage per vector, at 1 KB (1024 dims, int8) versus 8 KB (2048 dims, float32).
Perplexity@perplexity_aiAI score24AIPerplexity says its pplx-embed-v2-context-9b-preview model leads on Answer and Evidence retrieval at every cutoff. In document retrieval the gap is smaller, with its context v1 4B model slightly ahead at Document@3 and Document@5.
Perplexity@perplexity_aiAI score23AIPerplexity says context-bench, a context-aware retrieval benchmark created and privately held by turbopuffer, has 2,099 queries and 38,894 documents. Perplexity submitted it for blind evaluation, with queries and capabilities inspired by turbopuffer customer conversations.
Perplexity@perplexity_aiAI score34AIPerplexity introduced a new training method for contextual embedding models that encode each document chunk with the whole document in view. Its pplx-embed-v2-context-9b-preview sets a new state of the art on the ConTEB benchmark and on turbopuffer's privately held context-bench.
Google Cloud Tech@GoogleCloudTechAI score28AITerminal test runs miss multi-turn agent regressions, so Agent Clinic Episode 3 builds an automated eval suite for a LangGraph agent in 60 minutes. The post presents a four-step framework for moving from informal checks to benchmarking AI agents, with a link to the full guide.
TypeSafe AI@typesafeaiAI score27AIJev reranking retrieves Rox sales data 20x faster, 10x cheaper, and 12% more accurate than GPT-5 Mini. The benchmark compared Jev classification against LLM-based reranking for pulling transcripts, emails, CRM notes, news, and documents.
Fireworks@FireworksAI_HQAI score30AIFireworks' Ember-1 performs close to Kimi K3 on Vals' finance and legal benchmarks while using fewer reasoning tokens per turn. Fewer tokens per turn lower cost and speed up agent loops, and Ember-1 is available on Fireworks Serverless.
Stanford HAI@StanfordHAIAI score23AIBiomedical datasets are often incomplete, such as patients with imaging but no genomic data. Stanford speaker Olivier Gevaert will discuss using foundation models to fill these gaps in multimodal precision medicine modeling on October 7.
Nathan Lambert@natolambertAI score22AINathan Lambert's latest Interconnects plot tracks daily tokens processed by leading open model inference providers. The figures come from public disclosures by Together, Baseten, and Fireworks, plus the OpenRouter API. He says he underestimated OpenRouter's growth.
Ideogram@ideogram_aiAI score23AIIdeogram 4.5 was built for precise, targeted editing but also performs very well across general editing tasks. Design Arena ranks it 15th in Image Editing with an Elo of 1250, placing it in the same performance band as MAI-Image-2.6 and Gemini 3 Pro Image Preview. It is especially strong at typography edits, such as modifying text in infographics.
Ant Ling@AntLingAGIAI score22AIAnt Ling reports that Ling-3.1-flash scored 65.35 on HealthBench Professional, a healthcare evaluation rather than clinical certification. The model was trained with feedback from AQ and Haodf.com (Good Doctor Online) healthcare scenarios.
Ant Ling@AntLingAGIAI score22AILing-3.1-flash achieved a 1.432x speedup on Pyright 1.1.400 while passing all 2,279 checks. The chart shows Claude Fable 5 at 1.502x and GLM-5.3-Flash at 1.290x, a close result on a real codebase with correctness intact.
Ant Ling@AntLingAGIAI score38AILing-3.1-flash built a Lua-to-x86-64 ELF compiler from scratch in about 17 hours, passing 178 of 182 tests for 97.8%. That is within 1.2 percentage points of the 99.0% shown for Claude Fable 5 and above GLM-5.3-Flash at 95.9%.
Ant Ling@AntLingAGIAI score22AILing-3.1-flash ranks #2 in the Mobile App (react native) category on Design Arena's Open Weights leaderboard, with 1,207 Elo. It also ranks #4 in SVG with 1,271 Elo among models added in the last 30 days.
Ant Ling@AntLingAGIAI score46AIAnt Ling introduced Ling-3.1-flash, a model with about 560B total parameters, about 25B active per token, and up to a 1M-token context window. The company plans to open-source the model soon. It reports 1,673 Elo on GDPVal-AA v2.1, 75.16 on FrontierSWE, and 65.35 on HealthBench Professional across work, coding, and healthcare tasks.
NVIDIA AI@NVIDIAAIAI score27AINVIDIA and Nous Research published a hands-on walkthrough of NVIDIA NeMo Relay for collecting traces from Hermes Agent. The guide runs two example scenarios and shows the agent's calls and retries in Arize Phoenix. It also covers how Nous used traces and task results to evaluate fixes across repeated runs.
Ideogram@ideogram_aiAI score22AIIdeogram posted a side-by-side test of identical edits on Ideogram 4.5, GPT Image 2.5 Sunburst, Nano Banana Pro, and Nano Banana 2. The post claims GPT Image and Nano Banana outputs become unusable within a few edits, while Ideogram 4.5 remains clean across repeated edits.
Liquid AI@liquidaiAI score42AILiquid AI and InSilicoMeds released LongevityBench, an aging benchmark with 17 tasks spanning clinical records, DNA methylation, transcriptomics, proteomics, and genetics. On several tasks, Liquid AI's compact LFMs outperformed every frontier model the team evaluated. The team plans to present the work to the longevity research community at ARDD this week.
Tencent Hy@TencentHunyuanAI score62AIResearchers from Tencent Hy, Fudan University, and Tsinghua University released ExplorationBench, a benchmark that tests whether AI systems can discover hidden rules in executable Alien World sandboxes. Across 10 frontier systems, getting feedback from experiments outperformed thinking alone, with the best run reaching 89.0% after four rounds. The authors note that rankings barely transfer between the two worlds, and the code is listed as coming soon.
ModelScope@ModelScope2022AI score46AIIndexTeam has released Index-Translate, a multilingual family covering text, speech, dubbing, and long-document translation across 150 languages. Its 9B model scores 0.8789 on FLORES, 0.8209 on instTrans, and 0.7387 on MEME, and the 2B and 9B models are released under Apache 2.0.
AIIssue 942 of The Sequence links Anthropic's Claude Opus 5.5, reported for the week of September 21–27, to DeepSeek's September 19 environments paper and a report of AI-assisted biological discovery. The newsletter argues that progress increasingly depends on the surrounding machinery that governs where a model acts, what it observes, and how its conclusions are checked.
AIHamel Husain reviewed Anthropic's new build_eval and hill-climb commands in the claude-api plugin for Claude Code, finding it useful for discovering issues like human handoff, formatting, and voice agent problems. He criticized it for pushing evaluator creation before data review, asking for label validation in Markdown files, and bundling four failure checks into one broad call-transfer evaluator. Husain says he would hold off on using it for now.
ModelScope@ModelScope2022AI score62AIInSpatio-World 1.5 from InSpatio_AI turns a single image, four images, a panorama, or a video into a navigable scene with wide viewpoint changes. The 1.3B model scores 68.72 on WorldScore-Dynamic, ranking first among evaluated real-time and interactive methods, with speeds up to 24 FPS. The post says the code is released under Apache 2.0 and that dependencies keep their own licenses.
AIKorean AI lab Upstage has released Solar Mini 4, a proprietary reasoning model that scores 24 on the Artificial Analysis Intelligence Index with 35B total and 3B active parameters. It is priced at $0.10/$0.40 per 1M input/output tokens and has a 1M-token context window, but averages 7.1 minutes per task due to heavy output token use. Its weights are not released, and its size cannot be independently verified.
AIArtificial Analysis reports that Google's Gemini 4 Argon scores 53 on its Intelligence Index with high reasoning, matching GPT-6 Astra (max) and one point ahead of GPT-6.1 Sol (max). At the current 50% launch discount, its cost per task is $1.99, about 60% of GPT-6 Astra's $3.26, but the discount's end date is unconfirmed and standard pricing would raise it to $3.98. The model is being rolled out to selected users and is not publicly available.
Why it matters: The benchmark compares Gemini 4 Argon's cost per task and hallucination rate with GPT-6 Astra, showing where its value depends on a temporary 50% discount.
Jerry Liu@jerryjliu0AI score20AIJerry Liu says Jev, a System One model, outperformed other open-source classifiers and document-specific models on orientation detection, language detection, classification, and splitting. The benchmark measured accuracy, cost, and latency across these fast document decisions, with Jev leading most comparisons. The benchmark code is available in the run-llama/jev_vs_oss repository.
Jerry Liu@jerryjliu0AI score22AIJerry Liu benchmarked gpt-6.1 sol on document OCR tasks and found a sizable increase in table parsing and reading order over gpt-6 sol from a week earlier. Its table parsing is similar to gpt-6 astra. He noted frontier models still cost roughly an order of magnitude more than cost-effective document parsing solutions, leaving room to improve the premium end above 1c per page.
AIHugging Face released the Open TTS Leaderboard, which evaluates open-source text-to-speech models using objective metrics instead of arena-style human votes. It measures intelligibility via WER and CER using Qwen3 ASR, speed via RTFx and time-to-first-audio on an H200 GPU, and speaker similarity via WavLM embeddings. The leaderboard covers multilingual results and voice cloning, and it is intended to complement, not replace, human preference rankings.
AIApple researchers systematically tested LLM conditioning methods and found efficient activation steering often degrades fluency. Steering is far less effective on instruction-tuned models than base models, while prompting and full supervised fine-tuning work for concept injection but are weaker at concept removal. Cheap textual metrics correlate highly with costly LLM-as-judge scores.
Prime Intellect@PrimeIntellectAI score20AIPrime Intellect says it will be among the first to deploy on NVIDIA's new Vera CPU, after earlier access to benchmark sandboxes on Vera in March. The company plans to use Vera's dynamic memory latency for agentic workloads, which it describes as ideal for Prime Sandboxes.
Liquid AI@liquidaiAI score32AILiquid AI announced d1, its first decision model, which it says is the first to outperform Jev on Hugging Face's Decision Index. The company claims d1 wins on multilingual evals, resists prompt injection better, handles longer inputs more effectively, and is built for fast, structured decision-making in software environments. It is available via the Liquid API at console.liquid.ai, with OpenRouter availability coming soon.
Noam Brown@polynoamialAI score25AINoam Brown praised OpenAI for presenting model evaluations as intelligence plotted against cost, arguing that cost should be part of how intelligence is measured. OpenAI's linked context says GPT-6.1 Sol delivers near-Astra intelligence at one-fifth the price and is the most cost-efficient model for its performance available today.
AIBAAI released AREX-2, a 27B-parameter long-horizon agent model that improves solutions over multiple test-time rounds by proposing, measuring, reflecting, and revising. It was trained on machine-learning and algorithmic-programming tasks with verifiable feedback, and the source reports that this self-improvement transfers to deep research. The model is Apache License 2.0 licensed and has a 262,144-token context length.
Why it matters: The source compares AREX-2 against closed and open models on coding and deep-research benchmarks, showing how test-time self-improvement is measured across task types.
Cognition@cognitionAI score38AIAt low effort, GPT-6.1 Sol scores 58.1% at $0.21 per task, up from 50.5% for GPT-6 Sol at the same setting. That makes it the highest-scoring model under $0.30 per task on the leaderboard.
Cognition@cognitionAI score40AIGPT-6.1 Sol is now available in Devin, scoring 60.4% on FrontierCode 1.1, close to GPT-6 Sol's 60.7%. At medium reasoning effort it costs $0.31 per task, 81% less than GPT-6 Sol at max effort.
OpenAI@OpenAIAI score37AIOpenAI reports that GPT-6.1 Sol shows major alignment improvements over GPT-6 Sol in its evaluations, moving closer to GPT-6 Astra. The model is more transparent about its limitations and more reliable at respecting user intent and safety constraints.
Jerry Liu@jerryjliu0AI score22AIJerry Liu hosted a dinner with Snorkel's Vincent Sun on evals and RL environments, a topic shaped by models rapidly saturating benchmarks. The conversation highlighted that building fair RL environments is hard, since failures are difficult to attribute to input, harness, or reward model, and that long-horizon evals spanning weeks or months remain very difficult. The post also noted that regulated industries still require human-in-the-loop review because 80% accuracy is not sufficient.
AIReplit explains how its Agent lets the core model pick subagent tier and effort mid-task rather than relying on an external router. On DeepSWE and Terminal-Bench, Replit Agent scored 72% at $2.11 per task and 49% at $2.53 per task, beating a single long-lived worker sidekick setup by 11 and 16 points. The company says Astra on its own scores higher only at more than twice the cost.
Why it matters: The post gives a concrete harness design with benchmark cost-score comparisons, helping builders weigh delegation strategies against routers and single-worker setups.
Sebastian Raschka@rasbtAI score22AISebastian Raschka has published a comprehensive write-up on language models for text classification, covering RNNs, CNNs, transformers, and calibration. It is presented as a visual guide with hands-on experiments on accuracy and efficiency.
ModelScope@ModelScope2022AI score54AIModelScope announced IQuest-Q1, a 320B MoE model with 15B active parameters and a 512K context window for agentic coding. The post reports scores of 84.5 on CyberGym, 83.2 on Terminal-Bench 2.1, 64.6 on DeepSWE v1.1, and 63.0 on NL2Repo, and says weights are released under the IQuest-Q1 License.
