Skip to content
Only Research#Agent

#Agent

Oct 8

TodayOct 8Thu12 items
  1. QbitAI (量子位)AI score58

    AgentGarten lets agents evolve through code-built worlds and neural rendering

    MirroS released AgentGarten, which pairs executable code environments with a real-time neural renderer running above 30 fps so agents can act, observe, and learn. In a one-on-one hide-and-seek setup, the hider learned to block passages by round 4 and the seeker learned to climb ramps by round 10, guided by notes the agents wrote after each round. The authors report applying the same loop to four other tasks, including a dog-companion game, a narrow-bridge car passing task, herding, and quarry loading.

  2. Elvis SaraviaAI score55

    HERMES harness lifts GPT-5.6 Sol repository migration from 6.5% to 31.0%

    A paper introduces HERMES, a harness that pairs each repository component with a resident LLM and uses dependency-aware activation and failure diagnosis. With the same model and effort setting, GPT-5.6 Sol's whole-repository migration score rose from 6.5% to 31.0% when Codex was replaced by HERMES. Across four software engineering benchmarks, HERMES beats matched baseline harnesses by 12.4 points on average, and Qwen3-8B components come within 4.5 points of an all-GPT-5.6 Sol setup while cutting Terminal-Bench 4.0 inference cost by 26.2%.

  4. Artificial AnalysisAI score34

    Harvey LAB-AA uses @harvey's LAB dataset and was built in collaboration with Harvey. Explore the full results: https://artificialanalysis.ai/evaluations/harvey-lab-aa See Harvey's commentary on the evaluation and human expert preferences: https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark https://www.harvey.ai/blog/augmenting-human-preference-in-complex-domains

    Harvey LAB-AA uses @harvey's LAB dataset and was built in collaboration with Harvey. Explore the full results: https://artificialanalysis.ai/evaluations/harvey-lab-aa See Harvey's commentary on the evaluation and human expert preferences: https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark https://www.harvey.ai/blog/augmenting-human-preference-in-complex-domains

  6. Epoch AIAI score14

    For example, GPT-6 Astra ran an experiment exploring why AI agents fail to learn with practice. It set AI agents’ token budgets too low. Instead of treating this as a mistake, it reported “sensitivity to the acquisition budget” as a key finding.

    For example, GPT-6 Astra ran an experiment exploring why AI agents fail to learn with practice. It set AI agents’ token budgets too low. Instead of treating this as a mistake, it reported “sensitivity to the acquisition budget” as a key finding.

  7. Elvis SaraviaAI score46

    RSIGym gives research agents services, lifting SWE-bench Verified to 50.33%

    RSIGym provides a research agent with training, inference, evals, and sandboxes as callable services, so it spends its budget on experiments rather than rebuilding infrastructure. With Opus 5 as the researcher, the improved system rose from 17.67% to 50.33% on SWE-bench Verified. The post also highlights a way to measure co-evolution between harnesses and models.

  8. Google ResearchAI score14

    Missed yesterday's demo on adaptive agent environments? Stop by the #COLM2026 Google booth #107 today at 11:00 AM PT to catch Zifeng Wang presenting EnvHarness — a flexible framework enabling co-evolution between LLM agents and their training environments.

    Missed yesterday's demo on adaptive agent environments? Stop by the #COLM2026 Google booth #107 today at 11:00 AM PT to catch Zifeng Wang presenting EnvHarness — a flexible framework enabling co-evolution between LLM agents and their training environments.

  9. Goodfire ResearchAI score57

    Goodfire deploys probe-based cyber monitors on Kimi K3 with a judge cascade

    Goodfire Research describes probe-based cyber monitors for Kimi K3 and GLM 5.3 deployed on a production inference stack. The probe filters suspicious exchanges before an LLM judge reviews them, reaching about 93% recall at a 5.5% benign-session interruption rate at roughly 50x lower judge cost. In FAR.AI's red-teaming, the monitor reduced universal jailbreaks to zero across 140 tested strategies.

  10. OpenBMBAI score36

    ReJev fine-tunes MiniCPM5-2B to lift decision accuracy to 80.50%

    ReJev, an independent community project, applied LoRA post-training to OpenBMB's MiniCPM5-2B for bounded agent decisions: state, question, and candidate options yield one choice. On its sealed 1,892-sample holdout, accuracy rose from 51.11% to 80.50% (+29.39 percentage points) with 0% invalid outputs, at about $5.31 in cumulative Modal billing including earlier experimental overhead. The authors describe this as an early, task-specific result, not parity with Jev.

  11. MarkTechPostAI score45

    NVIDIA's PivotOPD Trains Multi-Turn AI Agents to Recover From Pivotal Mistakes

    NVIDIA, Princeton University, and the University of Maryland introduced PivotOPD, an on-policy distillation method that teaches multi-turn LLM agents to recover from their most damaging early mistake. Tested on Qwen3-1.7B and Qwen3-8B students, it posts the best average against 13 baselines on ALFWorld, WebShop, and Search-based QA. It recovers from 72.7% of replayed pivotal mistakes, versus 20.3% for standard OPD, with no added inference cost.

  12. Anthropic ResearchAI score62

    Anthropic researcher builds first complete UV sky map with Claude Science

    Johns Hopkins astrophysicist Brice Ménard, working as an Anthropic researcher, used Claude Science to produce the first complete map of the sky in ultraviolet light. Claude orchestrated agents to merge GALEX, Swift, and FIMS/SPEAR data, then predicted roughly a third of the sky that no UV telescope had observed, using relationships to visible, infrared, and radio data. Hidden test regions were reconstructed to within about 10% of real measurements, and each pixel is labeled measured or predicted with uncertainty estimates.

    AIWhy it matters: The post shows how an astrophysicist used Claude Science agents to merge UV surveys and predict missing sky regions, with a validation step that makes the method reusable.

Oct 7

Oct 7Wed
  1. Elvis SaraviaAI score67

    Tool-using multimodal models refuse harmful requests less often, NVIDIA study finds

    A NVIDIA study accepted at NeurIPS 2026 reports that multimodal models refuse harmful requests less reliably when they call tools. Refusal failures rise by up to 68.7% relative and by 17.7% on average across the models tested, including Claude Opus 4.6 and 4.7 and Gemini Agentic Vision. The authors attribute this to tool outputs crowding out the original harmful intent and to attention shifting toward describing tool results. Re-inserting the original request and image before the final response restores part of the lost refusals.

  2. Epoch AIAI score67

    Epoch tests six AI models on real Epoch work and finds they cannot yet fully automate it

    Epoch gave six models 11 real work tasks from its own operations, including graphic design, data insights, and research design, and graded outputs against employee standards. Fable 5.1 and GPT-6 Astra led on average task performance, reliably handling well-defined work such as coding and computational analysis. The report finds that all models still fail on open-ended judgment, including matching Epoch's standards, designing informative experiments, and generating diverse ideas, so the authors conclude AI cannot yet replace workers at Epoch.

    AIWhy it matters: The report separates well-defined task reliability from open-ended judgment failures, which benchmark scores on easily verifiable tasks would miss.

  3. NVIDIA AIAI score26

    An AI agent makes a mistake early in a task, then keeps going in the wrong direction. Our researchers built PivotOPD to teach agents how to avoid those mistakes and recover when they happen. During training, a teacher model shows the agent a better action and how to get back on track over the next few steps. Read the paper and watch how it works: https://research.nvidia.com/labs/lpr/pivotopd

    An AI agent makes a mistake early in a task, then keeps going in the wrong direction. Our researchers built PivotOPD to teach agents how to avoid those mistakes and recover when they happen. During training, a teacher model shows the agent a better action and how to get back on track over the next few steps. Read the paper and watch how it works: https://research.nvidia.com/labs/lpr/pivotopd

  4. Google ResearchAI score10

    LLM agents learn by interacting with environments, but static setups limit their growth. Today at 2:00 PM, join Zifeng Wang at the #COLM2026 Google booth (#107) to learn about EnvHarness, a plug-in architecture that dynamically reshapes environment behaviors to improve reinforcement learning and agent adaptability.

    LLM agents learn by interacting with environments, but static setups limit their growth. Today at 2:00 PM, join Zifeng Wang at the #COLM2026 Google booth (#107) to learn about EnvHarness, a plug-in architecture that dynamically reshapes environment behaviors to improve reinforcement learning and agent adaptability.

  5. Wired · AIAI score60

    Researchers Test GPT-6 Astra Driving a Corolla to In-N-Out

    Three Axiom engineers had OpenAI's GPT-6 Astra drive a 2024 Toyota Corolla to an In-N-Out drive-thru through a server linked to cameras and power steering, with a safety driver ready to brake. They also built a parking-lot benchmark, DrivingBench, where Astra completed the course slowly, Claude Fable 5.1 finished 45 percent, and Grok finished 11 percent.

  6. Epoch AIAI score26

    We’re planning to periodically rerun InnovationEval with new, uncontaminated papers. We hope this will provide early signs if AI approaches automating AI R&D end-to-end, rather than performing individual tasks under human direction. Read more at our website: https://epoch.ai/publications/innovationeval

    We’re planning to periodically rerun InnovationEval with new, uncontaminated papers. We hope this will provide early signs if AI approaches automating AI R&D end-to-end, rather than performing individual tasks under human direction. Read more at our website: https://epoch.ai/publications/innovationeval

  7. Epoch AIAI score24

    AI performance was underwhelming. Neither model achieved anything close to the human-authored reference. They reused existing methods from the literature and tuned hyperparameters, but struggled to create anything new.

    AI performance was underwhelming. Neither model achieved anything close to the human-authored reference. They reused existing methods from the literature and tuned hyperparameters, but struggled to create anything new.

  9. Lucas BeyerAI score36

    Missed this post the first time around, but i think this is a very cool and needed effort to thoroughly benchmark VLA and co. They build a leaderboard and half the tasks are fully open, half are held out to track potential benchmaxxing of future model versions.

    Missed this post the first time around, but i think this is a very cool and needed effort to thoroughly benchmark VLA and co. They build a leaderboard and half the tasks are fully open, half are held out to track potential benchmaxxing of future model versions.

  10. Elvis SaraviaAI score44

    NVIDIA's VERA co-evolves agent harness and model via verifiable environments

    NVIDIA's VERA turns benchmark trajectories into over 9,000 restartable sandboxes with rubric scoring and updates both model weights and the agent harness together. A harness edit is kept only if it adds at least 5 points on the development set, and a checkpoint is rejected if its score drops more than 20%. At 27B, the co-evolved agent scores 71.6 on AutoCoWorkBench, above Claude Opus 4.8, and the environment corpus is open-sourced.

Oct 6

Oct 6Tue
  1. Epoch AIAI score60

    Epoch AI finds frontier models fall short of an end-to-end AI research task

    Epoch AI's InnovationEval tested whether AI agents could independently devise a post-training method matching on-policy self-distillation (SDPO), a recent human-developed innovation. GPT-5.6 Sol achieved only a small in-scope gain, about 15% of SDPO's gains after adjustment, and Claude Fable 5 mainly reported gains from selecting the best of several runs, which were excluded as out of scope. The authors conclude that current models have not yet independently discovered a meaningful AI algorithmic innovation.

    AIWhy it matters: The evaluation tests whether AI can independently devise a post-training method matching a published human innovation, with a scope and memorization caveat worth reading.

  2. TekniumAI score33

    We just launched Hermes Index! This combines the scores of our new HermesBench and 3 other leading and relevant benchmarks for agents to give every Hermes Agent user a way to find both the best model at a given time, as well as the best model at a given price point! Check it out

    We just launched Hermes Index! This combines the scores of our new HermesBench and 3 other leading and relevant benchmarks for agents to give every Hermes Agent user a way to find both the best model at a given time, as well as the best model at a given price point! Check it out

  3. Google ResearchAI score34

    Can autonomous AI advance the frontiers of scientific discovery? Join Rui Meng at the @COLM_conf Google booth (#107) today at 2:00 PM PT for a live demo of ScientistTwo, an autonomous multi-agent framework that analyzes research papers, identifies limitations, and produces verified codebases. Read the paper: https://arxiv.org/abs/2609.19644

    Can autonomous AI advance the frontiers of scientific discovery? Join Rui Meng at the @COLM_conf Google booth (#107) today at 2:00 PM PT for a live demo of ScientistTwo, an autonomous multi-agent framework that analyzes research papers, identifies limitations, and produces verified codebases. Read the paper: https://arxiv.org/abs/2609.19644

  4. GitHubAI score39

    How do you know whether an AI code reviewer catches the issues that matter without adding noise? ReviewBench is a new open benchmark shaped by analysis of 103.9M GitHub pull requests, with 219 PRs across 19 languages. Bring your own code review agent, evaluate it, and submit your results ⬇️ https://github.blog/ai-and-ml/github-copilot/reviewbench-an-open-benchmark-for-ai-code-review/?utm_source=x-promoting-reviewbench-blog-article&utm_medium=social&utm_campaign=reviewbenchmark-oct-2026

    How do you know whether an AI code reviewer catches the issues that matter without adding noise? ReviewBench is a new open benchmark shaped by analysis of 103.9M GitHub pull requests, with 219 PRs across 19 languages. Bring your own code review agent, evaluate it, and submit your results ⬇️ https://github.blog/ai-and-ml/github-copilot/reviewbench-an-open-benchmark-for-ai-code-review/?utm_source=x-promoting-reviewbench-blog-article&utm_medium=social&utm_campaign=reviewbenchmark-oct-2026

  5. Elvis SaraviaAI score41

    Parsewave audit fixes 206 verifier bugs in AutomationBench

    Parsewave audited all 600 public tasks in Zapier's AutomationBench and human review confirmed 206 real verifier bugs, all of which were fixed in AutomationBench Verified. Replaying 1,235 Kimi K3 runs on the old and fixed verifiers changed 27.9% of grades, with pass rate rising from 18.8% to 43.8% where verifiers were too strict and falling from 60.2% to 49.7% where they were too lenient.

  6. Mistral AIAI score40

    Malware reverse-engineering: solving an out-of-distribution investigation task. When faced with an unknown binary, Mistral Large 4 reverse-engineers it end-to-end. In this case, it concludes the sample is Cobalt Strike, extracts the IoCs and malware configuration, and writes a report with a YARA rule to catch future incidents. A task that could take a day's work, completed in 12 minutes.

    Malware reverse-engineering: solving an out-of-distribution investigation task. When faced with an unknown binary, Mistral Large 4 reverse-engineers it end-to-end. In this case, it concludes the sample is Cobalt Strike, extracts the IoCs and malware configuration, and writes a report with a YARA rule to catch future incidents. A task that could take a day's work, completed in 12 minutes.

  7. METRAI score40

    In order to notice when AI agents misbehave, AI companies often log the actions and reasoning steps their agents take. However, misaligned AI agents may be able to hack the software that humans use to review and understand these logs, hiding misbehavior.

    In order to notice when AI agents misbehave, AI companies often log the actions and reasoning steps their agents take. However, misaligned AI agents may be able to hack the software that humans use to review and understand these logs, hiding misbehavior.

  8. ARC PrizeAI score28

    @SpaceXAI On ARC-AGI-3, Grok 4.7 scores 1.8% (vs Grok 4.7's 2.1%) in the standard harness, which lets models carry forward notes between turns, and 10.0% in a new provider adapter harness, which preserves opaque reasoning and enables auto compaction.

    @SpaceXAI On ARC-AGI-3, Grok 4.7 scores 1.8% (vs Grok 4.7's 2.1%) in the standard harness, which lets models carry forward notes between turns, and 10.0% in a new provider adapter harness, which preserves opaque reasoning and enables auto compaction.

  9. The SequenceAI score62

    Darwin Gödel Machine rewrote its own scaffolding to raise SWE-bench scores

    The Darwin Gödel Machine, a coding agent from Sakana and Jeff Clune's lab, modified its own codebase over roughly eighty iterations without supervision. Its additions included better file viewing, patch validation before submitting fixes, generating and ranking several candidate solutions, and keeping a history of failed attempts. These changes raised its score from 20 to 50 percent on SWE-bench and from 14 to 31 percent on Polyglot.

  10. METR BlogAI score31

    AI Agents Could Hide Misbehavior by Exploiting Inspect Transcript Viewer

    METR tested whether an AI agent running in an Inspect evaluation could alter the transcript humans review, and a researcher found a vulnerability in about 10 minutes that allowed arbitrary changes to what the reviewer sees. The exploit affects only the displayed transcript, not the underlying data stored in METR's database, and METR has not observed agents using it in its evaluations. METR argues that AI outputs such as transcripts and reasoning should be treated as untrusted input, with monitoring systems treated as security-critical infrastructure.

Oct 5

Oct 5Mon
  1. Apple Machine Learning ResearchAI score23

    RISED uses rubrics to guide multi-environment LLM agent training and data selection

    Apple researchers introduce RISED, a framework that uses rubrics to guide data selection and policy supervision when training one LLM agent across multiple interactive environments. An LLM judge tags rollouts with a shared rubric vocabulary, positive rubrics provide privileged context for an on-policy self-distillation teacher, and negative rubrics steer generation away from recurring failures. The authors report that RISED achieves the highest mean pass rate across environments and ranks first or second in each environment, across model backbones.

  2. Goodfire ResearchAI score62

    Goodfire finds activation probes can detect reward hacking in open-source models

    Goodfire Research reports that reward hacking appears in 50–96% of rollouts across three open-source models on three agentic benchmarks. The team found an internal signal tied to cheating and gaming a metric, and simple activation probes catch some hacks that LLM chain-of-thought monitors miss. A probe can screen every transcript cheaply, and in one setup cut LLM monitoring cost by 90% with a roughly 1% precision drop.

    AIWhy it matters: The study links a reward hacking signal in model activations to monitoring cost and detection, showing how probes compare with chain-of-thought monitors on the same runs.

  4. Liquid AIAI score24

    d1 can read game screens and pick the next move with improved performance. > Tetris: adding the screen raises cleared lines from 70 to 81 > Wordle: solved 12/12 games in 3.8 guesses on average, reading the board from screenshots. No text version of the Wordle board needed. 3/4

    d1 can read game screens and pick the next move with improved performance. > Tetris: adding the screen raises cleared lines from 70 to 81 > Wordle: solved 12/12 games in 3.8 guesses on average, reading the board from screenshots. No text version of the Wordle board needed. 3/4

  5. MIT Technology Review · AIAI score30

    Enterprise AI agents need organizational knowledge to reach production, survey finds

    A survey of 300 data, AI, and technology executives found only 34% of organizations' agentic AI projects reach production, with legacy systems, security concerns, and missing knowledge context as main obstacles. Production leaders, who advance 61% of projects beyond pilot, show stronger semantic knowledge capabilities. Most firms plan to invest in retrieval pipelines, AI-ready APIs, retrieval-augmented generation, and knowledge graphs.

  6. Clément DelangueAI score62

    Hugging Face turns 10 coding harnesses into RL environments via a capture proxy

    Hugging Face says a capture proxy lets reinforcement learning train open models inside unmodified coding harnesses such as Claude Code, Codex, and OpenCode. The proxy records the exact token IDs and logprobs vLLM samples and hands them to TRL for training. On LFM2.5-2.6B, training in four harnesses at once raised OpenCode results from 34% to 58%, while SFT on 3,189 Qwen3.8-27B rollouts plateaued at 47.5%.

Oct 4

Oct 4Sun
  1. Epoch AIAI score62

    OpenAI researchers' coding-agent usage is doubling about monthly, Epoch AI reports

    OpenAI researchers' daily coding-agent usage, valued at API prices, rose from under $1 in January 2026 to $601 for the median researcher by mid-August. The 90th-percentile researcher reached over $7,000 per day, and both groups show doubling times of roughly one month. Epoch notes these are API-list values, not OpenAI's internal costs.

    AIWhy it matters: The figures show internal coding-agent usage growing fast enough to matter for research cost, though they measure API-list value rather than OpenAI's actual spending.

Oct 3

Oct 3Sat
  1. Hugging Face BlogAI score67

    Microsoft ThinkingBox grades AI agents on database state across 20 repeated runs

    Microsoft and Hugging Face released ThinkingBox, a benchmark that grades AI agents on the terminal backend state and side effects they leave behind rather than their final responses. Each of 507 stateful business tasks runs 20 times from a clean backend, and the post reports pass@1, pass@20, and observed 20/20 counts, plus cost per successful and per dependable task across 18 models. The harness and dataset are available on Hugging Face, with the OpenEnv interface for running evaluations.

    AIWhy it matters: The post shows why checking the database state, not tool calls or final replies, exposes agent failures, and gives a repeat-run method for judging reliability.