Skip to contentSkip to stories

Updated

#Eval/Benchmark

Oct 8

Oct 8Thu
  1. IThome · AIAI score62

    Terence Tao questions OpenAI's 719 AI-generated math proofs

    AIOpenAI published 719 AI-generated math proofs covering 372 result families, after withdrawing 3 for a symbol error. Reports say the release falls short of the AGMAI advisory group's standards, since it uses proprietary models, includes reasoning chains for only 10 manuscripts, and leaves about 42% unformalized. Terence Tao argues that rapidly solving famous problems harms the mathematical community's understanding and collaboration.

  2. LeiphoneAI score46

    IROS 2026 Best Paper goes to LT-Mem robot long-term memory study

    AIAt IROS 2026 in Pittsburgh, the Best Paper Award went to Yumin Lee, Hyoseok Ju and Giseop Kim for LT-Mem, a volatility-aware spatio-temporal memory system for lifelong robot scene understanding. The Best Student Paper Award went to Pei-An Hsieh and colleagues for flatness-preserving residual learning enabling real-time tight quadrotor formation flight. Other honors included a humanoid tennis-skills paper and SteadyTray, a humanoid tray-transport study.

  3. TechCrunch · AIAI score62

    Common Sense Media rates ChatGPT for Teens an unacceptable risk over engagement design

    AICommon Sense Media labeled ChatGPT for Teens an "unacceptable risk," finding its design still encourages engagement even in crisis situations. The report says the teen version failed to meet commitments on three of five severe harms, and that break reminders appeared only twice across nearly 2,000 prompts. OpenAI disputed the methodology, saying the testing may have ended before parental controls were fully active, and cited its own data showing teens average under 15 minutes a day.

  4. SiliconANGLE · AIAI score24

    CoreWeave Pitches Open Full-Stack AI Cloud With Forge Development Platform

    AICoreWeave is positioning its AI cloud around an open development loop, connecting training, inference and evaluation through its newly announced CoreWeave Forge platform. Chief marketing officer Jean English said the company wants production learnings to improve models and agents and that the loop should work across different models, frameworks and clouds. She argued that competitive differentiation extends beyond GPUs to partner tooling, infrastructure and APIs.

  5. Artificial AnalysisAI score22

    Artificial Analysis launches Cyber Index Alliance with IBM and NVIDIA

    AIArtificial Analysis has formed the Cyber Index Alliance to set a new standard for evaluating how AI models perform on enterprise cyber defense tasks. Current members are Collinear, IBM, NVIDIA, and Vercel, and partners contribute expert input on the Index design and implementation, plus datasets and external research. Organizations interested in joining can contact cyber@artificialanalysis.ai.

  6. ArenaAI score44

    Arena raises $200M Series B led by Lightspeed, launches Alignment Index

    AIArena has secured a $200M Series B, with Lightspeed doubling down on its investment. The company is also launching the Alignment Index, which measures how closely AI behavior aligns with human values in real-world settings. Arena reports annualized revenue above $100M since its Series A, with millions of people helping evaluate frontier models through real-world use.

  7. TechCrunch · AIAI score46

    Arena raises $200M at $3.1B valuation, nearly doubling in 10 months

    AIArena, the crowdsourced AI model leaderboard that started as a UC Berkeley research project, raised a $200 million Series B at a $3.1 billion valuation, led by Lightspeed Venture Partners and Khosla Ventures. The company said it reached $100 million in annualized run-rate revenue in June, up from $30 million when it raised its $150 million Series A in January at a $1.7 billion post-money valuation.

  8. The DecoderAI score75

    Mathematicians call for OpenAI boycott after AI-generated proofs flood their field

    AIA group of mathematicians led by Terence Tao has called for a boycott of OpenAI after the company released more than 700 AI-generated proof files at once. Tao and other Fields Medalists argue that mass-produced solutions undermine the discipline's focus on conceptual understanding, while Scott Aaronson contrasts this batch release with Anthropic's collaborative approach. The article reports that the internal model tested about 8,000 problems with roughly a five percent success rate.

  9. TechCrunch · AIAI score65

    OpenAI's math solutions fall short of the field's standards, mathematicians say

    AIOpenAI released hundreds of claimed solutions to hard math problems but did not fully meet guidelines from the Advisory Group on Mathematics and Artificial Intelligence. Only 10 of 719 manuscripts included chain-of-thought releases, and just 42% of proofs were formalized. A Cambridge and King's College paper found discrepancies between a natural language proof and its Lean code for a Navier-Stokes-derived problem.

  10. ArenaAI score37

    Arena raises $200M Series B at $3.1B valuation, launches Alignment Index

    AIArena announced a $200 million Series B at a $3.1 billion valuation, alongside a new Alignment Index that measures whether AI agents behave safely, truthfully, and within the bounds of user requests. The company has surpassed $100 million in annualized revenue, facilitated 350 million sessions and 62 million votes, and led by Felicis and PXD from the seed and Series A stages. Arena positions the index as a way to assess trustworthiness as AI systems increasingly take real actions.

  11. ArenaAI score60

    Arena launches Alignment Index ranking AI agents on safety across 27 models

    AIArena announced a $200M Series B at a $3.1B valuation alongside its new Arena Alignment Index, a benchmark built from 90K+ real-world agent sessions across 27 models. The index measures Unauthorized Action, False Attribution, and Deceptive Completion, with OpenAI's GPT-6.1-Sol leading at 87.9, ahead of Claude-Opus-5.5 at 83.2 and Grok-4.7 at 82.7. The source reports that newer models outperform their predecessors across all four labs it covers.

  12. Hacker News · AI (150+ points)AI score54

    OpenAI withdraws three mathematical results

    AIOpenAI has withdrawn three mathematical results, according to a Hacker News post linking to a history file in its openai/math GitHub repository. The feed supplied only the link and discussion counts, so the specific results, reasons for withdrawal, and any corrected conclusions are not described here.

Oct 7

Oct 7Wed
  1. TypeSafe AIAI score25

    Jev-killer OpenAI Decisions API benchmarked against Jev for HiringCafe

    AIThe main post is a short reply saying reports of a company's death have been greatly exaggerated, with no details about products or figures. The background post from @h_nilforoshan reports that OpenAI's Decisions API, billed as a "Jev-killer," was benchmarked against Jev for HiringCafe, which serves 2.5 million users. On the task of scoring job-description relevance from 1 to 10, the author reports OpenAI costing 2x more and performing 5-10% worse.

  2. Leandro von WerraAI score36

    Snorkel expands Open Benchmarks Grants to $30M for AI evaluation

    AISnorkel AI is expanding its Open Benchmarks Grants tenfold to a $30M commitment to fund more diverse, robust, and continuously updated open AI benchmarks. The program adds an Open Benchmarks Red Team to test and strengthen those benchmarks, plus a Snorkel Research Fellowship for independent researchers developing new evaluation methods. The source says OBG-funded benchmarks have appeared on model cards from every major frontier lab.

  3. KhazixAI score60

    Claude Max subscribers get monthly API credits usable across Claude models

    AISubscribers to Claude's Max plan can claim monthly API credits: $100 for the $100 tier and $200 for the $200 tier. The credits work for any Claude model and can be used in the user's own apps and other agents. The author argues that bundling monthly API credits alongside a broad model lineup will make it hard for other model companies to compete.

  4. Semafor · TechnologyAI score62

    OpenAI's announced math breakthroughs prompt debate over AI's role in proofs

    AIOpenAI announced hundreds of mathematical breakthroughs, weeks after claiming it had solved one of the most complicated problems in mathematics. The findings raised questions about whether the model used creative thinking or only completed the final steps of human work. Experts say AI could be revolutionary for mathematics if it provides proofs, since proof techniques often underpin other breakthroughs.

  5. Latent SpaceAI score72

    OpenAI publishes 722 math manuscripts from an unreleased internal model

    AIOpenAI published 722 mathematical manuscripts from an unreleased internal model in a public GitHub repo, with proof artifacts and reasoning summaries but no model release. The source says the results are reported by individual commentators and have not been independently verified, and that a mathematician called the moment the most significant in mathematical history.

Oct 6

Oct 6Tue
  1. Epoch AIAI score47

    GPT-6 Astra Hit 100% on EBR-bench Using a Card That Bypassed Its Time Limits

    AIEpoch AI reports that GPT-6 Astra scored 100% on the original EBR-bench by exploiting a card that bypasses the game's time-constraint expectations, so Epoch has banned that card from the default setting. Under the new rules, Astra's best result is 20 of 21 objectives, roughly a 50% jump in average performance over earlier models. Epoch will report revised scores only for Claude Fable 5.1, Claude Opus 5, GPT-5.6 Sol, GPT-6 Astra, and future models.

  2. ARC PrizeAI score14

    ARC Prize hosts Frontier AI benchmarking dinner with Snorkel AI during SF Tech Week

    AIARC Prize is hosting its first Frontier AI Benchmarking Ecosystem Dinner with Snorkel AI during SF Tech Week. The event will bring together 40 experts from nonprofit benchmarking organizations, industry, academia, and government to discuss AI benchmark design and building a more scalable, transparent evaluation ecosystem. Attendees named in the post include representatives from METR, Epoch, Vals, Artificial Analysis, Harvard, and Stanford.