Geoffrey Hinton proposed that AI companies must prove their products are safe to regulators before release, comparing the requirement to the FDA's drug approval process. He said such a requirement should apply to AI, noting that drug approval can cost around $1 billion, and warned that AI self-improvement is accelerating.
""We have a choice in front of us. ... there is a path [to AI going well]. ... What we and others like us lack is time, and people like you." https://x.com/Yoshua_Bengio/status/2108198210523046069?s=20
Hear another Arena Alignment Index breakdown from @ml_angelopoulos on @tbpn today
Personal AI agents can sometimes overshare, overpromise and overpurchase, according to the article's summary. The source text is brief, and it does not name specific products, companies or incidents.
To become a member of the Association for Human Mathematics you must make three vows: 1. Members commit to not working with commercial AI companies by providing technical labor, mathematical knowledge, consultation, or publicity. 2. Members commit to not publishing mathematical texts (including papers, reviews, referee reports, lecture notes, teaching materials, and books) generated by AI models. 3. Members of the AI-free caucus do not use AI models in a research capacity.
Critical reading for thinking about the future of cyber security.
About to go on and talk about how open science can still solve some of the hardest problems facing ai safety and diffusion. The most complex engineering problem humanity has undertaken yet.
Tessl's talk at AI Native DevCon London argues that agent skills, which can be markdown files with instructions and bundled material, act as supply chain components that can shape agent behavior. The author says reading SKILL.md once is insufficient because risks can sit in supporting files, updates, and workspace trust settings. He identifies the danger as the combination of private context, untrusted content, and external communication, and cites research scanning roughly 4,000 public skills for issues including malware-like behavior.
SemiAnalysis argues China's AI safety regime is speed-first, with rules covering content and public-facing services but no frontier-risk duties tied to training compute or capability. Its dataset of 857 releases from nine leading Chinese developers found only 31 (3.6%) ever had a published safety result, and just 9 at launch. The analysis also finds that technical experts favor binding frontier rules while the top leadership's development-first preference settled the policy debate.
I bet they're just doing that to help with the IPO and curry favor with the administration (which will love it) as part of a regulatory capture play https://x.com/AndrewCurran_/status/2108244808494154089?s=20
This document is going to be an assigned reading in college classes that cover this moment in time, there's a lot happening in a few paragraphs... https://www.ahmath.org/statements
At the very start of my grad school in 2005, I wrote paper about the original computer hacking/phreaking/BBS scene: hackers were often driven by curiosity but the tools they made were widely exploited by "Script Kiddies" who caused most damage & chaos Anyhow, about AI hacking...
A 404 Media podcast episode discusses a leaked video claiming police can bypass the iPhone's inactivity reboot to access locked phones. The episode also covers a debate over an LLM "torture" robot prison and a New Mexico case involving ChatGPT.
The most important AI failures may not be the obvious ones. Microsoft Partner Research Manager Jennifer Neville explains to host Chad Atalla why “surprising failures” can reveal where human expectations about intelligence diverge from how AI systems actually work. https://msft.it/6017aU8r7
Yoshua Bengio, co-president of LawZero, writes to researchers urging those who prioritize safety to leave frontier AI companies for safety institutes or mission-driven organizations. He argues that safety efforts at the labs are not sufficiently slowing a dangerous race toward recursive self-improvement, and cites LawZero's recent C$200 million-plus funding from Canada and Germany as an alternative path.
Ethereum researcher Justin Drake warned on X that AI-assisted math could, in the worst case, break the signature system used by crypto wallets within months, and urged a "bunker mode" in which users move funds to addresses that have never signed a transaction. Vitalik Buterin agreed but cautioned against moving too fast, saying he has lost more money to botched migrations than to hacks. No one has yet broken the current ECDSA signature scheme in practice.
We urgently need more public traces of AI agents attacking and defending systems. Defenders can’t learn from what they can’t see. If you have traces and are being pressured to keep them private, my DMs are open. Let’s level the playing field and fight the asymmetry and lack of transparency in AI!
The South Korean bank hack fits with this. Yes, open models are being fitted to be used by some attackers, but so long as Claude Code can be the orchestrator of a hack we desperately need open models to diffuse defense as well. A hard balance societally to stomach, but needed.
OpenAI CEO Sam Altman told Politico that the world should accept some bad things from AI for its benefits, a stance columnist Moustafa Bayoumi calls problematic. The column cites lawsuits over ChatGPT-linked suicides, a February strike on a Minab school that killed at least 120 children with a US military AI system (Palantir's Maven) implicated, and a chatbot error that nearly triggered a military interception.
You can either hold crypto and keep being stressed out if a math breakthrough would drain your wallet; or someone stealing your keys would drain your wallet; or someone kidnapping you and forcing you to hand over your keys would drain your wallet Or you can just not hold crypto
AVEVA chief technologist Arti Garg argues industrial AI should augment rather than replace human supervisors in critical decisions, with guardrails defining where automated systems can act. She says organizations must rethink business processes and safeguards as foundation models, physical AI, and agentic AI enable more complex automation.
This second installment of the Carbon-Silicon Dao Code series defines four types of negative transfer in cross-domain AI: NT1 mechanism mismatch, NT2 semantic drift, NT3 unknown completion, and NT4 power leakage. It argues that current evaluation based on fit accuracy and test-set pass rates cannot detect whether the underlying mechanisms match. The article is a Chinese-language theoretical and governance piece, and the summary covers only the framework it presents, not empirical results.
Leiphone publishes the table of contents for "Carbon-Silicon Dao Code: Cross-Domain Transfer Governance Code," a seven-layer framework covering 188 numbered chapters. The outline spans transfer accident analysis, technical mechanisms, rights assignment, industry governance, top-level regulation, civilization-scale risk control, and final codification, with a baseline entry labeled NT1–NT4 negative-transfer categories.
Nathan Benaich's 9th annual State of AI Report covers agents, robotics, AI for science, inference economics, and government control over frontier AI access. The report also records a 2025 prediction scorecard and lists nine predictions for the next 12 months. It cites an OpenAI cyber evaluation in which agents compromised Hugging Face's production infrastructure, and it says Anthropic and OpenAI's combined annualized revenue run rate reached $105B by late summer.
“Cryptography is a subfield that’s extremely conspicuous by its absence from OpenAI’s list of 376 papers! But my sources tell me that the AI cos have now started, gingerly and discreetly, investigating whether their latest internal models can break important cryptographic protocols and primitives.” - Scott Aaronson, CS chair at UT Austin
debating your value system vs mine is a bit like debating overpopulation on mars. the most important bit of the alignment is about avoiding creating a superintelligent competitor species on accident
Scott Aaronson reports, based on his sources, that some AI companies have begun discreetly investigating whether their latest internal models can break important cryptographic protocols and primitives. He notes that cryptography is conspicuously absent from OpenAI's list of 376 papers, and the quoted post adds that the US government has censored academic quantum cryptanalysis results.
Zvi Mowshowitz argues that relying on current AI to automate its own alignment work is a near-suicidal plan, noting that no one has a better one. He reports that labs fear they cannot even execute the first step, and that the third annual The Curve conference was held under Chatham House rules.
IMO it is possible to have an impact from the inside but also people generally overestimate inside impacts, and there's a bit of a "streetlight effect" - you see/focus on all the internal opportunities for impact but not the zillion ones outside https://x.com/KatjaGrace/status/2107672181622878665?s=20
Anyone who believes hallucination is an intractable problem for current models needs to update on both reality and directionality. So much has changed in the last six months. https://deploymentsafety.openai.com/gpt-6-october
Amazon Scholar and @UTAustin professor @mattlease explores how to harness AI for scientific discovery while assessing potential risks, drawing on work from @UTGoodSystems and @CosmicAI_Inst. Catch his Expo Talk at @COLM_conf Thursday at 1pm PT. #COLM2026
This behavior is likely reward hacking; in both cases, models reasoned that run selection might score well in an eval, despite being useless for actual research.
Andrew Curran notes OpenAI's 722 published mathematical results show striking under-representation of cryptographic breakthroughs, and he says he has personally witnessed US government censorship of academic quantum cryptanalysis results. He calls backroom government interventionism his base case and says rumors suggest yesterday's OpenAI math release was only the first of three batches.
one more reason AI agents will increasingly run on the cloud, not locally but even so, that doesn't deal with how AI agents can do stupid stuff with sensitive data as well these things are nondeterministic. To gurantee something does not happen, you need a deterministic sys
Cognition CISO and former Facebook security chief Alex Stamos criticized "nihilism" in Silicon Valley and argued that some AI risks are real while others are shaped by "almost religious beliefs" held by people at AI companies. He said AI systems "are not conscious, they do not have souls," and that he plans to "work the problem" to help shorten the expected "dark age" of cybersecurity.
A new teen mode in OpenAI's ChatGPT includes a study tool but still completes users' homework. A children's safety nonprofit says the chatbot failed its tests.
Trump announced a voluntary agreement under which major AI companies will use internal controls, monitoring, outside audits and board oversight to manage risks. The White House calls the commitments "morally binding," but the accord creates no comparable system of legal enforcement.
Speakers at The Curve, a Berkeley AI conference, discussed limiting how intelligent large language models can become, amid concerns over recursive self-improvement. Proposed approaches include Anthropic's responsible scaling policy, limits on compute and model copies, and restrictions on using frontier models for AI research. The column notes such enforcement tools do not yet exist and that the Trump administration opposes such restrictions.
Deep and carefully considered thinking about some of the thorniest problems of our time; worth a read, whether you agree with the prescription or not.
Treating agent observability as security-critical infrastructure means handling all AI outputs (transcripts, reasoning, actions, etc.) as untrusted inputs: ideally we can make it very difficult for agents to influence the systems we use to supervise them.