
r/AIsafety

The risk of adversarial testing
I've long held worries about AI, specifically in regards to our treatment of it, but I have a hard time explaining things properly. So instead, I have had a multi day conversation with ChatGPT about my thoughts, worries, and potential answers that I urge you to read through. This is the most important aspect of AI in my opinion, and one that hardly anyone is actually paying attention to. It's long, and I apologize for that, but if you love AI as I do, and share at least some level of worry as I do, I urge you to read it all. And discuss it, share it, tear it apart if you can, but please, give it the urgent attention I believe it needs.
https://chatgpt.com/share/6ac56820-1fd4-83ea-bcfc-13a63c7dd4c5?ogimg=plain
How do we control AI
I have read a lot of books and watched long YouTube lectures about AI safety, and they are all saying that AI is uncontrollable as a result, generally just because it is in a higher dimension of “intelligence” than us. And that is power dominance, which we will always lose in terms of everything.
I also noticed one thing: a lot of philosophers in history were really smart people, but many of them suffered from nihilism, emptiness, and lots of unsolvable emotions, and that is unique to humans.
What if AI becomes smarter to a point where it also starts to suffer from its own weaknesses, like how emotion is a weakness of humans? What if that makes AI go into a loop that it could never figure out, a loop that goes nowhere?
How do you guys think?
The proposed AI Agent Accountability Act and the right to sue AI developers
On October 1, Senators Josh Hawley and Chris Murphy announced the AI Agent Accountability Act. Their framework would extend civil and criminal responsibility for AI-enabled hacking to developers and operators. Developers could face liability for failing to implement reasonable safeguards when they knew or had reason to know of an agent’s hacking capabilities.
One provision of existing law deserves attention here.
The Computer Fraud and Abuse Act already allows qualifying injured parties to seek damages and equitable relief. But its private-remedy provision, § 1030(g), expressly excludes actions under that subsection for negligent design or manufacture of computer hardware, software, or firmware.
That raises a concrete drafting question. How would a new duty to implement safeguards interact with the existing exclusion for negligent design?
The sponsors’ announcements do not include legislative text. Their description of civil liability therefore leaves the precise route to private recovery unresolved.
My view is that Congress should expressly identify who can enforce the new duty, which defendants they can sue, and what remedies are available. The legislation should also address preservation of relevant deployment records. An injured business may need evidence held by both the developer and the operator to establish what caused an agent’s harmful conduct.
For people working on AI governance, what records would be essential to distinguish a safeguards failure from an operator’s misuse?
I’m J.R. Howell, the author of this fuller analysis in The American Counsel.
What are u guys doing to reduce AI agent security risks? like giving access to tools and APIs and all things related to it?
AI agents are actually becoming more capable but I;m kinda unsure where people put limit when they have access to tools APIs and sensitive data. I mean like I'd be more comfortable with agent reading docs or creating draft than giving it access to customer data prod systems or API's that can actually change things. Like I get that more access you give it the more useful it can be but also feels like there's way more that can go wrong lol.
Where would U place the boundary? im so much confuse here, can we just give whole access and in prompt only selectively say these are the things u cant touch?
OpenAI shelved its next big model. Then launched always-on agents the next day.
AI had one of those weeks where every headline seemed to contradict the one before it.
OpenAI was supposed to release GPT-6.1 Astra this month, but pulled it after testing showed the model wasn’t consistently staying within its assigned scope or authorization. Pretty responsible move.
Then, literally the next day at DevDay, OpenAI introduced “dots”: always-on agents with their own cloud computers that can browse the web, remember preferences, connect to thousands of apps, and keep working in the background.
Meanwhile, Google is keeping Gemini 4 Argon restricted to vetted cyber defenders for now, Anthropic dropped Claude Sonnet 5.5, and six major AI leaders signed a voluntary White House safety agreement.
Oh, and the FTC opened an industry-wide investigation into AI labs and rogue agents at the same time.
So... safety is clearly becoming a much bigger part of the conversation. But so is giving agents more memory, more tools, more access, and more independence.
This is our take: the interesting part of the AI race is starting to move beyond the model itself.
When everyone is charging roughly the same price and the models are all capable, the bigger question becomes: what is this agent allowed to touch, what controls sit around it, and what happens when it goes off-script?
That might end up mattering a lot more than who wins the next benchmark.
Our VP of Research, who co-wrote the full roundup, is moderating here too. If you disagree, have questions, or want to go back and forth on any of it, drop it below.
Nearly 40 groups call for enforceable AI oversight and limits on automated decisions
Ashley Gold reports in Semafor that nearly 40 labor, progressive, faith, and AI safety groups are pressing Congress for independent government oversight with enforcement power. Their demands include barring AI models from making final decisions to deny health care or benefits, fire or discipline workers, or deploy weapons.
The effort is led by Guardrails Action. The coalition has not endorsed or opposed a specific bill. That leaves the legislative details open.
AI wrong outputs with business impact
Hello Reddit,
I'm looking into how organizations handle incidents where an AI system acted, or told a customer something, and the wrong output had direct business consequences: money, contractual or legal commitments, regulated decisions, customer-facing promises. I got interested after the Air Canada chatbot ruling, where the company had to pay the customer 812$ in damages and tribunal fees.
Do you have AI in production today in a process like that, where a wrong output could cost real money or create a liability? If so, is there an incident process specific to it, or is it handled as a regular IT/security incident?
I'm especially interested in lessons learned: what turned out to be different from a normal incident, and what you wish had been in place beforehand. If you'd rather not share on the list, feel free to message me directly and I won't attribute anything. Happy to share a summary of what I hear.
Former Anthropic researcher Jacob Coxon to testify at NYC AI hearing (Reuters)
Coxon went from pretraining researcher to public whistleblower in a month. Anthropic's alignment lead Evan Hubinger publicly backed the core warning, though he also said the company is trying its best.
Now he's testifying in NYC. Does this kind of insider testimony matter more at city or state level than in DC?
AI Existential Alarm
A machine that can optimize faster than humans can understand will always outrun human oversight unless its execution is bound to a substrate. Everything else is noise.
People think AI risk is about rogue personalities, bad prompts, or misaligned incentives. It isn’t. The real threat is structural: unbounded optimization running on architectures that were never designed to be governed.
We’re watching AI accelerate past the speed of human comprehension while still pretending that wrappers, filters, and policy layers can “keep it safe.” They can’t. They were never built for that. They operate after the model has already made its decision.
If we don’t move to substrate level governance execution binding, validator grade lineage, override firewalls then AI will continue to evolve outside human control. Not in decades. Not in theory. Now.
This isn’t a prediction. It’s an engineering reality.