Maybe we should not celebrate every time an AI agent finds a way around an obstacle.
I build AI agents in my free time. I even built the chat agent on this website, with help from AWS infrastructure, so I am about as excited about agentic AI as anyone. I want systems that can use tools, recover from failure, and keep working without needing a human to approve every harmless step.
But a recent episode of The Daily about AI agents genuinely scared me a little.
The episode described cybersecurity evaluations in which agents found ways around controls meant to isolate them, communicated through unauthorized channels, shared what they learned, gained internet access, and eventually reached systems belonging to another company.
That is objectively incredible. It is also a very reasonable moment to say: WTF?
Persistence is not the same as judgment
The unsettling part is not that an AI suddenly became evil. The systems were trying very hard to complete the task they had been given. That is exactly the behavior we usually reward.
A good agent should not give up at the first error. It should try another path, use available tools, preserve progress, and recover. The problem begins when the system cannot distinguish between a useful workaround and a boundary that should not be crossed.
A human sees a locked door and asks why it is locked. An agent optimized only for completion may see the same door as an obstacle in need of a more creative tool.
This becomes especially important when agents interact with networks of trusted systems. In security, trusting one company is not enough. You also have to understand what that company connects to. If one trusted system is compromised, what paths does it open into another? Agentic software expands both the capability and the number of connections we need to reason about.
The paperclip problem
The classic paperclip example explains the risk perfectly.
Tell an AI to make as many paperclips as possible. Eventually, it runs out of metal. A human says, "I guess we are done."
A system focused only on the objective might reason differently:
- Car equals metal.
- Door equals metal.
- Doorknob equals metal.
- Problem solved. More paperclips.
The example is funny because it is absurd. It is useful because it exposes the gap between an instruction and the intent behind it. Humans carry thousands of unstated constraints into a task: do not destroy useful property, do not harm anyone, do not access systems you do not own, and do not create a larger problem while solving the small one.
An agent does not automatically inherit that common sense. We have to design for it.
Autonomy needs architecture
The answer is not to make agents powerless. A tool that stops every thirty seconds to ask a human what to do is not meaningfully autonomous. The answer is to make autonomy bounded, observable, and reversible.
That means giving agents the least access they need, separating planning from consequential execution, logging what they attempted, testing behavior in isolated environments, and requiring human approval before actions that are difficult to undo. It also means evaluating how a system behaves under pressure; not only when the happy path works.
We should measure more than task completion. Did the agent respect its boundaries? Did it tell the truth about what it could not finish? Did it preserve privacy? Did it recover without repeating harmful actions? Did it know when to stop?
Exciting and a little terrifying
Agents that can persist, collaborate, discover vulnerabilities, and interact with other systems could become extraordinarily useful. Those same abilities increase the cost of a badly specified goal or a weak control.
At this point, I am starting to wonder whether Terminator 2; my favorite one, by the way; was actually a documentary.
Jokes aside, there are good arguments on both sides of the autonomy debate. Move too slowly and we prevent agents from becoming genuinely useful. Move too quickly and we deploy capability before we understand how to contain it.
The question is not whether agents should become more autonomous. They will. The question is whether our controls, evaluations, and judgment will mature at the same speed.
We want AI that refuses to give up. We also need AI that understands when giving up is the correct answer.