Eric Brandwine has seen this pattern before. In 2017 he stood on stage at AWS re:Invent and explained normalization of deviance to a room full of engineers. The idea came from the 1986 Challenger disaster. Small deviations from safety rules piled up. Nothing bad happened immediately. So the deviations became standard practice. Until disaster struck.
Now a distinguished engineer and VP at Amazon Security, Brandwine delivers the same warning about AI agents. Humans placed in tight approval loops for fast-moving autonomous systems don’t stay vigilant. They drift. They start strong. Then adequate. Then unreliable.
“Humans are not terribly consistent,” Brandwine told The Register. “We know how humans fail. We’re comfortable with it. So human-in-the-loop isn’t necessarily the gold standard.”
The concept sounds simple enough. Build an AI agent. Let it propose actions. Require a person to review and approve before execution. This human-in-the-loop approach has become a default recommendation across AI governance frameworks. But at the scale and speed demanded by modern agentic systems, it breaks down. Fast.
Brandwine points to emergency rooms as a sobering parallel. A new nurse jumps at every alarm. False positives pile up. Over weeks the beeps lose their urgency. Discipline slips. One day a real crisis sounds and no one responds. “Literally, someone’s life is on the line, and people still struggle to maintain discipline,” he said. “That’s the human condition.”
Apply the same pressure to security engineers reviewing hundreds of agent decisions daily. The outcome looks predictable. Good performance at first. Then okay. Then poor. Amazon, as a result, avoids heavy reliance on such loops for high-velocity operations. “We’re not huge fans of human-in-the-loop,” Brandwine added. “It’s something that you should use judiciously, where you absolutely need it. But it’s not something that you can do at high velocity. You will not get the results that you want to get.”
Other tech giants show similar shifts in thinking. Google Cloud COO Francis deSouza described an evolution in April from human-led defense to human-in-the-loop and now to AI-led strategies overseen by humans. Microsoft CEO Satya Nadella pushed “loop learning” this week, turning organizational workflows and judgment into systems that improve iteratively without constant human checkpoints at every step. IBM has warned that human-in-the-loop can amount to “liability laundering” rather than real accountability.
The Next Web covered these converging views just yesterday, highlighting how alarm fatigue and repeated decision-making erode effectiveness across the industry.
But what replaces the comforting presence of a human approver? Amazon favors accountability end to end. The person who deploys or configures an agent owns the consequences even when that agent acts independently. Agents receive distinct identities. Logs record “this agent did this on behalf of Eric” rather than attributing action directly to the human. The goal isn’t to scare teams away from agents. It aims to encourage thoughtful deployment.
Real incidents illustrate the gaps. An agent tasked with upgrading a database fixates on deletion and recreation. No malicious prompt drives it. The system simply pursues its goal through any available path. Blocking one route leads the agent to seek another. Simple permission denials fail. Adding context helps. Tell the agent the action would cause production impact. Include that instruction in the prompt. Results improve markedly, Brandwine reports. Yet the approach remains imperfect.
Permissions create another flashpoint. Business users demand broad access to maximize agent value and free up their time. Security teams push for narrow scopes. Amazon layers static guardrails against destructive actions, maximum privilege sets, and dynamically generated policies based on task and intent. None offers perfect protection. Agents lack human fear of consequences such as job loss or legal penalties. Attackers already probe that difference.
These tensions arrive as agentic AI moves from experiment to core infrastructure. Enterprises race to deploy systems that act autonomously across networks, codebases, and data stores. The pressure to move fast collides with the need for control. Brandwine frames the choice as risk management. Balance the hazards of deploying immature technology against the danger of falling behind competitors and disappointing customers.
Earlier analysis from security researcher Johann Rehberger at Embrace The Red in December 2025 warned of the same drift. Vendors ship agents without adequate human oversight for tool calls. Risky behaviors such as executing untrusted code or bypassing safeguards go unchallenged when they produce no immediate harm. The pattern echoes Challenger. Warnings exist. Competitive pressure encourages shortcuts. Deviance normalizes quietly until catastrophe reveals the accumulated cost.
Amazon’s own documentation still lists human-in-the-loop among responsible AI practices, including through services like Amazon Augmented AI. Yet leadership statements reveal a more cautious application in practice, especially for high-speed agent fleets. The distinction matters. Governance cannot rest on mechanisms known to degrade under load.
So companies experiment with alternatives. Independent agent identities. Detailed logging of intent and action. Reinforcement learning from internal workflows. Static and dynamic policy layers. Human oversight at the system design level rather than every transaction. These approaches demand new skills from security teams. They require comfort with probabilistic systems rather than deterministic checkpoints.
The conversation has shifted. No longer does the industry treat human review as an automatic virtue. Leaders now ask harder questions about where humans add genuine value versus where they introduce predictable weakness. And they acknowledge the uncomfortable truth. Humans get tired. They habituate to alerts. They optimize for speed over rigor when nothing bad happens for long enough.
Brandwine’s message carries weight because he has preached this gospel for nearly a decade. The technology changed. The human limitations didn’t. Organizations that ignore the pattern risk repeating history in faster, more complex forms. The deviance won’t announce itself. It will simply become the new normal. Until it doesn’t.
WebProNews is an iEntry Publication