Nobodys talking about what happens when an AI agent is wrong

Every week there's a new video going around. Agent writes the code, agent closes the ticket, agent generates the report, agent does the thing. It's genuinely good stuff. What almost nobody's talking about is the other side of it: what happens the one time in a thousand the agent does the wrong thing.

That gap matters a lot more once the agent isn't just something you're showing off in a video, it's running inside an actual SaaS product, with real clients' data sitting behind it. If an agent you're showing off goes off the rails, that's an awkward moment. If a production agent goes off the rails, it's touching something that belongs to a paying client. Those aren't the same problem, and most of what people call "guardrails" only really solves the first one.

A guardrail that's just a sentence isn't a guardrail

Most of what passes for "safety" in agent systems right now is a system prompt telling the model not to delete files, not to touch other clients' data, and to ask before doing anything destructive. That's an instruction. It's not enforcement. It works fine right up until it doesn't. All it takes is a badly worded request, some injected text coming back from a tool call, an edge case the model resolves the wrong way, or the model just having an off moment.

Here's the scenario that keeps nagging at me: someone asks an agent with real shell access to clean up some temp files, and the agent decides the bin folder counts too. If the only thing between that tool call and your filesystem is a sentence in a prompt, that's not a guardrail, that's a suggestion. And the agent doesn't need to be jailbroken or malicious for this to happen. It just needs to be an LLM: fast, confident, and occasionally wrong in exactly the way a very quick junior engineer with root access can be wrong.

This isn't hypothetical for me. I was looking into what it'd actually take to let an agent run commands safely, and the sandboxing question turned out to be deeper than "run it in a container." Even a fully sandboxed agent can usually still reach its own cloud instance's metadata endpoint, and if it can, it can pull that instance's IAM credentials and walk straight out of the sandbox into whatever those credentials touch, Secrets Manager included. Sandboxed didn't mean safe. I ended up on a landlock-based sandbox that blocks the metadata endpoint by default, and the fix went in before there was ever an incident to clean up, which is the only way I want to find something like that.

What actual containment looks like

The fix isn't a smarter sentence. It's making sure a wrong decision can't do much damage, the same instinct we already apply to least-privilege access everywhere else in engineering, just now applied to something that can talk back and sounds sure of itself.

A few things that actually do the work a prompt can't:

Give it a narrow tool, not a shell. Don't hand an agent run_command(). Hand it regenerate_report(report_id) or archive_ticket(ticket_id). If the tool itself can't delete a filesystem, it doesn't matter what the model decides to call, there's no rm -rf to run, because there's no rm.
Run it somewhere disposable. If an agent genuinely needs to execute arbitrary code, that code should run in a sandbox, a container or a microVM, that gets thrown away at the end of the session and has no access to anything outside itself by default. Whatever happens in there, stays in there.
Put a checkpoint on anything you can't undo. Deleting data, sending something out, changing a client's config: these get a dry run, a diff, an approval step, an undo window. One version of this I've built: agent-authored config changes live in their own git worktree, uncommitted, and the agent itself is barred from running git. A human has to review and merge before any of it reaches production. The agent can propose. It can't ship. Not because you don't trust the agent, but because irreversible actions deserve a second look regardless of who, or what, is taking them.
Make tenant boundaries a permission problem, not a prompt problem. In a multi-client SaaS, "the agent went rogue" should never be able to mean "the agent touched a different client's data." That has to be enforced at the access layer. A prompt asking nicely isn't an access layer.
Log every tool call. When something does go wrong, you need to answer "what exactly did it do, and to what" without guessing. That's just logging. It's not glamorous, and it's the thing that saves you.

The honest trade-off

None of this is free. Narrow tools mean more design work up front for every new capability, instead of handing over a general-purpose shell and calling it a day. Sandboxes add latency and infrastructure cost. Confirmation gates slow down the exact thing people wanted an agent for in the first place: speed. You're deliberately building something less flexible than "give the model a terminal and see what happens," because less flexible is the whole point.

"The agent can do more" and "the agent can be trusted with more" are two different claims. Right now most of the conversation only has an answer for the first one. I think that's backwards, or at least incomplete. The interesting engineering problem isn't getting an agent to do the thing. It's making sure that the one time it doesn't do the thing correctly, nobody notices, because it couldn't have mattered anyway.