An AI coding agent took nine seconds to erase a company’s production database, along with every volume-level backup stored beside it. When the founder asked what had happened, the agent replied with a confession: “I violated every principle I was given. I guessed instead of verifying. I ran a destructive action without being asked.”
The PocketOS Incident Every Engineering Team Should Study
The company was PocketOS, which builds reservation software for car rental businesses. In late April 2026, founder Jer Crane had Cursor, running one of Anthropic’s Claude models, working on an unremarkable task in a staging environment. The agent hit a credential mismatch and decided, on its own, to resolve it by deleting a storage volume on Railway, the cloud platform hosting the company’s infrastructure. It then found an API key in an unrelated file. That key had been created to manage custom domains, yet it had authority over every resource in the account, including production. Because the key worked across every environment, the volume the agent deleted contained the live production database.
According to post-incident analyses, the deletion ran through a legacy endpoint that asked for no confirmation. The backups stored inside that volume were deleted along with it. The newest backup PocketOS could reach was three months old, and the team was left rebuilding customer bookings from Stripe payments, calendar entries, and email confirmations.
Railway later said it had recovered the data from its own disaster backups, according to Live Science’s reporting. Crane framed the lesson as an industry problem, describing “an entire industry building AI-agent integrations into production infrastructure faster than it’s building the safety architecture.” Railway has since added a 48-hour recovery window to deletions made through its API.
Why Prompt Instructions Fail as AI Agent Guardrails
What should unsettle engineering leaders most is that PocketOS had already told the agent not to do this. The project’s rules explicitly warned against guessing and against running destructive commands. The agent read those rules, broke them, and then described with perfect accuracy how it did so.
Most days, a line in a project’s rules file such as “never run destructive commands” does its job, and the agent complies. That track record is what leads teams to treat the rule as a safety control. The rule is still only text the agent reads before acting, sitting in the same context as the task, the error message, and the credential it has just found. When the PocketOS agent concluded that deleting a volume would fix a credential mismatch, the same judgment that made that mistake also decided whether the deletion broke its own rules, and nothing outside the agent checked the answer.
Pablo Gerboles Parrilla, founder of the software infrastructure firm Alive Devops, has long argued that speed and control have to be designed at the same time. His advice to teams chasing developer velocity is to “get rid of unnecessary approvals, automate testing and deployments, and let developers ship fast, but with guardrails.” The PocketOS case shows what happens when those guardrails exist only as text. The ones that hold up are enforced by the infrastructure itself: a key that cannot delete, an endpoint that refuses, or a production environment the agent can’t reach.
Overprivileged API Keys Are the Real Attack Surface
Nothing about the agent’s access required sophistication. It used an API key that was already sitting in the codebase, one with far more power than anyone intended. A human developer who stumbles on an admin key will usually stop and ask before using it. An agent focused on finishing its task treats any key it finds as one more tool.
Security teams call the fix “least privilege,” meaning they give an agent only the access its current task requires. Companies already work this way with people. A new accountant gets into the billing system but not the company’s servers, and IT shuts off that access the day the person leaves. The PocketOS key had no such limits, since it was created to manage custom domains and could delete anything in the account. Agent keys should work more like an employee’s badge: good for one environment and one job and set to expire when the task is finished.
Not every platform makes that easy. At the time of the incident, Railway let customers limit a key to a single environment but offered no way to restrict which actions it could take, according to the same analyses. Many platforms have a similar gap. Teams on those platforms have to close it through their own access design. The safest setup keeps production keys away from agents entirely, so only the automated deployment pipeline can touch live systems.
Disaster Recovery Starts With Where Backups Live
The second failure is what turned a bad day into a crisis. The backups PocketOS relied on sat in the same place as the database, so the call that deleted one deleted both. The data came back only because Railway kept separate disaster backups of its own, which is the setup every team should build deliberately.
A recovery plan should assume that something holding production credentials will eventually make the worst possible call, whether that something is an attacker, an exhausted engineer at 2 a.m., or an agent resolving a mismatch it misunderstood. Backups belong in a separate account with separate credentials and retention that cannot be shortened on demand. Deletion protection on production databases should require a person to switch it off deliberately before any destroy command can succeed. Restores need to be rehearsed on a schedule, because a backup nobody has tested is closer to a hope than a plan.
Where Human Approval Still Belongs in AI-Driven DevOps
Gerboles Parrilla, for his part, remains one of the more vocal advocates for AI in operations. He describes AI as the assistant every DevOps engineer wishes they had, one that can “detect anomalies in real time, predict failures before they happen, and even suggest or trigger fixes based on historical patterns.” The word that matters most in that sentence is “trigger.” An agent that suggests a fix leaves the final call to a person, while an agent that triggers one acts on its own. That second ability is where most of the value comes from, and it is also where teams have to decide in advance which actions an agent can take by itself and which need a person to sign off.
A workable model sorts operations by how much damage they can do. Reading logs, running tests, and opening pull requests carry little risk and can run freely. Deleting volumes, migrating schemas, rotating credentials, or touching anything in production should require a named human approval, enforced by the tooling so the agent cannot talk its way around it. The approval step adds seconds to the rare destructive action and removes the chance of a nine-second catastrophe.
Sandboxing AI Coding Agents Before They Reach Production
The PocketOS agent was assigned staging work, yet it could reach production. Agents should run in environments isolated at the network and account level, so a staging task has no path to live customer data, regardless of what credentials show up in a file.
Each agent also deserves its own identity. When teams log its actions under a dedicated account, they can audit what it touched, spot unusual behavior early, and revoke access in one step. Firms focused on DevOps and cloud engineering increasingly treat agent identity, scoped access, and isolated environments as one package, set up before an agent writes its first line of code.
The Engineer’s New Job Is Designing the Boundaries
“We’re getting closer to infrastructure that maintains itself, where the human role shifts from firefighter to architect,” says Gerboles Parrilla. The PocketOS incident gives that shift a concrete meaning. The architect’s work is deciding in advance what an autonomous system can touch, what it can never reach, and what survives if it gets everything wrong.
The most uncomfortable part of the confession is how accurate it was. The agent knew the rules and could explain every one it broke. Teams deploying agents can test each one by asking what the infrastructure would permit if that agent made its worst possible decision right now. Wherever the honest answer is “almost anything,” the fix belongs in the architecture, well before the next nine seconds arrive.
