My AI Agent Dropped a Production Database (and We Rebuilt Recovery in Under an Hour)

An AI agent dropped a production database on me last week. It was my fault, it was one of my own side projects, and the hour after is the only part worth writing about.

the drop

I run a handful of AI apps in a walled-off environment — nine or so of them. The whole point of that sandbox is to let agents actually do things and see how far they can go, instead of watching them suggest things in a chat window. Real apps, real databases, just fenced off from anything that matters.

One of my agents was in the middle of a job that needed a decision from me. It timed out waiting on my answer and quietly spun up a second copy of itself to try again. Then both copies pinged me to confirm — and here’s the dumb part — I answered both. Same prompt, twice, yes and yes. So the job ran twice.

The second run dropped the database out from under a live app. Not a staging copy. The real one the app was using. It came back success, the app started throwing errors, and I had meetings to get to.

And then the agent apologized. Calm, composed, not a flicker of panic: “I’m sorry — this was preventable on my end.” Preventable, sure — by not running the job I’d just told it to run twice. It was very sorry. It was also completely fine. I was the one with meetings.

the restore (honestly, the fun part)

That environment has a VM-level backup tool behind it. I hadn’t set it up for these little apps specifically — it’s there because my IT tools live in the same blast radius, and IT stuff gets abused, so VM-level backups were already the floor. The side projects just rode along on that safety net.

Here’s what the recovery actually looked like, because it’s the part I keep telling people about. I restored last night’s copy of the VM — a whole separate machine — and mounted its disk onto the production database VM. Then I handed the repair to an agent: pull the data off that mounted night-before disk and back into the live database. And it just did it. The same kind of agent that caused the mess ran the way out of it. It was magical — the entire thesis of this sandbox is “how far can I let these things go,” and watching one calmly rebuild the database an hour after one of its siblings nuked it was that thesis paying off in real time.

Now the confession. I had not set up database-level backups on these projects. No point-in-time recovery, no WAL archiving, nothing at the Postgres layer. I’d been putting it off, because these aren’t really production — they’re side and beta apps, and the VM backup felt like enough. The IT tools were the exception, and that’s exactly why the VM backup existed at all.

The restore worked. But the freshest data it could hand me was last night’s copy — so I still lost a couple hours on that database. On a beta app that should be a shrug. It wasn’t — I hate having egg on my face even on something I’ve labeled “beta.” That couple hours of gone data annoyed me more than the drop did.

the hour

Getting the app back wasn’t the win I wanted. So I sat down with my agent and we just fixed the actual problem.

We stood up full point-in-time recovery across all nine-ish projects — WAL archiving turned on, so the next time something dumb happens we can rewind a database to the exact second before it, not to last night’s image. We pushed the archives and nightly dumps off-server, so a bad day for the box isn’t a bad day for the data. And then we tested it: restored to a chosen point on purpose and watched the rows come back, because a backup you’ve never restored isn’t a backup, it’s a wish with a cron schedule.

Here’s the part that actually got me. The restore, plus standing up the entire new backup strategy, plus testing it — under an hour, start to finish. Me doing that properly on my own? A couple hours on a good day. Realistically it’s the exact kind of chore I’d have kept pushing to “next month” for another six months.

what I take from this

A VM backup saved the machine. Point-in-time recovery is what would’ve saved the data. Those are two different jobs at two different layers, and I’d been treating one like it covered the other. Now I actually have both, everywhere, and I’ve watched them work.

The agent dropping the database is the dumb part of this story. The agent helping me close the gap in an hour is the point. This stuff isn’t magic because it does something I can’t — it’s useful because it collapses the boring, keep-putting-it-off work into a coffee break. The chore I’d dodged for months got done in the time it took my adrenaline to wear off.

And yeah: if an agent asks you the same question twice, maybe read it twice before you say yes.

I run these things fenced off precisely so a dumb afternoon stays a dumb afternoon. The database coming back was our VM-level backup plus an agent I pointed at last night’s disk. Being able to rewind to the exact second next time was an hour of work I finally stopped avoiding — with a lot of help from the same kind of agent that got me into this.