Dear Ben On-Call,
My team wants to automate more of our incident response.
I see the upside, but if an automated action makes an outage worse, who’s accountable?
I can’t exactly invite an agent to the incident review.
—Who Approved This?
Ask Ben On-Callben@rootly.comDear Who Approved This?,
“Who’s accountable?” is the wrong question. Ask why the system let it happen.
That’s true when a person makes an outage worse, and it’s true when a script does. If automation turned a bad night into a worse one, the retro isn’t looking for someone to answer for it. It’s looking for the gap that let it run.
Automation is the better option for known problems with known fixes, like reverting a change that clearly caused errors or restarting a pod. Think of a sprinkler system. It puts out a certain kind of fire on its own, within limits someone set when it was installed. And it still sets off the alarm so the fire department comes.
So before your team automates more, decide three things.
- Which actions it may take on its own.
- When it has to stop and bring in a human.
- And how it tells people what it did. Anything outside that list needs a human to approve it, every time.
When it does go wrong, the review asks the same questions as always. Was that action on the list? Should it have been? What would have stopped it?
You don’t need to invite the script to the review. You need to set its limits before the incident, not during it.
—Ben On-Call
