Some problems earn your full attention the first time. An S3 bucket goes public and you want to know what happened: who created it, whether it was on purpose, what else that person has been touching. You learn something. By the tenth time it happens you've stopped learning anything, and by the time it wakes you at 2 a.m. you're just copying commands out of a wiki page and hoping the page is still accurate.
That's the situation this is meant to kill off: where a remediation is repeatable and safe to automate, we codify it as an AWS Systems Manager (SSM) Automation runbook so the response is consistent and fast.
It reads like a line lifted from a governance doc, and it is one. But the reasoning under it is ordinary. Ops teams rot in a specific way, and codified runbooks are one of the few things that slow the rot down.
Wiki runbooks don't age well
Most teams already have runbooks. They're just written as prose and parked in Confluence or a shared doc, and they say things like "confirm the alert, log into the account, run this command, check the output."
The problem with a page like that is it drifts out of sync with reality and nobody notices. Someone changes the process and forgets to update the doc. A flag gets deprecated. The console flow the screenshots describe stopped existing two releases ago. None of that is visible to the person following the page at 3 a.m., so they do exactly what it says and occasionally make things worse.
And even when the page is right, it leans on the operator being sharp. Skip a step because you're tired, paste the wrong resource ID, run the fix against staging when you meant prod because you had two tabs open. The runbook was fine. The hands weren't.
A prose runbook also can't do anything by itself. It sits there until a human notices the alert, digs up the page, and starts typing, which means your time-to-fix is capped by how fast someone can wake up.
Turn the runbook into code and those problems mostly go away, or at least turn into more manageable ones. It's version-controlled, so drift shows up in a diff instead of hiding. Execution is deterministic, so a tired operator can't fat-finger it. And you can fire it automatically, so the fix can land before anyone's read the alert.
What these things are, briefly
An SSM Automation runbook is a document, YAML or JSON, describing a series of steps AWS runs on your behalf. A step calls an API, runs a script, waits on a condition, or branches on the output of an earlier step. AWS ships a large library of ready-made ones (the AWS-* documents), and you write your own for anything specific to your environment.
Your own ones will mostly be short. Here's a fix that comes up constantly: switch public access back off on a bucket that shouldn't have had it.
schemaVersion: '0.3'
description: Enforce S3 Block Public Access on a target bucket
assumeRole: '{{ AutomationAssumeRole }}'
parameters:
BucketName:
type: String
description: The bucket to remediate
AutomationAssumeRole:
type: String
description: IAM role the automation runs under
mainSteps:
- name: BlockPublicAccess
action: aws:executeAwsApi
inputs:
Service: s3control
Api: PutPublicAccessBlock
PublicAccessBlockConfiguration:
BlockPublicAcls: true
IgnorePublicAcls: true
BlockPublicPolicy: true
RestrictPublicBuckets: true
That's the whole thing. No console, no trying to recall which of the four toggles you actually need. It behaves the same on the first bucket and the four-hundredth.
Nothing about it is clever, which is the point. You want your remediations boring. The thinking goes into deciding what's worth codifying, not into the runbook itself.
"Repeatable" and "safe" are carrying the weight
This is where people get themselves in trouble. Automation feels good, so the urge is to automate everything, and those two words in the policy line exist to stop you. Both have to hold.
Repeatable means the fix is the same every time: same trigger, same steps, same result. If the right response needs judgment ("depends whether this box is in the payment path"), it isn't repeatable, and forcing it into a runbook just hard-codes a guess. Sometimes a fix only becomes repeatable after you've handled it enough times to see the variations. That's fine. Codify it then, not before.
Safe to automate is really a question about blast radius. If this fires at the worst possible moment against the wrong resource, what breaks? Re-enabling bucket encryption is safe; worst case nothing changes. Automatically terminating an instance because something flagged it as compromised is not, because the flag might be wrong and you might have just wiped the state you needed for forensics. Same action, repeatable enough, but not something you let run unattended.
The stuff that clears both bars tends to be security and compliance drift. A security group opened to the whole internet. An unencrypted volume. A log stream someone switched off. An IAM policy granting more than it should. In those cases the correct state isn't up for debate, and getting back to it quickly matters more than mulling it over.
Something that clears only one bar doesn't get binned. It gets the version with a human in the loop, which I'll come back to.
What sets it off
A runbook a person still has to launch by hand beats a wiki page, if only because it runs the same way every time. The real payoff is wiring it to something that watches the environment for you.
The usual chain: AWS Config, Security Hub, or GuardDuty spots the problem; an EventBridge rule matches the finding; EventBridge starts the runbook with the offending resource handed in as a parameter. Bucket goes public, bucket comes back private a few seconds later, no human involved, and there's a full execution log sitting there for whoever wants to look in the morning.
Config also has native remediation that points straight at an SSM document, so for a lot of compliance rules you can skip the plumbing entirely: attach the runbook to the rule and let Config drive.
Keeping it safe once it runs itself
Handing a machine permission to change your infrastructure should make you a little uneasy. Good. Put that unease into guardrails rather than into avoiding the whole idea.
The ones that earn their keep:
Scope the role. The runbook runs as an IAM role, so give that role permission to do the one remediation and nothing else. The S3 fix does not need delete rights on your buckets. Whatever that role can do is the ceiling on how bad a bad day gets.
Add an approval step when you're not sure. SSM has an aws:approve action that pauses until a named person clicks approve. That's your release valve for fixes that are repeatable but sitting right on the edge of safe. The machine does the tedious assembly; a human makes the go/no-go call.
Make it announce itself. Every run should surface somewhere a person will see it, whether that's Slack, a ticket, or an SNS topic. Automation that runs silently is how a runbook quietly does the wrong thing for three weeks before anyone catches it.
Test what it does when it fails, not just when it works. What happens if the resource is already gone? If the API throttles? If it dies halfway through? A remediation that leaves things worse than it found them is worse than not having one.
The habit is worth more than the count
It's easy to start counting runbooks like they're the deliverable. They aren't. The thing you're building is the reflex, that moment when a fix shows up for the third time and someone says "that one's repeatable, let's codify it" instead of bolting another paragraph onto a page nobody reads.
That's what shifts on-call. The repetitive, brainless work slowly disappears into runbooks, and what's left for a person is the stuff that needs a person. Your better engineers stop spending nights retyping the same six commands.
There's a cost, and I'd rather name it than pretend it away. Runbooks are code, and code rots. One written against an API that AWS later changed will either fail or, worse, quietly succeed at doing the wrong thing. So they need owners, reviews, and the occasional cull, same as anything else you ship. A remediation that's drifted out of sync with reality is a hazard wearing a safety net's clothes.
Where to start
Don't start with the scariest incident you can picture. Start with the most annoying one, the small dumb fix that keeps reappearing and never teaches you anything. It'll be a short runbook, it'll clear both bars without an argument, and it'll pay for itself almost right away. Then do the next one.
And to be clear, none of this is about getting people out of the loop. It's just that nobody should be spending their night being a slow, error-prone version of a script. Keep the human attention for the incidents that need it.
