# Cloud incident response: process for AWS, GCP and Azure

URL: https://rootly.com/incident-response/cloud-incident-response
Pillar: Incident Response
Bundle: https://rootly.com/llms/incident-response.txt

> Cloud incident response for AWS, GCP and Azure: how shared responsibility changes the process, a step table, a provider-outage playbook and a checklist.

**Cloud incident response is the process engineering teams use to detect, declare, contain and recover from incidents in workloads running on AWS, Google Cloud or Azure. It follows the standard incident lifecycle, plus two cloud-specific jobs: working out whether the fault sits in your configuration or the provider's infrastructure, and keeping customers served when the provider itself is down.**

## Key takeaways

- Shared responsibility decides who fixes the fault. It never decides who owns the customer impact: that is always you.
- Check the provider's account-specific health view first. Public status pages often lag and rarely show single-account or single-zone problems.
- Write the "provider is down" playbook before you need it, including where your status page and incident tooling run.
- Run a retrospective after provider outages too. The follow-up work is about your dependency on the provider, and that is in your control.

## How cloud incidents differ from on-premises incidents

Three things change when the infrastructure belongs to someone else.

**Shared responsibility splits the fault domain.** Each major provider publishes a model of who secures and operates what. AWS describes its side as security "of" the cloud and yours as security "in" the cloud in its [shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/). Google Cloud calls its version [shared responsibility and shared fate](https://cloud.google.com/architecture/framework/security/shared-responsibility-shared-fate), and Microsoft documents how the split shifts across IaaS, PaaS and SaaS in [shared responsibility in the cloud](https://learn.microsoft.com/en-us/azure/security/fundamentals/shared-responsibility). The more managed the service, the more of the stack the provider owns.

**Provider outages and your own outages need different responses.** A bad deploy, an expired certificate or a too-tight IAM policy is yours to roll back. A regional control-plane failure is the provider's to fix, and your job becomes failover, degradation and communication. Most of the first 15 minutes of a cloud incident goes into telling these apart.

**Your visibility stops at the provider's boundary.** You can't read the provider's logs or page its engineers. You rely on its health dashboards, support cases and status updates, which arrive on its schedule.

| | Your own fault | Cloud provider fault |
| --- | --- | --- |
| Typical causes | Deploys, config drift, IAM or network rule changes, quota exhaustion, expired credentials | Regional or zonal service failure, control-plane outage, degraded managed service |
| Who fixes it | Your team | The provider |
| What your team does | Roll back, fix forward, scale | Fail over, degrade gracefully, queue work, communicate |
| Where evidence lives | Your logs, metrics, traces and change history | Provider health dashboards, support cases, your error rates by region |
| Retrospective focus | The change and the safeguards that missed it | Your dependency design: single-region risk, failover tested or not, alerting on provider health |

## Cloud incident response process: step by step

| Step | What happens | Cloud-specific checks |
| --- | --- | --- |
| 1. Detect | An alert fires from monitoring, a provider health event or a customer report. | Route provider health events to the same on-call path as your own alarms, so a provider issue pages someone. |
| 2. Declare | Open the incident, set severity, assign an incident commander and open the incident channel. | Set severity by your customer impact, not by the provider's label for the event. |
| 3. Scope the blast radius | Identify affected services, regions, zones, accounts and customers. | Compare error rates across regions and accounts. One region failing points at the provider; every region failing after a deploy points at you. |
| 4. Mitigate | Stop or reduce customer impact. | Roll back your change, or fail over, shed load, disable non-critical features and pause batch jobs if the provider is at fault. |
| 5. Communicate | Update internal stakeholders and your status page on a fixed cadence. | Say which provider and service are involved, what customers will see and when the next update comes. Link the provider's status entry internally. |
| 6. Recover | Restore normal operation and verify it. | Fail back deliberately, drain queues, reconcile data written during degraded mode and watch for a second wave as the provider recovers. |
| 7. Retrospective | Review the timeline, decisions and follow-up work. | Ask what would have reduced impact: multi-region, cached fallbacks, earlier provider-health alerts, a tested failover runbook. |

The [incident response lifecycle](https://rootly.com/incident-response/lifecycle-process) covers each stage in more depth. The steps above are the same lifecycle with the cloud checks added.

## When your cloud provider is down: a playbook

Use this when the evidence points at AWS, Google Cloud or Azure rather than your own change.

1. **Confirm on the provider's account-specific view.** Public pages are the [AWS Health Dashboard](https://health.aws.amazon.com/health/status), [Google Cloud Service Health](https://status.cloud.google.com/) and [Azure status](https://azure.status.microsoft/en-us/status). Events that only affect your account or project show up in [AWS Health](https://docs.aws.amazon.com/health/latest/ug/what-is-aws-health.html), Google Cloud [Personalized Service Health](https://cloud.google.com/service-health/docs/overview) and [Azure Service Health](https://learn.microsoft.com/en-us/azure/service-health/overview) first.
2. **Declare your own incident.** The provider's incident covers its infrastructure. Yours covers your customers, your decisions and your follow-up work.
3. **Open a support case.** Include resource IDs, regions and timestamps. A case gives you a channel for account-specific details that a public status page won't carry.
4. **Freeze changes in the affected area.** Deploys during a provider outage make it harder to tell which errors are whose.
5. **Fail over or degrade.** Move traffic to a healthy region if your architecture supports it. If it doesn't, turn off features that depend on the failing service and keep the rest of the product running.
6. **Post a status update.** Name the provider dependency in plain terms and give a time for the next update. Keep updating on that cadence even when nothing has changed.
7. **Track the provider's updates in your incident timeline.** Paste each provider status change with its timestamp, so the retrospective has a full picture.
8. **Recover in order.** When the provider reports recovery, verify with your own metrics before failing back. Drain backlogs gradually to avoid a self-inflicted second incident.
9. **Hold a retrospective.** Focus on the dependency: what single points of failure the outage exposed and which follow-up items reduce impact next time.

One thing to settle in advance: where your incident tooling and status page run. If they sit in the same region as the outage, you lose them at the moment you need them.

## Cloud incident response checklist

**Before an incident**

- [ ] Provider health events (AWS Health, Google Cloud Personalized Service Health, Azure Service Health alerts) route to your on-call path.
- [ ] Every production account, project and subscription is mapped to an owning team and service.
- [ ] Failover runbooks exist for your critical services and were tested in the last quarter.
- [ ] You know where your status page and incident tooling are hosted and whether they share a failure domain with production.
- [ ] Support plan level and case-opening steps are documented for each provider.

**During an incident**

- [ ] Incident declared, severity set by customer impact, incident commander assigned.
- [ ] Blast radius scoped by region, zone, account and service.
- [ ] Own-change versus provider-fault question answered and recorded.
- [ ] Changes frozen in the affected area.
- [ ] Support case opened if the provider is involved; case ID in the incident record.
- [ ] Status page updated on a fixed cadence.

**After an incident**

- [ ] Recovery verified with your own metrics before fail-back.
- [ ] Data written in degraded mode reconciled.
- [ ] Retrospective held, with provider timeline included.
- [ ] Follow-up items have owners and due dates.

## How teams do this in Rootly

Rootly connects to the cloud signals above and runs the incident from Slack or Microsoft Teams. Each item here is documented on [docs.rootly.com](https://docs.rootly.com/) and listed on [Rootly integrations](https://rootly.com/integrations).

- **AWS alarms and events.** The [AWS CloudWatch alert source](https://docs.rootly.com/integrations/aws-cloudwatch) receives alarm state changes through an SNS topic. Rootly opens an alert when an alarm enters `ALARM` and resolves it when the alarm returns to `OK`. For other AWS events, Rootly's [alert sources guide](https://docs.rootly.com/alerts/alert-sources) says to route EventBridge events to an SNS topic subscribed to the [AWS SNS source](https://docs.rootly.com/integrations/aws-sns), which is one way to bring AWS Health events onto the same paging path.
- **Google Cloud alerts.** [Google Cloud Monitoring](https://docs.rootly.com/integrations/google-cloud-monitoring) sends alerts to Rootly through a webhook notification channel. Rootly provides a non-paging endpoint for visibility and a paging endpoint that pages a user, group, escalation policy or service.
- **Azure alerts.** Azure is an alert source type under **Alerts → Sources** for routing and paging, as described in the [alert sources guide](https://docs.rootly.com/alerts/alert-sources).
- **Kubernetes context.** The [Kubernetes integration](https://docs.rootly.com/integrations/kubernetes) uses kubewatch to send pod, deployment, service and node events to Rootly, where they appear as pulses on the incident timeline.
- **Routing to the right team.** [Alert routing](https://docs.rootly.com/alerts/alert-routing) sends alerts to teams, services and escalation policies by conditions you define, so a CloudWatch alarm on a payments account pages the payments on-call.
- **AI investigation across your cloud.** Rootly AI SRE can read cloud evidence through read-only connectors for [AWS](https://docs.rootly.com/ai/connectors/aws) (an IAM role with service and region scoping, plus optional EKS support), [Google Cloud](https://docs.rootly.com/ai/connectors/google-cloud) and [Azure Monitor](https://docs.rootly.com/ai/connectors/azure-monitor). You choose the scope each connector can see.
- **Status updates.** [Publish incidents to Rootly status pages](https://docs.rootly.com/configuration/publishing-incidents) from the web, Slack or @Rootly.
- **Retrospectives.** [Retrospectives](https://docs.rootly.com/retrospectives/retrospectives) draft from the incident timeline, so provider status changes you logged during the incident appear in the review.

For the coordination side of these incidents, see [incident response communication](https://rootly.com/incident-response/communication). If a cloud incident turns out to involve unauthorized access, move it to your [security incident management](https://rootly.com/incident-response/security-incident-management) process.

## Frequently asked questions

### What is cloud incident response?

Cloud incident response is the process for detecting, coordinating and recovering from incidents in workloads that run on AWS, Google Cloud, Azure or another provider. It follows the usual incident lifecycle, with two additions: deciding early whether the fault sits in your configuration or in the provider's infrastructure, and planning for the case where the provider itself is down.

### How do I know if an outage is my fault or my cloud provider's?

Check three things in the first few minutes. Look at the provider's account-specific health view (AWS Health, Google Cloud Personalized Service Health or Azure Service Health), not only the public status page. Compare error rates across regions, zones and accounts. Then check your own change log for deploys, config changes and expiring credentials in the window before impact began.

### Should I declare an incident when AWS, GCP or Azure has an outage?

Yes, when the outage affects your customers or your ability to operate. The provider runs its own incident, but your customers, status page, failover decisions and follow-up work are yours. Declare it, set severity by your own customer impact, and link the provider's status entry in the incident record.

### What does shared responsibility mean for incident response?

The provider is responsible for the infrastructure that runs its services, and you are responsible for what you build and configure on top of it: identity and access, network rules, data, application code and, depending on the service, the operating system. Incidents in your half are yours to fix. Incidents in the provider's half are yours to mitigate and communicate.

### Which Rootly integrations help with cloud incidents?

Rootly has alert sources for AWS CloudWatch, AWS SNS, Google Cloud Monitoring and Azure, a Kubernetes integration that adds cluster events to the incident timeline, and read-only AI connectors for AWS, Google Cloud and Azure Monitor that Rootly AI SRE uses during investigations. Each is documented on docs.rootly.com.