On-call as code: manage schedules with Terraform

On-call as code means defining on-call schedules, rotations and escalation policies in version-controlled files, usually Terraform, and applying them through a pipeline instead of editing them by hand. Every change to who gets paged becomes a reviewed pull request and an auditable commit, and terraform plan shows the effect before it reaches anyone’s phone.

This guide covers why engineering teams move on-call configuration into code, a worked example with the Rootly Terraform provider, a CI pipeline that plans on pull requests and applies on merge, and how to handle drift when someone changes a schedule in the UI.

Why teams manage on-call as code

A schedule edited in a web UI changes who gets woken up at 3 a.m. with no review and often no record anyone reads. On-call configuration is production configuration, and the same three problems show up that infrastructure as code solved for servers.

Problem	In the UI	In code
Review	One person changes a rotation or escalation delay and saves it.	A pull request shows the diff and the terraform plan output, and a teammate approves it.
Audit	You reconstruct who changed what from activity logs, if they exist.	Git history records the author, the reviewer, the reason and the previous state of every change.
Drift	Similar teams end up with different delays, gaps and fallbacks depending on who set them up.	A shared module gives every team the same escalation shape, and a scheduled plan flags anything that diverges.
Recovery	A deleted escalation policy is rebuilt from memory.	You revert the commit and apply.
New services	Someone copies an existing schedule by hand.	A new team gets a schedule and escalation policy by adding one module block.

Code also makes on-call safer to change at scale. When a reorg moves engineers between teams, a reviewed change to a few files replaces a long session of UI edits, and the plan lists exactly which rotations and escalation levels change.

Worked example: a schedule, a rotation and an escalation policy in Terraform

The example below defines a weekly primary rotation for a Payments team and an escalation policy that pages whoever is on call, then pages the whole team if nobody acknowledges within 10 minutes. Every resource and argument comes from the provider’s published resource docs. The basics of installing the provider are on the Rootly Terraform provider page.

Provider and team

terraform {
  required_providers {
    rootly = {
      source  = "rootlyhq/rootly"
      version = "~> 5.23"
    }
  }
}

# The provider reads the API token from the ROOTLY_API_TOKEN environment variable.
provider "rootly" {}

data "rootly_user" "alice" {
  email = "alice@example.com"
}

data "rootly_user" "bob" {
  email = "bob@example.com"
}

data "rootly_user" "carol" {
  email = "carol@example.com"
}

resource "rootly_team" "payments" {
  name = "Payments"
  user_ids = [
    data.rootly_user.alice.id,
    data.rootly_user.bob.id,
    data.rootly_user.carol.id,
  ]
}

Schedule and weekly rotation

A rootly_schedule holds one or more rotations. The rotation below hands off every Monday at 09:00 New York time, and the three responders take turns in the order of their position.

resource "rootly_schedule" "payments_primary" {
  name              = "Payments primary"
  description       = "Primary on-call for payments-api. Managed in Terraform."
  owner_group_ids   = [rootly_team.payments.id]
  all_time_coverage = true
}

resource "rootly_schedule_rotation" "payments_weekly" {
  schedule_id                = rootly_schedule.payments_primary.id
  name                       = "Weekly primary"
  position                   = 1
  schedule_rotationable_type = "ScheduleWeeklyRotation"
  schedule_rotationable_attributes = {
    handoff_day  = "M"
    handoff_time = "09:00"
  }
  active_all_week = true
  time_zone       = "America/New_York"
}

resource "rootly_schedule_rotation_user" "alice" {
  schedule_rotation_id = rootly_schedule_rotation.payments_weekly.id
  user_id              = data.rootly_user.alice.id
  position             = 1
}

resource "rootly_schedule_rotation_user" "bob" {
  schedule_rotation_id = rootly_schedule_rotation.payments_weekly.id
  user_id              = data.rootly_user.bob.id
  position             = 2
}

resource "rootly_schedule_rotation_user" "carol" {
  schedule_rotation_id = rootly_schedule_rotation.payments_weekly.id
  user_id              = data.rootly_user.carol.id
  position             = 3
}

The provider marks rootly_schedule_rotation_user as deprecated in favor of a schedule_rotation_members block inside rootly_schedule_rotation. That block needs the schedule nesting feature on your account, so check which form your account supports before you standardize on one.

Escalation policy, path and levels

An escalation policy has one or more paths, and each path has numbered levels. delay is the number of minutes a level waits before the next level is notified. repeat_count reruns the policy if nobody acknowledges.

resource "rootly_escalation_policy" "payments" {
  name         = "Payments escalation"
  description  = "Pages the Payments primary, then the whole team. Managed in Terraform."
  group_ids    = [rootly_team.payments.id]
  repeat_count = 2
}

resource "rootly_escalation_path" "payments_default" {
  name                 = "Default"
  default              = true
  escalation_policy_id = rootly_escalation_policy.payments.id
}

# Level 1: page whoever is on call on the primary schedule.
resource "rootly_escalation_level" "payments_primary" {
  escalation_policy_id      = rootly_escalation_policy.payments.id
  escalation_policy_path_id = rootly_escalation_path.payments_default.id
  position                  = 1
  delay                     = 10

  notification_target_params {
    type = "schedule"
    id   = rootly_schedule.payments_primary.id
  }
}

# Level 2: nobody acknowledged within 10 minutes, so page every member of the team.
resource "rootly_escalation_level" "payments_team" {
  escalation_policy_id      = rootly_escalation_policy.payments.id
  escalation_policy_path_id = rootly_escalation_path.payments_default.id
  position                  = 2

  notification_target_params {
    type         = "team"
    id           = rootly_team.payments.id
    team_members = "all"
  }
}

Once this works for one team, move the schedule, rotation and policy into a module that takes the team name, members, time zone and handoff time as variables. Every team then gets the same escalation shape, and a reviewer only has to read the inputs.

CI workflow: plan on pull request, apply on merge

The pipeline below runs terraform plan on every pull request that touches the on-call configuration, posts the plan for review, and runs terraform apply only after the change merges to main. It assumes a remote state backend is already configured in the Terraform code, because CI runners do not keep local state between runs.

# .github/workflows/on-call-terraform.yml
name: on-call-terraform

on:
  pull_request:
    paths: ["oncall/**"]
  push:
    branches: [main]
    paths: ["oncall/**"]

permissions:
  contents: read
  pull-requests: write

concurrency:
  group: on-call-terraform
  cancel-in-progress: false

env:
  ROOTLY_API_TOKEN: ${{ secrets.ROOTLY_API_TOKEN }}
  TF_IN_AUTOMATION: "true"

jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: oncall
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init -input=false
      - run: terraform fmt -check
      - run: terraform validate
      - id: plan
        run: terraform plan -input=false -no-color -out=tfplan
      - name: Post the plan on the pull request
        uses: actions/github-script@v7
        env:
          PLAN: ${{ steps.plan.outputs.stdout }}
        with:
          script: |
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: "On-call terraform plan:\n\n```\n" + process.env.PLAN.slice(-60000) + "\n```",
            });

  apply:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: on-call-production
    defaults:
      run:
        working-directory: oncall
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init -input=false
      - run: terraform apply -input=false -auto-approve

A few choices in this workflow matter more than they look:

1. concurrency with cancel-in-progress: false. Two merges in quick succession queue instead of racing each other against the same state.
2. A protected environment on apply. Require a reviewer on the environment if you want a second approval before production paging changes.
3. Branch protection. Require the plan job to pass before merge, so nobody merges a change that fails validation.
4. A scoped token. Store a dedicated Rootly API token for the pipeline in CI secrets and rotate it like any other production credential.

Handling drift

Drift happens when the live on-call configuration no longer matches the code, usually because someone fixed a gap in the UI during an incident. Decide your policy before it happens.

1. Detect it on a schedule. Add a nightly job that runs terraform plan -detailed-exitcode. Exit code 0 means no changes, 1 means an error, and 2 means the live configuration differs from the code. Alert the owning team on exit code 2.
2. Find out why. Read the plan and ask whoever made the change. A manual fix during an incident is often correct and should be kept.
3. Adopt or revert. To keep the live change, update the HCL to match and merge it so the next plan is clean. To undo it, merge nothing and run terraform apply, which restores the code’s version.
4. Keep short-lived changes out of scope. Overrides, coverage requests and shift swaps change every week. Leave them in the on-call tool. In Rootly they are separate objects from the rotation, so a swap does not show up as drift on rootly_schedule_rotation.
5. Say it in the UI. Put “Managed in Terraform” in the schedule and policy descriptions, as in the example above, so anyone editing in the UI knows the change will be reviewed or reverted.

Bringing existing schedules under code

Import existing objects before you manage them, or Terraform will create duplicates. Write an import block for each schedule, rotation and escalation policy using its ID from the Rootly API or the dashboard URL, then let Terraform draft the HCL:

import {
  to = rootly_schedule.payments_primary
  id = "<schedule-id>"
}

import {
  to = rootly_escalation_policy.payments
  id = "<escalation-policy-id>"
}

terraform plan -generate-config-out=generated.tf

Review generated.tf, move the resources into your module, and apply only when the plan shows the imports and no other changes.

Who runs on-call as code

• Anton Payments defined its entire incident response in Terraform with the Rootly provider, going from two and a half months of trying other approaches to a complete deployment in 15 hours. Read the Anton story.
• Mistral AI treats alerting as code, with Terraform as the source of truth for which checks exist, how alerts route to services, and which escalation policy each service uses. Routing changes are reviewed like any other infrastructure change. Read how Mistral AI does it.

How teams do this in Rootly

Rootly’s official provider is rootlyhq/rootly. Beyond schedules and escalation policies it manages alert routes, teams, services, severities, workflows and status pages, so the path from an alert to the person paged can live in one repository. The Rootly Terraform docs cover setup, and the provider page has more examples, including linking a service to an escalation policy. To design the rotation itself before you encode it, start with schedules and rotations and escalation policies.

If you are moving from PagerDuty, its own provider uses pagerduty_schedule and pagerduty_escalation_policy. Schedules, rotations and escalation levels exist in both, so most of the work is translating resource definitions and checking the plan.