How Canary built incident management from scratch on Rootly while its engineering team quadrupled.

When Z Lee joined Canary to lead platform, the entire incident process was a Slack channel and the word "help." One on-call engineer would catch an alert, post into a single channel asking if anyone could help, or just try to fix it alone. If a page was noisy, the team muted it and moved on, and then a real outage could come through and go unnoticed. At thirty or forty engineers who all knew the systems by heart, that just about held together. Then Canary quadrupled its engineering team in eighteen months, and "post in a channel and hope" stopped scaling overnight. There was no better tool to switch to, because there was no tool. They had to build the practice from zero, mid-sprint.

"When I joined we had no incident management process at all, just some light Notion docs. We've quadrupled engineering since then. Rootly is the platform we built the whole practice on, and it scaled right alongside us with ease." -Z Lee, Director, Engineering

No process, in the middle of hypergrowth.

The cracks were the kind that don't announce themselves. Alerts got muted and missed; a noisy page would be silenced, and then a real one would come through unnoticed, so the team sometimes simply wasn't aware of outages. And nothing closed the loop; an issue would get patched in the moment, then recur months later because there was no process to drive it to a real fix, generating the same toil again and again. This wasn't a bad tool, it was the absence of a practice, exposed by growth.

Building the practice, with Rootly as the foundation.

Most of Canary's rollout wasn't technical. Starting from nothing, the team first had to design the practice itself: severity levels, the on-call process, the rules for when to even declare an incident, incident-commander and on-call duties, and how the status page should work. That groundwork, plus educating every engineer on the new process, is what made it a three month effort. The technical build inside Rootly took a couple of weeks due to the custom automation Canary wanted on top of the opinionated defaults.

The point is that Rootly was the platform they built the whole practice on, not a like-for-like swap, and it started paying off before the first incident arrived. The moment On-Call was enabled, it surfaced a batch of misconfigured alerts and incorrect thresholds the team didn't know were wrong in the first place. When the first real incident occurred, the process pulled the right people in immediately and ran cleanly from declaration to resolution.

“Rootly is the platform we built all our incident processes on because it was easy, it had everything we needed, and we knew we could continue to grow into it for years to come.” -Z Lee, Director, Engineering

What changed

Every alert, and the right pages

Canary now captures all of its alerts in Rootly instead of muting and losing them, and Z's read is that the pages coming through are the right ones, with an actual process to resolve them. Escalation went from guesswork to clarity, where the team previously had no idea who to escalate to, every team now has an escalation policy with the right people on it, and anyone can page another team straight from Slack.

A loop that actually closes

The incident timeline is one of the features Z calls huge. It auto-captures what happened and the actions taken, then feeds the RCA, so a responder heads-down on a fix doesn't have to reconstruct the story afterward. From there the team creates the follow-up tickets that stop the issue recurring, the difference from before, when the same error would quietly come back months later. The loop now closes, which is exactly how you take toil out of a fast-growing engineering org.

Telling customers first

Canary's over 20,000 global customers want to know about a degraded service before they experience it, and Canary would rather tell them than be told. The status page now notifies subscribers automatically when something is off, instead of the customer success team scrambling to inform individual customers after the fact. For a platform sitting at the hotel front desk, getting ahead of the message is its own kind of reliability.

A platform that grew with the company

Canary today is a very different company than the one Z joined; quadrupled engineering and a much wider service footprint. The incident practice they built on Rootly scaled through all of it without being rebuilt or replaced. That is the quiet proof point. The platform you choose in hypergrowth is either one you will outgrow, or one built for the company you are becoming.

"We went from no process to one that actually closes the loop, so the same issue stops coming back months later. Rootly is the platform we built our incident management on because we knew it would grow with us instead of needing to be replaced." -Z Lee, Director, Engineering

Proof points (Canary lightning round, 1-5)

• Time to first status update improved: 5
• On-call handoffs are clearer: 5
• Pager fatigue is lower: 4
• Stakeholder communications are more consistent: 4
• Customers are happier: 4
• Engineering morale is in a better place: 4

The most dangerous incident setup isn't a noisy one. It's a quiet, informal one in a company about to scale, where muted alerts and unclear ownership stay invisible until the team is suddenly four times bigger and the systems are wider than anyone can hold in their head. Canary got ahead of that by building a real practice from scratch on Rootly; every alert captured, ownership and escalation defined, a learning loop that stops repeat incidents, and customers told first. Best of all, the Rootly scaled with them rather than becoming the next thing to replace. If you're growing fast and your incident process is still a channel and a hope, that's the moment to build it on something that will grow with you.