# How to run a pilot that gives a real answer URL: https://rootly.com/why-rootly/pilot > A guide to piloting an incident response platform: how to scope it, how long to run it, the exit criteria to write down first, what to measure, and the questions to ask every vendor including Rootly. Why Rootly · Part two of four A pilot is only useful if it could end in a no. Here is how to set one up with clear pass or fail criteria. [Book a demo](https://rootly.com/demo) How to run a pilot that gives a real answer 1. What a pilot is actually testing 2. Scope it, and include your messiest team 3. Measure the length in rotations, not weeks 4. Write down what would make this a no 5. What to measure, and what to ignore 6. Questions to ask any vendor, us included 7. Questions to ask your own team at the end 8. Five ways a pilot can mislead you 9. What we do during yours 10. The switching guide ### On this page 1. What a pilot is actually testing 2. Scope it, and include your messiest team 3. Measure the length in rotations, not weeks 4. Write down what would make this a no 5. What to measure, and what to ignore 6. Questions to ask any vendor, us included 7. Questions to ask your own team at the end 8. Five ways a pilot can mislead you 9. What we do during yours 10. The switching guide ## What a pilot is actually testing Not whether the product works. It does, and so does the one you have now. A pilot tests whether it fits how your teams actually work, and there are three things a demo cannot show you. Whether it bends to the ways your teams differ from each other. What happens the first time you need something it does not do. And whether people reach for it at 3 a.m. without being told to. If nothing in your pilot could have made you say no, it was not a real test. ## Scope it, and include your messiest team Two or three teams, not the whole organization. One of them should be the one you would rather not think about: the strange escalation path, the rotation shared between two groups, the service nobody wants to own. A pilot run on your best-behaved team proves nothing, because every tool passes that. The point is to find the tool's limits while you are still evaluating, not eighteen months after signing, and limits always show up in the unusual cases. ## Measure the length in rotations, not weeks Four weeks means something different to every team. One full rotation means every responder has carried the pager at least once, which is the unit that matters: someone who was never paged during the pilot cannot tell you much about it. For most teams a full rotation lands somewhere between two and four weeks. If yours runs monthly, plan for longer rather than deciding on partial evidence, or shorten the rotation for the duration. Not everything has to wait for a real incident. Rootly can replay past alerts, so you can test routing, escalations, and AI investigations against your own history in the first week, and save the live rotation for what only real pages can show you: who responds, how fast, and whether they reach for the tool on their own. ## Write down what would make this a no, before you start Without written criteria, a pilot often ends with the loudest person in the room deciding. The fix is simple: write the exit criteria down before the pilot starts, put a name against each one, and set the date you will decide by. Write each one so it could fail. A criterion that always passes tells you nothing. Every responder was reached the way they expected Not “the page fired.” Reached: the right person, on the device they actually carry, without needing an escalation to find them. The awkward team works the way it needs to Whoever set it up, you or the vendor, that team’s unusual escalation path pages the way they expect. That is the setup you will repeat for every team you add. Every gap got an answer Every product has gaps, ours included. Write down each one you hit and how it was handled: solved by configuration, solved with the vendor’s team, or given a date on the roadmap. Retrospectives got written without chasing If they only got written because someone kept asking, that tells you about that person, not the tool. You got a person when you needed one When you needed help, a person answered, and you know how long it took. You know what it costs at your next size In writing, at twice your headcount, with the capabilities you just used still included. ## What to measure, and what to ignore Ignore mean time to resolution over a pilot. Across a few weeks that number is dominated by which incidents happened to occur, not by the tool you were using, so it cannot tell you much either way. It is misleading at any length: across Rootly's incident data, critical incidents resolve in about an hour at the median, but the mean is more than nine days, pulled out by a few that run for weeks. [Our report on great incident management](https://rootly.com/great-incident-management#after-the-incident) explains why. What is worth writing down instead: - **How deep escalations went before somebody acknowledged.** This is the best sign of whether pages reach the right person first. - **Pages per declared incident.** Every page beyond the people who had to act is noise. The best quarter of teams on Rootly stay under four, at every company size. - **Time to mitigate, not time to resolve.** How long until customers stopped feeling it. For SEV0 and SEV1, the best quarter of teams on Rootly do it in 44 minutes or less. - **How much coordination happened inside the tool.** If the real conversation moved to a side channel, the record is partial and every retrospective starts incomplete. - **How each gap was handled.** List what the tool did not do out of the box, and next to each one, whether it was solved by configuration, by the vendor’s team, or by a date on the roadmap. A gap with an owner and a date is different from one you are left to maintain alone. - **How long support took, and whether you got a person.** Raise something real during the pilot on purpose, so you learn this while you still have leverage. - **Whether retrospectives got written without anyone chasing.** Your own baseline is the main comparison. The two reference points above come from [Great, by the numbers](https://rootly.com/great-incident-management#great-by-the-numbers), and its [scorecard](https://rootly.com/great-incident-management#the-scorecard) is worth filling in before the pilot and again at the end: it shows what changed in how you run incidents, not only in the tool. ## Questions to ask any vendor, us included Ask all of these of everyone on your list, and get the answers in writing. Ours are below each one, so you can hold us to them the same way. ### Plans and pricing #### Which of the things we just used are on our plan, and which need an upgrade? Pilots usually run with every feature switched on. The plan you buy may not include all of them. **Rootly:** Here is what Essentials, the starting plan, includes: everything a team needs to start and grow. - Incident Response: the @Rootly AI agent, AI Scribe, AI retrospectives, and AI related incidents; response in Slack, Microsoft Teams, or Google Chat; incident roles, playbooks, action item tracking, reminders and nudges, and the communications module; one external and one internal status page. - On-Call: SMS, voice, and push paging that bypasses Do Not Disturb; alert routes, sources, and grouping; heartbeats; live call routing; 20 schedules and escalation policies; shadow rotations, overrides, coverage requests, holiday calendars, and schedule gap detection; Slack user group sync and the on-call pay calculator; 99.99% reliability. - Both: workflows, the catalog, metrics and insights, the API, CLI, webhooks, MCP server, Terraform provider, and mobile apps; SSO and SAML; SOC 2, GDPR, CCPA, and a BAA; data export; PagerDuty and Opsgenie migration; a dedicated success manager and a private support channel. AI SRE is a separate product that requires Incident Response, and Enterprise includes it. Enterprise adds custom forms and incident types, private incidents, custom roles, SCIM, audit logs, and unlimited organizations when you need them. #### Do people who only follow incidents count as paid seats? Engineering leaders, support, and executives need to see incidents without carrying a pager. Paying a full seat for each of them adds up fast. **Rootly:** No. Rootly has a free tier for stakeholders who follow incidents but never carry a pager. #### What does this cost at twice our headcount? Prices often jump at renewal, once you have grown and switching is harder. **Rootly:** Ask us and we will put it in writing before you sign the first one. ### On-call #### When several people are paged at once, does one acknowledgement cover the alert? If everyone paged has to acknowledge separately, people keep getting chased after the alert is handled, and your response times count the slowest person instead of the one who acted. **Rootly:** Yes. One person’s acknowledgement covers the alert for everyone paged with them, and your metrics record the person who acted. #### Can one escalation step page a team, a person, and a Slack channel at the same time? If it cannot, you build it from several steps with timing tricks, and every later change means checking that timing again. **Rootly:** Yes. Escalation steps page teams, users, schedules, and Slack channels directly, with no placeholder users or extra services to wire up. #### Can engineers arrange their own cover, and will the tool catch a gap before a page goes unanswered? If only an admin can edit the schedule, cover gets arranged in direct messages and the schedule stops matching reality. **Rootly:** Engineers request coverage themselves, schedule gap detection flags holes and falls back to someone, and Slack on-call groups stay in sync with the schedule, on every plan. ### AI #### Is the AI you are showing us generally available, or in a design-partner program? A pilot should compare what you can buy today, not a roadmap. **Rootly:** Generally available. The AI agents come with Incident Response, and AI SRE requires Incident Response and is included in Enterprise. #### Can I see the evidence behind a conclusion the AI reached? If you cannot see the evidence, you have to take the AI at its word, and during an incident there is no time to double-check it. **Rootly:** Every claim links to the artifact under it: this deploy, this log line, this past incident. Confidence is shown as a level rather than a made-up percentage, and the causes it ruled out are shown with the answer. #### Does the AI analysis sit in front of paging? If the AI has to finish before the page goes out, a slow analysis means a late page. **Rootly:** No. Analysis runs alongside your escalation policy and cannot delay it. ### Security and data #### Do you train on our incident data, and who processes it? Ask it plainly and get it in writing. This is the question your security review will ask you. **Rootly:** No. Nothing learns across customers. Your graph, your memory, and everything inferred about your environment stay yours. On Enterprise you can bring your own AI key (OpenAI, Anthropic, or Gemini), so model calls run under your own account. #### Where is our data stored, and how long is it kept? Data residency and retention are usually the first two questions from legal, and the answers decide whether a pilot can use real data at all. **Rootly:** Rootly hosts and processes your data in the US today, and EU hosting is on the roadmap. Rootly is SOC 2 Type II and covers GDPR, CCPA, and HIPAA, with a BAA on every plan, and Enterprise sets custom data retention and session timeouts. Every subprocessor is listed publicly with what it processes and where, and we will send the data processing agreement before your review starts. [Rootly’s subprocessors, what each one processes, and where](https://docs.rootly.com/configuration/subprocessors) #### How finely can we control who sees and does what? Security incidents, separate business units, and contractors all need narrower access than everyone else, and coarse roles force you to run them somewhere else. **Rootly:** Rootly has the deepest role-based access control of any incident management platform: built-in roles on Essentials, custom roles on Enterprise, private incidents, and SSO, SAML, and SCIM. [Security best practices: identity, roles, API access, and audit logs](https://docs.rootly.com/configuration/security-best-practices) ### Fit with your stack #### Does it work the same way in Slack, Microsoft Teams, and Google Chat? Most companies run more than one, through acquisitions or different departments, and a tool that only works well in one splits your response in two. **Rootly:** Response runs in Slack, Microsoft Teams, or Google Chat on every plan. #### Can we manage the whole setup as code? Teams that run their infrastructure as code want incident configuration reviewed the same way, not clicked together in a UI. **Rootly:** Yes. The Terraform provider, REST API, and CLI are on every plan. ### The company #### Who configures it, during the pilot and after? Setup encodes your own process, and it keeps changing as you add teams. Whether the vendor keeps helping after the pilot decides how much of that lands on your team. **Rootly:** We do it with you: the audit, the migration, and the setup of each team, with a dedicated success manager on every plan and PagerDuty and Opsgenie migration included. #### What happens when we need something the product does not do? At some point you will need something it does not do, and how the vendor handles that is what you will live with for years. **Rootly:** You get a dedicated success manager and a private support channel on every plan. If the answer is that we cannot do it, you hear it from a person, not a pricing page. #### What is the commitment that this product still exists in three years? Not hypothetical. Atlassian closed new Opsgenie sales in June 2025 and set end-of-life for April 2027, and its customers now have to migrate on Atlassian’s timeline. **Rootly:** Rootly is profitable and in control of its own direction. Incident management is its whole business, not one product among many, and Fortune 500 companies that have been around for decades run on it. Weigh that against how important this product is to the other vendors you are considering. #### If we leave, what comes with us? Ask on the way in, when you have leverage, rather than on the way out when you do not. **Rootly:** Your data, by API, whenever you want it. We would rather you stay because you want to, not because leaving is hard. ## Questions to ask your own team at the end These predict adoption better than a scorecard does, because they ask about behavior rather than opinion. Have your champion collect them from the people who carried the pager. - When it did not do something you needed, who helped, and how quickly? - Did you still check the old tool out of habit? For how long? - Would you be annoyed if we switched back on Monday? - Did it ever tell you something about an incident you did not already know? - Was there a moment you thought, “I would not want to do this at 3 a.m.”? The second one is the most useful. Habit is slow to move, and a team that stopped checking the old tool inside one rotation has told you something no survey would have. ## Five ways a pilot can mislead you You assume setup is your job alone Incident setup encodes years of your own process. Ask every vendor who configures the pilot, and who keeps configuring it after you sign. A vendor that does it with you, and keeps doing it as you add teams, is worth more than one that hands you a manual. Get it in writing: is that help part of the plan, or a services line item? It runs in a sandbox A sandbox only shows you the demo. Route real alerts from real services. Nobody owns it A pilot without a champion drifts. Nobody chases the setup, nobody raises the gaps, and nobody makes the case at the end. Name one person who owns it, give them the vendor’s direct line, and make sure the people who carry the pager have used it by the decision meeting. It is too short to catch anything A pilot with no real incident in it tested your configuration, not your response. Replay past alerts to test the setup, and if nothing real happened, extend it rather than deciding. You compare a fresh install to three years of accumulated setup Your incumbent has years of tuning in it and the new tool has two weeks. Keep that in mind, and say it out loud in the decision meeting. ## What we do during yours We start with a read-only audit of your current environment, so the pilot is scoped against what you actually have rather than what the org chart says. You keep the findings report whether or not you go further. Then a named person for the duration, help writing the exit criteria including the ones we might fail, and the awkward team set up first rather than last. If we come out of it thinking you are better served elsewhere, we will tell you. That is better for both of us than a contract you regret. > “Our time to acknowledge a critical alert went from hours to minutes. We evaluated every alerting tool out there, and nothing else matched Rootly on ease of use, scale, or price.” Pedro Espindula, SRE, [Cora](https://rootly.com/customers/cora) Write down what would make it a no before you start, and the pilot will tell you whether the tool fits. ## The switching guide 1. The case [Why teams switch to Rootly](https://rootly.com/why-rootly) What you get, what teams who switched say changed, and how Rootly compares. 2. Part two **How to run a pilot that gives a real answer** Scope, length, the exit criteria to write down first, and the questions to ask every vendor, us included. 3. Part three [How a migration to Rootly runs](https://rootly.com/why-rootly/migrations) The audit, the phased rollout, what we cannot promise, and what it costs your team. 4. Part four [Making the case inside your company](https://rootly.com/why-rootly/pitch) Seven conversations a champion has to win, and how to answer each one. [Book a demo](https://rootly.com/demo) · [Next: how a migration to Rootly runs](https://rootly.com/why-rootly/migrations)