Invite Responders to Incidents via Slack

๐Ÿ“ฎ Invite Responders to Incidents via Slack

Incidents are best tackled collaboratively and looping in the right responders fast is critical.

We've found today not only is it a manually intensive task but knowing who should be included is frustrating.

We are introducing a new Workflow Task to automatically add teammates into the incident Slack channel.

  • Supports both Slack users and Slack user groups (e.g. @oncall-infra, @legal)
  • Dynamically add people based on conditions set in Workflows (e.g. If postgres-db impacted then add Mary)
  • Users do not have to be signed up on Rootly in order to be added

๐ŸŒ New & Improved

  • ๐Ÿ†• New slash command for assigning roles via /incident assign or /rootly assign, eliminating the need to scroll up
  • ๐Ÿ†• Roles are exposed via the Rootly API
  • ๐Ÿ’… We simplified our syntax in URLs on Web
  • ๐Ÿ’… Made adding summary via /incident summary more obvious inside of incident channels
  • ๐Ÿ’… Continued to invest heavily in our own reliability and on-call structure, customers can page us if urgent
  • ๐Ÿ› Fixed typo in Slack integration
  • ๐Ÿ› Fixed issue where users weren't able to unassign from on-call roles

Severity Descriptions

๐Ÿฆท Severity Descriptions

What's the difference between a SEV0 and SEV1? Most organizations at best have this defined in an internal doc buried away.

However, that is not much use when you're under pressure and need to quickly create an incident.

We are introducing dynamic severity descriptions that change depending on what severity is picked to provide context.

  • Defined in Rootly > Configuration > Severities > Description
  • Dynamically changes depending on which severity level is picked
  • Reduce top of funnel friction during incident creation
  • Clearly and easily educate your organization on how to define severities


๐ŸŒ New & Improved

  • ๐Ÿ†• Action Items and Tasks have been merged into just Action Items. You can now decide if they are action items you do now (tasks) or do later (follow-ups).
  • ๐Ÿ†• Mitigated and resolved messages are displayed Slack incident channel for broader visibility
  • ๐Ÿ†• Integration with Dropbox Paper for postmortem
  • ๐Ÿ†• User settings are remembered if you toggle between grid or list view in Web UI
  • ๐Ÿ†• Markdown syntax support for severity description fields
  • ๐Ÿ†• Postmortems created in Google Docs can be written to shared Drive folders
  • ๐Ÿ’… Invested significantly into scaling our infrastructure and reliability as we continue rapidly onboard customers
  • ๐Ÿ’… Action items Web UI uses tags to indicate priorities instead of sections
  • ๐Ÿ’… Improved our SEO for better discoverability
  • ๐Ÿ’… Renamed Genius to Workflows for specific tasks that run in Slack (that should be them all, goodbye Genius!)
  • ๐Ÿ’… Added sign out link in Slack onboarding flow if users want to restart or use a different account
  • ๐Ÿ› Fixed issue where postmortem previews on Web UI dashboard looked incorrectly sized
  • ๐Ÿ› Fixed UI issue for connecting Slack accounts to Rootly

Reopening Incidents

๐Ÿฅก Reopening Incidents

We've all been there. Sometimes we get ahead of ourselves and close an incident too early before it's been resolved.

Today, we are excited to introduce the ability to reopen incidents and continue right where you left off without missing a beat.

Features:

  • Accessible in Slack via /incident status or /rootly status commands and in the Web UI (see above screenshot)
  • Metrics are updated accordingly to reflect the reopened incident to ensure accuracy
  • As always, you can go back and retroactively edit the state time of incident down to the minute
  • Ability to swap back and forth between all incident states (Started, Mitigated, and Resolved)


๐ŸŒ New & Improved

  • ๐Ÿ†• Pin to timeline feature now supports .docx format files
  • ๐Ÿ†• Rootly is now available for a 14-day trial with all enterprise features unlocked
  • ๐Ÿ†• Zoom meeting passwords are displayed in URL to make joining easier
  • ๐Ÿ†• We published our year in review for 2021 newsletter that recaps a few highlights
  • ๐Ÿ†• Ability to trigger manual Workflows at incident creation time
  • ๐Ÿ’… In Web UI we've made it more obvious when an incident is selected
  • ๐Ÿ’… Weekly summary email has been reformatted to be more spaced out
  • ๐Ÿ’… Improved loading of Opsgenie on-call schedules
  • ๐Ÿ› Fixed issue where deleting a Service required two clicks instead of one
  • ๐Ÿ› Fixed issue where occasionally unselecting severity in Web UI was stuck

SOC 2 Type II Certified

๐Ÿ›ก๏ธSOC 2 Type II Certified

We are ringing in 2022 with our SOC 2 Type II (2) certification!

We continue to be committed to our own privacy, governance, and security as our first priority to protect our customers.

How is this different from SOC 2 Type I?

  • Although SOC 2 Type I has it's benefits, it pales in comparison to SOC 2 Type 2.
  • A SOC 2 Type 2 requires companies to show an attestation of controls over a minimum 6 month period.
  • Whereas SOC 2 Type 1 is only an attestation of controls at a specific point in time.
  • SOC 2 Type 2 vendors continuously validate against tough security protocols and controls they remain compliant.

How did we achieved SOC 2 Type II

A large part of this success is with partnering with Secureframe, the best SOC 2 automation platform.

They integrate with our infrastructure and continuously scan and monitor our systems in addition to ensure any vendor we use is secure and compliant.

More details covered in our full customer case study with Secureframe.

Security @ Rootly

To learn more, please visit https://rootly.com/security where we highlight our security best practices. Everything from encryption to SSO.


๐ŸŒ New & Improved

  • ๐Ÿ†• Weihan Li joined our team as a Senior Software Engineer and will help accelerate our roadmap efforts and continue to ensure the platform is reliable
  • ๐Ÿ†• Added Slack command /incident status or /rootly status that allows you to pick from Mitigated and Resolved easier
  • ๐Ÿ†• Postmortem URL can be used as an incident variable
  • ๐Ÿ†• Action items can be assigned to teams and not just individuals, this is also supported in Workflows - Action Item as a condition if you want to filter Team A action item goes to Team A board
  • ๐Ÿ†• Pinned images and screenshots will show up in external timelines on Confluence and Google Docs
  • ๐Ÿ†• Search and pagination added to Invite Users from Slack modal
  • ๐Ÿ†• New Workflow Condition called Visibility that will let you trigger workflows based on whether an incident is private or public
  • ๐Ÿ†• New Workflow Task that allows you to create an incident independent of Slack automatically
  • ๐Ÿ†• Brand new coat of paint on our Slack Marketplace app as we continue to deepen our partnership
  • ๐Ÿ’… Private incidents are excluded from weekly email for all users regardless of admin privileges
  • ๐Ÿ’… All Slack integrations on Rootly by default ask users to connect their Slack account to make onboarding more seamless
  • ๐Ÿ’… Anywhere incident variables can be used we now link to a better resource for Slack markdown, more enhancements coming here soon
  • ๐Ÿ’… Manually running Workflows UI in Slack now says โšก Run instead of Submit
  • ๐Ÿ’… Improved carriage returns and spacing on status page events, in particular for long form updates
  • ๐Ÿ’… Improved status page styling, title, and added support for favicon
  • ๐Ÿ’… Slack integration page now includes references to Slack markdown for incident title
  • ๐Ÿ’… Removed /genius from Slack commands as it was replaced by Run Workflow option when an incident channel is created
  • ๐Ÿ› Fixed issue where PagerDuty list of users had an infinite scroll
  • ๐Ÿ› Fixed issue where there were inconsistent timezones on incident timeline
  • ๐Ÿ› Fixed issue where Opsgenie teams weren't importing
  • ๐Ÿ› Fixed UI issue where connecting to Slack account had a banner partially covering the connect button
  • ๐Ÿ› Fixed issue in Slack UI where duplicate emojis displayed
  • ๐Ÿ› Fixed issue where VictorOps occasionally did not page
  • ๐Ÿ› Fixed issue where scheduled maintenance timezone and system did not align correctly
  • ๐Ÿ› Fixed issue where occasionally manually clicking Run Workflow in Slack did not populate all workflows
  • ๐Ÿ› Fixed issue where occasionally editing or creating a Workflow did not work under specific conditions
  • ๐Ÿ› Fixed issue where alerts would ping old resolved incident channels

Trigger Workflows Anytime

โš™๏ธ Trigger Workflows Anytime

We are excited to release the ability to manually trigger Workflows independently and at anytime!

Workflows are the best way to automate your incident response and process. However, they rely on incident conditions being met (e.g. Started, Severity, Team, etc.).

There are plenty of use cases where you might not know the incident conditions but may still want to rely on a Workflow automation.

You'll now be able to trigger Workflows from both Web and Slack.

  • Functions independent of incident conditions
  • New Slack command /incident workflow or click Run Workflow button
  • New Trigger Workflows option in incident tab on Web
  • Accessible by any Rootly user
  • Quickly rerun Workflows if necessary (e.g. fetch Datadog metric)


๐ŸŒ New & Improved

  • ๐Ÿ†• Launched our first Whitepaper - The Ultimate Guide to SRE
  • ๐Ÿ†• Auto-populate and search through Slack channels on Rootly, no more scrolling necessary
  • ๐Ÿ†• Added rerun button next to any Workflows
  • ๐Ÿ†• Notification sent to users whenever they are assigned an action item (Slack + Email)
  • ๐Ÿ†• Trello integration now fetches labels for more granular Workflow control
  • ๐Ÿ†• Emojis in incident channel topics dynamically change whenever the incident is mitigated ๐ŸŸ  and resolved ๐ŸŸข instead of always showing ๐Ÿ”ฅ
  • ๐Ÿ†• Live refresh of Web UI, incident states are updated the second they are changed without needing to refresh
  • ๐Ÿ’… Workflows now display a warning message if the integration your Task is using has not been configured
  • ๐Ÿ’… Built even more redundancies across core services and platform for improved reliability as customer demand continues to scale
  • ๐Ÿ’… Polished UI on Slack for users that are running on a free trial
  • ๐Ÿ’… Revamped our internal on-call so we continue providing greater coverage for our customers
  • ๐Ÿ’… Workflow conditions default to One-Of instead of Is
  • ๐Ÿ’… Slack command /incident genius is rebranded to /incident workflows
  • ๐Ÿ’… Legacy genius command in help menu drop-down on Slack has been rebranded to Workflows
  • ๐Ÿ› Fixed issue where formatting incident events on status pages
  • ๐Ÿ› Fixed issue where duplicate resolved messages were sent on Slack briefly
  • ๐Ÿ› Fixed issue where if you reassigned a role that person was not notified or added to channel
  • ๐Ÿ› Fixed issue where adding timeline events UI in Web was hidden

Making Incident Creation a Breeze

๐Ÿ’จ Making Incident Creation a Breeze

Whenever customers adopt Rootly, our name quickly becomes synonymous with the easiest way to create incidents.

Today, we are making incident creation even easier by introducing a new slash command /incident on Slack.

  • /incident will function exactly the same as /rootly
  • Both slash commands can be used at the same time and interchangeably

Why did we do this? Although /rootly was great, many organizations that had a homegrown bot had already defaulted to using /incident.

Not only is it more intuitive, it also means less behavioural changes for our customers which is critical to adoption!


๐ŸŒ New & Improved

  • ๐Ÿ†• Workflows for PagerDuty now support paging escalation policies in addition to schedules and individuals
  • ๐Ÿ†• Customers who use VictorOps can now create Workflows to page
  • ๐Ÿ†• Ability to pin more file formats to incident timeline
  • ๐Ÿ†• Add events to incident timeline in Slack with /rootly timeline or /incident timeline command
  • ๐Ÿ’… Improved Web UI for action items so at a glance much easier to see what is open
  • ๐Ÿ’… Assigning users for action items are now ordered alphabetically
  • ๐Ÿ’… Integrations when configured prompt users to navigate to Workflows to complete their setup
  • ๐Ÿ’… Adding items to incident timeline from Web users can use Ctrl + Enter as a keyboard shortcut
  • ๐Ÿ’… Invested into additional monitoring for the Web platform and integrations
  • ๐Ÿ’… Relaxed rate limit requirements for bulk updates and configuration
  • ๐Ÿ› Fixed issue where PagerDuty did not page the Admin who originally configured it in Rootly
  • ๐Ÿ› Fixed issue where incident timeline occasionally displayed the incorrect date, one day in advance. Yes timezone is hard
  • ๐Ÿ› Fixed issue where occasionally Workflows did not trigger on incident update but did on incident creation

Workflows 2.0

๐Ÿ”ฎ Workflows 2.0 - Next Gen Incident Automation

We are excited to introduce the next generation of our Workflows feature to help automate the entire incident process in a consistent way!

What has changed since Workflows 1.0? Well, now all integrations are supported by our simple but powerful automation engine!

  • Zoom / Google Meet
  • Confluence / Google Docs
  • PagerDuty / Opsgenie
  • Jira / Zendesk / Linear / Asana / Trello / Shortcut / ServiceNow / Airtable

Use cases with Workflows 2.0 that you couldn't before? A few examples:

  1. ๐Ÿ”ฎ Create Jira tickets on multiple project board depending on which team is impacted
  2. ๐Ÿ”ฎ Open a Zoom or Google Meet bridge for high severity (>SEV1) incidents for high bandwidth conversations
  3. ๐Ÿ”ฎ Automatically page the Infrastructure team via PagerDuty or Opsgenie whenever the postgres-db is impacted
  4. ๐Ÿ”ฎ Use different Confluence or Google Doc postmortem templates if the incident was security related
  5. ๐Ÿ”ฎ ...thousands of other combinations to fit your exact incident process!

We make getting started simple too. When you first configure an integration we'll automatically create the Workflow template for you in 1-click. You can go in and modify anything you want later.

Create Jira Task

We are excited to see what you come up with and build!


๐ŸŒ New & Improved

  • ๐Ÿ†• Buyers guide for whether or not you should build or buy an incident management platform: https://rootly.com/buy-vs-build
  • ๐Ÿ†• Add new item to incident timeline via /rootly timeline instead of only using an emoji ๐Ÿ“Œ to pin items
  • ๐Ÿ†• More detail provided in Opsgenie alerts when paged via Rootly to give responders better context
  • ๐Ÿ†• Significant loading and performance improvements on Web UI, especially noticable in incident creation
  • ๐Ÿ’… Delayed execution of Workflows now support less than 1 min time increments (e.g. wait 10 seconds)
  • ๐Ÿ’… Default Slack channel names use a sequential ID instead of UUID when there is a duplicate to keep name cleaner
  • ๐Ÿ’… UI improvement to show whenever a Workflow fails making it more obvious for users where to debug
  • ๐Ÿ› Fixed issue where modifying mitigate timestamp saved but did not refresh the UI
  • ๐Ÿ› Fixed issue where some Workflows occassionally double triggered
  • ๐Ÿ› Fixed issue where Linear priorities were not loading when configuring and mapping to Linear
  • ๐Ÿ› Fixed issue where Linear ticket generation did not show up in dedicated incident channel when created

API and Infrastructure as Code Ready

๐Ÿซ API First and Infrastructure as Code Ready

Our API is now stable and using OpenAPI Specification. This will allow you to programmatically control the platform.

As an API-first company, everything you can do in the UI can be done via our REST APIs without clicking around. View our documentation here.

Now, integrate into existing Infrastructure as Code workflows or build new ones with Rootly.

The use cases are endless. For example:

  • Create new incidents automatically from internal tools alerting (e.g. SLOs)
  • Export and offload incident metrics to your BI tool of choice
  • Automate updating services and creating new teams

Have a custom use case or integration in mind with our API? Reach out to support@rootly.io or on Slack and we'll help you build it.


๐ŸŒ New & Improved

  • ๐Ÿ†• Ability to specify which fields are required and optional at incident creation time on Slack, previously was only optional by default
  • ๐Ÿ†• Pinning messages to incident timeline displays who sent the message instead of who pinned
  • ๐Ÿ†• Ability to sort and reorder Workflows, making it easier to stay organized
  • ๐Ÿ†• Redesigned our own Rootly internal admin to make it easier to support customers
  • ๐Ÿ†• Built internal tools to support wiping demo data easily for customers that are done testing and moving over to production
  • ๐Ÿ’… Cleaned up how selecting marking incident as Mitigated and Resolved looks in Slack
  • ๐Ÿ’… Incident timeline now displays system messages coming from Rootly instead of who created the incident
  • ๐Ÿ’… Link incident variables to new documentation, stay tuned for V2 where you'll no longer need to leave Rootly to browse
  • ๐Ÿ› Fixed issue where scheduled maintenance windows would occasionally display as an active incident in UI only
  • ๐Ÿ› Fixed issue where action items created in Rootly sometimes did not get created in Linear
  • ๐Ÿ› Fixed issue where Zoom room links were not getting attached to incident notification emails
  • ๐Ÿ› Fixed issue in Pulses (used for deploy tracking) would occasionally display duplicates

Slack Enterprise Grid & Multi-Workspace Ready

๐Ÿญ Slack Enterprise Grid & Multi-Workspace Ready

We are excited to announce Rootly is the only incident management platform that is Slack Enterprise Grid ready.

Our support for Slack Enterprise Grid means organizations with multiple workspaces can unify how they manage incidents.

Slack Enterprise

What does this mean for multi-workspace customers?

  • Ability to create incidents from any workspace without needing to switch
  • Users are seamlessly redirected to the incident channel after creating, no manual switching required
  • Continue using /rootly commands from any workspace
  • Get notified of incidents across all workspaces (through shared channels)
  • All Workflows support multi-workspaces (e.g. page team in workspace A and B)
  • Workaround locked down Slack permissions for creating new channels with incident specific workspaces

In Action


๐ŸŒ New & Improved

  • ๐Ÿ†• New Task in Workflows that allows you to invite Teams to incidents
  • ๐Ÿ†• Display who pinned events to the incident timeline
  • ๐Ÿ†• Added granular trigger events for Workflows beyond incident_updated, such as any variable like severity_updated, teams_updated, status_updated, and more
  • ๐Ÿ’… UI improvements in Workflows that make adding Tasks more obvious
  • ๐Ÿ’… UI improvements to status pages that ensure a status page has been selected first before publishing
  • ๐Ÿ’… UI improvements to how Task names were labeled inside of Workflows to be even more clear
  • ๐Ÿ’… Users that are logged into rootly.com they are automatically redirected to the Dashboard
  • ๐Ÿ› Fixed issue where logging into PagerDuty via SSO put users in an endless loop
  • ๐Ÿ› Fixed issue where Workflow runs where mislabeled as failed when it was actually canceled
  • ๐Ÿ› Fixed issue where the incident variable {{ incident.title }} did not populate properly
  • ๐Ÿ› Fixed issue with Email integration where sending an email did not always trigger an incident
  • ๐Ÿ› Fixed issue with Pulses where we weren't properly ingesting team variables in some cases

Quick Start Shortcuts

โšกQuick Start Shortcuts

We are introducing a faster way to get started with Rootly.

Whenever you type /rootly and hit enter, we will populate the most popular shortcuts to help you get started.

The command is context aware and will show different results depending on what channel you are in.

Running /rootly in non-incident channels

Toolbar1

Purposefully designed to help you create incidents faster and learn about Rootly.

  • Create Incident - declare an incident
  • All Commands - all available /rootly commands to pick from
  • Open Tutorial - getting started guide
  • Learn More - custom redirect to your 1-pager otherwise goes to docs.rootly.com

Running /rootly in incident channels

Toolbar2

Purposefully designed to help you manage and move incidents along after they've been declared right at your finger tips.

  • Add Summary - periodically summarize the latest customer impact
  • Escalate - directly page someone on e.g. PagerDuty, OpsGenie
  • Resolve - mark the incident as resolved
  • Update Incident - edit incident attributes such as severity or service impact
  • All Commands - all available /rootly commands to pick from


๐ŸŒ New & Improved

  • ๐Ÿ†• Introduced suite of new tools that monitors and tests Slack app functionality that alerts us even earlier
  • ๐Ÿ†• Page a team via PagerDuty or Opsgenie through Workflows whenever a team is added to the incident
  • ๐Ÿ†• Custom redirect for any Learn More button inside of Rootly (e.g. to your internal 1-pager)
  • ๐Ÿ†• Select incident causes from the Postmortem tab even if you have a Confluence or Google Docs integrated
  • ๐Ÿ’… Improved method to remove Workflows from database record when users requests deletion
  • ๐Ÿ’… Faster paging via VictorOps that requests one less call to their API
  • ๐Ÿ’… Renamed Genius to Workflows inside of the incident overview section
  • ๐Ÿ’… Improved test coverage for when a Workflow fails and outputs error to user and notifies us via Sentry
  • ๐Ÿ’… Updated walk-through content on /rootly tutorial
  • ๐Ÿ’… Update branding for Google Docs integration inside of Slack
  • ๐Ÿ’… Renamed the project management integration Clubhouse to Shortcut
  • ๐Ÿ’… Added incident title to your Chrome tab instead of the generic incidents label, easier to browse with multiple Rootly tabs open
  • ๐Ÿ› Fixed issue where Slack channel not notified if Confluence or Google Doc callback misfires
  • ๐Ÿ› Fixed issue where Confluence did not load all pages when encountering > 100 space keys
  • ๐Ÿ› Fixed typo for importing Confluence playbooks
  • ๐Ÿ› Fixed issue where auto resolve task in Workflows would be inconsistent
  • ๐Ÿ› Fixed issue where alerting tools like PagerDuty would occasionally post duplicate alerts in the alerts channel


Rootly helps you respond to incidents faster as a team and learn to prevent them in the future.
wave