We Stopped Shipping Features for a Quarter

When incidents pile up fast enough, every part of the company bleeds: support is fielding angry customers, AEs are on apology calls, and engineering is burning cycles on retrospectives instead of shipping.

For Eran Kampf, VP of Engineering at Twingate and co-founder of Monday.com, that was the moment he made a call most engineering leaders won't: stop all feature work for a quarter and fix reliability.

In this episode, Eran walks through what actually triggered that decision, why best-in-class engineering practices (full test coverage, Terraform provisioning, comprehensive monitoring) turned out to be just the baseline for four nines, not the destination, and what it took to rebuild trust internally and with customers. The conversation goes further: how Twingate moved from a single regional cluster to active-active multi-region, why sharded production is the next frontier for containing self-inflicted outages, and how a team of 30 is navigating agentic AI coding without letting speed erode the reliability they fought hard to earn.

Key Topics Discussed

(00:00) Why Twingate's reliability problems hit harder than most: if the product is down, customers have no network at all
(05:30) Why best-in-class engineering practices are the baseline, not the destination, and the mindset shift required to deliberately pursue four nines
(08:00) How Twingate ran its reliability sprint: scoping the system, ranking projects by severity and lift, and executing without stopping the business entirely
(10:30) From single cluster to active-active multi-region to sharded production, the architectural roadmap for reducing blast radius over time
(14:40) Why running lean (30 engineers, 3 on DevOps) is a reliability advantage, and how constraint forces the automation discipline larger teams buy later
(18:10) A framework for AI in high-stakes systems: judgment density times risk, and why production incidents will always need a human in the loop
(21:25) Agentic coding, PR review, and why shipping fast without human ownership is the new reliability risk

Where to Find Eran Kampf

Transcript

What's Twingate, and why does reliability hit differently for you than for most companies?

Twingate provides zero trust network access, essentially replacing VPNs. The important context is that we're the networking infrastructure for our customers. If Twingate is down, they don't have a network. They can't reach production, databases, any of it. So reliability isn't a nice-to-have: it's the product.

We started right around when COVID hit and saw strong early traction. Then we had a cluster of big incidents. Some were self-inflicted: software bugs. Others weren't: Google had load balancer and regional issues that cascaded into us. That period forced a fundamental shift in how we think about building and operating the system.

You made an unusual call: pausing all feature work to focus on reliability. What triggered that, and how did you sell it internally?

Every major incident taxes the entire company, not just engineering. Customers escalate, executives go on apology calls, you're scheduling reassurance Zoom calls, writing incident reports, showing root cause. It's not just engineering time: support, AEs, every customer-facing team gets pulled in. And beyond the operational cost, there's a trust problem: customers lose confidence in the product, and internally, customer-facing teams start losing confidence in engineering.

The goal of the reliability push was to rebuild that trust and make sure it didn't break again. To do that seriously, we had to stop everything else for at least two quarters and communicate clearly to the whole company: we're investing in reliability now, full stop.

You described Twingate as already being in the top percentile of engineering practices. So what was actually missing?

We had good processes across the board: everything provisioned through Terraform, nothing touched manually in production, full test coverage including unit, integration, and smoke tests running against live environments, Prometheus and Grafana for monitoring, Elasticsearch for logs, comprehensive alerting. We genuinely believed that if you do all the engineering best practices correctly, you won't have production issues.

That was the mindset shift we had to make. All of that is the baseline. Getting to four nines, consistently, quarter after quarter, requires thinking deliberately about reliability as a first-class concern. It can't be a byproduct of good process.

How do you actually operationalize that? What does "reliability as a first-class concern" look like in practice?

The most concrete change: every engineering spec now has a reliability section. Just having that header forces engineers to ask the right questions during design. What breaks here? What happens if the database is down? If the cache fails? If a downstream service misses its SLA? It prompts thinking about fallbacks, retry mechanisms, and graceful degradation before a line of code is written.

How did you run the reliability sprint itself with such a small team?

We got all the engineering leads in a room and went through the entire system. We mapped every problem we could see, then ranked them on two axes: severity (how acute is this?) and lift (how hard is it to fix?). That gives you a clear prioritized list: critical low-hanging fruit goes first, and you work down from there. We started executing against that list and got through the core work in under two quarters.

The other thing we did: that group of leads didn't disband after the sprint. We formalized it as a recurring reliability team that meets every six weeks, reviews the backlog, and keeps reliability work flowing through the pipeline alongside feature work, with equal weight, sometimes more.

What did the architectural work look like? Where did you go after the initial sprint?

The central organizing principle behind every reliability project is blast radius reduction: if something breaks, how do we limit what goes down with it?

When we started, Twingate ran on a single Kubernetes cluster, highly available across multiple zones, but a single backend cluster in one Google region. One of the first major efforts was moving to active-active across multiple regions with automatic failover. That protects against cloud vendor issues. Even outside major outages, cloud providers have small regional incidents all the time. Being multi-region means we can automatically shift traffic away from a troubled region and investigate without risking customer traffic.

But that doesn't solve self-inflicted problems. If we deploy a bad code change, it goes to all clusters simultaneously and breaks all of them. So the next step, what we've been working on for the past year, is sharding production. Instead of one production environment, we'd have many, and new deployments would roll out gradually across shards. A bad deploy would only impact one shard and a fraction of customers, not everyone at once.

There's a bonus: sharding also enables data locality. We can have a Europe shard where customer data stays in Europe, or a dedicated shard for specific compliance requirements. That turns a pure reliability project into something with sales and product value, which makes it much easier to get the whole company behind it.

You're doing this with 30 engineers total, 3 on DevOps. Is that a disadvantage?

I actually see it as an advantage, especially for reliability. When you're lean, you're forced to automate everything. No one has access to production, not even me. If there's an issue, you can't just SSH in and patch it manually. You fix it in Git, open a PR, push it through the automation. In the short term that's painful. In the long term, every change is documented, there's a history of what happened and why, and the system becomes more resilient because humans can't take shortcuts.

Small teams also have simpler communication structures. Decisions move faster, context stays shared. You have to be thoughtful about every hire and every system design choice because you can't throw people at problems.

How has AI changed on-call and incident response for you?

We recently adopted an AI SRE tool that automates the initial triage step. When an alert fires, you mention the AI and it already has access to the runbooks, logs, and Kubernetes. It does an initial diagnosis and returns findings. It's not always right, but when it's wrong, you immediately know you need to page a human. When it's right, you've saved the on-call engineer significant investigation time.

Before this, we had a Notion page with runbooks. Something breaks, you search for the error, find the relevant runbook, follow the steps, ping the right person if needed. That still worked, but people would skip the Notion and go straight to pinging someone. The AI automates the runbook lookup and initial investigation, which is the part that was most inconsistently followed.

On code, it helps us write more and review faster.

That raises a concern: if AI handles the easy incidents, do engineers lose the knowledge they'd build from working them?

I think production incidents will always need a human in the loop, regardless of AI capability. But the broader question, when to let AI automate versus keeping a human involved, comes down to two factors: judgment density and risk.

When judgment density is high and risk is high, you always want a human in the loop. Production incidents sit squarely in that quadrant. If the AI makes a mistake and takes the system further down, that's catastrophic. So even if AI can triage and propose a fix for a trivial incident, a human should still review before anything is applied.

When judgment density is low and risk is low, automation makes sense. Updating dependencies, fixing CVEs: the test suite will catch regressions, the blast radius of a mistake is small, and the repetition cost of having a human do it manually every time is high.

What's your policy on AI-generated code and PR reviews?

The foundation before policy: whoever owns a piece of code owns it regardless of how it was written. If you assign our bot to a Linear ticket and it opens a PR, you own that code. You are accountable for what it does in production.

Our policy: every PR requires the code owner to review it first, not the AI that wrote it, and then a second human reviewer. The second reviewer isn't just looking for bugs. The primary goal is that someone else knows that piece of the codebase. If you're on vacation, if you leave the company, we need people who understand the logic and can maintain it. That organizational knowledge transfer is the reason PR review exists, and AI-generated code doesn't change that.

There's a gap between what you see on Twitter and what's actually happening in production. What's your read?

Most of what you see, 20 agents running autonomously with nobody looking at the code, doesn't survive contact with a real production environment. The open source demos don't work when you try to install them. The code doesn't match the documentation. There's a difference between shipping a proof of concept and shipping production-ready software.

We're not at a stage where autonomous agents can build entire product features without human review and ownership. Where agents work well right now is on lower-judgment tasks: dependency updates, CVE fixes, changes where the test suite is a reliable safety net and the blast radius of a mistake is contained. For anything else, I treat AI like an intern or junior engineer. You can get a lot done, but you have to manage them closely. They'll write more code than necessary, duplicate logic, and sometimes break a test and just update the test to pass. The guardrails themselves aren't safe from the thing you're trying to guard against.

We Stopped Shipping Features for a Quarter — Here's Why It Saved Us