
When should AI be allowed to approve a pull request?
We examined 3,661 pull requests to assess our human review process and propose which changes could qualify for AI approval.
When should AI be allowed to approve a pull request?
On this page
Nine in ten PRs on our main repo were approved without a single human review comment. In almost half, the last approval was attached to an earlier commit than the PR’s final head.
Now that agents write most of our code, I wanted to understand what happens between a PR being ready for review and someone approving it, and whether that approval still gives us what we expect.
In October 2025, Rootly’s main repository merged 356 pull requests. In October 2026, it merged 2,053. Across the whole GitHub organization, the number went from 530 to 4,187.
And shipping keeps accelerating. In just two months, from August to October 2026, monthly approvals per person jumped from roughly 28 to 70. But we had only two more engineers approving PRs.
Our AI reviewer already reads the code. I want to let it approve low-risk changes without waiting for a human. Which changes could we trust it to approve, and what would it miss if we did?
Across the whole GitHub organization, monthly merged PRs rose from 530 to 4,187 between October 2025 and October 2026.
The cost of waiting for review
I pulled every PR merged into master between August 1 and November 5, 2026: 3,661 of them, written by people or by agents on their behalf, and approved by someone other than the author.
The median wait from ready for review to first human approval was 1.7 weekday hours. The slowest tenth waited more than 39 weekday hours. Added across PRs, that is 2,316 weekday PR-days in the queue. Several PRs can be waiting at once, so that total isn’t engineering time lost.
When a PR stalls, people ask in chat. I found 570 merged PRs that someone asked about in public channels. A quarter were approved within five minutes of the ask, almost always with no comment. The asks usually mention size and speed. Risk rarely comes up.
What a human approval tells us
The finding that bothers me most is the 43.3% of PRs approved on an earlier commit than the final head. We compared the latest approval’s commit with the PR’s final head before the squash merge. We don’t dismiss stale approvals after a push, so a new commit can reach master without an approval attached to it.
Written feedback was sparse too: 90.5% of PRs were approved without a human review comment, and 1.1% received a human change request.
We also checked how quickly silent approvals arrived. For 24.4% of PRs, the interval from ready for review to approval was shorter than a read-through at 500 changed lines an hour. SmartBear’s study of a Cisco team recommends inspecting fewer than 500 lines an hour. Our interval includes queue time; it doesn’t measure how long anyone spent reading.
The fast approvals included high-risk PRs. PRs with a written review needed a follow-up fix within seven days about as often as those without one. None of this tells us how carefully someone read a particular diff. A silent review can be careful, but the approval record gives us little to inspect afterward.
A second repo, with 178 PRs in the same window, showed a similar pattern: 91% of approvals had no human comment. Human change requests were three times as common and the approval wait twice as long, but silent approvals still dominated.
Our AI reviewer already grades PRs
Our AI reviewer runs on every PR to the main app. It scores each PR from 1 to 5, states a risk level, and leaves inline comments. 98% of PRs carry a score.
The median time to its first score is 6 minutes, compared with 1.7 weekday hours to human approval. Those are different milestones, but the bot is already giving authors feedback while they wait.
On 552 PRs, an initial score of 3 or lower rose to 4 or 5 after further pushes. Humans left review comments on roughly 350 PRs and formally requested changes on about 40. Those counts describe different activities, not defects found or fixed.
The bot also reviews again after every push. Our human approvals stay valid after new commits. That is one concrete reason I want to give the reviewer more responsibility.
My AI approval proposal
I’d let the reviewer approve a PR only when all three conditions hold:
risk:low, confirmed by the reviewer- a review score of 5 on the current head commit
- no sensitive paths touched, including migrations
Everything outside the rule, including a truncated diff, failed review or incomplete result, would wait for a human. A score of 4 would be a later expansion, after evaluation.
The risk assessment needs an independent check. 69% of risk labels are applied under the author’s account, mostly by the coding agent that opened the PR, seconds after opening it.
On the 1,865 PRs where both the author and reviewer stated a risk level, about 51% of the full sample, they agreed about 77% of the time. The reviewer rated risk higher about 22% of the time and lower about 2%. When it raised a risk:low PR to medium, that PR needed a follow-up fix more often. One reverted PR that would have qualified on the author’s low-risk label alone was in that group.
Where both sides stated a risk level
1,865 PRs with both author and reviewer assessments.
The policy would keep the higher assessment. A label a person set would never be lowered. The reviewer would replace its own earlier assessment when it reviews a new commit. A mistaken escalation costs another human review; a mistaken low-risk classification can let a defect through.
What the historical check missed
I argued in May that the small PR rule won’t survive AI, because line count is a poor proxy for risk when an agent writes the diff. But our historical check found a miss that makes this choice harder.
That check tested a broader rule: a score of 4 or 5, without sensitive-path exclusions. Nineteen PRs qualified despite human pushback. The AI fully raised the same issue once and partly raised four. It missed three substantive issues. The other eleven were style, preference, or not about code.
Two substantive misses touched a migration or authentication code, so the proposed exclusions would send them to a human. The third was large and fell outside those paths. A size cap would have kept it with a human. Starting without one means accepting that this kind of miss may recur.
The check doesn’t establish how the narrower score-5 rule would perform. Nor do the two PRs we traced as causes of incidents settle the question. Both were risk:high, touched sign-in code, and had a human approval with no comment. Our exclusions would have sent both to a human, just as the current process did.
The safeguards I’d require
We build incident management. When our customers are down, Rootly is how they page their on-call and run the response. If we break, we break during someone else’s worst hour.
Before enabling automatic approval, I’d require:
- Human ownership of the policy. The model supplies risk, score and findings; a deterministic rule checks eligibility. Changes to that rule and the review instructions require human review. The coding agent that wrote a PR cannot approve it.
- Human review for sensitive paths. Migrations, authentication and permissions, billing, CI and deploy config, and dependency changes. No score overrides those exclusions.
- Approval tied to a commit. A new push invalidates the bot’s approval and requires a fresh, complete review.
- The existing deploy gates. CI, staged rollout and feature flags apply to auto-approved PRs too.
- A way to detect misses and stop. Track reverts, follow-up fixes within seven days, incidents linked to auto-approved PRs, and disagreements found through monthly human sampling. Keep a human-review label and a single off-switch. A serious incident linked to an auto-approved PR suspends the policy until we understand why.
What other teams have tried
The pressure on review isn’t unique to Rootly. Researchers at Carnegie Mellon and Stanford followed 802 developers and about 196,000 PRs at one company with an AI productivity mandate (paper, Semgrep’s summary). PR volume grew 3.1x, the reviewer pool grew 1.5x, and human-written review comments fell from 39% to 21% of PRs. The median reviewer’s silent approvals roughly doubled. Revert rates didn’t rise as written feedback declined, so watching reverts alone wouldn’t reveal that shift.
Faros AI found a similar bottleneck across more than 10,000 developers in 2025: high-adoption teams merged 98% more PRs and had 91% longer review times.
Some teams have already enabled AI approval. Rewind set 90% human agreement as its rollout threshold and reported about 99% agreement on bot approvals that also received human review, with around 40% of PRs classified for auto-approval. It calls those early numbers directional. It also routes diffs over roughly 10,000 lines to a human because of review capacity.
Ona reported median time to first approval falling from almost three hours to under four minutes, alongside two other workflow changes. Intercom reports over 19% of PRs auto-approved. Its AI-authored backend code had a 0.53% revert rate versus 5.39% for human-authored code, though authorship doesn’t isolate the effect of automatic approval.
One difference matters for our proposal: Ona keeps a human clicking merge, and Rewind requires one for agent-authored PRs. We would allow a qualifying PR through the deploy pipeline without a human merge step. Their results don’t establish that our policy would be safe.
What we need before enabling it
We haven’t started shadow mode. That comes first: record what the bot would approve while keeping human approval required.
We need the eligible share, human disagreements and substantive misses for the score-5 rule, alongside the PR count. We also need to see where author and reviewer risk assessments disagree beyond the 51% of PRs with paired assessments in this analysis, and whether large changes keep producing misses outside the sensitive paths.
The control needs assessment too. The AICPA Trust Services Criteria include change management under CC8.1. Our current process requires review by someone other than the author. We haven’t taken this proposed replacement through an audit.
For each merged PR, the evidence should show who approved, whether the approval covered the final head, whether required checks passed, and who authorized any bypass. For AI approvals, it should also retain the policy version, reviewer identity, score, findings and risk reasoning, with named owners for classification, overrides and monthly sampling. Rewind’s control mapping is useful input, but its human merge requirement differs from our proposal.
The historical analysis gives me reasons to test this rule, not enough evidence to turn it on. Before rollout, we need to decide what results would justify it, including which misses would stop it. A score of 4 and expansion to other repos come later.




