“Human in the Loop” is the new “Pay Attention” for AI Ops

The phrase “human in the loop” is frequently invoked these days to suggest that we can work much more efficiently with AI and still maintain our model of human accountability. Throughout the twentieth century, various efficiency improvements in industry sped things up, made things more complex, and it was assumed that people could be held responsible as long as they “pay attention.” The former is basically a re-packaging of the latter, and neither one is good strategy.

Two Incident Management Odors

Once upon a time, when a software system produced an undesirable outcome, the attribution of guilt wasn’t assigned to the system itself but to “human error.” This allowed us to preserve the overall operation of the system, which was good, because changing the system was difficult and unpredictable. It wouldn’t be disturbed with operational downtime, inspection, or redesign. The much more malleable and expendable humans who used the system could accept blame and be handled according to the magnitude of their offense.

Many books, frameworks, and PhD theses confidently delineate the types of “human error” and when and why they show up. We aren’t interested in re-hashing those frameworks, but they use descriptors like: unintentional, omission, competency, procedural, execution, planning, slips, lapses, perceptual, etc. – to give you an idea of what some ontologies look like.

In Incident Analysis, we don’t really have time for all that classification, so “human error” narratives generally imply that someone either didn’t know what they were doing, or they weren’t paying attention to the thing they should have been paying attention to.

In the present day, enlightened by blameless postmortems and other practices adjacent to Resilience Engineering, we recognize that didn’t-know-what-they-were-doing and weren’t-paying-attention aren’t very useful if we want to improve system reliability. Changing the system is difficult, but it is the only way to improve the system. This gave rise to two Incident Management odors. Much like certain programming practices leave behind “code smells,” didn’t-know-what-they-were-doing and weren’t-paying-attention often leave behind an odor. That odor shows up in action items that rhyme with: needs more training, and pay more attention.

Anytime a safety incident occurs, in software or any other industry, notice whether the response includes some phrase about training or re-training the people who are proximate to the incident. This is an Incident Management odor. It is possible that indeed more training would benefit the system, but it’s more likely an admission by the system owner that it would take too much effort to make the system operate the way human operators want and expect it to, so the flexibility of the human operators will be further strained and exploited to absorb that disparity.

Paying Attention

Let’s specifically look at the second odor: paying attention. First, we need to acknowledge that “paying attention” isn’t a real thing. There is no currency in which someone can pay, no quantifiable link to neurotransmitter volume, no corresponding synaptic frequency. When we use the phrase, we mean something closer to, “try to notice things that are important.” But how can you tell someone to notice something important, if their brain has to determine whether or not it is important before they notice?

Actually, our brains can do this. Sort of. It’s why you can hear your name being called in a crowded room. We cannot listen to every conversation simultaneously and decide in realtime if they are relevant to us. Instead auditory ‘data’ goes in, and if it roughly matches a pattern (like your name), we pass it on up for processing. Otherwise we drop that data to the floor and never process it.

So that is like a pattern-matching filter for noticing. A human cannot ‘pay’ more attention. There is no reserve of attention hiding in the safe that can be squeezed for additional liquidity. There is no fee you can pay to extend your organic token limit. You simply notice things because they match a pattern, or not.

Whether or not someone notices a thing can be studied. I won’t spoil it for the reader, but there are some real bangers out there demonstrating that focusing attention on one pattern can make a person essentially blind to what would otherwise be a very easy second pattern to notice. Elevated excitement and depressed mood and activity both inhibit our ability to notice. Exposure to the context also has a bearing on what you notice: prolonged exposure to a set of factors can make it difficult to notice small aberrations, while being in a new situation can cause a person to notice too many things, many of which are inconsequential.

Given the myriad factors that can influence whether or not someone notices something in realtime that in hindsight we can all agree is obvious, does it make sense that we rely on human brains to do the noticing? Ideally, no. There are things we can do to support optimum noticing ability: make sure your on-call people are well-rested, expose people to system failure so they can build up their pattern-matching capability, avoid additional stimulation [like management watching over your shoulder] during an incident response. These things can help, but it still leaves a highly fallible faculty responsible for critical system operation.

The Problem is also the Solution

We can’t simply pump more juice into a human cortex (or out of it) and get better system results. What’s to be done, then? When you hear anything that presents a whiff of the weren’t-paying-attention odor, ask the following three questions:

The people who weren’t paying attention were doing something. Whatever they were doing, why did that make sense to do at that time?
How did the system produce what we don’t like, depending on the people to notice what they were not-noticing?
How can the system be changed so that it is better aligned to produce what we want, given what makes sense to the people operating it?

The first question flips the script on the antiquated “human error” way of thinking. The people operating the system are also users of the system. Understanding your users is the first step in any design. You need to meet them where they are.

The second question sketches the disparity between what the users think/believe/do, and how the system actually operates. The users [operators] wanted the system to produce good stuff [business continuity] and instead they got bad stuff [an incident.] Understanding how the system produced undesirable output gives us leverage for change.

Now, if you stop here, you’ll end up with another signal that you want the users to pay attention to, and maybe you’ll add an alert and then add a dashboard and then wish and hope and dream that the users will know to look up that dashboard when they run into a similar problem again. Recall that human brains don’t have a spare token stream for attention that they can tap into when the time is right. Giving the users yet another thing to pay attention to will make the system less reliable, not more reliable.

[This is not an argument against either alerts or observability, both of which are absolutely required tools. It’s an argument against using those tools as a bridge to a utopia where the user will surely notice the right thing at the right time. That would be nothing more than wishful thinking.]

The third question brings us to the solution. If the users got bad stuff instead of good stuff from the system, then you can blame the users or you can blame the system. One of those things will get you nothing in the long run, and the other can lead to significant improvement.

Design a better system. Yes, it’s more work and requires more resources to redesign and change the system than just putting more pressure on the squishy-brained humans. It also leads to a better system.

What’s Past is Prologue

All of which brings us to “human in the loop” (HITL). It’s a popular phrase right now for two reasons. First, “reinforcement learning from human feedback” (RLHF) has proven to be a useful mechanism for improving AI performance, and while technically that is a form of HITL, it isn’t the load-bearing operational task connotated by the phrase. Second, we are increasingly invoking HITL because it once again allows us to pass the buck for poor system design onto the user.

Before we even get started, let’s acknowledge that automation bias makes humans particularly bad at HITL when AI is involved, because we know that most humans will quickly learn to click “approve” or “continue” without judgment. This is human nature. We know this about ourselves.

Specifically, we see HITL invoked as a policy in software engineering because that industry is being eaten by AI right now. Let’s talk about when agents or LLMs write code and produce a bug, or cause an outage. HITL tries to solve the accountability problem by designating a human to be in the loop, and thus responsible for the non-humans in that loop. The way we currently deploy agents, responsibility is an intractable problem.

Let’s bring back our questions and apply it to HITL.

The people who weren’t paying attention were doing something. Whatever they were doing, why did that make sense to do at that time?

The humans are directing agents to write code to satisfy business requirements. Of course that makes sense. There’s a tension here, because agents can write, read, modify, and re-write code much faster than any human can possibly see. They can also build software in ways that humans can’t possibly understand. AI can change word choice and order to ‘fingerprint’ text so that it knows it was written by an AI, but humans can’t detect the fingerprint. AI can do the same with code. It can make code work in ways a human will simply never think to notice.

How did the system produce what we don’t like, depending on the people to notice what they were not-noticing?

The humans continue pushing the agents to write code, because the business rewards the increased velocity immediately. But the humans cannot track the increased risk of failure, because they can’t know what they don’t know about the code they are pushing to production. In the volume of code being produced, it just isn’t possible to notice bugs. So this system produces bugs, relying on humans to notice, when they can’t possibly notice.

How can the system be changed so that it is better aligned to produce what we want, given what makes sense to the people operating it?

This is the hard part. Redesign the system. Specifically, redesign the system so that HITL isn’t the faux-crutch for reliability. There are AI-centric ways to do this, bringing in more agents to track intention, context alignment, write tests, etc. There are qualitative ways to do this, measuring operator experience, customer experience, etc. And there are empirical ways to do this, building continuous verification frameworks to prove that the system output is the desired output within given parameters.

It’s not that we can’t solve this problem, it’s just different, and difficult. But one thing is for sure: HITL isn’t going to get us there. “Human in the Loop” is just the new “Pay Attention.”