The Intermediary –- June 2026 - Flipbook - Page 44
T E C H N O L O GY
Opinion
Human-in-theloop, or meaningful
oversight?
W
hen the story
broke about a
robotaxi being
swept away
aer driving
into a flooded
river recently, the dominant reaction
was predictable: this is why AI isn’t
ready to drive cars on its own.
The incident involved a Waymo
vehicle in San Antonio. It detected
the flooded road, slowed down – and
then drove into it anyway, before
being swept into a creek. Waymo
subsequently recalled 3,800 vehicles.
Fair enough. But my first thought
was actually the opposite, because it’s
not hard to find examples of humans
doing exactly the same thing.
In some cases,
the human becomes
less effective precisely
because the AI becomes
more reliable”
In 2023, a driver in Murcia followed
their sat nav down a boat ramp and
into a lagoon. Warning lights flashing.
Boats visible. They kept accelerating.
In Hawaii, two tourists drove a hire
car straight off a boat ramp into a
harbour while smiling and following
GPS directions – windscreen wipers
still going as the car tipped into
the water.
In Queensland, three students drove
a rental car into the sea trying to reach
an island the sat nav assured them was
accessible by road. It wasn’t.
In all these cases, the system gave
bad advice – and humans trusted it
more than their own eyes. So, when
42
The Intermediary | June 2026
we ask whether AI is ‘ready’, we
should also ask: ready for what? Ready
to be perfect? Clearly not. But humans
aren’t perfect either.
Waymo’s own data suggests its
vehicles are involved in significantly
fewer serious injury crashes than
human drivers. So, the uncomfortable
question might actually be: if AI
systems make fewer mistakes overall,
are we prepared to accept that they’ll
still sometimes make mistakes
humans wouldn’t? I suspect most
people would say yes, in principle.
But emotionally, I’m not sure we
react to machine mistakes the same
way we react to human ones. And
that’s where this gets interesting.
Because the harder problem isn’t
‘can AI can make decisions?’, it’s ‘what
happens to humans when AI gets good
enough that we stop questioning it?’
Too much trust
There’s a well-documented effect
called ‘automation bias’. This explains
the tendency for people to trust
recommendations or decisions made
by automated systems too much,
especially when those systems are
usually correct, instead of actively
checking the system’s output.
Over time, this reduces critical
thinking and independent
verification. When systems are right
most of the time, humans gradually
stop examining the decisions. Not
because they’re lazy. Because their
brain adapts.
Oversight becomes habit. Habit
becomes rubber-stamping. And
eventually, when the edge case
arrives, nobody is really paying
aention anymore.
The Murcia driver probably
wasn’t reckless. They had just
been conditioned by thousands of
correct directions to trust the system
MARK GILLIS
is technology director at TAB
automatically. And honestly, most of
us probably would.
This is something I spend a lot
of time thinking about – not in
robotaxis, necessarily, but in financial
services, where AI is increasingly
involved in underwriting,
monitoring, compliance, and
customer interactions.
A phrase you hear constantly
is ‘human-in-the-loop’. But I’m
increasingly convinced that
having a human involved is not the
same thing as having meaningful
human oversight.
In some cases, the human becomes
less effective precisely because the AI
becomes more reliable.
That creates some difficult design
questions. How do you stop oversight
becoming passive approval? How do
you keep people critically engaged
when the system is usually correct?
Should AI be designed to surface
uncertainty more aggressively, even
at the expense of user confidence?
And at what point does ‘human
accountability’ become mostly
performative?
At TAB, we think a lot about where
AI should sit in decision-making
workflows. This is a question the
wider industry needs to wrestle with:
not just what models are capable of,
but how humans interact with them
over time.
Personally, I think that
interaction layer may end up being
more important than the models
themselves. As AI systems become
more capable, that is what good
human oversight actually look like
in practice. ●
