That "Accurate" AI Checking Your Face? Regulators Just Called It High-Risk Anyway

Here's something that will probably break your brain a little: the exact same facial comparison software — same code, same algorithm, same accuracy score — can be classified as "low risk" in one situation and "high risk" in another. Not because it changed. Because the situation changed.

That idea seems backwards at first. We're used to judging tools by how well they work. A good thermometer reads the right temperature. A good scale shows the right weight. Surely a good facial recognition system just needs to match faces correctly, right?

Not quite. And understanding why not — really understanding it — will make you permanently smarter about every AI system that touches your identity. Whether that's a hiring platform, a bank's verification app, or a camera at an airport gate.

The Prescription Drug You Never Thought About

Think about how prescription drugs get approved. The same active molecule can sail through as a routine medication in one context and face years of extra trials in another — not because the chemistry changed, but because the patient population changed. A blood pressure drug approved for healthy adults gets re-evaluated entirely when doctors want to prescribe it to pregnant women. Same pill. Different stakes. Different rules.

UK AI regulation works almost exactly like this. The algorithm is the molecule. The use case — what decision it influences, whose life it affects, whether someone can push back — determines the risk category. According to Dentons, one of the world's largest law firms, the UK takes what's called an "outcome-based" approach to AI regulation. Regulators aren't primarily asking "how does this algorithm work?" They're asking "what happens when this system gets something wrong, and can anyone stop it?"

That single shift in perspective — from algorithm to outcome — is the thing most people miss entirely.

The Four Questions Regulators Actually Ask

When a UK regulator, say the ICO (the Information Commissioner's Office — the UK's main data privacy watchdog), looks at an AI system that handles identity or faces, they're not primarily running accuracy benchmarks. They're working through four questions. Each one can change the risk rating on its own. This article is part of a series — start with The Ai Rule That Decides If Your Job Loan Or Face Gets A Hum.

1. Purpose: What decision does this result feed?

There's a massive difference between "this system flags a possible match for a human investigator to review" and "this system automatically bans someone from a venue." Same face match. Completely different consequences. The first one informs a human. The second one replaces one.

Regulators pay close attention to this distinction. Under the EU AI Act — which the UK watches closely even though it has its own separate framework — systems that perform real-time biometric identification (scanning faces in public spaces continuously) are banned outright in many contexts. But a targeted facial comparison used by an investigator who then makes their own judgment? That sits in a different legal zone entirely, as the ICO's AI and Biometrics Strategy makes clear.

2. Data: What kind of personal information is involved?

Not all personal data is treated equally. Your name and email address? Protected, but relatively standard. Your face geometry, your fingerprints, your iris pattern — what UK law calls "biometric data" (the physical and behavioral characteristics that can uniquely identify you as a specific human being)? Those trigger an entirely separate set of legal safeguards called "special category" protections.

Here's the kicker: the UK's Data (Use and Access) Act relaxed some rules around automated decision-making — but explicitly excluded special category data from those relaxations. So if an AI system processes your face to identify you, the rules don't get easier just because the government loosened things elsewhere. They stay strict. Your face geometry is, legally speaking, one of the most protected data types in existence.

3. Decision Power: Who actually decides what happens next?

This is the question that separates a useful tool from a dangerous one. Is the AI making a recommendation that a trained human then acts on — or is the AI's output directly triggering a consequence with no human in the loop?

Regulators call this the "human oversight" question, and reporting from Biometric Update in June 2026 noted that the ICO's upcoming code of practice for AI and biometrics will put human oversight at its center. When a human must review an AI result before anything consequential happens, risk goes down. When the AI result is the decision — when there's no human hand on the wheel — risk goes up, hard and fast.

4. Human Review: Can you challenge the result and get an explanation?

This one surprises people most. It's not enough for the AI to be accurate. The system around the AI needs to let affected people know a decision was made, understand why, and dispute it if it's wrong. According to A&O Shearman's analysis of the ICO's strategy, organizations deploying AI systems that touch biometric data must document their governance — why they're running comparisons, whose data they're using, and crucially, what happens if the result is wrong. If there's no mechanism for someone to say "wait, that's not me" and be heard — that's a high-risk system by definition, regardless of the accuracy score. Previously in this series: A Robot Rejected You For That Job New Law Says You Can Deman.

The Misconception That Trips Almost Everyone Up

It's completely understandable why people assume that accuracy is the main thing. That's how we judge almost every other tool in our lives — does it work? And it's not wrong to care about accuracy. An AI system that matches the wrong face 20% of the time is obviously a problem.

But here's what that framing misses: a 99% accurate facial comparison system is still categorized as high-risk if it makes automated decisions about real people's lives without a human check. Accuracy is a technical property of the algorithm. Safety is a property of the entire system around it — the data collection, the human review step, the explanation you can demand, the appeal you can make.

Think of it this way. A surgeon's scalpel is extremely precise. But you wouldn't call a surgery "safe" just because the scalpel is sharp. You'd want to know: was the right patient on the table? Did a second doctor review the plan? Is there a process for something going wrong? The scalpel's precision is just the starting point.

"The ICO's approach focuses on the results of an activity rather than the specific rules used to achieve those results, emphasizing what is being delivered and holding organizations accountable for results with a focus on areas of greatest potential harm." — A&O Shearman, on the ICO's regulatory approach

That quote is the whole thing in one sentence. "What is being delivered" — not "what algorithm was used." Outcome. Context. Consequences.

Why This Actually Matters for You

You might be thinking: "I'm not a regulator. I'm not building AI tools. Why does any of this change anything for me?"

Because now you know the right question to ask.

Whenever an organization uses AI to check your identity — a background screening service, a bank's onboarding flow, an employer's hiring platform — most people ask "does it work?" The sharper question, the one that actually tells you whether you're protected, is: "What happens if it gets me wrong?" Up next: Roblox Age Verification Kids Apps Privacy Parents.

Can you find out an AI flagged your identity? Can you ask a human to review it? Can you see the reasoning? Is there any record of why a decision was made? Those questions map directly onto the four dimensions regulators use. And if the answer to most of them is "no" — that's a system operating at high risk, whether the organization running it knows it or not.

This is exactly why tools built for professional investigators — the kind used in legal and insurance contexts, for instance — are increasingly designed around these same four pillars. At CaraComp, facial comparison work is built with court-ready reporting and documented human review steps not as extras, but as the architecture. That's not a compliance checkbox. That's what it looks like when someone builds a tool that has genuinely internalized what regulators are measuring.

So here's the thought to sit with: by Q1 2026, EU regulators had already issued 50 enforcement actions totaling €250 million — mostly for AI systems deployed without proper governance around them. Not for inaccurate algorithms. For missing the four things around the algorithm that make it safe to use.

The companies getting fined built tools that worked. They just forgot to build in the part where a human could say "wait."

Next time you hear someone say an AI identity system is trustworthy because it's accurate, you'll know exactly what question they forgot to ask.