Facial Recognition False Arrest: Robert Williams, ACLU, and Police Tech

A man in St. Louis spent 17 months in jail because a computer looked at a blurry photo of someone wearing a mask and said, "close enough." He wasn't convicted. He wasn't even properly investigated, according to the lawsuit he's now filed. He just got unlucky enough to resemble a low-quality image that an algorithm flagged — and then everyone downstream treated that flag like it was a fact.

Here's the thing that should bother you, whether or not you've ever been anywhere near a police station: this isn't really a story about one unlucky guy. It's a story about how easy it is for all of us to hand our judgment over to a machine the second it sounds confident.

What Happened in the St. Louis Case

The case, first reported by Biometric Update, centers on a man named Gatlin who was arrested and held for 17 months after facial recognition software was used to compare a low-quality, partially obscured photo of a suspect against a database of faces. The lawsuit doesn't just blame the police department. It names the technology itself, arguing that a defective system, combined with sloppy police work and inadequate training, is what actually put an innocent man behind bars.

That's a big deal legally. Up until now, most of the accountability conversation stopped at "the officers should have double-checked." This lawsuit is asking a harder question: what if the tool itself is part of the problem, not just the humans using it wrong? This article is part of a series — start with Your Rewards Points Just Became A Bribe For Your Face.

To understand why that matters, you need to know how these systems actually behave outside of a lab. Under perfect conditions — clear, well-lit mugshot-style photos — leading facial recognition systems get it wrong about 0.1% of the time. That's genuinely impressive. But real life isn't a mugshot booth. When you feed these systems "in the wild" images — grainy security footage, a masked face, bad lighting, a bad angle — the error rate jumps to 9.3%, according to research cited by the Center for Strategic and International Studies.

Read that again. A 90-fold jump. And here's the part that should really land: an officer looking at a "match" on a screen has no way to know which situation they're in. The software doesn't say "hey, this was a grainy photo, treat this with extra suspicion." It just spits out a name and a confidence score, and confidence scores feel like facts even when they're closer to educated guesses.

Why Facial Recognition Became a Problem

The vendor itself, as far back as 2020, recommended that police only rely on matches when the confidence score hit 95% or higher. Sounds responsible, right? Except a 95% confidence score is a statement about how sure the software is that it found a good match in its own database — not a statement about whether the person is actually guilty of anything, or even whether the underlying photo was clear enough to trust in the first place. A system can be very confident and very wrong at the same time. Ask anyone who's ever been "very confident" they parked in aisle C at the mall.

Most police departments know this, at least on paper. Standard guidance tells officers that a facial recognition result is not a positive identification — it's a lead, one piece of a bigger puzzle that needs actual investigation before anyone gets arrested. That's the theory.

"When facial recognition is used for investigation, most investigators know that the vast majority of matches will be false, and the point is to return a broad range of potential candidates of whom the vast majority will be discarded." — Expert analysis via Biometric Update

That's the theory. In practice, according to the Gatlin case, a poor-quality match became "investigative momentum." Momentum became probable cause. Probable cause became 17 months in a cell. Somewhere along that chain, the warning label got ignored — not because anyone was evil, but because a computer sounding certain is a psychologically powerful thing to argue with. This is called authority bias: when something looks technical, official, and confident, we stop questioning it, even when questioning it is exactly our job. Previously in this series: That Prove Youre 18 Pop Up Is About To Be Everywhere And Fak.

How Facial Recognition False Arrests Happen

If this were an isolated incident, you could chalk it up to bad luck. It isn't. The American Civil Liberties Union has documented more than a dozen wrongful arrests tied to police reliance on facial recognition software. Separately, the Innocence Project has tracked at least seven confirmed cases of mistaken identity from these systems — six of them involving Black people who were wrongly accused.

Of the ten publicized false arrests connected to facial recognition matches in the U.S. so far, nine involved Black people. That's not a coincidence and it's not a footnote — it's the pattern. These systems have historically performed worse on darker skin tones and on women, which means the "9.3% error rate" isn't spread evenly across all of us. Some people are carrying a lot more of that risk than others.

That third point deserves a beat. You'd think a human double-checking the computer's answer would catch the mistake, right? Not necessarily. The ACLU has pointed out that when a face recognition system flags someone, that photo tends to look more like the actual suspect than the other random photos put in a lineup — because that's the entire point of the software. So a witness is nudged toward picking the machine's pick, even if the machine was wrong from the start. The "independent check" isn't as independent as it sounds.

What You Can Actually Do With This

Okay, so here's the part where I stop being alarming and start being useful. If you've ever wondered whether a photo, a profile, or an "identity match" claim about you or someone you know is actually accurate, that worry is completely reasonable — it's the exact question this entire category of technology exists to answer, and answer badly, if nobody's checking its work. Up next: Digital Identity Verification Three Layer Process Explained.

The one thing worth remembering, and actually using, is this: a match score or "confidence percentage" is never the end of a story — it's the start of one. If you, or someone you love, is ever told "the computer identified you" as the reason for a serious decision — an arrest, a firing, a frozen account, a denied benefit — the right response isn't panic, it's a very specific, very calm question: what independent evidence do you have besides the match? A time-stamped receipt. A second witness. Phone location data. Video from a different angle. If the honest answer is "just the computer," that's not evidence. That's a hunch wearing a lab coat.

The Real Shift Here

Technology companies will tell you, not unreasonably, that a computer match was only ever supposed to be a starting point for human investigators — not a verdict. That's true. It's also incomplete, because it assumes humans reliably resist the pull of a confident-sounding machine, and the Gatlin case is 17 months of proof that they don't always resist it. The tool didn't put him in jail by itself. But it made the wrong outcome a lot easier to reach, a lot faster, with a lot less friction — and that's exactly the point of a warning like this one.

---

Seventeen months is roughly the same amount of time it takes to raise a puppy into a fully grown dog, or watch a toddler learn to speak in full sentences. That's how much of one man's life got quietly erased because a piece of software was 95% sure of something it had no business being sure about. The next time someone tells you a computer "identified" a person — any person, in any context — the only sane follow-up question is the one nobody in that St. Louis case apparently asked loudly enough: sure, but identified them how confidently, and compared to what?

Why NYPD-Style Police Departments Keep Using Facial Recognition Anyway

NYPD and departments like it argue that facial recognition technology helps them work faster on real cases, and that's not a dishonest claim — the technology can genuinely narrow a huge pile of security footage down to a short list of names in minutes instead of weeks. The problem isn't that police use facial recognition technology. The problem is what happens next: does an officer treat that short list as a starting point for real investigation, or does the confidence score alone become the reason someone gets arrested? The Gatlin case suggests that inside at least one department, the second thing happened, and nothing in the process caught it before an arrest occurred.

How Arrested Suspects Can Challenge a Facial Recognition Match

If you were arrested after a facial recognition match, you have the right to ask, through a lawyer, exactly how that match was generated: what photo was compared, what confidence score came back, and whether any human corroboration happened before the arrest. Courts increasingly want to see that record, because a wrongful arrest built entirely on a face match with no supporting evidence is a much weaker case than one where officers also had a witness, a location record, or physical evidence. Being arrested on the strength of a computer's guess is not the same as being arrested on the strength of proof, and that distinction is exactly what these lawsuits are trying to force into daylight.

What Wrongful Arrests Teach Us About Trusting the Technology

Wrongful arrests tied to facial recognition keep sharing the same shape: a low-quality photo, a confident-sounding score, and a police process that skipped the slow, boring work of actually confirming identity before making an arrest. That pattern isn't a coincidence, and it isn't limited to one city or one department. Every wrongful arrest tied to this technology adds pressure on lawmakers and courts to require independent verification — not just a second glance at the same computer output — before facial recognition technology can justify taking away someone's freedom.

The Role of Police Training in Preventing These Errors

Police officers are typically trained that a facial recognition result is a lead, not proof, but training on paper and training in practice are two different things. When a department treats a match as strong enough on its own, officers under time pressure often skip the additional investigative steps that policy actually requires. Better police training would mean building in a mandatory pause: no arrest based on a facial recognition technology match alone, without at least one independent piece of corroborating evidence gathered and documented first.

Recognition technology like this isn't going away, and pretending otherwise won't help anyone. What can change is how much weight a recognition technology output is allowed to carry inside a police investigation, and whether an arrest can happen before a human actually verifies the machine's guess. Facial recognition technology works best as a filter, not a finish line — and every wrongful arrest on record is what happens when a department treats it as the finish line instead.

Privacy advocates have pushed for laws that would limit how police can use facial recognition technology in the first place, arguing that the technology's error rate on real-world images is too high to justify the risk to people's rights. Some cities have passed rules requiring public disclosure whenever police relied on facial recognition technology to develop probable cause for an arrest, so that defense attorneys and courts can actually examine how the match was made. Surveillance systems that feed these facial recognition databases raise a separate but related concern: the more footage a city collects, the larger the pool of images an algorithm can misread. None of this requires banning the technology outright, but it does require treating a computer's guess with the skepticism any guess deserves under the law.

Robert Williams is the name most people associate with the first widely reported case of face surveillance leading to a wrongful arrest, and his case set the pattern that later lawsuits, including the Gatlin case, still follow. Williams was wrongfully arrested in front of his family after a facial recognition technology match, and he was held for hours before anyone seriously questioned whether the match was accurate. His case became a reference point for the ACLU and for journalists trying to explain, in plain terms, how a confident-looking computer output can turn into a real arrest warrant with almost no independent checking along the way.

What made the Robert Williams case so important is that it showed the problem wasn't a one-time glitch. Police facial recognition tools had already been flagged by researchers as less reliable on Black faces, and Williams is Black, which meant his case lined up exactly with what the error-rate research had already predicted. When the ACLU took up his case, it wasn't just advocating for one wrongfully arrested man; it was using his experience to argue that the law needed to catch up with the technology, since existing rules didn't require officers to treat a facial recognition match with the caution the science says it deserves.

Under current practice, an arrest warrant can sometimes be built on little more than a facial recognition lead plus a thin layer of follow-up that looks like investigation but isn't. That's part of why the ACLU and defense lawyers keep pushing courts to require documentation: what confidence score came back, what photo was used, and whether police did anything beyond the computer output before asking a judge to sign off. Without that paperwork, it's nearly impossible to tell the difference between an arrest warrant grounded in real evidence and one grounded in not much more than a lucky-looking algorithm guess.

The law hasn't fully caught up with what facial recognition technology can and can't do, and that gap is exactly where wrongfully arrested people keep falling through. Some states have started passing laws that limit how police facial recognition results can be used, requiring that a match alone can never be enough to establish probable cause for an arrest. Other jurisdictions have no such law at all, which means the same facial recognition technology can lead to very different outcomes depending on which side of a city line a person happens to live on.

Rights groups argue that this patchwork of law is part of the problem, not a side issue. If a wrongfully arrested person in one state has a clear legal remedy while a wrongfully arrested person in another state has almost none, then the technology's risks aren't just about accuracy — they're about whether the law protects people equally once the software gets it wrong. Advocates for reform want a baseline law nationwide: no arrest warrant issued on facial recognition alone, full disclosure to defense attorneys, and a documented human review before police move forward.

It's worth being precise about what "incorrect" actually means in this context, because the word gets used loosely. An incorrect facial recognition match doesn't mean the software malfunctioned in some obvious, detectable way. It means the algorithm did exactly what it was built to do — scan a database and return its best statistical guess — and that guess happened to point at the wrong person. The system doesn't know it was incorrect, and if nobody checks its work, nobody else finds out either until real harm has already happened.

That's why an incorrect result is so dangerous compared to an obvious computer error. A crashed program or a garbled printout gets noticed immediately. An incorrect facial recognition match looks exactly like a correct one on the screen: a clean name, a clean photo, a clean confidence score. The only way to catch an incorrect match before it becomes a false arrest is to build in the kind of independent verification that the ACLU, the Innocence Project, and now the Gatlin lawsuit are all pushing courts and departments to require.

Facial recognition technology sits at the center of all of this, and it's worth restating plainly: the technology itself is not illegal, and using it isn't automatically reckless. What turns facial recognition technology into a false-arrest machine is the absence of a law requiring real verification, combined with police culture that treats a confidence score as good enough. Robert Williams, the Gatlin case, and the dozen-plus wrongful arrests the ACLU has documented all point at the same fix: better law, better training, and a permanent rule that a computer's guess is a lead, never a conviction.