A 99.7% Accurate Face Search Can Still Finger 3,000 Innocent People — Including You

A face search system can be ninety-nine point seven percent accurate and still point at three thousand innocent people. Not because it's broken. Because that's exactly what the math does when you search a database of a million faces. And any one of those three thousand could be you.

If you've ever had a driver's license photo taken,

If you've ever had a driver's license photo taken, your face is probably sitting in a searchable database somewhere. That's an uncomfortable thing to sit with, and I'm not going to pretend otherwise. But the fear most people carry — that a computer will just wrongly declare you guilty — misunderstands what's actually happening. Today I want to teach you the single most important question to ask about any face-match claim you ever hear. It's a question that changes what those big accuracy numbers even mean. So why does a system that's almost perfect produce thousands of wrong answers?

There are two completely different jobs we lump together under the phrase "facial recognition." The first is one-to-one verification. That's your phone unlocking. The system holds up two images and asks a yes-or-no question. Is this the same person? The second is one-to-many search. That's a detective feeding a blurry security camera still into a database of five hundred thousand mugshots and asking, who is this? Same technology underneath. Wildly different math on top.

Comparing your selfie to your passport photo is like asking whether a fingerprint matches the one already on file. You're testing one guess. Searching a million faces is like asking whose fingerprint is on a doorknob. You don't get an answer. You get a ranked list of maybes. And many of those maybes will look genuinely similar and still be completely wrong.

The numbers

Now the numbers. According to research summarized by the biometrics testing community, one-to-one verification is the easier task by far. Top algorithms miss real matches less than two times in a thousand. But flip to database search and the errors multiply, because every single face in that gallery is another chance to be wrong. Run a system that's ninety-nine point seven percent accurate against a hundred thousand faces, and you generate roughly three hundred false candidates. Scale to a million, and it's about three thousand. The algorithm didn't get worse. The haystack got bigger.

For the person watching a case unfold, that means a candidate list is a starting point, never a conclusion. For the rest of us, it means being in the results doesn't mean you did anything.

The National Institute of Standards and Technology has tracked this improvement for years. Back in twenty ten, N.I.S.T. found the best algorithm could correctly identify someone in a database of one point six million criminal records about ninety-two percent of the time. By twenty eighteen, the leading systems were searching twelve million faces with error rates under two-tenths of a percent. That's a genuine, enormous leap forward. And it still doesn't rescue you from the arithmetic of scale.