How Does Facial Recognition Work: 512 Numbers, Wrong Arrests

Somewhere right now, a computer could describe your face without storing a single picture of it. It would use a list of roughly five hundred numbers. And when it compares your face to someone else's, it never actually recognizes you. It just measures the distance between two lists of numbers.

If you've ever been photographed at an airport, a

If you've ever been photographed at an airport, a store, or a traffic light, your face may already have been turned into numbers like these. That can feel unsettling, and that feeling makes sense. But most of that fear comes from not knowing what the machine is really doing. So in the next few minutes, you'll learn how a face becomes math. You'll learn why a system called ninety-nine percent accurate can still point at the wrong person. And you'll see exactly where the real decision gets made. So how does a face turn into numbers in the first place?

First, the system finds your face and measures how it's arranged. How far apart your eyes sit. The curve of your jawline. How deep the bridge of your nose sits compared to your cheekbones. Then it compresses all of that into a fixed list, usually between a hundred and twenty-eight and five hundred and twelve numbers. Engineers call that list a vector. It's basically a mathematical description of your face's geometry. It isn't a photo, and it isn't pixels. It's a pattern.

Why not just use fewer numbers? Shorter lists run faster, but they start to blur different people together. Around five hundred and twelve numbers gives the system enough room to keep people apart as the database grows. For you, that means the thing stored about your face is a set of measurements, not your selfie.

Once those numbers exist, matching begins

Once those numbers exist, matching begins. Picture a library so enormous that no human brain could picture its full size. Every book gets squeezed down into a single dot on a giant grid. Books by the same author land close together. Books by different authors land far apart. Your face is one of those dots. A match is simply a question. How far apart are these two dots? The system measures a straight line between them. If that distance falls under a cutoff, it calls a match. That cutoff is called the threshold, and a person picked it. Set it too loose, and strangers start counting as matches. Set it too tight, and the system misses the right person entirely.

Which brings us to that phrase you've probably heard. Ninety-nine percent accurate. It sounds like one mistake in a hundred. People hear it that way because that's how percentages work in daily life, like a test score or a weather forecast. But those numbers usually come from one-to-one tests. Two photos. Same person, or not? Police searches often work differently. They upload one photo and search it against millions of faces. A one percent error rate across ten million faces could mean about a hundred thousand wrong faces clearing the bar. Small mistakes, multiplied by millions.

According to N.I.S.T., the U.S. agency that tests these systems, the top algorithms perform remarkably well in the lab. Its ongoing testing covers roughly two hundred algorithms and photos of more than eight million people. On clean, high-quality visa photos, the best systems falsely match strangers only about once in ten thousand comparisons. Real searches look different. In one search of twelve million photos, described in a National Academies report, an algorithm did find the right person. But it ranked fifteen strangers as more similar first. The true match came in sixteenth. For an investigator, that's fifteen wrong leads. For everyone else, it's the chance that your face is one of those fifteen.