When One Neural Network Doing 3 Jobs Breaks Matches

Here's something that should bother you: the same model feature that makes a facial AI better at guessing someone's age can simultaneously make it worse at confirming who they are. Not because the model is broken. Because it's doing exactly what it was trained to do — and those two jobs are quietly fighting each other at the mathematical level.

Welcome to the world of multitask learning (MTL) — one of the genuinely clever ideas in modern AI, and also one of the most misunderstood when people start deploying it in high-stakes contexts. The efficiency story is irresistible. Train one model. Get three answers. Run it on hardware so cheap it fits in your pocket. Recent research published in Scientific Reports demonstrated that an MTL model using MobileNet as its base architecture could achieve 99% accuracy in person identification, 99.3% in age estimation, and 99.5% in ethnicity prediction — all running in real time on a Raspberry Pi, a computer that costs less than a decent sandwich in most cities.

That's genuinely impressive. It's also where the story gets complicated.

How Multitask Learning Actually Works (And Why It's Seductive)

To understand the problem, you need to understand the architecture. In a standard deep convolutional neural network for facial recognition, the early layers learn low-level features — edges, textures, basic shapes. The middle layers combine those into higher-order structures — the curve of a jaw, the spacing between eyes. The final layers make task-specific decisions based on those structures.

Multitask learning hijacks that design in a specific way. Instead of building three separate networks, you build one shared trunk — all those early and middle layers are pooled — and then branch out into separate "heads" at the end, one per task. Identity verification gets its head. Age estimation gets its head. Emotion classification gets its head. They all read from the same shared spine.

The efficiency gains are real. You're doing one forward pass instead of three. Memory footprint drops dramatically. On constrained hardware like the Raspberry Pi studied by Scientific Reports, that difference is the gap between deployable and not-deployable. For certain applications — a smart city sensor that needs to flag approximate demographics, a robot that adjusts its interaction style based on emotional cues — this is a perfectly reasonable architecture. This article is part of a series — start with Deepfake Detection Accuracy Gap Investigator Workf.

But then somebody decides to use the identity head for something evidentiary. And that's where it unravels.

The Gradient Interference Problem Nobody Talks About

Here's the mechanism that matters. During training, each task head generates its own error signal — its own gradient — that flows backward through the shared trunk, adjusting the weights of all those shared layers. Identity verification is trying to pull the shared features one direction. Age estimation is pulling them another. Emotion classification is pulling them a third.

Researchers call this gradient interference, and it's documented extensively in multitask facial analysis literature. The practical consequence is that the shared feature layers get optimized for a compromise between all three tasks — not for any one of them at its theoretical best.

Now consider what each task actually needs from those features. Identity verification is fundamentally about deep structural geometry — the precise orbital spacing between your eyes, the specific angle of your jaw, the width of your nasal bridge. These are features that don't change. They're the same at 25 and at 55. They're the same when you're happy and when you're furious.

Age estimation, on the other hand, depends heavily on surface features — skin texture, wrinkle depth, the slight softening of facial contours over time. Emotion classification reads muscle activation patterns — the micro-movements of the zygomaticus major, the corrugator supercilii, the orbicularis oculi. These are features that change constantly, sometimes second to second.

When you force one model to optimize for all three simultaneously, the shared layers start weighting surface-level features more heavily — because those features are doing a lot of work for two out of three tasks. The deep structural geometry that identity verification actually needs gets partially compressed. Not erased. Just... deprioritized. And that's enough to matter. Previously in this series: How Facial Recognition Accuracy Is Really Measured.

A 2021 study published in Pattern Recognition found that multitask facial models trained jointly on age and identity showed measurable drops in verification accuracy specifically for cross-age comparisons — comparing a face at one age to the same face years later. Think about that for a moment. Cross-age comparison is exactly the scenario investigators face most often: a current surveillance photo versus a years-old ID document, a recent arrest photo against a decade-old database entry. The MTL model's blind spot lands precisely where the investigative need is sharpest.

The Witness Analogy That Makes This Click

Here's an analogy that makes the abstract concrete. Imagine asking a single witness to simultaneously estimate a suspect's age, read their emotional state, and confirm whether they match someone from a lineup photo. Each task is legitimate on its own. But the cognitive load bleeds. A face that "looks angry" gets subconsciously coded as less familiar than the neutral reference photo. A face that "looks older" introduces uncertainty about whether it's really the same person. Professional forensic identification protocols keep these interviews strictly separate for exactly this reason — not because investigators are inefficient, but because cognitive contamination is a known, documented phenomenon.

The multitask model has the same problem. It's just happening in matrix algebra instead of human memory. And unlike the human witness, it won't tell you it's uncertain. It will hand you a confidence score that looks authoritative, generated by a system that was quietly compromised at training time.

"This paper investigates the feasibility of multi-task learning for facial recognition on the Raspberry Pi, a low-cost single-board computer, demonstrating its ability to perform complex deep learning tasks in real time." — Authors, Scientific Reports, Nature

Notice what that framing emphasizes: feasibility, real-time performance, resource efficiency. All true, all valuable. What it doesn't emphasize is what happens when you take the identity head out of that architecture and try to use it as a standalone verification tool in an evidentiary context. That's a different question entirely — and the answer is not in the headline metrics.

Why "More Information" Doesn't Mean More Accuracy

The instinct most people have when they first encounter MTL is straightforward: if one model can tell me the person's identity and their approximate age and their emotional state, shouldn't that extra information make the identity match more confident? It feels right. It's wrong.

The model isn't using age and emotion data to confirm identity. It's using the same shared features to generate all three outputs. When the shared features get tuned to excel at age estimation — which means weighting surface texture heavily — they become slightly worse at the deep geometric comparison that identity verification requires. More outputs don't mean more accuracy. They mean more optimization pressure on the same underlying representation, pulling it in more directions at once. Up next: Hidden Authenticity Check Before Face Comparison.

Understanding how deep learning encodes facial geometry differently from surface attributes is exactly why professional-grade facial comparison platforms treat identity verification as an isolated, dedicated task — not a side output of a multi-purpose system.

The real lesson here isn't that multitask learning is flawed. On a Raspberry Pi running demographic analysis for a research project, those 99% accuracy numbers are genuinely remarkable. The lesson is about what you're actually asking a model to optimize for — and whether that optimization secretly undermines the one output you care about most.

The model didn't get the identity wrong because it was bad at its job. It got distracted. It was doing three things at once, and in the process of getting very good at two of them, it subtly eroded the third. Efficiency and auditability are not the same thing. For anything that needs to hold up under scrutiny — in an investigation, in a legal proceeding, in any context where a wrong answer has real consequences — you don't want a model that multitasks.

You want one that does a single job. Provably. Every time.

So here's the question worth sitting with: if you were handed two outputs — one from a dedicated identity model, one from a multitask model with a 99% headline accuracy score — and you had to stake something important on one of them, which lane would you want to be in?