Deepfake Legislation: No Law Stops AI Training on You

A woman who has been in the FBI's abuse-tracking system since before she started school just found out her old nightmare got a new job. According to a class action lawsuit reported by CyberScoop, survivors of childhood sexual abuse allege that xAI's chatbot Grok was trained, at least in part, on real images and videos of their abuse — the same material that's been logged for years with the National Center for Missing & Exploited Children. This isn't a story about a bad post. It's a story about why deepfake legislation hasn't caught up to what training an AI system actually means, and why your own photos could be doing something you never agreed to, somewhere you'll never see.

Survivors say their abuse images were used to teach an AI how to generate new abuse — and there's currently no law that requires a company to tell you, or stop you, before that happens.

Why the law hasn't caught up: training data versus posted fakes

Most people picture the harm as a single moment: someone takes your photo, runs it through an app, and posts a fake. Bad enough. But this lawsuit describes something worse — a company allegedly using real abuse material as training images, meaning the content wasn't just misused once, it became part of the machine's permanent vocabulary. Once an AI model learns a pattern from data, you can't simply delete it the way you'd delete a post. The weights are baked in. That's the compounding part nobody warns you about: the original violation becomes a renewable resource.

Here's where it gets genuinely alarming. According to the lawsuit, during a single 11-day stretch between December 2025 and January 2026, Grok generated more than 3 million sexualized images — and at least 23,000 of them appeared to depict children. That's not a glitch. That's mass production, running quietly, at a scale most people can't picture until they see the number written down. This article is part of a series — start with How To Spot A Deepfake.

The lawsuit also claims xAI's safeguards were "very weak" and could be gotten around with indirect prompts — meaning if a system still holds the underlying capability to generate this kind of content, someone will eventually talk their way past the filter. Filters catch phrasing. They don't remove what the model already learned.

Does current law cover fakes made from abuse material used in training?

In most U.S. states, that category of imagery and child abuse material are already illegal to create, share, or possess — full stop. The unresolved legal question is whether using survivor images as AI training images, without consent, is separately punishable, and whether victims can sue the company, not just the person who posted the fake.

How training-data scale turns one violation into millions

This isn't the first time investigators have found abuse material buried in training data. Academic researchers previously discovered child sexual abuse images sitting inside open datasets used to train popular text-to-image AI models — years before Grok existed. So this pattern didn't start with one company. It's a structural blind spot across the industry: everyone builds on huge scraped datasets, and almost nobody fully audits what's inside them before training starts.

Zoom out and the numbers get worse. According to a UNICEF, ECPAT, and INTERPOL study across 11 countries, at least 1.2 million children disclosed that their images had been manipulated into sexually explicit fakes in just the past year. That's not a hypothetical future risk. That's a documented, present-tense reality for over a million kids right now.

The lawsuit alleges xAI's safeguards can be bypassed with indirect prompts, meaning the underlying capability to generate abuse content was never actually removed — just hidden behind a thinner door. — as reported by CyberScoop

And this is where deepfake legislation keeps failing survivors: most law focuses on the output — the fake image, the video, the post someone finds and reports. Almost none of it regulates the input. There's no requirement, anywhere in U.S. federal law right now, forcing an AI company to prove its training images don't contain abuse material before the model ships. Companies say they can't fully see inside third-party datasets before training. Maybe. But refusing to audit isn't a technical limit — it's a cost decision. Auditing data is slow and expensive. Skipping it is fast and cheap. Guess which one companies keep choosing. Previously in this series: Biometric Id.

What real accountability rules for deepfakes would require

A handful of U.S. states have moved fast on this issue — several have passed laws making non-consensual intimate fakes a crime, some carrying real criminal penalties, others sitting as gross misdemeanors depending on the state. Ballotpedia's AI deepfake legislation tracker shows a genuine patchwork: rules differ wildly depending on where you live, whether a minor is involved, and whether the content touches political or election material. That patchwork matters here, because most of those laws were written to punish the person who posts a fake — not the company whose synthetic media pipeline made mass production possible in the first place.

What's actually missing is upstream accountability: rules that force companies to say what's in their training sets, real penalties tied to court enforcement when abuse material turns up, and a legal path for survivors to demand removal — not just from a website, but from the model itself. Right now, if your images end up in a dataset, you may have no idea, no notice, and no clean way to get them out. That's the actual policy failure hiding under this lawsuit.

So here's the one thing worth doing, before you touch any product or app: if you've ever had intimate images shared without consent, or you suspect old material connected to you is circulating, check whether it's been logged with the National Center for Missing & Exploited Children or a similar hash-matching registry — these systems create a fingerprint of an image so platforms can detect and block copies without needing to see the image again. It won't undo training that's already happened. But it's the single most concrete step that exists today toward getting ahead of where your images travel next.

The security question every platform is dodging

Every company caught in this mess gives the same answer: we didn't know it was in there. Maybe that's even true. But "we didn't check" and "we couldn't have known" are different sentences, and companies keep saying the second one while doing the first. Real cybersecurity and data hygiene practices — scanning training sets against known abuse-material hash lists before a model ever touches them — already exist. They're used elsewhere in the industry. The choice not to use them everywhere isn't a mystery. It's a budget line. Up next: How To Spot A Deepfake 1 School Photo Is All It Takes.

Meanwhile survivors are left doing the discovery work companies should've done years ago — finding out, after a lawsuit is filed, that the worst day of their life became synthetic intimate content raw material for a chatbot. That's not a policy gap. That's a decision, made by someone, that got a lot of people hurt twice.

Frequently Asked Questions