That "Study" You Just Read? 66% of Its Sources Don't Exist

Here's something that should make you stop scrolling: In 2025, more than 100 fake citations slipped through peer review at one of the world's top AI research conferences — NeurIPS (that's the Super Bowl of machine learning conferences, where the best researchers in the world submit their work). Not one reviewer. Not two. These fabricated references passed three or more expert reviewers each. The experts read the papers. They just didn't check the footnotes.

That's the thing nobody tells you about AI-generated fakes. We're all worried about the face in the photo or the voice on the call. But sometimes the most dangerous fake isn't a manipulated image — it's a polished PDF with a convincing title, clean formatting, and a reference list full of papers that don't exist.

The Counterfeit Invoice Problem

Think about a fake invoice. Not a sloppy one — a good fake. Proper letterhead. Real company name. Plausible-sounding product codes. The right font. You'd probably pay it. Most people would, because we've trained ourselves to read formatting as a signal of legitimacy. Professional-looking equals trustworthy. It's a mental shortcut that works fine 999 times out of 1,000 — until the one time it doesn't.

A fake research paper works exactly the same way. Clean academic formatting. An abstract that sounds authoritative. A bibliography that runs two full pages. Your brain registers all of that and files it under "credible." But here's the catch: none of those visual signals actually prove the underlying evidence is real. The invoice fails the moment you call the vendor and ask, "Did you send this?" The paper fails the moment you try to look up citation number 14 and find that it doesn't exist anywhere on Earth.

The problem is that most people never make that call. This article is part of a series — start with Your Face Is Now Your Train Ticket And Nobody Asked You Firs.

---

How AI Builds a Fake That Passes the Eye Test

When an AI model generates a fake citation, it doesn't just make up a random string of words. It constructs something that sounds right — and that's a very different thing. According to a cross-model audit of AI-assisted academic writing on arXiv, GPT-3.5 fabricated 55% of its citations outright, and even the more capable GPT-4 fabricated 18%. Those aren't typos or confused attributions. Those are invented references.

Here's what makes them so hard to catch: the fabrication is layered. The AI pairs a real author's name — someone who genuinely exists and publishes in that field — with a title that sounds exactly like something they might have written. The journal name is real. The year is plausible. Everything checks out individually. It's only when you stack all the checks together that the deception collapses.

A sixfold rise in two years. That's not a slow drift — that's an avalanche. And the fabrications don't stay in one paper. Research from an analysis of compound deception in elite peer review on arXiv found that the same fake citation appeared across up to 16 independent papers. Once a ghost reference enters the record, other researchers find it, trust it (because it's already been cited — by someone else who didn't check), and cite it again. The fabrication compounds. It grows roots.

There's also a pattern worth knowing: the AI hallucinates more when the topic is obscure. Fabrication rates hit 28–29% for niche subjects where fewer real papers exist to contradict the model, compared to roughly 6% for well-covered, major topics. The model fills in the gaps with invented plausibility. The less real data exists, the more it makes up — confidently.

---

The Misconception That Gets People Burned

Here's what almost everyone gets wrong: "If it looks professional and I recognize the author's name, the citation is probably real."

It's a completely understandable mistake. We've spent years learning that professional format signals legitimate effort. Academic papers have gatekeepers — editors, reviewers, institutions. So when something arrives dressed in that clothing, our brain relaxes. We've been trained, reasonably, to treat presentation as a proxy for trustworthiness. Previously in this series: Your Friends Doorbell Just Scanned Your Face And You Cant Ta.

But AI broke that rule. It can produce the costume without the person underneath. And according to the arXiv compound deception analysis, 66% of the fabricated citations found in elite peer-reviewed work were total inventions — not corrupted versions of real sources, not confused attributions, but papers that were simply made up from scratch. They didn't fail on one check. They were designed to pass each check individually while failing all of them together.

"AI-hallucinated citations don't fail on just one front — they layer semantic plausibility onto fabricated content AND incorporate identifier hijacking simultaneously, making every single fabrication a multi-layered deception." — Research synthesis, arXiv compound deception analysis, NeurIPS 2025

That phrase "identifier hijacking" is worth unpacking. An identifier — like a DOI (that's a Digital Object Identifier, basically a permanent address for a published paper) — looks like an official stamp of legitimacy. Fake citations often include one. It looks valid. It may even be formatted correctly. But when you actually follow it to the source database, it resolves to a different paper entirely, or to nothing at all. The identifier is borrowed clothing on a fictional body.

---

The Three-Step Evidence Chain

So what does actual verification look like? This is the behind-the-scenes work that most people skip — the equivalent of calling that vendor to confirm the invoice. It's not glamorous. But it's the only thing that actually works.

Step 1: Citation Audit

Find the first author of the cited paper. Look them up on ORCID (a free academic registry where researchers list their published work) or Google Scholar. Does THIS paper appear in their publication history? Not a paper with a similar title. This exact paper. If it's not listed anywhere in their record, treat it as fabricated until proven otherwise. According to Sourcely's verification workflow for AI-generated citations, author-profile cross-referencing is one of the most reliable first filters precisely because the AI fabricates the paper but can't retrofit it into a real person's actual publication history.

Step 2: Author-Title-Venue Cross-Check

This is where compound deception hides. The author is real (check one passes). The journal exists (check two passes). The year is plausible (check three passes). But does THIS author publish in THAT journal, and does THAT journal publish on this topic, and is the volume number consistent with THAT year? You have to check all of those simultaneously. Any single check run in isolation will feel fine. The mismatch only appears when you line them up together — like three puzzle pieces that each look like they might fit until you actually press them against each other. Up next: Ai Facial Recognition Doorbell Cameras Lawsuits Privacy.

Step 3: Identifier Verification

Take the DOI or the citation link and follow it directly to the source database — not just to any landing page, but to the actual journal record. Verify that the paper title, author name, volume number, and publication date all match what was cited. A fake citation can contain a real DOI that leads to a completely different paper. The link works; the destination is wrong. That's the deception. You need all four data points to confirm the same source.

At CaraComp, we spend a lot of time thinking about the difference between what something looks like and what something actually is — it's the core challenge in facial recognition work, where a convincing image and a verified identity are two completely different things. The same gap exists with documents. A face that matches a photo isn't automatically the right person. A citation that matches a real author's name isn't automatically a real paper.

Here's the thought worth sitting with: we've spent decades building intuition about what trustworthy information looks like. Formal language. Proper citations. Institutional names. Clean layout. AI just learned to replicate all of that surface texture — without any of the underlying substance. The confidence that formatting used to signal is now detachable from the evidence itself.

So the next time someone hands you a document as "proof" — whether it's a research paper, a background check, a medical report, or anything dressed up in official clothing — ask yourself: am I trusting the formatting, or am I trusting the source trail? Because those are two completely different things. And only one of them is actually hard to fake.