Deepfakes first spread as a tool of a specific and devastating kind of abuse: nonconsensual sexual imagery. Early iterations often were technically crude, with obvious doctoring or voices that didn’t quite sound real. What’s changed is the engine behind them. Generative artificial intelligence has made convincing imitation faster and cheaper to create and vastly easier to scale—turning what once took time, skill and specialized tools into something that can be produced on demand. Today’s deepfakes have seeped into the background of modern life: a scammer’s shortcut, a social media weapon, a video-call body double borrowing someone else’s authority. Deception has become a consumer feature, capable of mimicking a child’s voice on a 2 A.M. phone call before a parent is even fully awake. In this environment, speed is the point: by the time a fake is disproved, the damage is already done.
Hany Farid, a digital forensics researcher at the University of California, Berkeley, has spent years studying the traces these systems leave behind, the tells that give them away and why recognizing them is never the entire solution. He’s skeptical of the AI mystique (he prefers the term “token tumbler”) and even less convinced of the idea that we can simply filter our way back to truth. His argument is plainer and harder: if we want a world where evidence still counts, we must rebuild the rules of liability and go after the choke points that make digital deception cheap and profitable. Scientific American spoke with Farid about where deepfakes are headed and what works to blunt them.
[Live event: Life in the Age of AI. Join SciAm for an insightful conversation on the trends and innovations shaping AI in the year ahead. Learn more.]
On supporting science journalism
If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
An edited transcript of the interview follows.
When you say “trust infrastructure” in the age of generative AI, what are its core layers right now?
What we have been living with for the past 20 years in terms of disinformation on social media is now being driven by generative AI: more sophisticated bots, fake images, fake video, fake everything. Here you have to think about the intersection of the ability to generate images and audio and video of anybody saying and doing anything and the distribution channels of social media coming together. And by “trust,” I’m referring to the question of how you trust anything that you see online.
There’s another aspect of trust, which is in the courtroom, for example. How do you trust evidence in a civil case, a criminal case, a national security case? What do you do now? I mean, I deal with this almost every day. Some lawyers are like, “Well, we got this recording, and we have this image, and we have this closed-circuit TV video. All right, now what?”
And then there’s the fact that chatbots are going to go from sitting off to the side to being fully integrated. So what happens when we start building the next generation of everything—from self-driving cars to the code we write—that is now infused with AI, and how do we trust those systems anymore when we’re going to turn them over to critical infrastructure at some point?
What do you think most people misunderstand about today’s generative AI?
I think the biggest misconception is that it’s AI. My favorite term for it is “token tumbler.” What they’ve done is grab huge amounts of text, collapse words into numeric tokens and then do a sophisticated auto-complete: “Okay, I’ve seen these tokens. What’s the next token?” It is artificial, but it’s certainly not intelligence.
Here’s the other thing people have to understand: most of the “intelligence” is not in the computer—it’s actually humans. Scraping data and building a token tumbler doesn’t get you to ChatGPT. The way you get to ChatGPT is by then bringing tons of humans in who human-annotate questions and answers and say, “This is a good answer; that is a bad answer.” That is what’s called the fine-tuning and the reinforcement learning.
What are the biggest harms you’re seeing right now?
So, the nonconsensual intimate imagery, or NCII, is awful. Child sexual abuse, sextortion, kids talking to chatbots and the chatbots convincing them to take their own lives—which has happened, and that’s what the lawsuits are. Fraud is now being supercharged by generative AI in terms of voice scams at the individual level—Grandma getting a call, the CEO getting a call. I would say the disinformation campaigns, the poisoning of the information ecosystem.
And because I’m a university professor, I’ll say you shouldn’t underestimate the impact on education. I mean, there is not a single student who is not using this AI. And you can’t say, “Do whatever you want.” We have to fundamentally rethink how we teach students, not only to prepare them for a future where these tools will almost certainly be sitting side by side with them but also to figure out what they need to learn.
For nonconsensual intimate imagery, what’s the best removal playbook right now—and what’s the weakest link?
There’s blame up and down the stack, from the person with their hands on the keyboard, to the product that was made, to the companies that are hosting it, and then of course to the social media companies that allow all this stuff to spread. Across the board, everybody gets blamed in varying amounts.
Is hash matching, based on the identification of digital “fingerprints” in media files, meaningfully effective, or is it whack-a-mole at this point?
I was part of the Microsoft team that built the image-identification program PhotoDNA back in the day for doing hash matching for child sexual abuse. And I’ve always been supersupportive of the technology. In the child sexual abuse space where it’s real children being exploited, it actually works fairly well because we know that the same images, the same videos circulate over and over.
The NCII stuff today is AI-generated, which means you can produce it en masse. The problem with hash matching is, “All right, you’re going to catch this image, but I can make 100 more in the next 30 seconds.” So the hash matching gets you only to a certain level, and because people can now make these things so fast, I don’t think you’re going to be able to keep up.
What should lawmakers stop doing in deepfake bills, and what should they do more of?
For full disclosure, I worked on the early incarnations of the TAKE IT DOWN Act with law professors Mary Anne Franks and Danielle Citron. I would say it was a pretty good law when it started, and it is a terrible law on the way out.
If you’re the creator of a Nudify app, it doesn’t actually hold you accountable. It’s got a 48-hour takedown window, which is ridiculous because it’s the Internet, which means everything happens in the first 90 seconds—and it’s the mother of all whack-a-moles. And the other issue is that there are no penalties for creating false reports, which is why I think the law will be weaponized.
So what they should stop doing is passing bills like that—completely ineffective. You can’t go after the content. You have to go after infrastructure: the couple dozen companies out there that are hosting it; the Apple and Google stores; the Visa, MasterCard and PayPal systems that are enabling people to monetize it. You have to go upstream. When you’ve got 1,000 cockroaches, you’ve got to go find the nest and burn it to the ground. And by the way, right now the burden is still on the victims to find the content and send the notices.
“What happens when we start building everything with AI? How do we trust those systems anymore?” —Hany Farid U.C. Berkeley
What has changed as generative AI has improved, and how is your company GetReal responding?
When we started in 2022, we were focused on file-based analysis: somebody sends you a file—image, audio or video—and you determine as much as you can about its authenticity. But then we started seeing real-time attacks where people were getting on Zoom calls and Teams calls and impersonating other people. So we started branching out to say, “We can’t just focus on the file. We have to start focusing on these streams.”
And what has happened is what always happens with technology: it gets better, faster, cheaper and more ubiquitous.
We take a digital-forensics-first approach. We ask: What are the artifacts you see not just in this one Sora video but across video generators, voice generators and image generators? We find a forensic trace we believe we will be able to measure even after the file has been recompressed and resized and manipulated, and then we build techniques to find that artifact. When I go into a court of law and testify, I don’t tell the judge and the jury, “Well, I think this thing is fake because the computer told me so.” I say, “I think this thing is fake because we look for these specific artifacts—and look, we found that artifact.”
Two years from now what would have to be true for you to say we’ve built workable trust infrastructure?
There are two types of mistakes you can make. You can say something real is fake—we call that a false positive—and you can say something fake is real, which we call a false negative. And the hardest thing is keeping those false positives really low. If every time you get on a call the technology’s like, “Oh, Eric’s fake, Hany’s fake,” you’re just going to ignore it. It’s like car alarms on the street.
So false positives have to be low. Obviously, you need to keep up with the tech, and you need to catch the bad guy. It has to be fast, especially on a stream. You can’t wait 10 minutes. And I think it has to be explainable. You can’t go into a court of law or talk to folks over at the Central Intelligence Agency or the National Security Agency and say, “Well, this is fake because we said so.” Explainability really matters.
Now, the good news is that, I think almost paradoxically, we will get streams before we get files. In a stream, the bad guy has to produce the fake in real time. I can wait five seconds—that’s hundreds of frames. With a file, my adversary can sit in the quiet of their home and work all day long creating a really good fake and then launch it into the world. At GetReal we have a product that sits on Teams and Zoom and WebEx calls, and it analyzes audio and video streams with very high fidelity.
If you could change one thing about platforms or apps to protect people the fastest, what would it be?
First I’d create liability. The laws aren’t going to do it. You create a product that does harm, and you knew or should have known it did, and I’m going to sue you back to the dark ages the way we do in the physical world. We haven’t said that to the digital world.
Aren’t these platforms protected under Section 230, the law that shields Internet platforms from liability for content posted by their users?
Section 230 most likely doesn’t protect you from generative AI, because generative AI is not third-party content. It’s your content. You created it. You made an app that’s called Nudify. Your chatbot is the one that told the kid to kill himself and not tell his parents about that conversation. That’s your product.
And, by the way, I would love to have 230 reform to hold the Facebooks and Twitters and TikToks responsible.
Another good protective step is what Australia did, which is ban social media for children younger than 16. Social media for kids was an experiment. It didn’t work. It’s a disaster. The evidence is overwhelming.
What do you tell families about voice-cloning scams?
I love safety words. My wife and I have one. It’s an analog solution to a digital problem. It’s low tech.
The other advice we give to everybody is to stay aware. Know that this is happening. Know that you’re going to get a call at two in the morning from your son, who’s saying something terrifying—so hang up, call him back. This situation is like everything in cybersecurity: don’t click on links. Public awareness doesn’t solve the problem, but it minimizes the impact, and it makes it less efficient for the bad guy.
Do you and your wife use a safe word in every call, every digital exchange?
Only if something dramatic happens. This isn’t hypothetical: I got attacked with a voice clone. An attorney I was working with on a very sensitive case got a call from my number, talking about it in my voice. At some point he got suspicious and called me back and said, “Was that you?” I said, “What are you talking about?” So he and I made a code word for the rest of that case. For me and my wife, it’s “I’ve been in an accident,” “I’ve been kidnapped”—that kind of thing.
Between those who fear AI as an existential threat and those who think the current wave is all hype, where do you land?
If you talk to people in the technology space, it seems like there are two basic anti-AI camps. There’s the camp with computer scientist Geoffrey Hinton, an AI pioneer, that’s like, “Oh, God, we’re all going to die. What have I done?” And then there’s cognitive scientist Gary Marcus and his camp that’s like, “This is all bullshit, and I’ve been telling you it’s bullshit for 10 years.”
I think they’re both wrong. I don’t necessarily think we’re all going to die, but it’s clear something is shifting the world. The next few years are going to be very interesting. We have to think seriously about the future we want and put the systems in place now. Otherwise we will have a repeat of the past 20 years.