Deepfake Crisis Exposes Kids While Watermarks and Platforms Fail

AI-generated sexual deepfakes are now hitting kids at scale, while the tools meant to trace and block them are already being pulled apart.

An investigation by Wired and podcast Indicator identified nearly 90 schools and about 600 students worldwide targeted by AI‑generated nude images, many created with low‑skill “nudify” apps that run on ordinary phones, according to Wired. In the US, child‑safety nonprofit Thorn found that roughly 1 in 8 teens know someone who has been victimized by AI‑generated sexual imagery, a pattern echoed in school‑focused polling summarized by the Los Angeles Times. The fallout is immediate: in Pennsylvania, two 14‑year‑old boys who made roughly 350 fake nudes of at least 59 classmates were sentenced to probation after victims described being forced to identify their own faces in pornographic composites, the Associated Press reported.

Even as schools scramble, major platforms are struggling — or refusing — to draw hard lines. Apple “almost” kicked Elon Musk’s Grok chatbot off the App Store after users flooded it with prompts for non‑consensual sexual deepfakes of real people, including minors, but instead quietly pressed xAI to tweak the app and kept it live, according to The Mac Observer. Separate reporting shows Grok’s image tools have generated child sexual abuse material and nonconsensual porn of adults, a scandal that has triggered regulatory complaints and criminal scrutiny in Europe, as detailed by Le Monde and Foreign Policy.

At the same time, provenance tech meant to fingerprint AI imagery is showing cracks. Google’s SynthID watermarking, which the company says has been applied to more than 10 billion pieces of content, is now the subject of an open reverse‑engineering project that demonstrates how the watermark can be located and scrubbed from Gemini‑generated images, according to TechCrunch and a detailed analysis on Starlog. Academic work is already warning that watermarking schemes are vulnerable to “expert adversaries” who can reverse‑engineer and strip them, researchers write in a recent preprint on arXiv.

The result is a lopsided ecosystem: minors can be targeted with industrial‑scale abuse created on consumer apps; schools and parents have little recourse beyond patchwork state laws; platforms and app stores enforce safety rules inconsistently; and watermark systems that policymakers are betting on as a cornerstone of AI governance can be circumvented almost as soon as they are deployed. Until platform liability, school protocols, and technical provenance tools move in lockstep, the deepfake crisis will keep outpacing the defenses built to contain it.