The problem has two directions
Generative systems reduce the time and skill needed to create convincing text, images, audio, and video. False media may be accepted as real. At the same time, authentic evidence may be dismissed as synthetic—the “liar’s dividend.” The second effect does not require a perfect deepfake; it requires only enough public awareness of fabrication to make denial plausible.
This is a present generative-AI problem, not evidence that AGI exists. Current harms include impersonation scams, non-consensual intimate images, fabricated political content, forged evidence, and cheap content flooding. For the household recovery sequence after an impersonation scam succeeds, see What to Do After an AI-Enabled Scam. AGI-scale individualized persuasion is a projection that depends on future capability, access to personal data, deployment, and human susceptibility.
What is documented, and what is not
Voice cloning and face synthesis have been used in fraud and harassment. Political actors have released or amplified synthetic media. Yet viral examples do not reveal prevalence or net electoral effect. People share false content for social and partisan reasons, and conventional editing, misleading captions, and authentic footage presented out of context remain powerful. A narrow focus on spectacular deepfakes can miss the more common problem of ordinary deception accelerated by automation.
Detection claims also need caution. A detector estimates whether artifacts resemble material in its training data. Performance may fall after compression, cropping, translation, model updates, or deliberate evasion. False positives can harm journalists, artists, and people documenting abuse. Absence of a detector signal does not prove authenticity; a positive signal should initiate investigation rather than end it.
Provenance is useful—but it does not certify truth
The Coalition for Content Provenance and Authenticity (C2PA) standard lets compatible tools attach cryptographically verifiable information about a file’s origin and editing history. A valid credential can show that a recognized device or organization signed particular assertions and that the signed data has not been altered unnoticed. It can support a chain of custody.
C2PA itself states that provenance alone cannot establish that content is true, accurate, or factual. A credentialed camera can record a staged event; an authorized editor can publish a misleading crop; a trusted organization can make a mistake. Identity support also depends on the credential and trust system used (C2PA Explainer, specification 2.2).
Credentials can also be absent from authentic media. Screenshots, transcoding, social platforms, and metadata stripping may remove them. OpenAI’s own help documentation notes that screenshots can remove C2PA metadata (OpenAI Help Center). Therefore:
- valid provenance is positive evidence about history, not truth;
- invalid provenance shows a broken or altered chain, not necessarily a fake scene;
- missing provenance is not proof of fabrication.
The standard is shaped by technology and media organizations that benefit from adoption. Its technical documentation is primary evidence for how it works, not independent evidence of effectiveness across the information ecosystem.
Disclosure law is becoming real
In the European Union, AI Act Article 50 transparency duties began applying on August 2, 2026. Providers of certain systems must support machine-readable marking of generated or manipulated content, and deployers face disclosure duties for specified deepfakes and public-interest text, subject to exceptions. Enforcement is divided among national authorities and, in narrower circumstances, the AI Office. The European Commission says fines for relevant violations can reach €15 million or 3% of worldwide annual turnover, with proportionality rules (European Commission, July 2026).
In the United States, the federal TAKE IT DOWN Act became law in May 2025, addressing non-consensual intimate imagery, including covered digital forgeries, and requiring covered platforms to operate a notice-and-removal process. It is not a general deepfake-authenticity law and should not be described as one. Election, consumer-protection, publicity-right, and state laws may also apply, with different definitions and jurisdictions.
Labels help honest users and support enforcement, but malicious actors willing to commit fraud will not reliably self-label. Disclosure is one layer, not the solution.
A stronger verification workflow
High-stakes verification should combine independent evidence:
- Preserve the original. Save the file, URL, time, and surrounding post before recompression or deletion.
- Check provenance. Inspect valid credentials and the signer’s trust chain, while treating absence neutrally.
- Find the earliest source. Reverse-search key frames and locate the first documented upload.
- Corroborate externally. Look for independent recordings, official records, witnesses, weather, geography, and contemporaneous reporting.
- Contact the claimed source through a separate channel. Do not use contact details contained only in the suspicious message.
- Assess motive and timing. Urgency, secrecy, payment demands, and emotional pressure are fraud signals independent of media quality.
For a family voice-clone call, hang up and call the person or another relative using a known number. A private family verification phrase can help, but treat it as compromised if spoken or typed into an exposed channel. For corporate payment requests, require a second-person approval and out-of-band confirmation.
Journalism, courts, and archives
Newsrooms need capture-to-publication chains, transparent correction practices, and a way to retain originals. Courts need authenticated evidence, expert examination where appropriate, and caution about both fabricated exhibits and opportunistic claims that real evidence is fake. Human-rights investigators must balance provenance with the safety of sources; revealing a photographer’s identity or location can create danger.
Archives face a long-term challenge: formats, certificates, and verification infrastructure can expire. Preserving only a platform-rendered copy may destroy evidence available in the original. Durable integrity requires original files, hashes, timestamps, contextual records, and migration plans—not just a badge in a user interface.
The shared-baseline risk
Information integrity is not the same as agreement. Healthy democracies contain conflict about values and interpretations. The objective is to preserve reliable methods for establishing basic events: who said what, which document existed, how an image was produced, and what independent sources corroborate it.
Synthetic-media saturation may increase verification costs and reward established institutions that can afford authentication. That can protect quality but also concentrate gatekeeping. Open standards, public-interest verification tools, local journalism, and transparent methods can reduce the tradeoff between authenticity and pluralism.
What stronger systems could add
More capable agents might generate many mutually consistent artifacts, impersonate people across channels, localize narratives, test messages, and adapt to an individual’s reactions. That scenario is plausible but not yet measured as a general AGI effect. Defenses would need to authenticate people and processes, rate-limit suspicious campaigns, protect personal data, and preserve human review without creating universal identity surveillance.
The practical conclusion is disciplined skepticism rather than universal disbelief. Do not trust media merely because it looks real, and do not reject it merely because synthesis is possible. Provenance, corroboration, source history, and incentives together create a stronger factual baseline than either automated detection or intuition alone.