A percentage is a judgment, not a measurement
When a researcher says there is a 10 percent chance that advanced AI causes human extinction, the number is not a frequency measured across repeated histories. It is a subjective probability summarizing beliefs about capability, deployment, institutions, conflict, and many uncertain steps. Another expert may use “existential risk” to include permanent human disempowerment, while a headline reports only extinction.
Such estimates can inform decisions, especially when consequences are enormous, but they should not be presented as established scientific constants. The speaker, date, exact question, time horizon, outcome definition, evidence, and uncertainty all matter. A famous scientist’s personal estimate remains a personal estimate.
Define the outcome before comparing numbers
At least four outcomes are commonly blended:
- severe harm: large casualties, economic loss, or rights violations;
- global catastrophe: harm affecting much of humanity or a specified fraction of deaths;
- human extinction: no surviving human population;
- existential catastrophe: extinction or irreversible destruction of humanity’s long-term potential, sometimes including permanent severe disempowerment.
Forecasts may ask whether AI causes the outcome, is a necessary contributor, or merely appears somewhere in the chain. They may cover ten years, the year 2100, or all future time. A one-percent estimate over ten years cannot be compared directly with a five-percent estimate over a century.
Definitions of the technology also vary: current general-purpose AI, “high-level machine intelligence,” AGI, superintelligence, or any future AI. Always preserve the original wording.
What the large researcher survey found
The 2023 Expert Survey on Progress in AI invited authors from leading AI venues and obtained responses from 2,778 researchers. Its published analysis found broad disagreement on timelines and impacts. Most respondents expected good outcomes from superhuman AI to be more likely than bad, yet many assigned nontrivial probability to extremely bad outcomes. The paper also found that responses changed with question framing (Grace et al., “Thousands of AI Authors on the Future of AI”).
This does not support “AI experts agree extinction is likely.” It supports a narrower claim: a substantial portion of surveyed researchers considered extreme outcomes plausible enough to assign meaningful probability, alongside optimism and wide dispersion. For a closer look at why the extinction-specific responses in this survey and others diverge so widely, see why expert extinction estimates diverge so widely.
Survey limitations include nonresponse, differing expertise in risk analysis, unstable definitions, and difficulty translating intuitive concern into calibrated probabilities. Publishing at a top AI conference establishes technical experience, not forecasting skill about geopolitics or century-scale social change. Median answers also conceal polarization and correlations among respondents.
Forecasting tournaments reveal another disagreement
The Existential Risk Persuasion Tournament brought together 80 subject-matter experts and 89 superforecasters with records on shorter-term forecasting. Participants estimated AI, nuclear, biological, and other risks under common definitions, exchanged arguments, and updated (Forecasting Research Institute XPT).
The resulting peer-reviewed analysis found large differences between domain experts and superforecasters, especially for AI (Karger et al., International Journal of Forecasting). This is valuable evidence against false consensus. It does not prove the lower or higher group correct.
Superforecasters have demonstrated calibration on many resolvable, shorter-horizon questions. Century-scale extinction questions cannot be scored within participants’ careers, and rare probabilities below one percent are particularly hard to validate. Domain experts may understand mechanisms better but can be selected into a field because of concern and may lack forecasting calibration. The groups bring different strengths and biases.
Why experts disagree
Capability assumptions: Will current methods produce robust autonomy, scientific discovery, and strategic planning, or will reliability and embodiment remain hard?
Timeline assumptions: Risk accumulates differently if transformative capability arrives in five years versus fifty. Institutions have more time to adapt under slower progress, but more actors may gain access.
Alignment assumptions: Researchers disagree about whether advanced systems will develop persistent goals, whether current training will control them, and how much observable precursor evidence should move forecasts.
Power and deployment assumptions: A capable model without tools or authority differs from a widely deployed agent controlling money, code, laboratories, or infrastructure.
Governance assumptions: Competition, military conflict, concentrated ownership, regulation, and international coordination may dominate technical uncertainty.
Moral definitions: Permanent loss of human control may count as existential harm for one forecaster and not another.
A headline number compresses this entire model. Good reporting expands it again.
Calibration is difficult but not optional
A calibrated forecaster who assigns 20 percent to many events should be correct about one-fifth of the time. Calibration can be measured only across a set of comparable, resolved questions. We cannot directly calibrate one forecast of extinction by 2100 today.
Forecasters can still improve discipline by decomposing the pathway into nearer, observable questions: model autonomy, research automation, safeguard failures, deployment scale, incident rates, and governance adoption. Those forecasts can resolve sooner and reveal which assumptions were wrong.
Use base rates carefully. Humanity has no historical sample of AGI transitions. Analogies to nuclear weapons, pandemics, cybersecurity, industrial accidents, or software adoption offer partial evidence, not a direct reference class. Mechanistic models also risk compounding speculative probabilities across dependent steps.
The scientific synthesis does not assign one probability
The 2026 International AI Safety Report, written through an international expert process, separates documented malicious use and reliability failures from hypothesized loss-of-control scenarios. It states that current systems lack capabilities needed for loss of control, that relevant abilities are improving, and that experts disagree substantially about likelihood and severity (International AI Safety Report 2026).
That is not a “consensus probability.” It is a consensus that the evidence contains serious risks and major uncertainty. Scientific assessment can identify mechanisms, observations, gaps, and safeguards without pretending to know a single chance of catastrophe.
How to present a public estimate
A responsible entry should look like this:
In [date], [person or group] assigned [probability or range] to [exact outcome] by [time], under [technology definition], based on [main reasons]. This is [personal judgment, survey aggregate, or structured forecast], not an observed frequency.
Include the distribution when possible. Report response rate and sample selection for surveys. Distinguish mean from median and conditional from unconditional probability. Preserve revisions so forecasts can be evaluated rather than quietly overwritten.
Do not average incompatible questions. Combining a survey’s “permanent disempowerment” estimate with a tournament’s “population below 1,000” estimate creates a number with no coherent meaning.
Decision-making under disagreement
Policy does not require certainty. Low-probability, high-consequence risks can justify research, monitoring, resilience, and reversible safeguards when those measures have benefits across scenarios. The cost and side effects of intervention matter too. A precaution that entrenches monopoly, suppresses legitimate science, or increases geopolitical instability may raise other risks.
Robust measures include independent evaluations, incident reporting, model security, emergency response, biological defense, cybersecurity, human control over nuclear use, and clear accountability. They remain useful across a wide probability range.
Different risk tolerances are legitimate political questions. Experts can inform the evidence but cannot alone decide how society trades innovation, liberty, distribution, security, and uncertain future harm.
Updating rather than defending a tribe
Forecasts should change when evidence changes. Relevant updates include sustained long-task autonomy, validated AI-generated research advances, real-world safeguard failures, successful oversight, major laws, or persistent capability plateaus. A forecaster should state what would move the estimate up or down.
Intellectual diversity improves assessment. Include technical skeptics, safety researchers, social scientists, forecasters, domain experts, affected communities, and policymakers. Disagreement should be mapped by assumptions, not reduced to “optimists” and “doomers.”
The practical conclusion
Expert estimates show that catastrophic AI risk is taken seriously by many qualified people and doubted by others. They do not establish a measured probability or unanimous scientific view. Surveys reveal beliefs; tournaments impose forecasting discipline; theory explains pathways; evaluations test components; deployment supplies real-world evidence.
The most honest public position is structured uncertainty: define the outcome, show the range, expose assumptions, track calibration where possible, and choose safeguards that remain worthwhile under several plausible futures.