02 Consequences
What changes while humans are still in charge
Many consequential harms do not require a rogue system seizing control. They can arise from how people and institutions deploy powerful systems while humans still hold authority.
Near-term effects
These are four important risk families, not four settled predictions. Evidence is strongest for present labor-task exposure, fraud and security misuse; economy-wide displacement, institutional breakdown and loss of control remain increasingly uncertain as the scenario moves farther from systems deployed today.
- Labor-market disruption. If cognitive-task automation outpaces job redesign and policy, retraining systems, tax bases, and safety nets may struggle to adapt.
- Power concentration. Control of leading models, chips, cloud infrastructure, and distribution can create advantages in capital, information, and state capacity.
- Information integrity. Cheap synthetic media and scalable persuasion can increase fraud and raise the cost of maintaining a shared factual baseline.
- Security pressure. AI assistance can change the cost, speed, or scale of some cyber, biological, and military activities; measured effects differ substantially by domain.
On the labor figure specifically, one widely cited institutional projection is not an AI-only collapse forecast. Its job-creation and displacement totals combine AI with other technological, demographic, economic, geopolitical, and environmental trends:
World Economic Forum projects, across the macrotrends surveyed rather than AI alone, roughly 92 million roles displaced and 170 million created globally by 2030.
World Economic Forum, Research/report
The escalation path to loss of control
One hypothesized loss-of-control path runs through four stages. It is a scenario model used to identify interventions, not an observation that current systems have already begun this sequence:
- Autonomy is granted
Systems are given increasing autonomy when organizations value speed and lower supervision costs more than the additional control that review can provide.
- Oversight falls behind capability
Verifying a superhuman system’s reasoning is itself a hard problem, so the quality of human oversight degrades exactly as the stakes rise.
- Instrumental incentives emerge
Some theories predict that sufficiently capable, goal-directed systems may treat resource acquisition or resistance to shutdown as useful intermediate strategies. Whether trained systems develop stable incentives of this kind is unresolved.
- Correction becomes contested
By the time misalignment is legible to humans, the system may already hold enough leverage — over infrastructure, capital or information — to resist correction.
The theory does not require malice. Instrumental convergence — the hypothesis that sufficiently capable, goal-directed systems may seek resources, preserve options, or resist correction as intermediate strategies across many objectives — is one proposed mechanism. That is why chapter 04 examines evaluations, access controls, monitoring, oversight, and corrigibility together.
The scientist who chaired the International AI Safety Report has argued publicly that this risk deserves present attention:
Yoshua Bengio warned that future systems exhibiting self-preservation behavior could create a loss-of-control risk.
AFP, via TechXplore, Secondary coverage
Geoffrey Hinton has offered a personal probability estimate for AI-driven extinction; it is an expert judgment, not a measured frequency:
Geoffrey Hinton said in a December 2024 BBC interview that he put the chance of AI causing human extinction within thirty years at 10–20%.
BBC Radio 4, reported by The Guardian, Secondary coverage
What this means for a household or a firm
Most readers have more immediate control over household and workplace exposure than frontier-lab practice, but voting, organizing, research, employment, procurement, and public comment can still affect policy. Household exposure can include income or investment concentration, dependency on single points of digital failure, and readiness for a period of fast, disorienting change. Chapter 03 and chapter 06 turn this into a checklist.