Formal authority can survive after practical control is gone
A parliament can remain elected while relying on systems legislators cannot evaluate. A minister can remain legally responsible while receiving recommendations no civil servant can reproduce. A citizen can retain a formal right to appeal while facing an automated decision whose data, model, vendor contract, and reasoning are inaccessible. Nothing in that picture requires a coup or a conscious machine ruler. Authority becomes ceremonial because the practical capacity to choose differently has eroded.
This is a governance risk before it is an AGI risk. Governments already use algorithms in benefits, taxation, policing, migration, education, healthcare, and public administration. More capable systems could improve consistency, translate services, find fraud, summarize evidence, and make expertise accessible to small agencies. They could also deepen dependency and make public decisions harder to contest.
Existing international instruments provide a starting vocabulary, but their legal force differs. The Council of Europe Framework Convention on Artificial Intelligence is designed as a binding treaty for parties and centers human rights, democracy, and the rule of law. Its official status page listed no entry-into-force date as of September 11, 2026, so signature must not be described as a state already being legally bound (CETS No. 225 status). The UNESCO Recommendation on the Ethics of Artificial Intelligence is a non-binding recommendation addressing oversight, transparency, accountability, and environmental and social effects. Neither instrument solves post-AGI governance. Both show that capability and efficiency are not the only public values at stake.
Agency means being able to change the outcome
“Human oversight” is often treated as a staffing diagram: place a person somewhere after the model and the decision is human. Real oversight is a set of capabilities.
The reviewer needs enough time and information to understand the case. They need training to recognize model limitations. They need legal and organizational authority to reject the recommendation. A different choice must not trigger punishment merely because it departs from an automated score. The institution must retain records that support audit and appeal.
When those conditions are absent, a human click is a ritual. High automation can even make review worse: as people see fewer unusual cases and practice less independent judgment, their expertise decays. When the system finally fails outside its normal distribution, the nominal overseer may be least prepared to intervene.
The practical test is counterfactual: can a qualified person reach a different decision, explain why, and have that decision stand?
Four dependencies that can hollow out government
Vendor dependency
A public agency may lack the staff, compute, data rights, or contract terms to operate without one provider. Switching looks possible in procurement documents but would require retraining systems, migrating data, rewriting workflows, and accepting a long service interruption. That gives a private supplier influence even without explicit political power.
Contracts should therefore address portability, audit access, incident disclosure, subcontractors, data retention, security, service continuity, and exit assistance before a system becomes essential. The cheapest initial bid can be expensive if it purchases permanent dependency.
Epistemic dependency
An institution can lose the ability to know whether an output is good. If AI performs most analysis, drafting, simulation, and verification, the human organization may preserve final signature authority while losing independent models of the world.
Redundancy is necessary. Agencies need internal expertise, competing analyses for high-consequence questions, records of uncertainty, and protected channels for dissent. Some functions should be periodically performed without the primary system, not because manual work is always superior but because an untested fallback is not a fallback.
Infrastructure dependency
Advanced services depend on electricity, networks, identity systems, cloud infrastructure, and upstream models. An outage or commercial dispute can propagate through many public functions at once. Concentrating several agencies on one model or cloud can create correlated failure hidden behind separate contracts.
Dependency maps should identify shared providers and single points of failure across government, not merely within one department. Recovery exercises should test degraded operation, data restoration, manual prioritization, and public communication.
Legitimacy dependency
People accept difficult decisions partly because they understand who has authority, which rules apply, and how to challenge mistakes. A statistically accurate but inscrutable system can still undermine legitimacy if citizens experience it as arbitrary.
Explanation alone is insufficient. A generated explanation may sound plausible without revealing the actual basis of a result. Meaningful contestability requires the relevant evidence, applicable rule, responsible institution, and available remedy. It also requires a human office capable of changing the decision rather than merely explaining why the system produced it.
Democracy needs public technical capacity
Legislatures, courts, regulators, auditors, unions, and civil society need expertise in evaluation, security, procurement, data governance, and model evidence. Professional staffs, independent laboratories, inspectors general, standards bodies, and adversarial proceedings can provide that depth.
The NIST AI Risk Management Framework provides a voluntary structure for governing, mapping, measuring, and managing AI risks. Its value is procedural: organizations must identify context and affected people rather than treating benchmark accuracy as the entire risk assessment. A post-AGI institution would need stronger, enforceable versions of those habits, paired with democratic authorization.
Elections are necessary but not sufficient
Transformative AI could affect democracy through persuasion, synthetic media, surveillance, policy analysis, campaign operations, and control of information infrastructure. Focusing only on fake images misses the broader issue: who chooses the objectives of systems mediating political attention, and what data are used to optimize influence?
Disclosure can help, but labeling content “AI-generated” does not establish whether it is true or manipulative. Provenance can show parts of a file’s history without proving its claims. Political resilience also needs privacy protection, independent journalism, campaign transparency, platform accountability, secure election administration, and citizens able to verify important information through trusted channels.
Government use deserves special restraint. A state with powerful prediction and persuasion systems could personalize public communication helpfully—or make dissent easier to identify and behavior easier to shape. Legal prohibitions, warrants, minimization, audit logs, and independent oversight matter more as analytical capability increases, not less.
Preserve plural sources of power
No single company should be the sole source of public analysis, identity, communication, and administrative infrastructure. No central government should be the only institution able to use advanced capability. No universal scoring system should determine access to employment, credit, benefits, speech, and mobility.
Pluralism creates friction, and friction is sometimes protective. Federal systems, local governments, courts, independent regulators, universities, unions, professional associations, open technical communities, and civil society can expose failures one central system would suppress or miss. Competing institutions also give people somewhere to appeal and organize.
Pluralism is not the same as uncontrolled proliferation. High-risk capabilities can require access restrictions while decision-making remains institutionally diverse. The design problem is to avoid both private monopoly and total state centralization.
Decisions that should remain politically contestable
AI can inform public decisions without defining the public objective. Questions such as how to distribute income, which risks are acceptable, whose rights take priority, whether to use force, what children should learn, and how much surveillance is permissible are not optimization problems with neutral target functions. They are political choices among values and people.
Delegating implementation can also reshape the objective. A benefits system optimized to reduce processing time may deter complex applicants. A policing system optimized for predicted incidents may amplify patterns in historical enforcement. A health system optimized for measured cost may undervalue outcomes that are difficult to record. Democratic control requires reviewing what a system actually rewards, not merely approving its stated purpose.
Normative proposal: a minimum agency guarantee
For consequential public uses, people should be able to identify:
- that an AI system materially influenced the decision;
- the institution and official legally responsible;
- the governing rule and evidence used;
- important limitations, uncertainty, and data provenance;
- a timely human appeal with power to change the outcome;
- an accessible route to a court, regulator, ombudsman, or elected body;
- a non-digital path for people who cannot safely use the system.
These guarantees will not eliminate errors or political disagreement. They preserve the difference between administration and domination: the person affected remains a participant in a system of reasons and remedies, not merely an object being classified.
The post-AGI constitutional question
The deepest issue is not whether an AI receives a formal government title. It is whether humans and their institutions retain the competence, independence, and leverage to revise consequential systems. If removing one model would stop healthcare, finance, communications, and government at once, legal sovereignty offers little practical comfort.
Democratic survival after transformative AI therefore depends on capacity built before the dependency becomes complete: public expertise, interoperable infrastructure, enforceable rights, plural institutions, independent evidence, meaningful appeals, and tested ways to operate when automated recommendations are unavailable or rejected. Those are not anti-technology measures. They are the machinery that lets a society use powerful technology without surrendering the ability to choose its own purposes.