Start with the function, not the label
An autonomous weapon is generally understood as a system that, once activated, can select and engage targets without a human approving each individual strike. Definitions remain contested because autonomy is not a single switch. Humans may choose the mission area and target class while software navigates, identifies an object, recommends action, or decides the exact moment to fire.
This spectrum includes long-standing systems such as defensive interceptors as well as newer loitering munitions, drone swarms, AI-assisted targeting tools, and armed ground robots — the wheeled, tracked, and legged platforms surveyed in this manual’s military humanoid and ground robots page. Risk depends on the environment, target, time pressure, predictability, communications, and consequences of error. A tightly bounded system intercepting incoming munitions is different from one searching a city for people.
AI-assisted decision support also deserves scrutiny even when a human formally pulls the trigger. If an operator receives hundreds of machine-generated targets under severe time pressure, approval may become a rubber stamp. “Human in the loop” says little unless the person has information, authority, time, and training to challenge the system.
What is documented—and what remains disputed
A 2021 United Nations Panel of Experts report on Libya described Kargu-2 loitering munitions and other systems used against retreating forces, noting that the systems were programmed to attack targets without requiring data connectivity between operator and munition (UN Libya report, paragraph 63). The passage is frequently called the first autonomous-drone attack on humans. However, the public report does not establish conclusively that a Kargu-2 independently selected and killed a person. Experts dispute the operating mode and outcome. It is evidence of deployment in a relevant context, not proof of a confirmed autonomous kill.
More recent conflicts have featured extensive drone use, computer vision, automated intelligence processing, and AI-supported target recommendations. Public reporting often lacks enough technical access to determine which model performed which function or how much control operators retained. Military claims, vendor descriptions, and press accounts should be labeled rather than merged into one certainty.
The direction of development is nevertheless clear: armed forces are investing in systems that can operate when communications are jammed, process sensor data quickly, and coordinate many inexpensive platforms. Greater autonomy may be tactically attractive because a continuous remote-control link is vulnerable. Whether that produces safer precision or faster, less accountable violence depends on design and doctrine.
Three different escalation pathways
Speed escalation occurs when automated warning, targeting, or response compresses decision time. In a crisis, leaders may fear that waiting allows an opponent’s machine-speed system to gain an advantage. That can create pressure to preauthorize actions. A false sensor reading, model error, or ambiguous maneuver might propagate before people understand it.
Scale escalation occurs when autonomy makes it cheaper to coordinate many systems or generate targets. A human organization that could carefully review dozens of decisions may not meaningfully review thousands. Greater volume can increase accidental encounters, civilian harm, and demands for retaliation even if each system is individually more accurate.
Perception escalation occurs when states misread an adversary’s system, doctrine, or level of control. Secrecy makes it difficult to know whether a weapon is defensive, whether an action was intentional, or whether a human can stop it. An AI-generated recommendation may appear objective to one side and aggressive to another.
These are plausible mechanisms, not evidence that AI will inevitably start a war. Automation can also improve warning, reduce operator overload, intercept threats, and support de-escalation. The outcome depends on testing, communications, doctrine, and shared constraints.
Nuclear systems deserve a higher bar
Connecting AI to nuclear command, control, communications, or early warning raises exceptional stakes. Machine learning could help analyze large sensor streams, but opaque errors, adversarial manipulation, and distribution shift are dangerous where false confidence can be catastrophic. The risk is not necessarily an AI independently “launching nukes.” It includes leaders receiving compressed, poorly calibrated advice in minutes.
Maintaining human responsibility over nuclear-use decisions is therefore more than symbolism. Systems should provide uncertainty and provenance, preserve independent channels, resist spoofing, and allow leaders time to verify. No single model output should become a decisive indicator. Exercises should include AI failure and deliberate manipulation, not only equipment breakdown.
Accountability cannot end at the algorithm
International humanitarian law applies regardless of the technology used. Distinction, proportionality, and precautions in attack remain human and state obligations. Yet autonomy complicates investigation: a harmful outcome may involve training data, sensor limitations, a commander’s parameters, a software update, or operator misuse.
That complexity is not an excuse for an accountability gap. States can require traceable logs, legal review, auditable testing, clear command responsibility, and limits on environments where a system may operate. Developers and commanders should define foreseeable failure modes and stop conditions. A system too unpredictable to review should not be deployed merely because responsibility is distributed.
Bias and context matter as well. A vision system tested on clean imagery may fail in weather, rubble, camouflage, or unfamiliar populations. Adversaries can deliberately deceive sensors. Accuracy averaged across a test set does not answer whether the system is reliable for a particular target class and environment.
The international policy landscape
States have debated lethal autonomous weapons within the Convention on Certain Conventional Weapons for years without agreeing on a binding instrument. The United States launched a nonbinding Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy in 2023, committing endorsers to principles including senior oversight, rigorous testing, and the ability to disengage systems that behave unexpectedly (US State Department declaration).
The declaration can build norms and practical cooperation, but endorsement is not enforcement. Civil-society and state proposals range from prohibiting systems that target people without meaningful human control to regulating particular functions and requiring positive obligations. Disagreement persists over definitions, verification, and military advantage.
Useful near-term measures do not have to wait for a comprehensive treaty. States can exchange incident information, prohibit autonomous nuclear-launch decisions, require weapons reviews, set testing standards, protect human control over person-targeting, and establish communication channels for AI-related accidents.
What meaningful human control requires
The phrase should be operational. A human needs a comprehensible objective and target boundary; reliable information about the system’s confidence and limitations; enough time to intervene; a usable stop mechanism; and legal authority to refuse. Interfaces should resist automation bias rather than encouraging rapid confirmation. After action, records must support investigation and remedy. This manual’s companion page on ground robots, humanoids, and the laws of war examines how this human-control requirement is actually being implemented — or not — in fielded armed ground robots today.
Control also begins before an engagement. Political leaders decide procurement and doctrine; commanders set parameters; engineers select data and tests. Focusing only on the final button press overlooks these consequential choices.
A measured conclusion
Autonomy in weapons is already a policy and battlefield reality, but public evidence about particular engagements is often incomplete. The Kargu-2 report should not be inflated into a proven autonomous killing, and supervised targeting should not be mislabeled as fully independent warfare.
The catastrophic concern rests on credible pathways: faster decisions, mass coordination, fragile human review, and interaction with strategic warning systems. None makes escalation inevitable. Strong human authority, auditable design, legal accountability, shared norms, and explicit limits around nuclear and person-targeting functions can reduce risk while evidence continues to develop.