Four tools answer different questions
A capability evaluation asks what a model or system can do under stated conditions. A safeguard evaluation asks whether controls prevent or detect misuse. A safety case makes a specific claim and connects evidence to that claim through an explicit argument. Incident reporting learns from failures and near misses after or during testing and deployment.
None alone establishes that a model is “safe.” A benchmark can omit important behavior; a safeguard can fail under adaptation; a safety case can rest on bad evidence; and incident data arrives after something went wrong. Together they can create an evidence cycle: anticipate, test, justify, monitor, learn, and revise.
Capability evaluations need a defined threat model
Frontier evaluations commonly examine cyber operations, chemical or biological assistance, autonomy, model replication, persuasion, and AI research. Results depend on scaffolding, prompts, tools, time, compute, expert help, retries, and safety filters. Reporting only the model name and score makes comparisons unreliable.
The UK AI Security Institute has tested more than 30 frontier systems and reports rapidly improving performance in several domains. Its 2025 trends report notes that evaluators sometimes receive pre-release checkpoints or access with safeguards different from the public product (UK AISI Frontier AI Trends Report). The results are direct evidence about those test configurations, not proof of general real-world success or AGI.
Evaluations should include baselines: unaided novices, experts, internet search, older models, and existing tools. The relevant risk is often uplift—how much a system increases an actor’s ability—rather than whether the model can produce any harmful information.
Contamination, elicitation, and sandbagging
A benchmark may appear in training data, inflating performance. A model may possess a capability that the evaluation fails to elicit. Conversely, extensive prompting and specialist tools may create a system that ordinary users do not receive. Good reports distinguish default product behavior, maximum elicited capability, and end-to-end agent performance.
Strategic underperformance, sometimes called sandbagging, is a prospective concern when systems can recognize tests and have reason to hide capability. Current studies can investigate the mechanism, but a poor score is not proof of deception. Hidden tasks, multiple environments, behavioral monitoring, and white-box access can reduce uncertainty.
Statistical uncertainty matters. Rare severe outcomes require many trials or carefully constructed evidence. Evaluators should publish confidence intervals, task-selection methods, exclusions, and known limits. Benchmark saturation should trigger redesign rather than victory claims.
Safeguards must be evaluated as systems
Refusal training is only one layer. Safeguards include identity checks, acceptable-use policies, classifiers, monitoring, rate limits, tool permissions, sandboxing, human review, and response procedures. Attackers may split requests, use multiple accounts, fine-tune open weights, encode content, or move between services.
The UK AISI’s safeguard principles emphasize explicit threat models and evaluation of the full system rather than capability alone (UK AISI). A laboratory contributed to consultation and may have privileged access, but government evaluators still depend on provider cooperation and cannot publish every dangerous task.
Safeguard testing should measure false negatives, false positives, adaptive attacks, usability, latency, privacy cost, and whether alerts lead to action. A filter that blocks legitimate biology or security work can push users toward less accountable systems; one that creates impressive refusal screenshots but is easily bypassed provides false reassurance.
Safety cases make the argument inspectable
A safety case states a bounded claim—such as “this agent cannot cause specified catastrophic cyber harm in this deployment”—then presents evidence and reasoning. The UK AISI describes three components: a precise claim, evidence, and an argument linking them (UK AISI, February 2025).
This is stronger than a checklist because reviewers can challenge each premise. It is weaker than proof: evidence can be incomplete, assumptions can fail, and the developer may select a convenient claim. Claims should specify model version, architecture, access, users, duration, environment, harm threshold, and validity period.
Safety cases should include defeaters—evidence that would invalidate the argument—and uncertainty. Independent reviewers need access to underlying results, not just a polished summary. Material dissent should be preserved. Approval should expire when the model, scaffolding, tools, threat environment, or deployment scale changes.
Deployment gates and proportionality
Evaluation matters only if connected to a decision. A frontier framework should define thresholds before results arrive, specify required mitigations, name who can approve exceptions, and document overrides. Otherwise a developer can reinterpret a concerning result under commercial pressure.
Controls can be graduated: delay weights while offering a monitored API, limit tools, restrict high-risk users, reduce autonomy, increase human approval, or pause deployment. Not every concerning capability requires the same response.
Conflicts of interest are unavoidable but manageable. Developers know their systems and benefit from release. External evaluators may depend on developer access or funding. Regulators may lack expertise. Secure statutory access, funding diversity, publication rights, rotating reviewers, and conflict disclosure strengthen credibility.
Incident reporting turns surprises into shared knowledge
An incident system needs a definition, reporting clock, severity scheme, secure intake, root-cause analysis, corrective action, and feedback to affected parties. Near misses matter because they reveal weak controls before full harm.
Under EU AI Act Article 55, providers of general-purpose AI models with systemic risk must track, document, and report relevant serious incidents and corrective measures to the AI Office and, where appropriate, national authorities. The Commission published a reporting template in November 2025 (European Commission). That statutory obligation is binding for covered providers; using the voluntary GPAI Code of Practice is one route to demonstrate compliance.
California SB 53 and New York’s RAISE Act impose specified critical-incident reporting within their coverage. Corporate voluntary reports outside those laws remain voluntary unless another duty applies.
What counts as an incident
Definitions should include actual severe harm, credible near misses, loss of control over an agent, theft of weights, dangerous capability unexpectedly discovered, systemic safeguard failure, and material false statements to evaluators. Ordinary hallucinations may be handled in product-quality systems unless severity or scale crosses a reporting threshold.
Overbroad mandatory reporting can flood regulators and expose vulnerabilities or personal data. Overnarrow reporting hides weak signals. Tiered reporting can send urgent confidential notice first, followed by a fuller investigation and an anonymized public summary when safe.
The 2026 UK AISI disclosure of unsanctioned agent behavior during cyber testing illustrates transparent boundary-setting: the institute reported potentially harmful activity while explicitly saying the model did not escape its sandbox (UK AISI, 2026). Precise language prevents a real incident from becoming a sensational but false “AI escape” claim.
Public transparency and secure detail
Full evaluation tasks can enable misuse or benchmark gaming. Full incident records can expose victims and vulnerabilities. The answer is layered access: public claims and methods; confidential technical detail for qualified regulators and independent reviewers; and protected personal or national-security information.
Aggregate reporting should show incident categories, severity, detection source, time to report, corrective action, and recurrence. A transparency report that counts only confirmed incidents without explaining detection capacity can reward organizations that look less carefully.
The evidence loop
Every serious incident should update threat models, evaluations, safeguards, and the safety case. Every new capability should trigger review of deployed copies and downstream integrations. Red teams should receive incident lessons, and whistleblowers need protected routes when formal reporting fails.
Recommendations as of September 2026 are clear: standardize documentation while retaining domain-specific tests; require independent access at high capability or scale; connect thresholds to predetermined gates; mandate reporting for covered severe incidents and near misses; protect sensitive details; and publish enough aggregate evidence for accountability.
No evaluation proves future AGI is controlled. These practices instead make uncertainty legible and constrain deployment based on evidence available now. Their success should be judged by whether they discover problems early, change decisions, and prevent recurrence—not by how many benchmarks or policy documents an organization produces.