High-Consequence AI: When Outputs Affect the Physical World

When AI governance needs to go further

Most small businesses use AI to draft text, summarise documents, answer questions, generate images or automate routine administration. For these uses, a sensible AI governance policy — approved tools, data boundaries, human review, named accountability — provides reasonable protection.

Some AI uses are different. They produce outputs that can:

• Directly control physical systems
• Make or shape medical decisions
• Influence infrastructure or public safety
• Automate financial or legal consequences
• Affect employment decisions
• Design physical objects that will be manufactured
• Take irreversible actions without human approval

These uses deserve a different standard of control. This guide explains how to identify them and what stronger governance looks like in practice.

The central question

What makes an AI use high consequence?

Examples of high-consequence AI uses

High-consequence AI is not limited to frontier research laboratories. Businesses may encounter it in:

• AI agents that can send external communications, make payments or change system configurations without approval
• AI tools used in healthcare settings to inform clinical decisions
• AI tools used in financial advice, credit assessment or insurance underwriting
• AI tools used in employment decisions such as shortlisting, grading or dismissal recommendations
• AI tools that control physical devices, machinery or infrastructure
• AI tools that design objects, materials or biological systems intended for physical manufacture
• AI tools that can take irreversible actions such as deleting records, cancelling accounts or publishing legally significant content
• AI tools integrated with supply chains, logistics or safety-critical systems

Enhanced controls for high-consequence AI

1. 1CLASSIFY FOR ENHANCED REVIEW — Identify the use case explicitly as high consequence in your AI Tool Register. Document what makes it high consequence.
2. 2REQUIRE NAMED ACCOUNTABILITY — Assign a specific named person who is responsible for the AI's actions and outcomes. This person should be identifiable in any audit or incident review.
3. 3RESTRICT ACCESS — Limit access to only those staff whose role requires it. Apply user verification. Consider whether the tool should be available to junior staff without supervisor approval.
4. 4DEFINE HUMAN APPROVAL — Specify in writing which actions require a human to approve before the AI's output is acted on. Human approval must be meaningful — not a rubber stamp.
5. 5ASSESS DOWNSTREAM SYSTEMS — Map every system, supplier and physical process that can be triggered by the AI's output. Each link in the chain is part of the governance scope.
6. 6OBTAIN SPECIALIST ADVICE WHERE APPROPRIATE — Some domains require domain-specific expertise: legal, medical, financial, engineering, biosecurity. Standard IT governance advice may be insufficient.
7. 7MONITOR AND LOG — Ensure every significant action is recorded in a format that can be reviewed after the fact. Monitoring is not optional for high-consequence deployments.
8. 8REVIEW WHEN CAPABILITY CHANGES — A tool approved for one purpose under one capability level should be re-assessed when the underlying model is updated or when the tool's functionality expands.

The digital-to-physical boundary

One reason high-consequence AI deserves specific attention is that some AI outputs can cross from digital information into physical reality.

For most AI uses — text generation, image creation, document summarisation — the output stays in the information domain. The harm from an error is still real, but it is contained to information.

Some AI outputs can become physical objects, physical actions or physical consequences:

• An AI that designs a component which is then manufactured
• An AI that designs a biological sequence which is then synthesised
• An AI agent that executes a payment or modifies an infrastructure configuration
• An AI that controls machinery or physical access
• An AI that automates a legal, clinical or employment decision

Why this matters for everyday businesses

The AI-designed bacteriophage research published in August 2026 is an extreme example — most businesses will never work with biological AI. But the governance lesson is widely applicable.

The research demonstrated that generative AI can produce designs that cross from digital output into functioning biology. The same principle — that AI output can now trigger physical consequences — applies at much smaller scale in everyday business contexts.

Read: AI Has Designed Working Viruses — Why This Matters →

An AI agent with payment authority, a customer-service bot with the ability to issue refunds or cancel accounts, an AI that modifies access permissions — these are all examples where digital AI output creates real-world consequences that deserve proportionate controls.

Governance should match consequence

Generating an email and generating a biological design require radically different controls. Governance should be calibrated to what the output can cause, not just what the tool appears to do.

The Operational Heartbeat for high-consequence AI

High-consequence AI uses should be reviewed on a scheduled cycle — not assumed to remain safe because they passed an initial assessment. Capabilities change, models are updated, access controls drift and new risks emerge.

A recurring review should check:

• Approved purpose — is the AI still being used for what it was assessed for?
• Capability — has the model or tool changed since last review?
• Users — who has access, and is that still appropriate?
• Data — what information is entering the system?
• Tools — what can it call or trigger?
• Downstream actions — what can happen as a result of outputs?
• Human approval — are the right decisions still requiring human sign-off?
• Supplier safeguards — are the provider's controls still adequate?
• Incidents — have there been near-misses, errors or complaints?
• Research and regulatory changes — does new guidance apply?
• Specialist review — has appropriate expert input been obtained?
• Next review date and owner