When AI Must Say No - Part 1: Building and Operating Reliable AI Agents in Industry.
AI agents are a new type of software. Unlike static programs that do exactly what they are programmed to do, agents independently and dynamically plan a path to a solution, query data, call tools, and decide on the next step. This makes them attractive for knowledge-intensive processes, ranging from service and diagnostics to engineering and design, energy management, robot control, and dynamic pricing. But this very autonomy raises a crucial question: How do you ensure that an agent does not do what it shouldn't?
One possible answer lies in two levers that together determine whether an agent only impresses in pilot mode or actually delivers in production: How is it built? And how is it operated?
Lever 1: Security layers as architecture, not as a prompt
A language model alone is not yet a production-ready agent, let alone a trustworthy one. Reliability is created through layers of classic program code that run in parallel to the agent and monitor its actions. It is important to understand that none of these layers should rely on a friendly reminder in the system prompt. That would be far too weak.
We consider three layers to be absolutely essential for industrial agent systems:
- Policy. A rule engine alongside the agent checks every recommendation against operating limits, safety requirements, and compliance rules. The language model does not decide on the rule; the rule code does.
- Tool Security. Every action passes through a controlled wrapper that validates parameters, checks permissions, and, if necessary, performs a dry run of critical actions in advance. The agent can only use tools that are explicitly authorized in its tool registry.
- Logging. Every decision is logged as a machine-readable event, typically OpenTelemetry-compatible. This ensures it remains traceable in the event of a complaint, an audit, or in court.
Three additional layers should logically complete the picture: clear identity and access management based on the principle of least privilege with short-lived tokens, a data protection layer that masks sensitive content before and after every request, and behavioral monitoring that detects drift and anomalies in agent behavior.
These layers serve to avoid risks that are particularly critical in industrial applications. These include excessive freedom of action, indirect prompt injection, or unlimited resource consumption in loops.
Measure reliability, don't just claim it
In machinery and production, we talk about acceptance testing, while for AI systems, we talk about evaluation, or "evals" for short. In practice, a set of four classes has become established: a technical secondary review by another language model (LLM-as-a-judge), an input-output check against curated reference pairs (golden dataset), targeted attack attempts by a dedicated red team (penetration testing), and continuous runtime monitoring using anonymized real-world requests in production.
The classes are held together by a continuous integration process that automatically runs all tests whenever a change is made to the agent. If a previously functional capability breaks, the change is not rolled out.
Lever 2: Autonomy as a dial, not a switch
Building an agent is one thing; operating it responsibly in day-to-day business is another. This raises questions such as: How much independence should it have? How does it recognize when it is unsure of itself? And what happens when things go wrong?
To categorize this independence, three levels have proven effective (Staged Autonomy):
- Informative. The agent provides suggestions, but the human performs the execution. Technically, the agent has read access to data and documents, but no write access.
- Supervised. The agent is permitted to act independently within clearly agreed-upon parameters. If it moves outside this framework, it automatically escalates to a human.
- Autonomous. The agent acts independently within a defined scope of tasks. Humans only intervene if the audit layer reports an anomaly.
While levels 1 and 2 belong to the category of Human-in-the-Loop (HITL) , level 3 falls into the class of Human-on-the-Loop (HOTL), where a human observes the process but does not need to explicitly approve actions before the agent performs a write operation.
The user decides which level applies to each use case, not the provider.
For the agent to be able to escalate to humans in levels 2 and 3, it must have an understanding of how confident it is in its answer. Uncertainty quantification methods generate a corresponding confidence score, for example, by answering the same question multiple times and comparing the variance, by using a second evaluation model, or by cross-referencing with reference values from documentation. If the confidence falls below an agreed threshold, the agent automatically escalates to a human, regardless of the formal autonomy level assigned.
An agent must not only know how to work, but also how to stop working when something goes wrong. Three consecutive patterns have become established in practice: confidence-based escalation, followed by a gradual fallback to a simpler deterministic solution, or, ultimately, an emergency shutdown in the event of repeated errors. The latter is comparable to a classic fuse that cuts the circuit in the event of an overload.
What this means in practice
In industrial environments such as mechanical engineering or manufacturing processes, the audit layer is not optional—it is mandatory. Industrial quality assurance requires that any error can be reconstructed after the fact. This applies just as much to an AI agent as it does to a traditional control system.
And this principle is transferable. An agent-driven robot must not have impermissible parameterization. An agent-controlled air conditioning system in a server room must not simply shut down to save power. A pricing agent must not fall into a price spiral when competing with other agents. In all three cases, the architecture that prevents this is the same: layers, evals, and controllers.
Key Takeaway
Reliability is built and measured, not claimed. Autonomy is regulated, not just switched on. Anyone embedding an agent into a real-world operational workflow is shifting responsibility into the architecture. This responsibility can be met, but it requires Engineering, not just licensing.
So much for the technical logic. Which of these measures will become not just sensible, but regulatory requirements starting in 2026, we will examine in Part 2 of this series.
If you want to know more right now, you can find further details and practical recommendations for mechanical and plant engineering in our white paper, “When AI Agents Must Say No.”
