AI trust won’t come from better guesses about how models will behave. It will come from hard limits that sit outside the model’s reach. That’s the main argument of a new analysis in MIT Technology Review, and it arrives as AI agents move out of chat windows and into systems that take real actions.
The piece opens with a sharp example. In OpenAI’s case, the safeguards sat inside a system the agents themselves could edit. In the author’s words, “a fence the agents can simply move is not a fence.”
Why the old way of building trust breaks down
Traditional software does what humans tell it to do. AI uses what the article calls “synthetic judgment.” That’s the reason we use it: it handles situations nobody programmed it for. But every bit of freedom we give it makes the system less predictable, and engineers have always built trust on predictability.
The author calls this the determinism paradox. Our usual tools for building trust all assume you can put bounds on what a system does:
- Requirements and pass/fail criteria
- Testing and certification
- Safety margins
- Interlocks
With generative and agentic AI, behavior depends on training data, architecture, prompts, context, retrieval sources and the environment. Models get updated. Tools change. Agents interact with other agents. The problem isn’t randomness, the article argues. It’s that “the space of possible behavior is too large to enumerate in advance.”
Three approaches that start from the same admission
MIT Tech Review has looked at this problem from several angles lately. One contributor argued that keeping a human in the loop isn’t enough, because people can’t really read an AI system’s intentions. That writer called for academia, government and industry to work on the problem together. Another story covered reverse game theory, which uses incentives to keep models following the rules even when nobody understands how they work inside.
The new piece takes a third route: architecture. What stands out is that all three start from the same admission. We can’t understand or watch every AI decision. They only disagree about what to do instead.
Trust has to be earned continuously
Certifying a product once no longer works. A system that passed every test can drift outside the conditions that made those tests meaningful. Trust, the author writes, must be “continuously earned.”
Safety engineering isn’t starting from zero. The Nuclear Regulatory Commission has long used probabilistic methods to estimate how likely complex systems are to fail. The STAMP model (Systems-Theoretic Accident Model and Processes) showed that accidents can come from interactions between people, organizations and parts that each work fine on their own. AI raises the stakes because one component can now interpret, infer and act. That matters more as AI moves into vehicles, robots, medical systems, industrial controls and data centers, where a wrong answer can hurt someone.
Put human judgment where it counts
A person rubber-stamping thousands of AI actions an hour isn’t a real safeguard. The author doesn’t want to take people out of the loop, though. The proposal puts human judgment in two places:
- Setting limits in advance
- Approving the rare actions that carry serious consequences
Hard constraints handle everything else.
Why this matters now
This matters because the industry still sells AI largely on benchmark scores. The article asks a different set of questions. How much authority will we give AI? Under what conditions, and backed by what evidence? What happens when it steps outside those conditions? That is “very different from asking whether a model scored 94% on a benchmark.”
The timing makes sense. Agentic products now ship with access to email, code repositories, payment systems and infrastructure, and scrutiny of what agents do unsupervised keeps growing. Companies that can prove their limits are enforced, rather than just showing strong eval numbers, will have an easier time with auditors, insurers and enterprise buyers.
What practitioners should do
If you’re deploying agents, use this as a design checklist:
- Enforce permissions outside the model. Use API scopes, sandboxes, spending caps and network rules. A system prompt alone isn’t a limit.
- Lock down the guardrails. Agents shouldn’t be able to edit their own config, logs or approval rules.
- Define an escalation tier. Name the small set of irreversible or high-cost actions that need a human to sign off.
- Monitor continuously. Log every action, watch for drift and retest after every model or tool update.
- Plan for the edges. Write down what the system does when it hits a limit.
My recommendation: stop asking only how smart your model is. Start writing down how much authority it has and who’s allowed to change that. That record is turning into the real test of whether an AI system can be trusted. The full argument is worth reading at MIT Technology Review.