Walk into most boardrooms and the AI conversation is about models: Which one, build or buy, how fast can we roll it out. Reasonable questions, but a fraction of the problem. Boston Consulting Group puts numbers on it: Across hundreds of companies, about 10 percent of the value from AI comes from the algorithm, 20 percent from technology and data and 70 percent from the people and processes around it. The model everyone argues about is the easy part.
You see the problem after the pilot (I call it the “plausibility problem”). With capability everywhere, a team can summarize a contract or fire up a prototype in an afternoon. But a pilot only has to satisfy the team that built it. True production-ready solutions must satisfy everyone else from the data owners, legal, security, compliance to the executive who’ll take the heat when it goes wrong. For the first time, we’re seeing a technological era where access to the tech isn’t the constraint. Operational readiness is.
What Is the Key to Scaling Enterprise AI?
Governance and control, not the algorithm, are the key differentiators for scaling AI in the enterprise. While models account for only 10 percent of AI value, 70 percent comes from the people and processes around them. Successful implementation requires operational readiness built on three core properties:
- Visibility: Tracking step-by-step agent actions and data in real time.
- Predictability: Ensuring consistent behavior through explicitly codified norms and judgment.
- Control: Setting boundaries and human intervention limits that hold even under automated tasks.
‘Trust’ Is a Trap
The industry has settled on “trust” as the goal — trustworthy AI, responsible AI. I’d push back on that.
Trust is a human thing. It’s a quality you extend to something that has its own interests and could act against yours. A language model has no such interior life, and worse, it’s built to be agreeable. It’s trained to be helpful and deferential, some might even say sycophantic. It tells you what you want to hear, says it with confidence and folds the moment you push back. A fluent model earns misplaced trust easily. A careful one never feels trustworthy no matter how good it is.
The useful question is whether you’ve built the system around the model so that trust is beside the point. Three properties do that work.
Visibility
Can you see what the agent is doing, step by step? This includes the actual actions and data, not its own summary of them.
Predictability
Given the same situation, does it do the same thing? Predictability has a precondition most teams miss. An agent acts only on what’s been made explicit; it can’t absorb your norms by watching processes the way a new hire does. When a customer-facing agent meets a pricing exception no one codified, it doesn’t stop. Instead, it improvises and drifts off your goals. Undocumented judgment leads to undefined behavior.
Control
Can you direct it, bound it and stop it, and have those limits hold even when the model’s eagerness to help pushes against them?
None of the three requires trusting the model.
Governance Is a Design Problem
Because of my background in design, I’ve watched governance get filed as solely a technical problem. It’s a design problem, too. The most capable AI system on earth is worthless if the person using it can’t tell what it did, can’t anticipate what it’ll do, or doesn’t know when to step in. Most enterprise AI conversations dwell on infrastructure, models and architecture. Far fewer dwell on the experience around them, which is where visibility, predictability and control either materialize or stay opaque.
What does that look like in practice? Provenance has to be clear at the moment of decision, not buried in a log. When an agent recommends denying a refund, you need to see which policy it applied and the records it read right next to its recommendation. The system has to signal when it’s inside its competence and when it’s improvising so the user knows how much weight an output can bear. An answer drawn from governed, certified data should look different from one a model improvised. And the interface has to make plain where human judgment is still required along with a real way to intervene.
As AI moves closer to business-critical decisions, that design layer decides whether people can govern the system or merely hope it behaves. Transparency, accountability and oversight live in the interface or they don’t live at all.
The AI Question Everyone Avoids
Moonshots are sexy, but the projects that create value start small. This might be an app that reads invoices and routes discrepancies to a person or one that flags network disruptions in real time. When the problem is narrow and the judgment is clear, you can build solutions in days, not months. But the speed is earned as your teams learn how to build judgement into the system.
“How do we keep a human in the loop?” has become a reflexive response. But without a deliberate framework for deciding which loops actually need a human — and what that human needs to do — oversight doesn’t scale. It just risks turning every role into a glorified babysitter.
A cleaner approach is to map each task by the cost of an error and by whether it runs on explicit data or hard-to-codify tacit knowledge. Low-cost and explicit data — like categorizing expense reports, routing support tickets, matching invoices to purchase orders — let AI run unattended. High-cost or heavy judgment — like approving a non-standard promotion discount, responding to and angry customer or anything a regulator might question — always delegate to a person.
I’d add a third axis rarely considered: reversibility. A wrong answer you can roll back tomorrow is a different animal from one that’s cheap but permanent like a mispriced quote already sent or a deleted record. Agents make reversibility crucial because they act before anyone sees their decisions.
The Real Differentiator Isn’t Trust
The conversation around AI has often centered on which models are winning. That conversation will matter less over time. Models will improve. Capabilities will expand. Access to AI will become increasingly democratized.
Governance and control will remain the differentiator.
Organizations that establish trusted data foundations, clear oversight mechanisms, transparent decision processes, and effective human-in-the-loop controls will be able to scale AI confidently across the enterprise.
That isn’t trust. It’s something sturdier. And unlike trust, that’s something you can actually build.
