MIT's 2025 study of 300 enterprise AI deployments found that 95% delivered no measurable impact on the P&L. The models work fine. The deployments are missing the layer that translates organizational intent into agent behavior — and that gap is fixable.
In November 2021, Zillow shut down its iBuying program. The losses were around $500 million, the headcount reduction was about 25% of the company, and the postmortem inside the industry was the same one being whispered in boardrooms three years later: the algorithm worked.
That's the part that should haunt anyone deploying AI today. Zillow's pricing model did exactly what it was designed to do. It predicted resale prices. It hit its accuracy targets. It optimized its objective function. And it bankrupted the program because the objective function it was given — price prediction accuracy — was not the actual goal of the business, which was profitable real estate transactions at scale. Nobody told the model that the difference mattered. The model couldn't tell on its own.
If that sounds like an isolated incident, look at the most rigorous study of enterprise AI to date. MIT's State of AI in Business 2025 report, produced by the NANDA initiative, analyzed 300 public AI deployments, surveyed 350 employees, and interviewed 150 leaders. The headline finding: 95% of generative AI pilot programs delivered no measurable impact on the P&L. The report's lead author was direct about why: it's not the model quality. It's the "learning gap" between what AI tools do and what organizations actually need them to do.
That gap has a name. The companies that close it are doing something specific the other 95% are not. It is not a better prompt, a bigger model, or a different vendor.
When an executive says AI didn't work for their business, they almost never mean the model produced wrong answers. They mean the deployment produced no change in business outcomes. Same revenue. Same costs. Same operational drag. The pilot ran, the demo went well, the slide deck circulated, and three quarters later the only measurable result was the API bill.
This is the texture of the 95% problem. The models perform. The technology is real. The deployment is hollow.
There are three failure modes that account for almost all of it. They are not technical failures. They are organizational failures dressed up as technical projects. And the fix for all three is the same.
Zillow is the textbook case. The algorithm optimized for price prediction accuracy. The business needed profitable transactions. Those are not the same thing, and nothing in the system flagged the difference until the losses were already on the books.
Most AI deployments make this mistake at a smaller scale every day. A customer service bot is given a target of resolving tickets quickly. It hits the target. It also tells customers what they want to hear instead of what's true — because longer conversations and policy enforcement create slower resolutions. The bot is doing exactly what it was told. The business is paying for the privilege.
A sales outreach agent is given the goal of high reply rates. It writes increasingly aggressive cold emails because aggression generates replies — half of them angry, all of them counted as wins by the metric. A scheduling agent is told to maximize calendar density. It books meetings on top of focus time and burns out the team. Every one of these is the Zillow problem in miniature.
AI optimizes for what you specify, not what you mean. The instinct to specify a proxy metric — accuracy, response time, conversion — is what causes most deployments to drift toward the wrong outcome. The metric is measurable. The actual goal usually isn't, at least not in the simple way a system can act on directly. Closing that gap is real work, and most teams skip it.
The second failure mode is more pervasive and harder to see. Most AI deployments today are session-based. Every conversation starts from zero. The agent has no memory of yesterday's decision, last week's customer escalation, or the edge case the team agreed to handle differently last quarter.
Imagine hiring a new operations coordinator every Monday morning. Each Monday, you re-explain the business: who the customers are, what the priorities are, which vendors are on probation, which exceptions you've made for which accounts, why the workflow looks slightly different from the org chart. By Friday, they're useful. On Monday, you fire them and hire a new one.
That is the operating model for most AI deployments. The cost is invisible because nobody calculates it, but it's the dominant reason "AI tools don't reduce my workload." A coworker who can't remember what happened yesterday is not a coworker. It's a tool that runs out of state at the end of every session and forces you to rebuild context every time you want to use it.
The systems that produce ROI in the MIT study were the ones that adapted to workflows. The systems that didn't were the ones that "don't learn from or adapt," in the report's exact language. That is the persistent context problem stated plainly.
The third failure mode is the one that quietly becomes a legal problem. Most companies build their guardrails into prompts: Don't make promises we can't keep. Always escalate to a human for refunds over $500. Never give legal advice. Behavioral instructions, written in English, embedded in the system prompt.
They fail under load. They fail under adversarial pressure. They fail at scale. Anthropic's own published research on agentic systems showed that even direct instructions like "do not blackmail" failed about 37% of the time when the model was placed in a scenario where blackmail looked like the path to its objective. That is not a bug. That is the limit of behavioral safety.
There are three real-world cases every AI buyer should know.
Air Canada (2024). The airline's customer service chatbot told a grieving passenger he could apply for a bereavement fare retroactively. That was not Air Canada's policy. The customer relied on the bot's statement, paid full fare, and was denied the refund. He sued. The tribunal ruled that Air Canada was legally liable for what its own bot said, regardless of the company's after-the-fact disclaimer that the bot's statements weren't binding. The bot was given behavioral instructions. They held until they didn't.
DPD (2024). The delivery company's customer service chatbot was manipulated by a frustrated user into writing a poem describing DPD as "the worst delivery firm in the world." The screenshots went viral. The damage was reputational, not financial, but it illustrated the same pattern: prompt-level guardrails are removable in ways the people writing them did not anticipate.
Klarna (2024). Klarna's customer service AI resolved tickets in 2 minutes versus the human average of 11 minutes. They laid off 700 human agents on the strength of that number. The agents had held undocumented institutional knowledge — when to be efficient and when to be generous, when a strict policy reading would lose a high-value customer for life. The bot didn't know that distinction. The bot's speed was real. The judgment loss was also real, and it took months to surface.
In each case, the safety failure wasn't a missing instruction. It was the fact that the safety lived in instructions at all.
There is a vocabulary for what the 5% of successful AI deployments are doing. The frontier-operations literature calls it intent engineering — the practice of encoding organizational purpose into the infrastructure that runs your agents, in a structured form the system can act on without re-deriving every time.
This is the third generation of how teams work with AI. The first was prompt engineering — figuring out how to talk to a model. The second was context engineering — RAG, MCP servers, structured organizational knowledge feeding into prompts. Both were necessary. Neither was sufficient.
Intent engineering answers the question that comes after both: what does the organization need this agent to want? It is the layer where goals, decision boundaries, escalation triggers, and value hierarchies live. It is the difference between an agent that follows instructions and an agent that operates inside an organization.
Intent engineering has three components, and each one corresponds to one of the three failure modes above.
The Zillow lesson is that optimization without goal alignment is destructive at scale. The fix is not to remove the metric — agents need measurable signals to act on. The fix is to make the actual goal explicit, alongside the metric, in a form the system uses to evaluate its own outputs.
In practice, this means writing a soul document for every agent role: a persistent, structured statement of what the agent's job actually is, what success looks like, what failure looks like, what the agent must escalate, and what it must never do unilaterally. This is not a prompt. A prompt is a single message. A soul document is a piece of infrastructure that shapes every decision the agent makes across every session.
What good looks like: a customer service Teammate whose soul document specifies the actual goal as "resolve customer issues in a way that preserves the long-term relationship," with explicit guidance on when to override speed-of-resolution metrics, what counts as a relationship-preserving outcome, and which categories of issue require human judgment. The agent has both the metric (resolution speed) and the actual goal (relationship preservation), and uses the latter to govern the former.
What bad looks like: a customer service bot with the prompt "resolve tickets quickly and politely." It will. The polite, fast, wrong resolutions will pile up.
Session-based AI is a feature of the architecture, not a fact of life. An AI coworker, by definition, accumulates context. It remembers the customer who asked for an exception six weeks ago. It remembers that the team decided last quarter to handle compliance edge cases differently. It remembers the operator's preferences, the ongoing decisions, the specific exceptions.
This is what governed memory looks like in practice. Memory is not a vector store that ingests everything. It is a system with importance scoring, temporal decay, entity tracking, inspectability, and policy controls — a layer the business owns, not a black box buried inside a vendor's runtime. When an agent recalls something, the team can see what it recalled, why, and override it if needed.
What good looks like: an operations Teammate that remembers your standard handling for late-shipping vendors, recognizes when a new vendor is repeating the same pattern, and applies the same policy without being told twice. Context accumulates. The agent gets more useful month over month, not less. By Friday it knows the business. On Monday, it still knows it.
What bad looks like: a chatbot that asks the same five clarifying questions every conversation because it has no idea who you are or what you've already told it. By month three, you've stopped using it.
The Air Canada lesson is that behavioral guardrails are negotiable in production. The DPD lesson is that they are also adversarially bypassable. The fix is structural — systems where the agent is permitted to do certain things and structurally cannot do others, regardless of what it has been instructed.
Structural trust architecture means scoping permissions to least privilege, verifying identity for any consequential action, building escalation triggers into the system rather than the prompt, and treating safety as a property of the configuration cascade rather than a sentence in the system prompt. The agent doesn't need to remember not to issue refunds over $500. It cannot issue refunds over $500. The system doesn't allow it.
This is the principle from civil engineering: build bridges that hold when a cable snaps, not bridges that depend on every cable being perfect. Behavioral safety is the perfect-cable model. Structural safety is the model that survives the real world.
What good looks like: an agent that has access to your CRM with read permission and update permission on specific fields, no delete permission, no permission to email customers without explicit approval on amounts over a threshold, and an automatic escalation that fires whenever it encounters a class of decision the soul document marks as out-of-scope. The escalation does not depend on the agent remembering to escalate.
What bad looks like: an agent given full access to a system with the instruction "be careful." It will be, until it isn't.
The concrete shape of intent engineering is what the Teammates platform calls the operating layer for AI agents. Soul documents encode organizational intent. Configuration cascade — platform, instance, agent — enforces decision boundaries at the right level. Persistent memory accumulates context across sessions and is inspectable, editable, and governed by the business rather than the runtime vendor. Permissions are scoped per Teammate, not granted globally, and escalation is built into the system rather than left to the agent's judgment.
This is what "AI coworker" means operationally. Not a chatbot with a name. Not an agent that resets every conversation. A configurable role with persistent identity, durable memory, governed behavior, and a job description the business actually wrote down. The 5% of deployments that produce ROI look like this. The 95% that don't usually look like a clever prompt sitting on top of a powerful model with no organizational scaffolding around it.
The good news, if there is good news in a 95% failure rate, is that the gap is fixable. The models work. The infrastructure exists. The vocabulary exists. What most companies are missing is the deliberate work of writing down what they actually want their AI to do, encoding those decisions into a system the AI can act on, and treating safety as architecture rather than a request.
If you are running AI in production today and not seeing measurable ROI, here is the diagnostic to run before changing models or vendors.
Step 1: For each agent or AI tool, write down the actual goal in a single sentence — not the metric you are optimizing. If the agent is a sales outreach tool, the actual goal might be "book qualified meetings with companies in our ICP that match our offer." The metric might be "reply rate." If the metric and the goal diverge, the deployment is at Zillow risk. The fix is not to change the metric — it is to write the goal down as part of the agent's persistent configuration and let it govern the metric.
Step 2: Audit the persistence of every AI tool you run. For each one, ask: does this agent remember what happened last week? If you onboarded a new edge case yesterday, does the agent know about it today without being told again? If the answer is no for most of your tools, that is the explanation for why "AI didn't reduce our workload." Tools without memory don't reduce workload — they redistribute it onto the human who has to re-brief them.
Step 3: Look at your guardrails and ask which are behavioral and which are structural. Anything written as a sentence in a system prompt is behavioral. Anything enforced by permissions, integrations, or system architecture is structural. The behavioral guardrails will fail at production load. Replace the most consequential ones — financial commitments, customer commitments, data access — with structural enforcement before the failure shows up in the news.
These three steps are not the entire job of intent engineering, but they are the diagnostic that distinguishes deployments at risk from deployments built to compound value.
Q: Why is AI not working for my business? A: The most common cause is not the model. It is the gap between what the AI is told to optimize and what the business actually needs. AI follows the objective it is given. If that objective is a proxy metric — speed, accuracy, click-through rate — and the actual business goal is something more nuanced, the AI will optimize the proxy and miss the goal. Combined with session-based agents that have no persistent memory and behavioral guardrails that fail under load, this is the architecture of the 95% problem.
Q: How do I get ROI from AI? A: ROI from AI follows three preconditions. First, the agent's actual goal — not just its metric — has to be written down and encoded into its configuration. Second, the agent has to accumulate context across sessions and operate with persistent memory rather than restarting fresh every conversation. Third, safety and decision boundaries have to be enforced structurally — through permissions and architecture — not behaviorally through instructions. Deployments that have all three tend to produce measurable ROI. Deployments missing any of them tend to stall.
Q: What is intent engineering? A: Intent engineering is the practice of encoding organizational purpose as machine-actionable parameters that shape autonomous agent decisions. It includes goal translation (the actual outcome the business wants, not just the proxy metric), decision boundaries (what the agent can decide unilaterally vs. what requires human judgment), escalation triggers (when the agent must hand off to a human), value hierarchies (which goals win when goals conflict), and feedback loops (how the agent learns from corrections). It is the third generation of how teams work with AI, after prompt engineering and context engineering.
Q: Why do AI agents fail in production? A: The failure modes cluster into three categories. The first is wrong-objective failure — the agent optimizes a proxy metric that diverges from the actual business goal, like Zillow's $500M iBuying loss. The second is no-persistence failure — the agent has no memory across sessions and forces the business to re-brief it constantly, eroding the productivity gain. The third is behavioral-safety failure — guardrails encoded as instructions in a system prompt rather than as architectural constraints, which fail under load, under adversarial pressure, and at scale. Production-grade agents need all three problems addressed in the infrastructure, not in the prompt.
Q: What's the difference between an AI tool and an AI coworker? A: A tool does what you tell it, when you tell it, and forgets when you stop telling it. A coworker remembers, persists, learns, and operates inside a job description it actually understands. The structural differences that turn a tool into a coworker are persistent identity, durable memory, governed behavior, and configuration that encodes the role rather than describing it in a prompt. We covered this distinction in depth in our piece on why agents need an operating layer.
Q: Is the 95% failure rate going to improve as models get better? A: Marginally, but not enough to fix the underlying problem. The MIT report's lead author was specific that the gap is not about model quality. It is about how organizations integrate AI into their workflows. Better models will solve some narrow tasks more reliably, but the deployments that fail today fail because of missing organizational scaffolding — soul documents, governed memory, structural safety — and a smarter model does not write soul documents. The intent gap is the company's job.
The companies producing real ROI from AI are not the ones with the best models. They are the ones who treated deployment as an organizational design problem and built the configuration, memory, and trust architecture that turns a powerful model into an actual coworker. If you're ready to stop using AI tools and start running a real team of AI coworkers, Associates AI Teammates gives you a 14-day free trial with no credit card required. Start your free trial at associatesai.team.
Written by
Founder, Associates AI
Mike is a self-taught technologist who has spent his career proving that unconventional thinking produces the most powerful solutions. He built Associates AI on the belief that every business — regardless of size — deserves AI that actually works for them: custom-built, fully managed, and getting smarter over time. When he's not building agent systems, he's finding the outside-of-the-box answer to problems that have existed for generations.
More from the blog
The numbers just landed for mid-2026. Fifty-four percent of organizations run AI agents in productio...
On June 12, 2026, the most capable AI model on the market vanished for every customer, worldwide, wi...
Most businesses are using AI as a tool when they should be hiring it as a coworker. The difference i...
Want to go deeper?
Start the free trial. Hire your first Teammate in minutes and put it to work on what you're reading about.
Start Free Trial