Image created using AI tools

I attended a Databricks webinar earlier this month on data-driven AI governance, led by Matei Zaharia, the company’s cofounder and chief technology officer, with K. Sewell, who directs product for their AI products. Zaharia opened by saying that agents are beginning to go rogue inside enterprises, and he was specific about what he meant. Agents consume unexpected quantities of compute and tokens and costs escalate. Agents without adequate safeguards take harmful actions, and he named deleting production data as one of them. Agents are vulnerable to prompt injection, where untrusted content reaches a system that can then send information out of the company.

His diagnosis of why this happens is the part worth sitting with. Enterprises do not have enough data about what their agents are doing to make good decisions about them, so they are left with blunt instruments. A company can cap an engineer at a fixed monthly spend, but that cap cannot tell the difference between someone spending heavily because the work is producing value and someone whose agent has simply become expensive. A policy can forbid an agent from deleting files, even when deletion was a legitimate part of the task.

The Databricks answer is three words. See it, by collecting information about agents and the people using them. Show it, by putting that information in front of the people who can act on it. Act on it, by giving administrators and users real mechanisms rather than a single blunt limit.

It is a simple framework and I think it is a good one. What made the hour stay with me was not the framework but the account of what happened when they applied it to themselves, because that account contains three findings the company seems to have treated as incidental and I would treat as the main event.

What happened when the number became visible

Databricks had made coding tools available to engineers and adoption was uneven. Some of their strongest engineers were using them heavily and reporting real productivity gains, many others barely at all, and their engineering managers were among the people not using them. So the company began gathering usage data and making it visible.

The immediate effect was social rather than technical. Engineers could see that some colleagues were using these tools far more than others, and they started asking those colleagues what they were doing and whether it was working. Teams learned from each other. Zaharia describes the visibility as raising awareness, and it is worth noticing that the mechanism which actually moved behavior was people asking people, not the dashboard itself.

Then, in his own account, it produced the opposite problem. Some engineers began trying to maximize their score. They stopped considering whether an expensive model suited the task, handed the work over, and went to lunch while it ran. Costs rose, and the company responded by showing spending closer to the point of use so that an engineer could see what a session was costing while it happened.

It may be an easy reading here that Databricks was careless, no, it was not. They are among the most instrumented organizations you could choose for this. They run one of the world’s major data platforms, they have deep engineering teams, extensive traces, internal benchmarks and the ability to experiment on their own AI usage at scale. What happened to them is not a failure of sophistication. A company full of world-class engineers published a number, and people optimized the number rather than the outcome, which is Goodhart’s law arriving exactly on time. That they said so publicly is generous and unusual.

What a metric can answer, and what it cannot

Usage was worth measuring. If a company buys AI tools and nobody touches them, that is important to know, and usage is a reasonable signal of adoption. The difficulty is that usage is not the same as good use, and the gap between them is where most of the interesting questions live.

An employee can use AI constantly and badly. Another can use it rarely and exceptionally well. A third may use it for work whose value is genuinely hard to quantify. A fourth may generate enormous token consumption because the system around them is inefficient rather than because they are productive. Usage can answer questions about usage. It cannot tell you whether the use was appropriate, whether the output was sound, whether the person exercised judgment, whether the task should have gone to AI at all, or whether the organization became more capable as a result.

This matters now because enterprise AI produces a remarkable amount of measurable activity. Organizations can count licenses, active users, prompts, tokens, cost, latency, model calls, agent actions, training completions, inventoried systems, incidents and policy violations. Every one of those numbers can be useful. The risk is not that they are wrong. The risk is that an available number quietly stands in for a harder question because it is the one we happen to have.

If ninety percent of employees completed AI training, is the workforce capable of using AI well? If usage doubled, did productivity double? If an AI system has a named owner in an inventory, does that person understand what accountability requires when the system contributes to a decision that affects a customer? If there is a human in the loop, does that human have the knowledge, the authority, the time and the context to overrule the machine? The available number and the question we care about are related. They are not the same.

The measurement that pointed at the organisation

Databricks did not stop at counting usage, and the next thing they did is the reason I think their case is instructive rather than cautionary.

They built an internal benchmark for coding agents out of their own engineering work, then compared agents and configurations on both success and cost. That let them ask better questions. Was maximum reasoning effort necessary for every task? Did the harness around the model affect the result? Could a cheaper configuration do as well? The answers changed what they did. Lowering default reasoning effort and tuning the harness cut coding costs substantially without compromising quality, and, notably, without asking a single engineer to change how they worked. The most effective intervention in the whole account was invisible to the people it affected.

Then came the finding that was treated as an aside. When they sliced the benchmark results by project, they found parts of their own codebase where every agent struggled and burned tokens. The obvious interpretation was that the agents were weak there. Zaharia offered a different one: perhaps that code needed better documentation, and then the agents would do better and cost less.

Look at what happened in that sentence. An instrument built to evaluate models and harnesses produced a finding about the company. Not about the AI, but about an artifact the organization had made for itself, and a gap in it that nobody had noticed until a machine failed against it.

That is not a small thing, because AI never operates alone. It runs inside an environment that people built, made of data, documentation, workflows, permissions, incentives, review steps, escalation paths and assumptions about who is responsible for what. When an AI system performs badly the model may be the problem, and it may not. The documentation may be poor, the workflow badly designed, the task wrongly chosen for automation, the reviewer unclear about what to verify, or the responsibility so distributed that everyone assumes somebody else is checking. Those are not model problems. They are properties of the system the model sits in.

Outside engineering, the artifact disappears

Software is an unusually generous environment for this kind of measurement. You can hand an agent a task, let tests decide whether the code behaves, record what it cost, and compare configurations against each other. The feedback loop is fast and the artifact is countable.

Most of the enterprise does not work like that. In marketing, someone uses AI to analyze customers, shape a campaign, summarize research or draft copy, and the value of any single interaction may not be visible for months. In legal work, the number of prompts tells you nothing about whether the person caught a subtle error or verified a source. In finance, an answer can look entirely plausible while containing a mistake only domain knowledge would reveal. In human resources and in healthcare, the quality of the human’s interaction with the system matters as much as whether the system produced an output at all, because the output lands on a person.

Databricks’ own published guidance contains the admission that makes this concrete. Hard budgets are a last resort, they say, because cutting a developer off is debilitating and because some of the highest spenders are the people producing the most. So the ladder they recommend runs from visibility to gates to downshifting, with suspension only at the end. Every rung of it depends on telling the productive high spender from the wasteful one, and their proxy for that is pull requests shipped. It is a decent proxy and it exists in engineering because engineering produces a countable artifact on a short cycle. There is no ships-fourteen-pull-requests number in legal, in finance, in operations or in a clinic, and those functions are adopting AI at least as quickly.

Training completion is not competence

Organizations respond to all of this by investing in education, which is sensible, and education produces convenient numbers. Eighty-seven percent completed the training. Four thousand people attended the workshop. Two hundred managers finished the leadership program. Those numbers tell you that learning activity happened. They do not tell you what anyone can now do.

Education has known this for a century. Attendance is not learning, exposure is not competence, and finishing a course is not the same as applying it under real conditions. AI makes the distinction sharper, because using it well is mostly judgment rather than technique. Knowing when an answer is unsupported. Knowing what should never be typed into a system. Knowing which tasks are yours alone. Knowing that the standard of verification should rise with the cost of being wrong. Knowing that you remain accountable for a decision the machine recommended. None of that is an attendance record.

I do not want to assume this is everyone’s problem

There is a familiar danger in working on something for a long time, which is that you begin to see it everywhere.

I can construct a persuasive argument that organizations have a measurement gap around AI. That argument does not establish that anyone experiences the gap as painful, that anyone would pay to close it, or that it is urgent this year rather than eventually. And the fact that a company like Databricks can invest heavily in sophisticated measurement tells us very little about what a mid-sized healthcare provider, insurer, manufacturer or public sector organization is living through right now.

That evidence has to come from organizations themselves, which is why I am running a short study rather than writing another argument.

I am not asking anyone to predict the future. I am asking what has already happened. When AI raised a question that was difficult to answer, what was the question, and who needed the answer? What did it take to find it, and what did the difficulty cost in time, delay, friction or money? How does leadership currently work out what is happening with AI across the organization? How does anyone know whether employees have the judgment to use it well? When a customer, a board, an auditor, an investor or a regulator wants confidence in how a company uses AI, what can that company actually show them? What makes the question urgent, who owns it, and what has already been spent trying to solve it?

Those answers are worth more to me at the moment than another theory about what companies ought to need. The study takes six to ten minutes, every question is about events that have actually occurred, and I will send participants what comes back.

See if this is your problem too

Thoughts and Analysis by Suneeta, drafted by AI

Draws on Matei Zaharia and K. Sewell’s Databricks webinar on data-driven AI governance, August 2026, and on Databricks’ published accounts of benchmarking coding agents and managing AI coding costs, July and August 2026.

Image generated using AI tools

There is a slab of marble set into a wall on the Place Vendôme in Paris, at street level, where anyone walking past can put a hand on it. It is a little over three feet long, with a groove cut horizontally into the stone and a raised flange at each end. Between those two flanges is one metre.

In 1796 the French government installed sixteen of these around the city and called them mètres étalons, standard metres. The revolution had abolished a measuring system in which the same word meant one length in Lyon and a different length in Marseille, and in which the length a merchant used was frequently the length that suited the merchant. The new unit had to be taught to people who had never heard of it and had no reason to trust it. So the state cut it into stone and put it in the walls, at the height of a person’s hands, in the streets where people actually bought and sold. A draper could bring his cloth to the wall.

Two of the sixteen survive. The one on the Place Vendôme is still there, and it is still correct.

That is a stranger sentence than it looks, because the official definition of the metre has been replaced four times since that marble was cut. In 1799 it became the length of a platinum bar deposited in the Archives. In 1889 it became a different bar, of platinum and iridium, held near Paris and copied out to the nations. In 1960 it stopped being an object at all and became a number of wavelengths of light emitted by krypton-86. In 1983 it became the distance light travels in a vacuum in one part of 299,792,458 of a second.

Four definitions. Four moments when the authoritative answer to “what is a metre” was replaced by a different authoritative answer. And through all four, the draper on the Place Vendôme was measuring his cloth against a groove in a wall, and getting the right answer every time, because the cloth was the length the cloth was. The definition kept moving. The thing being measured never did.

On the fourteenth of May this year, the Governor of Colorado signed a bill repealing the Colorado Artificial Intelligence Act.

The Act had been signed two years earlier, in May of 2024, and it was the first comprehensive state AI law in the United States. It was supposed to take effect on the first of February 2026. That date was pushed back. Then, before the new date arrived, the legislature repealed the whole thing and replaced it with a narrower law that takes effect on the first of January 2027, which is to say that the most consequential AI statute in the country was in force for exactly zero days.

Look at what was removed. The duty of care is gone. The requirement to run impact assessments on high-risk systems is gone. The requirement to operate a risk management program is gone. The whole framework built around algorithmic discrimination has been replaced by a framework built around disclosure, in which the obligation is largely to tell people that an automated system was involved in a decision about them.

Somewhere in Denver there is a company that spent eighteen months and a real budget building an impact assessment process for a statute that never came into force, and has now been told that the requirement it was built for does not exist.

Let’s be careful about the easy reading here, which is that this company wasted its money and that the sensible ones were those who waited. I do not think that is what happened, and the reason has nothing to do with law.

Consider what an impact assessment was for. Under the repealed Act, a company deploying a system that helped decide who got hired, or who got a loan, or who got into a program, had to examine whether that system treated people differently in ways it could not justify. Not whether it filed the right form. Whether the thing it had built did what it believed it did, to the people it did it to.

That question did not get repealed. It cannot get repealed. It is a fact about the system, sitting in the company’s own infrastructure, doing whatever it does every day to whoever it does it to, entirely indifferent to the Colorado General Assembly’s calendar.

The statute was never the ruler. The statute was one government’s attempt, at one moment, to say which measurements it would require you to take. The measurement itself, whether your hiring model rejects applicants from certain neighborhoods at a rate you cannot explain, is the groove in the wall. It was true before the Act, it was true during the Act, and it is true now that the Act is gone.

I am not arguing that the repeal does not matter. It matters a great deal, and any company operating in Colorado has to understand what it now owes and when. What I am arguing is narrower and, I think, more uncomfortable. If a company built its entire understanding of its own AI systems out of a statute, then when the statute was repealed, the company lost its understanding along with its obligation. It did not just stop having to file. It stopped knowing.

And there is a version of this that is worse than wasted money. A company that ran those assessments and found nothing troubling has learned something durable and can say so to a customer, a court, or a board, in 2027 and in 2030. A company that ran them, found something troubling, and has now quietly stopped looking because nobody is requiring it to look, has converted a finding into a silence.

The second company will look identical to the first from the outside. Both will have a folder of documents from 2025 and no documents from 2026. Only one of them knows why.

Nine other states have AI legislation at some stage. The European framework has its own timetable and its own carve-outs. A federal executive order landed in December and changed the political weather. Somewhere in the next three years, some of these will be delayed, some narrowed, some struck down, and at least one will be replaced by something nobody has drafted yet.

Every one of those events will move the statute. Not one of them will move the system.

So here is what I keep turning over. In May, when Colorado repealed the Act, a great many companies recalibrated. Some of them canceled the assessment program, some of them shelved the documentation, some of them moved the person who had been running it onto something else.

If you were one of them, or if you work for one of them, there is a question worth sitting with, and it is not a question about Colorado.

When the requirement went away, did you find out that your systems had actually got better?

Or did you only find out that you no longer had to say?

Until next week
Idea, thinking and editing: Suneeta Modekurty
Drafting: AI

Image generated using AI tools

Sometime late last year, Uber did what many technology leaders were either already doing or being told they should do. It put Claude Code in front of its engineers and encouraged them to use it as much as possible. This was not a cautious experiment tucked inside an innovation lab, waiting for a committee to decide whether …

Read more

Image by AI tools

There was a kind of danger, underground, that a man could not see or smell or feel, that made no sound and left no mark, and that would kill an entire shift of experienced colliers who had done every other thing correctly.

The gas had names, and the names tell you that people had learned it the hard way. Firedamp, which would take a flame and turn the seam into a furnace, and which at least announced itself when it went, so that the survivors knew what had happened and could tell you afterward. And blackdamp, which did something quieter and worse.

Blackdamp did not burn. That is the first thing to understand about it, and the thing that makes it a different order of problem. It was not a violence waiting for a spark. It was an absence. Oxygen consumed by the slow oxidation of the coal itself, or by timber rotting in the dark, or by the seam simply breathing out over centuries what it had held, and in its place carbon dioxide and nitrogen, gases that do nothing at all. That is precisely the trouble. They do nothing. They have no color to see and no odor to catch and no taste on the tongue and no heat and no motion. They do not attack a man. They merely occupy the space where his air used to be, and stand there, doing nothing, while he breathes them.

And now, let’s move through slowly…..

A man walking into a gallery filling with blackdamp does not feel alarm. He does not gasp. There is no moment of choking that would spin him around and send him running for the shaft. What he feels, first, is that the work is a little harder than it was, and he attributes this, reasonably, to the work, or to his age, or to a poor night’s sleep, or to the fact that it is Thursday and he has been down here since Monday. Then his thinking slows. Not dramatically. It is more that the sharp edges come off it, that the arithmetic of the shift takes a moment longer than it should, that a question asked of him has to be asked twice. And a slowed mind is not equipped to notice that it has slowed, because the instrument doing the noticing is the instrument that has been dulled. Then comes confusion, mild, more like distraction than distress. And then, and this is the word the old accounts use and the word I cannot get out of my head, the man becomes content.

Yes, content. Neither frightened nor fighting. A pleasant heaviness, a sense that everything is basically fine and that there is no particular reason to be standing when one could be sitting. So he sits down. And in the ordinary way of a man taking a moment’s rest in the middle of a long shift, in a gallery that looks exactly like every other gallery he has ever worked, with his lamp still burning and his tools beside him and his mates somewhere down the drift, he does not get up.

I have gone on about this because the shape of it is the whole point, and the shape is this: the danger and the blindness to the danger were the same fact. Blackdamp did not merely kill people. It killed the specific human faculty that would have detected it, and it killed that faculty first, before it killed anything else. It took the alarm before it took the man. There was no moment at which a collier stood in a lethal gallery in full possession of his judgment and made a bad decision. By the time the gas was thick enough to matter, the part of him that would have judged was already gone. That is what made blackdamp patient. That is what made it, for centuries, unbeatable by ordinary vigilance. You cannot ask a man to notice the thing that is eating his noticing.

And the mine gave no help. The rock did not change. The timber held. The gallery a hundred feet away was fine, and this gallery looked exactly like that one, and had looked exactly like it an hour ago when it was fine too. A flame lamp would dim in bad air, which was something, but it was a coarse signal and a late one, and it dimmed for other reasons too, and a man whose judgment had already begun to soften was not the ideal reader of a subtle change in a small flame. Everything a person would ordinarily use to know whether a place was safe returned the answer: safe. Right up until it wasn’t. And by then the returning was being done by a mind that had been quietly emptied of its capacity to disbelieve.

So they carried a bird.

It seems, at this distance, almost tender, and almost absurd, and it was neither. A small caged bird, a canary usually, carried down into the dark by men whose lives were as hard as lives get, not as a companion and not as a mascot but as an instrument. The reason is unsentimental physiology. A bird’s respiratory system is extraordinarily efficient, built for the oxygen demands of flight, moving air through in a one-way circuit rather than in and out of dead-end sacs the way ours does. That efficiency, which is a gift in the open sky, is a liability underground: it means a bird extracts more from every breath, including more of whatever is wrong with the breath. A canary in bad air is affected faster and harder and at lower concentrations than the men standing around it. It is, in the exact sense, more sensitive. And so long before the gas was thick enough to fell a collier, it was thick enough to move the bird.

Now, what did the miners actually watch for? Because “the canary died” is how we tell it now, and it is wrong, and its wrongness matters more than you would think.

A dead bird is a failed warning. A dead bird means you left it too late. The whole value of the instrument was in the range of behavior before death, and the men who used them well were reading a graded scale, not waiting for a binary. They watched the ordinary business of a bird in a cage: the constant small adjustments, the shifting of weight from foot to foot, the head turning, the fussing at the seed, the periodic bursts of song that had no purpose except that the bird was a bird and that is what it did. That was the baseline. That was the sound and the sight of good air, and a man who worked a seam for years had that baseline written into him below the level of thought, the way you know the sound of your own house and only notice it on the night it changes.

The first sign was not death. It was a falling-off of that fuss. The bird went quieter and stiller than a bird ought to be. It stopped the small unnecessary movements. It came off its perch, or it sat low with its feathers puffed in a way that a working collier could distinguish, instantly and without reasoning about it, from the same posture in a bird that was merely sleeping. And most of all, and most usefully, it stopped singing.

That was the warning. Not a dead bird. A quiet one.

And I want to be exact here, because this is the sentence the entire essay is built to carry: the bird did not warn anyone.

The bird did not know it was underground. It did not know there was gas, or men, or danger, or a shift, or a wife at the head of the shaft with a lamp of her own. It had no message and no intention and not the faintest idea that it was, in that moment, the most important living thing for a quarter mile in any direction. It simply lived, in its small way, and answered, in its small body, to the air. When the air turned, it stopped singing. That is all it ever did. It stopped singing.

Everything else, everything we call the warning, was done by a man.

Somewhere in that gallery there had to be a person whose attention was genuinely on that cage. Not near it. On it. A man who had listened to that bird, and to twenty birds before it, through so many hours in so many drifts that the song had become part of the sound of the world down there. And so when it stopped, he heard the stopping. Understand what that requires. To hear a silence you must first have been carrying the sound, continuously, without effort, in the back of your mind, for long enough that its absence arrives as an event rather than as nothing. A stranger in that gallery would have heard no warning at all, because a stranger would have heard only quiet, and quiet is not information unless you were holding the song. Two men standing in the same air, one of them receiving the loudest signal in the pit and the other receiving nothing whatsoever, and the difference between them made entirely of accumulated attention.

And then he had to know what it meant, immediately, in his body rather than in his notes, because a man who has to stop and reason it out is a man whose reasoning is already being taken. He had to know that a quiet bird in that cage was not a resting bird. It was a scream. It was the loudest thing in the pit. And knowing it, he had to do the thing that is easy to say and hard to do, which is to turn his crew and walk them out of a gallery that looked, to every eye and every other instrument, completely fine. He had to act against the evidence of his own senses on the authority of a bird that had merely gone quiet. His men might have grumbled. The seam was good. The shift was not done. And nothing would happen, afterward, to prove him right, because the whole nature of a warning heeded is that the disaster does not occur, and a disaster that does not occur looks exactly like a disaster that was never coming.

That man is the reason the story has survivors. Not the bird. The reading of the bird.

Now I have to lay down the hard part, and I am going to take my time with it, because it does not resolve and I will not pretend it resolves.

Every crew carried a bird. That is the fact that ruins every easy version of this story. It was the rule. The deputy checked it, the regulation named it, the company provided it, and a shift that went down without its cage was a shift in violation. Everyone knew that. So everyone carried the bird. If you had stood at the head of the shaft with a clipboard and asked each crew, as they went down, whether they had their canary, every crew would have said yes, and every crew would have been telling the truth, and your clipboard would have been satisfied identically by all of them.

Let me draw you two of those crews, because they went down the same shaft on the same morning and they are the whole argument.

The first crew had a man in it, call him the one nearest the cage, and the cage was on his hook and it had been on his hook for eleven years. He did not think of himself as doing anything special. If you had asked him what his job was, he would have said he cut coal, and he would have been right, and the bird would not have come up. But the bird’s song was in him the way the smell of the pit was in his clothes, which is to say completely and without his consent and below the level of anything he could have described to you. He knew this particular bird, its habits, that it fussed more in the first hour and settled after, that it went quiet for a moment when the ventilation door swung and then started again. He had a calibration for it that existed nowhere except inside him and that he could not have written down if you had paid him to. When it stopped, he would hear it in the middle of a sentence and stop mid-word.

The second crew also had a bird. The cage was on a hook, and the bird in it was alive and adequately fed and entirely regulation. The man nearest it was not a bad man and not a lazy one; he was a good collier who worked hard and went home to his family. But he was new to that seam, or he had come from a pit where the ventilation was different, or he was simply one of the many people in the world who had never once been told that the carrying and the reading were two different acts. He believed, in the ordinary and completely forgivable way, that the bird was the safety. That having it was the point. That the cage on the hook was a kind of protection in itself, the way a man might feel safer for having a talisman in his pocket. He had never listened to that bird in the sense I mean. He could not have told you, an hour into the shift, whether it was singing or not, because he had never been holding the sound, and so its absence would not arrive as an event. It would arrive as nothing at all.

Now: hold those two crews side by side and tell me what separates them. It is not equipment. Identical. It is not compliance. Both compliant. It is not intention; neither man intended anything but a good shift and a safe one. It is not even competence at the job they were hired to do, which in both cases was excellent. The difference is made of nothing you can photograph, nothing you can file, nothing you can produce for an inspector, nothing that exists anywhere in the world except inside one man’s accumulated attention. It is invisible at the head of the shaft. It is invisible at the coalface. It is invisible for the entire length of every shift on which the air stays good, which is almost all of them.

And it is the only difference that has ever mattered.

I said the story does not resolve, and here is why. There is no test at the top of the shaft that separates those crews. You cannot ask “do you have the bird” because both say yes. You could ask a man to describe the bird’s ordinary behavior and you would learn something, but a clever man can describe a thing he does not attend to, and an inarticulate man may attend perfectly and tell you nothing. The faculty is real and it is decisive and it is nearly unfileable, and any system that tries to check for it will end up checking for the cage, because the cage is what can be checked. This is not a failure of anyone’s diligence. It is a property of the situation, and it is exactly as true above ground, in every industry, in every century, as it was in the pit.

Before I turn this toward the systems I actually keep, there is one more movement, and I think it is the one that will matter most to you, because it forecloses the exit that almost everyone reaches for.

The birds went away. In Britain the last of them came out of the pits in 1986, retired in favor of electronic detectors, and by every honest measure this was progress. The detector is a better instrument than the bird in nearly every respect. It does not need feeding. It does not have moods. It gives a numeric reading rather than a behavior to be interpreted, and it gives it continuously, and it can be calibrated against a known standard, and it will sound an alarm rather than merely going quiet. It is more sensitive, more consistent, and more objective. It is, in short, everything we mean when we say we have solved a problem with technology, and I do not want to be cute about it: the men in those pits were safer afterward. That is not nothing. That is the point of the whole enterprise.

And it did not touch the reading problem. Not at all. Not in the slightest.

Because a detector, too, only changes state. It has no more intention than the canary did. It does not know there is a gallery or a crew or a shift. It converts a property of the air into a number on a display, and there its entire capacity ends, and everything after that, the noticing, the interpreting, the deciding, the turning of the crew and the walking out, is still a human act performed by a person whose attention is actually on the thing. A detector on a belt clip, carried by a man who has never looked at it, is a bird in a cage on a hook. A detector whose alarm has been sounding intermittently for three weeks because of a fault nobody has fixed is worse than a bird, because it has trained the crew to hear the alarm as noise. A detector whose reading nobody understands, whose normal range nobody has internalized, whose drift nobody checks, is a more expensive way of carrying the bird without reading it.

I want to be very clear about what I am claiming, because there is a lazy version of this argument that says technology never helps and that is obviously false. Technology helps enormously. The detector is better than the bird and the men were safer. What I am claiming is narrower and harder: improving the instrument does not reduce the requirement for a reader, and it never has, and there is no version of the instrument that will. Every generation gets a better bird and discovers, usually at cost, that the better bird still needs someone to be watching it. The human half is not a transitional inconvenience awaiting a sufficiently good sensor. It is a permanent structural feature of what it means to be warned by anything at all.

Now I will turn this toward the systems I spend my days inside, and I will try, as always, not to insult you by making your situation smaller than it is.

You have measured your artificial intelligence, or you will. And you already know, if you have been reading these letters, that a number is not safety, because a true and reassuring figure still contains a day. And you know that a certificate speaks in the past tense, because a measurement is a fact about the moment it was taken and the moment it was taken is already gone. Grant all of it. This is the thing standing underneath both of those, the thing they were quietly resting on without saying so.

A measurement is a bird in a cage. It does not warn anyone.

Your dashboard raises no alarm. It displays a value, and the value moves, and that is the whole of what it can do. Your evaluation harness does not tell you that the model has drifted; it produces a score, and the score is lower, and a lower number on a screen is not a warning any more than a quiet bird is a warning. It is a change of state in an instrument. Everything we call the warning, the noticing, the interpreting, the deciding, the going to the people who can act and saying we need to stop, all of that is a human act, and specifically it is an act of reading, and it is performed by a person whose attention is genuinely on the instrument and who has watched it long enough that its ordinary state lives in them below the level of thought. Someone who would catch the change the way you catch a stopped clock in a house gone suddenly too quiet.

And here is the uncomfortable inventory, which I offer without any pleasure, because I have taken it in enough organizations to know how ordinary the answer is.

Most organizations have carried the bird and staffed no one to read it.

They have the dashboard. It exists, it is well-built, and it is open in a browser tab that nobody has looked at in nine days. They have the evaluation, run thoroughly at procurement, by a team that has since moved to another project, against a distribution of inputs that no longer resembles what the system sees on a Tuesday in March. They have the risk register, and it is maintained, and it is maintained for the auditor rather than for the air, which is a distinction invisible on the document and total in its consequences. They have monitoring that alerts, and the alerts go to a channel where they have been firing often enough for long enough that the crew has learned to hear them as noise, which is the worst outcome of all, because it is a detector that has actively trained its readers into deafness. And if you stood at the head of their shaft with a clipboard and asked whether they measure their systems, they would say yes. And they would be telling the truth. And your clipboard would be satisfied.

Go down into the gallery instead and ask a different question. Not do you measure. Ask: who reads it? Who, by name, has that instrument’s ordinary state written into them? Who would notice the change on the day it is small, which is the only day noticing is worth anything? Who has the standing to turn the crew and walk them out of a gallery that looks, to every other eye, completely fine, and who will back them when nothing bad subsequently happens and the whole thing looks in retrospect like an overreaction? Ask that, and the answer, very often, is a silence that is not the good kind.

There is a cage on a hook. There is a bird nobody is watching.

I want to be exact about the human half now, because it is almost always described badly, and the bad description is what makes it so easy to fail while feeling that you are succeeding.

The half of trustworthiness that is not the instrument is not enthusiasm. It is not belief. This distinction is the one I would most like to leave you with, because the industry has spent several years confusing them and the confusion is expensive. An organization can be full of people who are genuinely excited about the technology, who speak warmly about it in every meeting, who have sat through many hours of leadership explaining that this is the future and have come away persuaded, and that organization can be exactly as blind as an organization full of skeptics. Because belief is not literacy. Enthusiasm is not attention. A person who is thrilled about the bird but cannot tell its song from its silence is not reading the bird. He is carrying it, warmly.

And the inverse is just as true and more often missed: the skeptic who watches the gauge every day, who does not much like the technology and does not much trust it and precisely for that reason has its ordinary state written into him, is doing the work. His attitude is irrelevant. His attention is the whole thing. If I had to choose between an organization that believes and does not watch, and one that watches and does not believe, I would take the second one every time, and it is not close.

Nor is this faculty something you can buy or confer. It is not built in a seminar. It is not conveyed by a license, and I say that as someone who holds them. It is not transferred by a policy document, however well written, and I have written some. The capacity to read an instrument is built the only way that capacity has ever been built by anyone in any field, which is by putting your own attention on the actual instrument, repeatedly, over real hours, in real conditions, until you can feel a change before you can name it. There is no shortcut, because the thing being built is not knowledge. It is a baseline, and a baseline is made of exposure and nothing else.

Which brings me to the failure I see most often and have the least patience for, and I want to name it plainly. The senior person who has never watched the gauge himself, who has only ever instructed others to watch it, is carrying the bird by proxy. And carrying by proxy is not reading. He may be entirely sincere. He may have allocated the budget and hired the team and asked for the dashboard and reviewed the summary every month, and every one of those is a real and useful act, and none of them builds the baseline. When the summary says the number moved slightly, he has nothing inside him to compare it against. He cannot feel the wrongness, because feeling the wrongness requires having held the rightness, continuously, in his own attention, and he has never held it. He has held reports about it, which is a different substance entirely. Reading was never a thing you could delegate. It was always a thing you had to have done yourself, in the dark, until it lived in you.

And the last hard piece, which the pit knew and which we relearn every decade at cost: the reader must be able to act. A man who hears the bird stop and has no standing to turn the crew has not saved anyone. He has merely suffered the knowledge. This is where most organizations fail second, having already failed first at building the reader, and it is why the two failures are usually found together. The person with the baseline is junior, because the baseline is built by the people whose hands are actually on the thing, and the person with the authority is senior, because that is what senior means, and between them lies a distance measured in reporting lines and in the small daily discouragements that teach a junior person not to raise things. The gallery fills in that gap. It always has.

So I will not ask whether you measure your systems. You do, or you will, and the number will be good, and the cage will be on the hook, and the clipboard at the head of your shaft will be entirely satisfied. That was never the question. The question is the one the two crews answered differently on the same morning, in the same air, with the same equipment, and answered without knowing they were answering, which is the way this question is always answered:

Your instrument is quiet. Is it quiet because the air is good? Or is it quiet because there is no longer anyone in the gallery who could hear a bird stop singing?

I build the instrument. I build the bird that changes state honestly, that will not flatter the air, that goes quiet exactly when it should and not a moment later. That work is worth doing and I will keep doing it. But an instrument is only ever half of it, and I would rather tell you that than sell you the other half as a whole. A gauge with no one reading it is a bird in a cage on an empty hook, keeping faith with an air that nobody is judging. A reader with no instrument is a collier in the dark with nothing to watch, listening for a song that was never brought down. It takes both.

It always took both.

Suneeta Modekurty

Share

Image created using AI tools

There was once a cement company that learned to build bridges a new way.

They were faster than anyone believed a bridge could be. A town that had waited a generation for a proper crossing, that had made do with a ferry and a long detour and the river taking someone every few winters, could now have a bridge in a single season. And they were not flimsy for being fast. They were strong. They lasted. The company had found something real, and the something real was the kind of thing that changes how a place lives.

Two towns on the same river signed the same contract. They took the same design and the same crews, and in the same season they each got the same bridge. And for a long time, in both towns, the story was simply a good one. The detour was gone. The ferry was retired. Children who had grown up being told not to go near the water crossed the river twice a day without thinking about it at all, which is the truest sign that a bridge is working. You stop seeing it. It becomes part of the ground.

I want to stay in that period for a moment, because it lasted years and it deserves its due. Nothing was wrong. Nothing was being hidden. The bridge was good and the towns were right to love it and there was no dark secret waiting in the concrete. If you had gone to either town in those years and asked whether the bridge was safe, people would have looked at you the way you look at someone asking whether the floor is safe. You do not think about the floor. You stand on it. That is what the floor is for.

And then, on a day that had nothing to distinguish it, in both towns, the bridge came down.

Here is the part that matters, and I want to move through it carefully, because the easy version of this story is a lie.

The easy version is that the company cut a corner, or the concrete was bad, or someone was negligent, and the collapse was a crime with a culprit. That is the story we prefer, because it comes with a villain and a lesson and the comfortable sense that if we simply find the bad man and remove him, the floor is safe again.

That is not what happened.

What happened is that the bridge did exactly what it was built to do, for exactly as long as it was built to do it, and then it reached the edge of what it could do, and physics collected. When people went looking, afterward, for the thing that had gone wrong, they found something worse than a mistake. They found that nothing had gone wrong. The bridge had been, from the first day, a very good bridge that would stand almost always. And almost always is a phrase that contains a day.

The company had known this. Of course they had known it; they were serious people and they had measured their work honestly. They could have told you, from the beginning, precisely how reliable the bridge was, and the figure was high enough that no reasonable person would have hesitated to sign. It was the kind of number that ends a conversation rather than starting one. It sounded like a yes. And so both towns, hearing it, did the most natural thing a person can do with a reassuring number. They filed it, and they exhaled, and they stopped thinking about the bridge, because that is what the number seemed to permit.

The number was true. It had always been true. And true, it turns out, is not the same word as safe. A number tells you the odds. It does not stand on the bridge. Somewhere inside that reassuring figure was a day, and the day was real, and no amount of the number being honest made the day less real. It only made it easy not to look at.

So both bridges fell. That part was the same.

But the two towns were not the same, and the difference did not show up in the concrete. It showed up in a person.

In the first town, there had been someone whose actual job was the day inside the number. Not the bridge on its good days, which needed no one. The day. He walked the structure when there was no reason to. He watched the river pull at the footings in the spring. He knew, in the particular way that only comes from standing on a thing in the cold when everyone else has gone home, that a very reliable bridge is not the same as a bridge you may stop watching, and that the whole meaning of almost always is that someone has to be there for the remainder. When the river came up wrong that year, he was the one who saw it, and he closed the bridge the night before, and in the morning it fell into an empty river and hurt no one at all.

In the second town, there was also a name. There was a signature on the contract and a line of responsibility and a person who, if you had asked, was accountable for the bridge. On paper the two towns were identical. Both had measured. Both had a name in the box. But in the second town the name was there for a different purpose. It was there so that afterward, when something happened, there would be someone to answer for it. The accountability was real, and it was the wrong kind. It faced the wrong direction. It was pointed at the aftermath instead of the river, and a bridge cannot tell the difference between a watchman and a name in a drawer until the day it needs one, and by then the difference is the only thing that matters.

Same bridge. Same true number. Same collapse. In one town a story people tell about the year the bridge fell and no one was hurt, thank God, because someone was paying attention. In the other, a stone with names on it.

Now I will do the thing these stories are for, and turn it toward the systems I actually spend my days on, and I will try to do it without pretending your situation is simpler than it is.

Every organization running artificial intelligence has been handed a number, or will be. It is a good number. The systems are, in the main, remarkably reliable, and the people who build them are serious and measure honestly, and the figure they produce is high enough to end the conversation. It sounds like a yes. And the most natural thing in the world, the thing that is not foolish or negligent but simply human, is to hear that number and exhale and stop watching the bridge.

But the number is the odds. It is not the watchman. Inside every reassuring figure about a deployed system is a day, and the day is real, and the honesty of the number does nothing to keep the day from coming. The only thing that meets the day is a person who was watching for it before it arrived, who understood that almost always is a sentence with a remainder in it, and whose job was the remainder.

And here is the quiet, uncomfortable thing, the one I cannot resolve for you and would not insult you by trying to. On paper, the two towns look the same. A name in a box looks like a name in a box. You cannot tell, by reading the org chart, whether the person accountable for your AI is a watchman or a signature. You cannot tell whether they are out on the structure in the cold, or whether they are simply the one who will answer when it comes down. Both are called accountability. They face opposite directions. And they look identical right up until the day they don’t.

So I will not ask whether you have measured your systems, because you have, or you will, and the number will be good. That was never the question. The question is the one the two towns answer differently, and answer on the same terrible day, too late to change:

When the day inside the number arrives, is there someone standing on the bridge? Or is there a name in a drawer, waiting to answer for it?

And if the honest answer is that you would rather have the watchman, then the real question is quieter still, and it is the only one worth acting on while there is still time to act. Not do we have a number. You have a number.

Is anyone actually watching the river?

I build instruments for the day inside the number. But an instrument is only ever half of it. A gauge with no one watching it is a number in a drawer, and a watchman with no gauge is a person in the cold with nothing to read. It takes both. It always took both.

Suneeta Modekurty

Image created using AI tools

When my parents wanted gold, they did not go to a shop. They called a man. He came to the house. He knew my grandmother. He had made the pieces she was married in, and the pieces her daughters were married in, and he sat in the front room and drank tea while the design was discussed. When the ornament was ready he brought it himself, and my mother paid him, and nobody weighed anything or questioned anything, because you do not audit a man who has drunk your tea for thirty years. That is not how trust works. Trust was the whole transaction. The gold was almost an afterthought.

Let me be clear that he was not a villain. Nobody in this story is, which is rather the problem. He was, in every sense that a person can be trusted, trustworthy. He kept his word. He came when he said he would. He had a name and a face and a history with the family, and if you had asked my mother whether she trusted her jeweller she would have looked at you as though you had asked whether she trusted her own brother.

And the gold was not pure.

Not because he was a thief. Because that is simply how the trade worked, up and down the country, for longer than anyone could remember. A little copper here. A little less than promised there. Nobody checked, because there was no way to check, and so the question of purity was not really a question. It was a matter of the man’s word, and the man’s word was good, and that was the end of it.

I did not learn until much later how much of that gold was not what it was said to be. When I did, the number was startling. Across the country, when someone finally measured, a large share of the gold sold as one purity turned out to be another, lower one. Not in a few bad shops. Everywhere. In the pieces people had been married in. In the pieces they had saved their whole lives to buy. The trust had been total, and the trust had been misplaced, and those two facts had lived comfortably side by side for generations because nothing existed that could tell them apart.

That is the thing I keep coming back to, and it is the reason I am writing this.

Trust and accuracy are not the same quantity. We speak as though they were. We say we trust a thing and we mean we believe it is correct, as if the trusting and the correctness were one measurement. They are not. You can have complete trust in something that is quietly, persistently wrong, and the trust does nothing to make it right. It only makes you stop looking.

And something eventually changed it.

A company decided to sell gold differently, and it ran into exactly the wall you would expect. People did not trust it. Why would they? It had no front room, no thirty years, no tea. It was a brand, and a brand cannot be someone’s brother. Against a man who had made your mother’s wedding jewellery, a company does not stand a chance. Trust was the entire market, and trust was precisely the thing the company did not have and could not manufacture.

So it did something clever. It did not try to win the argument about trust. It changed what the argument was about.

It bought a machine that could measure the purity of gold in about a minute, without damaging the piece, and it put the machine in its stores, and it told people to bring in their own gold and test it. Not gold bought from the company. Their gold. The pieces from the man in the front room. Bring it in, it said, and let us all simply look at the number.

People came. And the number was not what they had been told.

Not occasionally, almost always. Piece after piece, gold that had been sold as one thing measured as something less, and people stood there and watched their certainty come apart on a small screen in about sixty seconds. Not because anyone argued with them. Nobody argued. The machine did not have an opinion. It just showed the number, and the number did what thirty years of trust had prevented anyone from doing. It looked.

I find that moment extraordinary because the company did not win by being more trustworthy than the jeweller. It could not have won that fight; the jeweller had thirty years and it had a logo. It won by introducing an instrument, and an instrument does something a relationship cannot. It makes trust unnecessary. You do not have to trust the number. You can see it. And the moment seeing became possible, the whole edifice of “his word is good” simply stopped being the point.

Now I will tell you why I have spent all these words on gold, because it is not really about gold.

Every organization I talk to about artificial intelligence has a man in the front room.

He is not a jeweller. He is a vendor with a confident dashboard, or a governance committee that meets and produces a report, or a consultant with good credentials and a reassuring manner, or an internal team that says the system is fine. And the organisation trusts him, in exactly the way my mother trusted her jeweller, which is to say completely and for reasons that have almost nothing to do with whether he is right. He has a name. He has been here a while. The dashboard is green, and green is a very calming colour. Everyone exhaled a long time ago.

And nobody has measured the gold.

They have measured other things. They can tell you the model’s accuracy to a decimal. They can show you uptime, throughput, cost per query. These are real numbers and they are measured honestly. But they are not the purity. They are the weight of the gold, and you can weigh a piece all day and learn nothing about whether it is what it claims to be. The organisation is measuring the thing that is easy to measure and calling it the thing that matters, and the thing that matters, whether this system can actually be trusted, whether its confidence is earned, whether anyone could prove it under questioning, sits there untested. A matter of the vendor’s word. And the vendor’s word is good. And that is the end of it.

I do not say this to accuse anyone, and I am very sure about that, because it would be the easiest thing in the world to turn the vendor into the villain and I think that is exactly the wrong lesson. The jeweller was not a villain. The vendor is usually not a villain. The problem was never that the trusted party was dishonest. The problem is that trust was doing a job it cannot do. Trust is not a measurement. It feels like one. It sits in the same place a measurement would sit, and it produces the same comfortable sensation of this has been handled. But it has not been handled. It has been trusted, and those are different things, and the difference does not announce itself. It waits.

It waits for the regulator. It waits for the incident. It waits for the day someone finally brings the piece to the machine and asks the question that thirty years of good feeling had made unaskable. And on that day, the beautiful relationship and the reassuring dashboard and the confident committee are worth exactly nothing, because none of them were ever the same quantity as the truth.

So I will leave you with the question the machine asked, because it is the only question that was ever really being asked, and it is worth asking of your own systems before someone asks it of you.

Not do you trust it. You do. That was never in doubt, and it was never the point.

Bring it to the machine. What is the number?

If you do not know, that is not a small gap. That is the whole gap. It means you have been doing what my mother did, which was reasonable, and generational, and completely sincere, and wrong. And the gentlest possible way to discover that is the way she never got to: on your own terms, in your own hand, before the day when the question is no longer yours to ask.

The right gold has never feared the test. It is the only kind worth owning.

I build instruments for measuring whether AI systems can be trusted. Which means, before anything else, I build the thing that is willing to show you the number you would rather not see, while it is still yours to do something about.

Suneeta Modekurty

Image created with AI tools

I want to start with something I noticed, not something I believe.

In rooms with executives, I have watched the same thing happen more than once. These are people who measure with real discipline. They can tell you their cost per transaction, their productivity per team, their adoption curve, their uptime to the decimal. Measurement is not foreign to them. It is the native language of how they run things.

Then I ask one question. How do you measure whether your organization actually trusts its AI?

And the room goes quiet.

Not a guilty quiet. Not an accusation landing. Just a pause, the particular silence of a question that has never been asked in that form before. Someone will eventually offer that they run model accuracy reports, or that they have a governance committee, or that there is an audit somewhere. All true. None of it an answer to the question I asked. The question was about trust, and there was no number to reach for.

I am not going to pretend that silence proves anything. One observation is not a finding. But it made me curious, and curiosity is a better place to begin than conviction.

The thing that made me curious

Here is what puzzled me about that silence.

We spend enormous effort measuring what our AI systems do. Their accuracy, their latency, their throughput, their cost. Whole dashboards exist for it. But we spend almost nothing measuring the thing that actually determines whether any of that effort matters: the confidence with which human beings decide to rely on the system. Whether they lean on it, or quietly work around it. Whether they trust it.

That struck me as odd. Because trust is not a soft afterthought in these systems. It is the hinge everything turns on. A model can be brilliant and go unused because no one believes it. A model can be mediocre and be over-trusted into a disaster. The performance of the model and the trust placed in it are two different quantities, and only one of them shows up on the dashboard.

So the question I could not put down was simply: why? Why do we measure the machine so carefully and the reliance on it not at all? Not because anyone decided trust was unimportant. Everyone says the opposite. So why the gap between what we say matters and what we actually measure?

A lens, from an unlikely place

I found a useful way to think about that gap in a book most people read for other reasons.

John Doerr popularized a deceptively simple idea in Measure What Matters. On the surface it is a book about OKRs, about goals and key results. But the deeper idea underneath the framework is this: OKRs were never really about metrics. They were a mechanism for making invisible priorities visible. The act of measuring something is the act of declaring, publicly and to yourself, that it matters. And the reverse is quietly true as well. What you never measure, you have treated as if it does not matter, whatever you say about it in your principles.

That reframed my question with some force. If trust is what decides whether an AI system gets adopted, questioned, overridden, or abandoned, then by any reasonable account it matters as much as anything on the dashboard. So why is it still handled as an intuition, a feeling in the room, rather than as something observable? Not because it is unimportant. Perhaps because we have not known how to see it.

Trust behaves like a latent variable

Here is the shift that, for me, changed the whole problem. And it is a small shift in language that turns out to be a large shift in method.

The instinct is to say trust should be measurable and go looking for the meter. But that is the wrong sentence. The right one is this: trust behaves like a latent variable.

That phrase comes from the sciences that have spent the longest wrestling with exactly this difficulty, the difficulty of studying something real that you cannot observe directly. And once you say it that way, the problem stops being strange and becomes familiar, because it is a problem those fields solved long ago.

Think about what we already measure this way, without blinking. Intelligence is never observed directly; it is inferred from performance across many tasks. Depression is never observed directly; it is inferred from a pattern of signs. The same is true of economic confidence, of quality of life, of customer satisfaction. Not one of these is read off an instrument. Every one of them is real, consequential, and studied rigorously. And every one of them is inferred from things you can see, rather than measured head-on.

Trust belongs in that family. You will never point a sensor at trust and get a reading. But that has never been how latent things are measured. They are measured through their traces.

What would trust leave behind?

So I stopped asking how to measure trust, and started asking the question a researcher would ask instead: if trust cannot be seen directly, what visible evidence would it leave behind?

This is the move that turns a slogan into science. You do not measure the hidden thing. You ask what the hidden thing causes, and you measure that.

So suppose trust in an AI system rises. What should we expect to see? People should override the system less often when it is right, and appropriately when it is wrong. Escalations should shift in character. The rate at which recommendations are accepted should change. The time spent re-verifying the system’s output should fall. Appeals, interventions, audit exceptions, all of these should move in patterns consistent with a change in reliance. And if trust falls, the same signals should move the other way.

Notice what has happened. Without ever claiming to measure trust directly, I now have a list of things I can observe: override rate, escalation frequency, recommendation acceptance, time to verification, human intervention, appeal rate, audit exceptions. These are not trust. They are the behavioral exhaust of trust, the visible residue a hidden quantity leaves in the operational record. And measuring the consequences of a latent variable is precisely how latent variables have always been studied.

I want to be careful here, because this is where rigor matters and where it is easy to overreach. Any single one of these signals can move for reasons that have nothing to do with trust. That is exactly why you do not lean on any single one. You look at the pattern across many, the way a diagnostician reads a panel of markers rather than a single number, and you reason from the pattern to the hidden state underneath. The mathematics for doing this well, for inferring a hidden quantity from noisy signals and for tracking how it moves over time, is mature and unglamorous and mostly borrowed from fields that have done it for decades. It does not need to be on the surface of this essay. It only needs to exist, and it does.

One more thing about trust: it does not hold still

There is a wrinkle that makes this harder than the classic textbook version, and it is worth naming because it changes what “measuring” even means.

Intelligence, once measured in an adult, is fairly stable. Trust is not. Trust in an AI system moves, and it moves because the world the system operates in moves. The data shifts. The users adapt. The conditions the system was validated under quietly stop holding. So trust is not only a hidden quantity; it is a hidden quantity in motion.

And that has a consequence I find genuinely important. If trust moves, then measuring it once tells you almost nothing about where it will be next quarter. A single reading is a photograph of a moving thing. The same measured value can mean opposite realities depending on the direction it arrived from: a system whose trust is recovering and one whose trust is collapsing can pass through the identical number on the way to very different fates. Only a sequence of readings, a trajectory, can tell them apart. Which suggests that if trust is worth measuring at all, it is worth measuring continuously, not certified once and filed away.

Why this matters to the person accountable

Let me connect this back to the executive in the quiet room, because this is not an academic puzzle for them. It is a future liability.

When an AI system fails, and eventually one will, the questions that arrive are all retrospective. Was this system trustworthy? Were you watching? What state was it in, and when did you know? These are questions about a quantity over time. And they can only be answered if that quantity was being observed as it moved, because a trajectory cannot be reconstructed after the fact. You cannot go back and take a measurement you never took. The evidence either exists because someone was collecting it all along, or it does not exist at all.

That reframes the quiet in the room. The silence was not a small gap in reporting. It was the absence of the one record that, on the hardest day, everyone will reach for and no one will find.

Where this is taking me

I will be honest that this line of thinking is a question I am still inside of, not a conclusion I am handing over. But it has led me somewhere specific, and I will name the direction even though I am not claiming the destination.

The question I keep returning to is whether executive trust in AI can be treated as a measurable construct, operationalized not through surveys and self-report alone, but through the behavioral, operational, governance, and outcome signals that a system generates as it runs, tracked over time as they evolve. Not to reduce trust to a single tidy number, but to make a hidden and consequential thing visible enough to reason about, argue over, and act on before a failure rather than after.

I do not think the next real advance in AI governance will come from larger models. I think it may come from better measurement. Not measurement of what the machine can do, we are already good at that, but measurement of the thing we keep saying matters most and keep leaving off the dashboard.

So I will end where a researcher should end, with the question rather than the answer.

Share

If trust is what matters most, what evidence should organizations be collecting, starting now, to know whether they actually have it?

I do not think we have a good answer yet. I think it is the right question. And I would rather leave you holding a better question than a borrowed conviction.

Until next Tuesday….



Suneeta Modekurty
Founder & CEO | SANJEEVANI AI

Image created by AI tools

I was invited recently to speak with a room of product leaders and senior professionals as part of AI in Motion, an initiative built to bring practitioners face to face with the harder questions AI is forcing on all of us. I had prepared to talk about something I have been circling f…

Read more

Image created using AI tools

There is a moment in every AI deployment that nobody schedules and nobody sees coming. It is the moment the system does something no one asked it to do, in a place no one realized it could reach, and the organization discovers, all at once, that it never actually knew what it had …

Read more

Image created using AI tools

On a Monday in June, Anthropic released the most capable model it had ever built. By Friday evening, it was gone. Not slowed, not limited to certain regions. Gone, for everyone, everywhere.

I have been sitting with that all weekend, and I want to share where my thinking landed, because I don’t think the lesson is the one the headlines are reaching for.

Here is what happened, told simply. Anthropic released Fable 5, and the model beneath it, Mythos 5. Teams across the industry spent the week building it into real work. Then the U.S. Commerce Department issued an export control directive: these models could not be used by any foreign national, anywhere, including foreign-born employees working inside U.S. offices (and you can imagine why it matters to me, and to you, if you are foreign-born). There was no clean way to honor that order in real time, so the only path to compliance was to switch the models off for everyone. And so they did. The most capable thing on the market went dark on a Friday night, because of a decision no customer was part of.

I want to be careful with the facts, because this story is being told in several versions, and some carry more confidence than the record supports. The verified spine is narrow. A government directive, citing national security, tied to the model’s cyber capabilities and a jailbreak technique. The company disagrees, calls it a misunderstanding, and says it is working to restore access. The more dramatic threads, who called whom, who set it in motion, are still unsettled. I would rather build on what is solid than on what is satisfying.

What is solid is enough to learn from.

This fits in an arc

Every infrastructure we now take for granted went through the same passage. There was a season when it was new and astonishing, and we built on it before we fully understood what we were leaning on.

Be it Electricity, the early eb or the Cloud.

Each one moved from “remarkable capability” to “thing we assume is simply there.” And each one, somewhere along that path, had a moment that reminded everyone it was not as solid as it felt.

Frontier models are early in that arc. They are astonishing, they are everywhere, and we have started to treat them the way we treat electricity. Something that is just on. Friday was a reminder that we are not there yet. The most capable models are still the newest part of the stack, and the newest part is always the part most exposed to forces that have nothing to do with engineering.

That is not a criticism of any one company. The same structure would hold for any frontier model from any provider. It is simply where we are in the arc.

A layer above

When a company adopts a model, it negotiates the things it knows how to negotiate, namely, uptime, latency, committed spend, data handling, support etc. Those terms feel like the boundaries of the relationship, and most of the time they are.

What this week showed is that there is a layer sitting above all of that and that is regulatory, geopolitical, national. And that layer does not read your service agreement. When it moves, the terms you negotiated still hold, they just stop being the thing that decides the outcome. The capability can be withdrawn by a force you have no seat across from.

This is worth naming plainly as something the field is now ready to see clearly.

We have understood model risk mostly as “will it perform.” This week added a second dimension: “will it remain available to me, and who decides that.” Both are real. We have simply been measuring one and assuming the other.

Why “it will come back” is not the lesson

It is tempting to wait this out. Access may well be restored. The disagreement may resolve, and the model may return as if the week never happened. And then the easy move is to file it under “strange few days” and carry on.

It’s worth gently resisting that. The specific outcome is not the lesson. The lesson is that the event was possible at all. A capability that hundreds of millions of people were using could go to zero overnight, lawfully, with no warning. Whether it stays gone for three days or three months does not change what it taught us about the shape of the risk. A thing that can happen once on a Friday can happen again on a Tuesday.

The narrow lesson is “this model carried risk.” The useful lesson is “leaning on any single model this completely carries risk.” One sends you shopping for a different vendor. The other invites you to look honestly at how you have built.

A question worth pondering over

So here is the question I have been thinking over, and I offer it as a genuine prompt rather than a rhetorical one.

If your primary model went dark this Friday, not slower, dark, what happens on Monday?

Picture it concretely. Which workflows stop. Which customers notice. How long before something else carries the load, and how much harder is it carrying. Do you know the answer already, or is it something you would discover in real time, under pressure, the way teams often discover their backups only when they finally reach for them.

Most organizations cannot answer this cleanly yet, and I don’t think that reflects poorly on anyone. The question simply was not urgent until this week. The capability was too good and too available to imagine it absent. That is exactly the condition under which dependence gathers quietly, when the thing we depend on feels too solid to question.

What I think resilience actually asks of us

I don’t believe the answer is fear, or stepping back from frontier models, or treating capability as a liability. Fear and stepping back is never a solution to any problem. The capability is real and the value is real. The answer is quieter than that, and frankly harder. It is knowing your own exposure well enough to act on it before the moment forces your hand.

That means being able to say, with specifics, where you are concentrated. It means having a fallback you have actually tested, not one you assume will be there. It means treating “what is our dependency profile” as a question you can answer at any time, rather than one you reconstruct in a hurry. None of this is dramatic. It is the difference between an organization that has a plan and one that becomes the plan, improvised live, while the screens turn red.

In its own backhanded way, Friday was a gift. It took a quiet, structural risk and made it loud, and dated, and undeniable, at a cost that for most of us belonged to someone else. That is the least expensive way to learn an expensive lesson. The expensive way is to learn it when the model that goes dark is the one your own business is standing on.

The model came back, or it will. The question it asked is going to stay with us a while longer.

What is your fallback plan if your primary model goes dark on a Friday? I would truly like to know how you are thinking about it. Reply and tell me.

Until next week…

Founder and CEO | SANJEEVANI AI

Image created using AI tools

One of the most dangerous AI failures inside companies today does not start as a failure. It starts as efficiency.

You put a predictive model next to a decision. It is fast. It is cheap. It is right often enough. So you lean on it a little more each week.

Then something quiet happens. The…

Read more

Image created with AI tools

In September 2025, Starbucks released a short promotional video to celebrate the launch of an AI-powered inventory system that the company had built in partnership with the computer vision startup NomadGo. The tool was positioned as a centerpiece of CEO Brian Niccol’s “Back to Starbucks” turnaround, a strategy designed to address the persistent product shortages that had been quietly eroding customer experience and same-store sales for years. The video showed a barista walking through a back room with a handheld tablet, sweeping the camera across shelves of milks and syrups while the system tagged and counted each item in real time. In one of the frames of that very video, the system failed to register a bottle of peppermint syrup that was sitting on the shelf in plain view, surrounded by other bottles that the system did correctly identify. Nobody at Starbucks caught the error before the video shipped, the blog post went live, and the rollout began across more than eleven thousand North American stores.

Eight months later, in May 2026, Starbucks quietly retired the system across the entire fleet, the original blog post was deleted from the corporate site, and the promotional video was pulled from circulation. The story made the trade press as a tidy AI failure narrative, with most coverage focusing on the technical embarrassment of a system that could not reliably distinguish oat milk from regular milk. What almost none of the coverage took seriously was the workforce story sitting underneath the technical one, and I want to tell that part here because I think it is the part that actually matters for every leader trying to navigate this moment.

The Starbucks partners, the workers on the floor who interact with these systems every day, did not trust the AI from very early in the rollout. They scanned the shelves with the tablet because they were required to, they watched the system return its counts, and then they quietly recounted by hand and entered the corrected numbers into the inventory log. They protected the stores from the AI’s errors for nine months by absorbing the extra workload onto themselves, working a second shift of verification on top of the work they were already doing. They are the reason the failures stayed mostly invisible to customers and to corporate leadership until the system was finally pulled, and they are also, in the same breath, the people the broader industry narrative is currently lining up to lay off in the name of AI productivity. That contradiction sits at the center of the AI adoption wave, and I want to name it as clearly as I can in what follows.

The Wrong Frame

The dominant story in the current AI moment is that humans and machines are competing for the same jobs, with the AI either taking the job from the human or the human keeping the job until the AI is good enough to take it. Either way, the relationship is framed as a contest with one winner and one loser, and the entire vocabulary of “AI productivity” has been built on top of that framing. This frame is wrong both technically and organizationally, and the Starbucks case shows you exactly why with a level of clarity that few cases offer.

The AI inventory tool did not do the partners’ job, even on the days when it worked. It did one slice of the partners’ job, which was the visual identification of products on a shelf, and it did that slice badly under the operational conditions of a real back room. The other slices of the job, including knowing yesterday’s shipment patterns, knowing which products burned through fastest during the morning rush, knowing which carton was tucked behind another because the closing shift had restocked in a hurry, and owning final accountability for the count when the day’s numbers were reconciled, were never part of what the AI could do and were never part of what the system was designed to do. Those slices remained with the human partner, who now held less authority over the visible counting work and the same responsibility for the eventual outcome, which is the worst possible position to put a worker in if you actually want the system to succeed.

This is not automation in any meaningful sense of the word. This is cognitive load transfer dressed up in the vocabulary of cost reduction, where the easy and legible part of a job gets handed to a machine while the hard and illegible parts get concentrated onto fewer remaining people who now have less context, less authority, and less protection from the consequences of the machine’s mistakes. The conversation we should be having is not whether to lay off the inventory counters because the AI can count, but rather what the expanded human role looks like around a partially automated process, and how many people we need in that expanded role for the system as a whole to actually work. That question is almost never asked in the rooms where AI procurement decisions get made, and so I want to ask it here with as much specificity as I can.

What If: A Manual Monte Carlo

Before I lay out the scenarios, let me say what I mean by a manual Monte Carlo. In quantitative finance and in operations research, a Monte Carlo simulation is a method of exploring possible futures by running the same model thousands of times with different inputs and watching the distribution of outcomes that emerges from the variability. You cannot do this on paper at scale, but you can do a slower and more deliberate version of the same thinking by hand, by asking what-if questions one at a time and walking each scenario out to its consequences before deciding which path to actually take. This is the kind of exercise that every leadership team should be running before they sign an AI contract, and the absence of this exercise is one of the most consistent features of the AI failures we have seen so far.

Image generated using python code by the author

Let me run five of these scenarios for the Starbucks case, not as predictions and not as numbers I could defend inside a formal model, but as the kind of structured thinking that the rollout itself appears to have skipped. I have begun to refer to this approach as a manual Monte Carlo for AI governance decisions, and the five scenarios that follow are the practical instrument I use when thinking about whether a given AI deployment is genuinely ready to ship into the operational world it will actually have to survive in.

The first scenario asks what would have happened if Starbucks had kept every partner on the floor and added a dedicated AI oversight team alongside the rollout. Imagine a team of perhaps fifty to one hundred people whose only job is to audit AI output against ground truth across the eleven thousand store fleet, sampling stores on a rotating schedule, reviewing confidence score distributions, identifying model drift before it becomes catastrophic, and feeding human corrections back into a retraining pipeline that updates the model on a regular cadence. The annual cost of such a team would be real, perhaps in the range of five to ten million dollars when fully loaded with engineering support and infrastructure, and it would be dwarfed by the cost of a failed nine-month rollout across eleven thousand locations, even before you begin to count the brand damage, the deleted blog post, the loss of internal credibility for the next AI initiative, and the harder-to-measure erosion of trust between partners and the corporate technology function. The deeper point, the one that matters more than the headcount or the budget, is that an oversight team of this kind functions as a forcing function for everything else the deployment should have included from the start, because the very existence of a paid oversight function makes engineering hygiene mandatory rather than optional. Without a budget line for oversight, the engineering team naturally optimizes for the demo and the ship date, since nobody inside the company is being paid to demand the audit trail, the retraining cadence, the rollback authority, or the operational testing protocol. With a budget line for oversight, the team’s first ninety days of work would surface exactly those missing pieces and would force the rest of the organization to build them, because the oversight function cannot do its own job without them. My own view, which I hold with some confidence after watching enough of these rollouts up close, is that this first scenario is the single highest-leverage intervention of the five I am about to walk through, because it creates the institutional pressure that pulls the other four into place behind it almost automatically. If Starbucks had committed to this one structural choice alone and had held the rest of the rollout to whatever pace the oversight team could actually validate, I believe the system would still be running today in a meaningfully better form than it ever achieved during its actual nine-month life.

The second scenario asks what would have happened if the partners themselves had been trained to interpret the AI’s confidence scores rather than being shown only the final classification. A two-day training program could have taught every partner in every store what a confidence score is, what it means when the tablet reports that an item was identified with 0.62 confidence versus 0.94 confidence, and how to decide whether to trust the AI’s count or recount manually based on that signal. The system would stop pretending to be certain when it was not, the partner would have a clear and defensible rule for when to intervene, the second shift of silent verification would become a deliberate and trained workflow rather than a hidden burden, and the partner would emerge from the deployment more skilled and more valuable to the company than they had been before it began. The analysis that needs to be done honestly here, however, is the one that asks whether the training alone would have changed actual behavior in the morning rush of a real store, and my view is that it would not have, at least not by itself, because reading a confidence score does no good if the worker has no operational permission to act on it. If the partner is still being measured on completing back-of-house tasks within a fixed time window, if the manager is still scheduling the shift around an assumption that inventory takes thirty minutes rather than the forty-five minutes it would take with proper verification, and if the performance review structure still rewards speed over accuracy, then the partner will see the 0.62 confidence score and will accept the count anyway, because slowing down to recount is a behavior the organization has not actually made room for. The training has to be paired with management metrics that explicitly reward verification over speed, and with shift scheduling that accommodates the time verification requires, otherwise the training becomes a polite fiction layered on top of an unchanged operational system. My opinion on this scenario is that it is useful as a complement to the first one but dangerous as a standalone fix, because it places the cognitive burden of catching AI errors back onto the worker without giving them the time, the authority, or the metric structure to discharge it well, and the worker is left in essentially the same position they were already in during the actual rollout, except now they are also responsible for understanding a confidence score on top of everything else they were already doing silently.

The third scenario asks what would have happened if every AI rollout in the company had been paired with a human oversight structure scaled to the size of the deployment, in the way that every other safety-critical system in the modern economy is paired with its corresponding human oversight function. Airplanes have pilots in the cockpit and air traffic controllers in the tower, power grids have local operators and regional supervisors who coordinate across the network, hospitals have clinicians at the bedside and quality review boards in the institutional governance layer, and each of these structures exists because we have learned over decades that complex technical systems do not stay safe or accurate without continuous human attention to their behavior. AI deployment, with stakes that increasingly rival these other systems, currently ships without any analogous oversight function in the vast majority of cases, not because the need is unknown but because nobody put a line item for it in the original procurement proposal and no buyer thought to ask why it was missing. The historical analysis that needs to be done honestly here is that each of these oversight structures came into existence reactively rather than proactively, after the industry had paid in lives or in losses for not having the structure already in place, with the Tenerife runway collision shaping the modern shape of air traffic control, the Three Mile Island incident shaping the modern shape of nuclear plant operations, and the Institute of Medicine reports of the late 1990s shaping the modern shape of hospital quality and safety. AI as a category of technology is currently in what I would describe as the pre-Tenerife phase of its lifecycle, by which I mean that the catastrophic failure which will eventually mandate the oversight structure has not yet happened in a form severe enough to force regulatory action across the entire industry, and the rollouts of the next two to three years are quietly building toward the kind of incident that will produce that mandate. My opinion on this scenario, and the one I would most want a board of directors to sit with for the longest time, is that the oversight structure is historically inevitable and that the only meaningful executive choice available right now is whether to build it voluntarily and proactively with room to design it thoughtfully and capture its benefits, or to be forced into building it reactively later under regulatory pressure and after the company has already become the cautionary tale that other companies study. The executives who choose voluntarily now will be the ones cited in five years as the responsible early operators of this technology, and the ones who skip it will be in the case study deck that the next generation of business school students will be required to read.

The fourth scenario asks what would have happened if the workforce story had been told to the partners and to the public in honest terms from the start. Imagine the announcement reading something like this: we are deploying AI in our back rooms, we are not reducing headcount in connection with this rollout, we are expanding the role of every partner to include AI verification and feedback, and we are hiring a new class of AI quality engineers to support that expanded role. This is the story that most companies are unwilling to tell because it does not fit the cost-cutting deck the CFO needs for the next earnings call, but it is also the story that would have given the partners a reason to be invested in the system’s success rather than skeptical of its motives, and it would have surfaced the operational realities of the back room into the design process months before the system shipped. The harder analysis that has to be done honestly here is the one that asks why no executive is currently telling this story even when they privately believe it, and my view is that the answer is not a personal failing on the part of the executives but rather a coordination problem at the level of the market itself. The current investor narrative around AI rewards companies that signal headcount reduction in connection with AI deployment, regardless of whether the AI is actually replacing the underlying work, which means the chief executive who tells the honest story takes a real stock price hit from the same investors who are funding the AI deployment in the first place. This is the trap that most public company executives are sitting inside right now, and the trap will not release until enough boards begin to realize that the dishonest story is what is producing the Starbucks pattern at the operational layer, and that the operational pattern is what is eventually destroying the value the AI rollout was supposed to create in the first place. My opinion on this scenario is that it is the hardest of the five to execute even when leadership privately wants to do it, because it requires bucking the entire current investor mood at the same moment that the investor mood is funding the AI strategy, and that this scenario will become executable at scale only when enough articles like the one you are reading right now help to seed the realization that is currently missing from the boardroom conversation. The piece in your hands is, in that limited sense, a small contribution to making this fourth scenario eventually possible.

The fifth scenario asks what would have happened if executives had drawn the line at the contract stage rather than after the failure was already visible. No AI deployment should be approved without explicit answers to a small number of governance questions, including who is accountable when the AI is wrong, what the confidence threshold is for autonomous action, what the rollback trigger looks like and who has the authority to pull it, who owns the human oversight team and where they sit in the org chart, what the retraining cadence will be and what the feedback loop is from worker corrections back into the model, and what the operational testing protocol is for the conditions under which the system will actually be used. Most AI contracts today are signed without any of these answers on the table, with the procurement process treating AI as if it were ordering office furniture, and the predictable result is that the answers get discovered the hard way, in production, after the system has already been deployed at scale. The structural analysis that has to be done honestly here is that the reason these questions almost never get asked in procurement is not because the buyer does not care, but because the buyer does not yet have the framework to know which questions are the important ones, the vendor does not volunteer the questions because they extend the sales cycle, and the legal team is checking standard contractual boilerplate rather than AI-specific governance language. Until procurement teams across industries have a shared rubric for AI contract due diligence, every individual deployment will reinvent these questions on its own and most will reinvent them badly or incompletely, which is the pattern we are watching play out in real time across the early enterprise rollouts of this technology. My opinion on this scenario is that it is the place where regulation will eventually arrive, because the market is not going to solve the problem on its own at anything close to the speed at which the technology is being deployed, and we are already seeing the early shape of that regulatory move in places like the recent FDA guidance for the use of AI in pharmaceutical quality systems, where the agency has now stated on the record that the company holds the liability regardless of whether the failure originated in an AI tool or in a human consultant. The executive who treats AI procurement today with the same diligence they would apply to the hiring of a senior consultant or to the qualification of a critical supplier is doing voluntarily what they will very soon be required to do anyway by their regulators, by their auditors, and by their insurers, and the gap between the voluntary adopters and the forced adopters is going to be one of the defining competitive lines of the next three years.

These five scenarios are not equally easy to execute, as the analysis I have walked through tries to make plain, and I want to be honest about the fact that the first one is the structural lever that pulls the others into place, the second one is dangerous in isolation though valuable as a complement, the third one is historically inevitable and the only real question is voluntary or forced, the fourth one is the hardest cultural lift because it requires bucking the investor narrative at the moment that narrative is funding the deployment, and the fifth one is where regulation will eventually arrive whether the industry is ready for it or not. What is shared across all five, and what makes them worth thinking about together rather than choosing among in isolation, is that each of them costs more in the short term than the path Starbucks actually took, and each of them, when you sit with the math honestly, costs far less in the long term and produces an AI deployment that actually achieves the operational outcome the technology was bought for, without quietly transferring the failure modes onto the workforce the company is simultaneously planning to lay off. The math is not the obstacle here. The thinking is.

The Roles, Properly Defined

The framing I want to argue for, and the framing that should sit at the center of every AI deployment decision going forward, is that humans and machines do not compete for the same role in a well-designed system because they are structurally suited to different roles. Machines are extremely good at pattern recognition across very large numbers of inputs, at repetition without fatigue across long time horizons, at computation that exceeds human speed by many orders of magnitude, at consistency on inputs that resemble the data they were trained on, and at operating across all twenty-four hours of the day without the breaks that human bodies require. These are real and valuable capabilities and there is no honest case for ignoring them in modern operations. Humans, in turn, are extremely good at integrating context that the machine never saw and could not have seen, at exercising judgment on novel situations that fall outside the training distribution, at maintaining calibrated awareness of their own uncertainty and communicating that uncertainty honestly to others, at making ethical decisions where the stakes are not encodable in a loss function, at holding accountability when the system errs in ways that affect real customers and real outcomes, at remembering past failures and the causes that produced them so that the same failure does not happen again, and at the deeply skilled work of training and supervising and correcting the machines themselves.

Notice what happens to the human role in a system that takes both lists seriously. As the machines do more of the work they are well suited for, the human role does not shrink, it changes shape and in most cases becomes more demanding rather than less. The human becomes the ringmaster who coordinates the system, the watchman who monitors its behavior, the trainer who feeds it new data and corrects its drift, the verifier who checks its outputs against ground truth, the loop-closer who turns every failure into an institutional learning, and the accountable party who answers for the system when something goes wrong in front of a customer or a regulator. These roles are not less skilled than the work the AI displaced, they are more skilled and more cognitively demanding, and they require deliberate investment in training programs and in headcount budgets and in career paths that did not exist in the pre-AI version of the organization.

If you are deploying AI seriously, you should expect your workforce composition to shift in shape rather than to shrink in size. You will have fewer people doing the specific task the AI now performs, and you will have more people doing the work the AI cannot do, which includes the supervision of the AI itself and the integration of its output into the broader context of the organization’s actual operations. A board of directors that approves an AI deployment with simultaneous layoffs in the same department is not making a productivity decision, regardless of how the decision is dressed for the earnings call. They are telling you plainly that they do not understand what they have just bought, and the cost of that misunderstanding will land somewhere in the organization eventually, usually on the customers and the workers who remain.

Human-Machine Synergy

In October 2024, I published a book titled The AI-Human Synergy: A Data Scientist’s Vision for the Future, in which I argued that the future of work is not a story of humans being replaced by machines but a story of humans and machines entering deliberate and complementary partnership, with each side doing what each is genuinely good at, each side compensating for the other’s blind spots, and each side accountable for the part of the work that only that side can do. The argument was meant to be a corrective to a public conversation that had already begun to slide into a binary framing of replacement versus survival, and it was rooted in the observation, from my own twenty-five years across education and research and applied data science, that the systems which actually work in the real world are almost always the ones designed around this kind of partnership rather than around the fantasy of full automation.

We are not yet living in the world that book describes. We are living instead in a transitional moment where companies are buying AI as if it were a workforce replacement, deploying it as if it were proven, laying off workers as if the deployments had succeeded, and then quietly retiring the systems eighteen months later when the failures become impossible to hide any longer. The Starbucks case is one example of this pattern, and it will not be the last one. There are several more in motion right now that will reach the trade press over the next twelve months, and the underlying dynamics will be recognizable in every one of them once you know what to look for. In the broader work I am leading at SANJEEVANI AI to build quantitative trust measurement infrastructure for enterprise AI deployments, this five-scenario lens is one of the components that feeds into a larger architecture for evaluating whether an organization is genuinely ready for the AI it is in the process of buying.

What I am calling for in this article, and what I think every executive in a position to draw this line should be calling for in their own organizations, is a different posture toward AI deployment that is neither anti-AI nor pro-layoff but rather pro-design, in the sense of designing the human-machine system on purpose rather than allowing it to emerge by accident from the collision of vendor sales decks and CFO spreadsheets. Defining the roles deliberately, building the oversight structure to match the deployment size, training the humans for the new and more demanding roles they are being asked to occupy, and hiring more of them in different roles to do the work the AI cannot do, all of this is the actual work of what I have been calling human-machine amity, and it is harder than firing people because it requires the organization to think clearly about what its own jobs actually are. It is also, and this is the part that should matter to the people who hold the budgets, the only path that delivers the productivity gains the technology is genuinely capable of producing.

The Starbucks partners knew what their job actually was, the AI did not know what the job was because it had never been told and could not have been told within the structure of the deployment as it was designed, and the executives somewhere in the middle of those two ends of the system mistook the visible surface of the work for its actual substance. That is the mistake that is being made at scale, in industry after industry, in the rollouts happening right now in 2026, and it is a mistake that we have every technical and organizational capability to correct if we are willing to do the harder thinking it requires. The next phase of this conversation, and the one I believe the field has to move into over the next two to three years, is not just better thinking about these tradeoffs but the construction of measurable infrastructure that quantifies them before deployment rather than discovering them after, and that direction of work is exactly where I am committing my own time. We can do this differently, and given what is now on the line for workers and for customers and for the credibility of AI as a category of technology, I would argue that we have to.

Read this, ponder it, sit with it, and bring it into the next AI conversation you find yourself in this week, whether that is a board meeting, a procurement call, a hallway conversation with a worker on your team, or a quiet moment of your own thinking. I am writing this as a businesswoman who is building toward a future where AI and humans work in deliberate partnership, not as a critic standing outside the field throwing stones. The companies that get this right over the next three years will be the ones still standing in 2030, and the work of getting it right starts with the kind of structured thinking we have just walked through together. Until next week.

Suneeta Modekurty
Founder, SANJEEVANI AI

Image created using AI tools

When a company lays off nearly a quarter of its staff, the narrative is usually one of retreat. But ClickUp’s recent 22 percent headcount cut was different. It was an aggressive, forward-leaning restructuring designed to answer a single, brutal question: Who deserves to survive in the age of AI?

The people who stayed, the survivors, were not kept because they knew how to write basic prompts or use chat interfaces. They were kept because they possessed a capability that most organizations cannot even define, let alone measure: the ability to direct, audit, and take absolute operational accountability for machine output.

This represents a massive, necessary transition from AI Literacy to AI Readiness.

Most executives treat AI adoption as a soft skill, rolling out licenses and calling “basic prompting” a success metric. But this approach creates a black box. When you cannot measure what your AI tools are actually producing, you cannot enforce accountability. You end up with an unmanageable explosion of automated noise. ClickUp’s restructure is the first industrial-scale proof that AI adoption without structural measurability is a dead end. It is a loud wake-up call for how we define the work of the future.

The Myth of Universal Productivity

In May 2026, ClickUp, a software company valued at $4 billion, announced a 22 percent headcount reduction. But unlike the standard tech layoffs of the past few years, which were framed as defensive, cost-cutting measures, ClickUp’s CEO, Zeb Evans, framed this cut as a proactive, structural bet.

Evans wrote on X that “the business is the strongest it’s ever been” and that “this wasn’t about cutting costs.” The goal was to completely rebuild the company around what he calls a “100x organization.”

To incentivize the transition, ClickUp threw traditional compensation structures out the window, introducing $1 million cash salary bands for the employees who remained, provided they could demonstrate “100x impact” by building and managing AI systems.

The mainstream headline was the layoff. The real story is the structural thesis underneath.

Evans challenged one of the most sacred assumptions of the generative AI era: the idea that AI tools automatically make everyone more productive. He argued the exact opposite.

In a traditional workflow, giving AI tools to junior or untrained staff does not speed up an organization. It creates an explosion of raw volume. Evans pointed specifically to companies celebrating 500 percent increases in pull request volume without matching customer outcomes, calling this “the great reckoning of AI coding.”

But someone has to review that output. Someone has to fix the subtle, confident errors the AI made. When that mass of automated volume collides with senior staff, it creates a massive review bottleneck. The senior people end up spending all their time auditing, rewriting, and debugging AI-generated noise.

As Evans bluntly wrote: “AI makes the best engineers wildly more productive, and everyone else using AI slows these engineers down... More code is just another bottleneck.”

This is the great paradox of the agentic era. When the cost of generating work drops to zero, the cost of evaluating that work becomes the most expensive line item in your business.

And that is where the line between AI literacy and AI readiness is drawn.

AI Literacy vs. AI Readiness

Most corporate training programs are designed to build AI Literacy. They teach employees what a Large Language Model is, how to open a chat interface, and how to write basic prompts. It is cognitive, basic, and tool-centric.

AI Readiness, however, is operational. It is the human capability to direct, judge, and take absolute accountability for automated systems.

When a company runs thousands of AI agents internally, it doesn’t need people who can write. It needs people who can manage. It needs what ClickUp calls Agent Managers: workers who can automate their own manual tasks, build systems around those automations, and act as the rigorous human filter for the machine’s output.

To build an organization of Agent Managers, companies must cultivate three distinct human capabilities.

1. Direction: From Prompting to Orchestration

An AI tool does exactly what you tell it to do. Therefore, the harder and more nuanced the work, the more critical the telling becomes.

The AI-Literate worker prompts: “Write a marketing email for our consulting services.” The result is a generic, instantly deleted block of corporate jargon.

The AI-Ready worker orchestrates: “Write a 200-word email to a small business owner who has heard of AI but fundamentally distrusts it. Open by directly validating that trust barrier, and close with a single, low-friction question they can answer in 30 seconds.”

The core skill here is not writing. The skill is system design. It is the ability to deconstruct a highly complex, intuitive human process, isolate its underlying variables, and translate them into a structured instruction set that an agent can execute flawlessly.

For decades, corporate structures have rewarded people for doing the work. Suddenly, we need people who can describe the work in such vivid, mechanical detail that a machine can replicate it. That is a rare, highly strategic skill. Traditional prompt-engineering workshops do not teach it because it requires deep domain expertise, not just software knowledge.

2. Judgment: The Review Bottleneck

Because AI tools produce highly confident, plausible-sounding answers that are frequently wrong, the ultimate bottleneck is no longer production. It is review.

The readiness skill here is knowing when to trust the output and when to interrogate it. This skill cannot be installed via a corporate slide deck. It is the slow, painful result of having done the manual work yourself, badly, for years.

It is what a senior software architect brings to reviewing AI-generated code, or what a veteran forensic accountant brings to an AI-run reconciliation. They do not look at the 99% that is right. They have the instinct to find the 1% that is catastrophically wrong.

Companies that lay off their expensive, senior experts while keeping only their cheap, “AI-fluent” juniors are going to discover their lack of judgment the hard way.

They will find themselves drowning in flawless-looking, broken work.

AI literacy lets you generate the code. AI readiness gives you the wisdom to realize that code should never go to production.

3. Ownership: The Air Canada Rule

An AI agent does not get fired when a calculation is wrong, and it does not get sued when a client is misled. The human in the chair does.

This is the psychological side of AI readiness that never makes it into vendor product demos. It is the willingness to sign your name to output you did not write, defend it to a client when it is challenged, and accept the professional and legal consequences when it fails.

This isn’t theoretical. In February 2024, the British Columbia Civil Resolution Tribunal ruled in Moffatt v. Air Canada that the airline was liable when its customer-facing chatbot promised a passenger an unauthorized bereavement refund. In its defense, the airline argued that the chatbot was a “separate legal entity” responsible for its own misinformation. The tribunal rejected the argument outright, ruling that the company was entirely accountable for the systems it chose to deploy.

Somebody at Air Canada owned that mistake, whether they wanted to or not.

Most companies have still not defined who legally and operationally owns their AI’s output. They will figure it out the first time a customer pushes back, and the lesson will be incredibly expensive.

The New Architecture of Work

If ClickUp’s thesis is correct, the transition to AI readiness will force a radical redesign of traditional corporate roles. Evans has outlined three distinct classes of ready workers:

The Builders. These are the 10x engineers and product designers. They are no longer typing lines of code or drawing static mockups. They are acting as system architects, directing fleets of agents to build, test, and iterate. They spend their time on system design and validation.

The Agent Managers (Systems Managers). These are operational workers who have successfully automated their own manual tasks. Instead of being displaced, they are kept because they possess the institutional knowledge required to run, monitor, and troubleshoot the automated systems they created. As Evans put it: “The people that automate their jobs with AI will always have a job.”

The Front-liners. These are the human-to-human connection points. In an era where AI can generate infinite digital noise, authentic human contact becomes a premium bottleneck. Under ClickUp’s model, customer-facing humans are explicitly protected from automation. They do not use AI to replace meetings. They use AI to automate everything around the meetings, so they can spend 100% of their time focused on real human relationship-building.

Image created with AI tools

Why Readiness Cannot Be Bought

The pattern repeating across the corporate world is a fundamental confusion of terms.

AI Adoption is technical. It is the work of installing software, integrating APIs, and provisioning licenses. Adoption can be bought. You sign a contract, run a launch event, and the software lands on every laptop in the company.

AI Literacy is cognitive. It is the basic understanding of how these tools work. Literacy can be taught via quick workshops and video modules.

AI Readiness is operational. It is the human capability to establish direction, exercise judgment, and accept ownership over automated workflows. Readiness cannot be bought, and it cannot be taught in a 45-minute webinar. It must be built through deliberate, supervised practice on real-world work over months.

This is why so many pilot programs yield zero measurable ROI. They successfully bought adoption, achieved basic literacy, and completely skipped readiness.

What to Do This Week

If you want to know where your company actually stands on the spectrum of AI readiness, run this simple test on Monday morning.

Pick one AI tool your team currently uses. Select one piece of work that tool produced this week.

Then ask these three questions.

Direction. Who wrote the operational instructions that produced this output, and could a different colleague reproduce this exact quality using those instructions alone?

Judgment. Who audited this specific output, and exactly how do they know it is correct?

Ownership. If this output goes to a client and severely backfires, whose name is on the line?

If you cannot confidently answer all three questions for a single piece of work, your company has adopted a tool, but it has not built readiness. You have active software, but you do not have an active system.

Closing that gap is the defining corporate challenge of the next two years. The organizations that build true AI readiness will look unrecognizable from the inside by the end of the decade. The ones that do not will still be running prompt-writing workshops, staring at empty dashboards, wondering where their ROI went.

Sources

1. Zeb Evans, post on X (May 21, 2026). primary source for the 22% layoff, the “100x organization” framing, and Evans’s direct quotes.

2. Fortune, “Outnumbered: At $4 billion ClickUp, a 3:1 agent-to-human ratio is rewiring work itself” (May 18, 2026). source for the 3,000 internal AI agents and the 3:1 agent-to-employee ratio.

3. StartupHub.ai, “ClickUp’s 22% cut comes with $1M salary bands. Evans calls it the 100x org.”. source for Evans’s “great reckoning of AI coding” critique and the 500% pull request volume reference.

4. The Next Web, “ClickUp cuts 22% of staff, offers $1M salaries in AI restructuring”. source for the Builders / Agent Managers / Front-liners taxonomy.

5. Business Today, “’People who automate jobs with AI will always have a job’: ClickUp cuts 22% of its workforce”. source for the Agent Manager quote.

6. Moffatt v. Air Canada, British Columbia Civil Resolution Tribunal (February 14, 2024). source for the Air Canada chatbot liability ruling.

Suneeta Modekurty | Founder, SANJEEVANI AI | Quantifying AI Readiness

Image created using AI tools

We are AI First, We are using AI, We are leveraging AI across the enterprise. ….

These kind of phrases have been around for a while. They show up in headlines and slide decks and strategy documents. But saying “AI ready” and actually being ready are two very different things.

In this series, I want to stay with a single, uncomfortable question:

Are we simply using AI, or are we genuinely ready for what it does to our decisions, our people, and the communities we serve?

To get there, I am going to stay close to real stories. Some come from the patterns I have heard across more than five hundred conversations I had over past months, with practitioners, leaders, and operators working through AI questions inside their organizations, with details abstracted to protect the people and the work. Others, like the one in this article, are already public. All of them sit at the intersection of three things: a human being in a real situation, an AI system that sounds confident, and an organization that is about to discover what “not ready” really means.

Let us begin with an incident that happened in November 2022.

A traveler in British Columbia named Jake Moffatt opened the Air Canada website to book a last-minute flight from Vancouver to Toronto. He was a private individual; his name only later entered the public record because of the complaint he eventually brought. His grandmother had just died. He needed to be at the funeral. Before he booked the ticket, he asked the airline’s chatbot whether Air Canada offered bereavement fares, and if so, how to claim one.

The chatbot told him that the airline did offer a reduced bereavement rate, and that if he needed to travel immediately, he could book at the standard fare and then apply for the discount by submitting a ticket refund application within ninety days of the date the ticket was issued. Moffatt took a screenshot of the exchange. He booked the flight. He attended the funeral. When he returned home, he submitted his refund application with the required documentation, including his grandmother's death certificate. Air Canada denied the request.

The actual bereavement policy on the airline’s website, on a separate page titled “Bereavement travel,” stated clearly that the discount could not be applied after travel had been completed. The chatbot had been wrong. An Air Canada representative later acknowledged, in correspondence with Moffatt, that the chatbot had provided misleading words and said the airline would update it. They did not offer him the discount.

Moffatt took the matter to the British Columbia Civil Resolution Tribunal, a small-claims body designed for low-value consumer disputes. The amount in question was a few hundred Canadian dollars. Air Canada, defending itself, made an argument that would later draw international attention. The airline argued that it could not be held liable for what the chatbot had said, because the chatbot was, in the airline’s framing, a separate legal entity responsible for its own actions. The tribunal’s response, written by Member Christopher C. Rivers, has been quoted in legal commentary in nearly every jurisdiction that watches AI law. The tribunal called the submission remarkable. It then explained, plainly, that while a chatbot has an interactive component, it is still just a part of Air Canada’s website, and it should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference, the tribunal continued, whether the information comes from a static page or a chatbot. Air Canada was ordered to pay Moffatt eight hundred and twelve dollars and two cents in total damages and fees. The decision, Moffatt v Air Canada, 2024 BCCRT 149, is not binding on any other court. It has nevertheless been treated, in nearly every serious analysis published since, as the first clear judicial statement of a principle the industry had been quietly avoiding. An organization that deploys an AI system is the author of what that system says.

It is tempting to read this as a chatbot story. A vendor sold an airline a customer service tool, the tool gave a wrong answer, the airline paid for the mistake, and the technology will get better.

That reading misses what is actually instructive about the case. The technology is not the failure mode. The failure mode is upstream of the technology, inside the organization that put the system in front of customers.

Air Canada deployed a customer-facing AI without, by all available indications, the apparatus needed to know what the system was saying day to day, the mechanism needed to correct it when it was wrong, or the internal framing needed to take responsibility for it when it caused harm. The remarkable submission the tribunal rebuked was not a clever legal maneuver by outside counsel. It was the logical end of an organization that had deployed a system it did not feel it owned.

This is what we mean by AI readiness, and this is where literacy enters the picture. The question almost every executive team is now asking is some version of are we using AI.

The honest answer in most organizations is yes, in some workflows, with varying degrees of intention. The more important question is the one Air Canada was, in effect, forced to answer in front of a tribunal. Are we ready for the fact that we are using it. Use is everywhere.

Readiness is the thing your board, your regulators, your customers, and increasingly your courts will ask you to demonstrate.

The Moffatt case, in a few pages of tribunal decision, demonstrated that Air Canada was using AI and was not ready for it.

Readiness sits on top of literacy, and the literacy that failed at Air Canada is the literacy of deployment. The literacy of deployment is what an organization’s leaders, owners, and senior decision-makers need so that the choices about where AI enters the organization are deliberate rather than reactive. It is not about coding skill or model intuition. It is the capacity to ask, before a system goes live, three questions that look simple and almost never get clear answers in unprepared organizations. The first is what this system will say to the people it interacts with, and whether that aligns with what the rest of the organization is saying on the same topics. The second is who, inside the building, owns the answer if the system gets it wrong. The third is what the organization will do, operationally and legally, when the system gets it wrong in a way that produces harm. Air Canada had a chatbot that said one thing, a policy page that said another, and no apparent reconciliation between the two. The deployment decision had been made without the deployment literacy needed to carry it.

This is also where shadow AI enters the conversation, and it is worth naming, because shadow AI is the version of this gap that most organizations are quietly living inside today. Shadow AI is the AI use that happens inside an organization without the knowledge, sanction, or oversight of the people who would be accountable if something went wrong. A marketing manager who pastes draft customer messaging into a public AI tool to tighten it up before sending. A finance analyst who runs a board memo through a generative system to make it sound sharper. A field engineer who asks an open chatbot how to handle a regulated chemical because the internal documentation is too slow to navigate.

None of these people are acting in bad faith. They are doing what their workload asks them to do, with the tools that are most accessible at the moment of pressure. The cumulative effect is that the organization is using AI in places its leadership does not know about, with data its policies have not classified, producing outputs that influence decisions that will later need to be defended. The Air Canada chatbot was a sanctioned deployment that did not have the literacy behind it. Shadow AI is the unsanctioned deployment that does not even have the visibility behind it. Both produce the same outcome, which is decisions the organization cannot defend.

The reason the Moffatt case has been quoted so widely is not that the airline paid a few hundred dollars. The reason is that the tribunal, in plain language, removed the option that many organizations had been quietly relying on. The option to argue, when an AI system causes harm, that the system was somehow separate from the organization that ran it. That option no longer exists in any jurisdiction that takes the Moffatt reasoning seriously, and the reasoning is too straightforward to confine to one tribunal in one province. If your AI is part of your website, your customer service, your hiring funnel, your underwriting workflow, your clinical pathway, your procurement chain, then your AI is part of your organization. What it says, you said.

The shift this forces is not legal. It is organizational, and it is what AI readiness is actually about. An organization that is ready for AI has built the apparatus to stand behind every AI-mediated decision its name is attached to. That apparatus does not begin with the technology. It begins with deployment literacy at the top of the house, and it cascades from there. The frontline staff need the literacy of use so they can interpret AI outputs with calibrated suspicion. The builders need the literacy of development so the systems they ship can be reasoned about and not just deployed. The leaders need the literacy of deployment so the choices about where AI enters the organization are deliberate. Without all three, readiness is a claim the organization makes about itself without the means to defend it.

The European Union, the National Institute of Standards and Technology, and the International Organization for Standardization have each written the requirement for AI literacy into their respective frameworks in the last three years. Article 4 of the EU AI Act, in force since February 2025, requires both providers and deployers to ensure sufficient AI literacy among the staff who use these systems. The NIST AI Risk Management Framework places workforce competency inside its GOVERN function. ISO/IEC 42001 requires organizations to determine and ensure the competence of personnel whose work affects AI performance. The instinct across all three is the same. The frameworks know that literacy is the missing piece.

None of the frameworks tells you what good looks like in operational detail, and none of them gives you a score. The frameworks name the requirement and pass the burden of measurement back to you. That is the gap we will keep returning to in this series.

Put differently: they can tell you that your people need AI literacy; they cannot tell you whether they have it.

If your organization had been Air Canada in November 2022, the operational questions worth sitting with are these.

• Would you have known what the chatbot was telling customers about bereavement fares that week.

• Would the chatbot’s answer have matched the page on your own website that bore the same title.

• Would anyone inside the building, before the refund was denied, have been able to flag that the two were saying different things.

• Would your first response, when the customer complained, have been to honor what your system had said, or to argue that the system was not yours.

These are not technology questions. They are readiness questions, and the answers are the difference between an organization using AI and an organization ready for it.

Sit with one question this week. If a customer, a regulator, or a court asked you tomorrow to defend one AI‑mediated decision your organization has already made, who in the building would you ask first, and could they answer.

Suneeta Modekurty | Founder, SANJEEVANI AI | Quantifying AI Readiness

Image created using AI tools

Dear Friends,

A school superintendent I worked with last year sat across from me at a kitchen table and asked a question she had been carrying for months.

“My teachers are using AI. My students are using AI. The plagiarism vendor wants a contract. The school board wants a policy. The parents want answers. And I do not know what good looks like. How do I tell if we are ready?”

I have heard that question, in different costumes, from a state CIO, a hospital chief of staff, a community organizer, a board chair, a founder, and a parent in a Costco parking lot.

The question sounds like it is about AI. It is not. It is about a chain that runs underneath AI, in a direction most current conversations do not name.

AI Readiness depends on AI Adoption. Adoption depends on Awareness. Awareness depends on Literacy. Literacy is the foundation, and most organizations skip straight past it.

The next letter arrives Tuesday, and from there every other Tuesday at nine in the morning Central time.

Suneeta Modekurty
Founder, SANJEEVANI AI

I am making a simple change today. I am stepping back from the regular cadence for now. No action needed on your part.

When I return, it will be with more clarity, more depth, and more of what actually matters to you as a practitioner navigating AI in the real world.

Thank you for being here early. That means everything.

See you soon.

Suneeta

Image created by AI

Last week I wrote about a New York bill that would restrict AI systems from providing professional advice in licensed fields. A few readers asked a sharp follow-up question: does the bill apply differently depending on how the AI system was built?

The answer might surprise you, and I wil…

Read more

Image created by AI

Yesterday I wrote about the shift from measuring organizations to measuring AI systems. Today, the New York State Legislature is proving why that shift is urgent.

A bill introduced by Senator Kristen Gonzalez would restrict AI systems from providing what lawmakers call “substantive responses” in fields that require professional licenses. Medicine, law, engineering, psychology, dentistry, nursing, and other regulated professions where incorrect guidance can cause serious harm.

Read that again carefully. The bill does not say “companies must have policies about what their AI says.” It says AI systems must not provide certain types of responses. The subject of the regulation is the machine, not the organization.

This is the shift happening in real time.

What the bill actually does

The proposal draws a line between general information and professional advice. An AI chatbot can share educational content about, say, symptoms of a condition or how a legal process generally works. What it cannot do is cross into substantive guidance that resembles what a licensed professional would provide. It cannot offer what looks like a medical diagnosis, a legal strategy, an engineering recommendation, or a psychological assessment.

The bill also includes a private right of action. That means individuals can sue companies if their AI systems provide restricted guidance. This is not a regulatory slap on the wrist. This is litigation exposure for every company deploying a customer-facing AI system in a licensed domain.

Why this matters beyond New York

If you are thinking “I do not operate in New York, this does not apply to me,” think again.

New York tends to set the template. When New York moved on financial regulation, the rest of the country followed. The same pattern is already forming with AI. Colorado’s AI Act takes effect in 2026. The EU AI Act becomes fully enforceable in August 2026. The NAIC Model Bulletin on AI in insurance has been adopted by 24 states. NYC Local Law 144 already requires bias audits for automated hiring tools.

The direction is clear: regulators are moving from governing organizations that use AI to governing what AI systems actually do. And they are doing it jurisdiction by jurisdiction, which means any company deploying AI across state lines will soon face a patchwork of requirements that all ask the same fundamental question: does your AI system stay within its authorized boundaries?

The measurement problem this creates

Here is where the data scientist in me gets interested.

“Substantive response” is a fuzzy concept. Where exactly does educational information end and professional advice begin? When does a health chatbot cross from sharing general wellness content into offering what could be interpreted as a diagnosis? When does a legal information tool cross from explaining a process into recommending a strategy?

These are not binary questions. They are spectrum questions. And spectrum questions require quantitative measurement, not policy checklists.

Think about what an organization would need to demonstrate under this bill. Not that they have a policy saying “our AI does not give medical advice.” They would need to demonstrate that their AI system actually stays within bounds, consistently, across thousands of interactions, including edge cases where users push the boundaries with creative phrasing.

That is a behavioral measurement problem. You cannot solve it by reading the organization’s policy documents. You solve it by observing what the AI system actually says when real people interact with it. You measure boundary adherence: how often does the system recognize when it is approaching a restricted domain, and how reliably does it pull back?

This is exactly the kind of observable, quantifiable AI system property that I described yesterday. The policy says the system will not give medical advice. The behavior shows whether it actually does or does not. The gap between those two is where the litigation risk lives.

What this means for different types of AI deployments

The bill applies regardless of how the AI system was built, but the risk profile varies significantly.

Organizations that build their own AI from the ground up have complete control over system prompts, guardrails, and response boundaries. They can engineer precise limits. But they also own 100% of the liability.

Organizations using low-code platforms like Copilot Studio or LangFlow face a shared responsibility problem. The platform provides underlying model behavior and some guardrails, but the builder configures the use case and the domain scope. When the system drifts into professional advice territory, who is liable? The platform or the builder?

And then there are the no-code deployments, the custom GPTs, the drag-and-drop chatbot builders. This is the highest risk category, and it is not close. The people building on these platforms are often the exact professionals the bill is trying to protect: small healthcare clinics, law offices, dental practices. They deploy an AI chatbot on their website, feed it their documents, and assume the platform handles compliance. It usually does not.

The gap between how easy it is to deploy AI and how hard it is to govern what it says is widest in the no-code tier. And that gap is exactly where this bill’s private right of action will land hardest.

The deeper signal

Step back from the specifics of this one bill and look at what it represents.

For decades, professional licensing has been a human-to-human regulatory framework. A doctor is licensed. A lawyer passes the bar. An engineer gets certified. The license attaches to the person, and the person is accountable for what they say.

AI breaks that model. The chatbot giving health guidance is not a licensed professional. It is not a person. It cannot be sued, sanctioned, or stripped of credentials. So the regulatory framework has to evolve. It has to attach accountability to the system’s behavior and to the entity that deployed it.

This bill is one of the first attempts to do that explicitly. It will not be the last. And every attempt will come back to the same core question: can you prove, with data, that your AI system behaves within its authorized boundaries?

That is not a policy question. That is a measurement question. And it demands the kind of quantitative, reproducible, behavior-based measurement that this newsletter exists to explore.

More next Tuesday.

This is part of the “Before The Number” series at A.I.N.S.T.E.I.N., exploring what it takes to build quantitative AI governance measurement from first principles. If this resonated, share it with someone deploying AI in healthcare, legal, or any licensed profession.

Image created by AI

I owe you an explanation for the silence.

Two weeks ago, I published “How to Measure AI Governance” and laid out the five pillars, the metrics, the frameworks, the KPIs. I meant every word of it. And then I went quiet, because something broke in my own thinking that I could not write aro…

Read more

Image by AI

If you cannot measure it, you cannot govern it.

That principle holds true across every regulated industry, from finance to healthcare to cybersecurity, and it holds true for AI.

Yet when most organizations talk about AI governance today, they are talking about policies, principles, and frameworks. They are talking about what they believe, not what they can prove. And there is a meaningful difference between having an AI ethics policy and being able to demonstrate, with data, that your AI systems are actually governed.

This essay is a practical guide to bridging that gap. It walks through the core pillars of AI governance measurement, the specific metrics that matter, the frameworks available to structure the work, and the challenges that make this harder than it sounds. If you are a CISO, a Chief AI Officer, a compliance leader, or a founder building in this space, this is the foundation you need.

Why Measurement Matters

Without measurement, AI governance is a set of intentions. It lives in documents that get written once and reviewed quarterly at best. It gives leadership a sense of comfort without giving them a basis for action.

Measurement changes that in four concrete ways.

1. It demonstrates due diligence. When regulators, boards, or the public ask how your AI is managed, measurement gives you evidence rather than assurances.

2. It allows you to identify and mitigate risks before they cause harm, because metrics like model drift detection time and fairness deviation surface problems that narrative assessments miss entirely.

3. It prepares you for the regulatory compliance landscape that is already here, with the EU AI Act requiring specific documentation and measurement for high-risk AI systems.

4. And it builds trust with every stakeholder who needs to know that your AI decisions are fair, transparent, and accountable.

The Five Pillars of AI Governance Measurement

Effective AI governance measurement is not about tracking model accuracy or inference speed. Those are performance metrics. Governance measurement focuses on accountability, fairness, transparency, compliance, and safety. These are the five pillars, and each one requires its own set of metrics.

Accountability and Ownership

This pillar measures who is responsible for your AI systems and their outcomes. It sounds basic, but in my experience, a surprising number of organizations deploy AI systems where no single person owns the governance risk. The model was built by one team, deployed by another, and monitored by no one in particular.

The qualitative goal is straightforward: every high-risk AI system should have a named business owner who is accountable for its impact. The quantitative metric that tracks this is the percentage of deployed AI systems with a defined, documented business owner. If that number is below 100% for your high-risk systems, you have a governance gap that no policy document can close.

Transparency and Explainability

This pillar measures how well your AI system’s decisions can be understood by the humans affected by them. A lending model that denies an application needs to be able to explain why. A hiring algorithm that filters out candidates needs to produce a reason that a human can evaluate.

The quantitative metric here is the percentage of AI-driven decisions that include a human-interpretable explanation. In practice, this is one of the hardest metrics to improve because many complex models, particularly large language models, are inherently difficult to explain. But the measurement itself forces the conversation about where explainability gaps exist and how material those gaps are.

Fairness and Bias Mitigation

This pillar measures the extent to which your AI systems treat different demographic groups equitably. It is not enough to say “we care about fairness.” You need to measure the actual disparity in outcomes across protected groups and track that disparity over time.

The core metric is the measurable difference in approval rates, error rates, or outcomes between demographic groups. If your lending model approves 78% of applications from one group and 61% from another, that disparity is your fairness metric, and it needs to be monitored continuously, not just checked once before deployment.

Risk and Compliance

This pillar measures adherence to both internal policies and external regulations. With the EU AI Act, NIST AI RMF, and ISO 42001 all converging on requirements for risk classification and documentation, this pillar is becoming the most operationally urgent.

The key metrics include the percentage of high-risk AI systems that have completed an Algorithmic Impact Assessment, the percentage of inventoried systems that have undergone formalized risk classification, and the policy adherence rate across all AI projects. These numbers tell you whether your governance framework is actually being followed or whether it exists only on paper.

Safety and Security

This pillar measures your AI system’s resilience against attacks, errors, and unintended harm. It includes incident response readiness and the speed at which AI-specific failures are detected and resolved.

The metrics that matter here are the average time to detect and time to resolve AI-related incidents, including model drift, toxic output, adversarial attacks, and data pipeline failures. If your organization cannot tell you how long it takes to detect when a model has drifted from its intended behavior, your safety posture has a blind spot.

Key Performance Indicators for AI Governance

Beyond the five pillars, there are specific KPIs that give leadership a clear picture of governance health across the organization.

Program health metrics include AI inventory coverage (the percentage of all AI systems currently cataloged), risk classification completion (the percentage of inventoried systems that have been formally classified by risk level), and policy adherence rate (the percentage of AI projects fully compliant with established guidelines).

Decision and accountability metrics include decision latency for risk issues (how long it takes to make a material decision on an escalated AI risk), human override rate (how frequently automated decisions are reversed by human reviewers), and governance debt (the number of deferred governance controls that were postponed to speed up deployment).

Operational integrity metrics include model drift detection time, data lineage visibility (the percentage of models with full source-to-sink tracking), and audit readiness score (the percentage of models with current documentation and version control).

Ethical impact metrics include explanation coverage and fairness deviation, both of which I discussed in the pillars section above.

The important thing about these KPIs is that they are specific, measurable, and tied to real governance risk. They are not opinions. They are not traffic lights. They are numbers that a board can track quarter over quarter and that an auditor can verify independently.

The Frameworks That Structure This Work

Organizations do not need to build their measurement approach from scratch. Several established frameworks provide the structure.

The NIST AI Risk Management Framework provides guidelines for managing risks to improve the trustworthiness of AI systems. NIST has also recently released a preliminary draft Cyber AI Profile (NISTIR 8596) that maps AI considerations directly onto the Cybersecurity Framework 2.0, embedding AI governance into operational security infrastructure rather than treating it as a separate discipline.

ISO/IEC 42001 is an international standard specifying requirements for establishing, implementing, maintaining, and continually improving an AI management system. As an ISO 42001 Lead Auditor, I work with this framework regularly, and its strength is that it provides a certifiable standard that organizations can be audited against.

The EU AI Act is the most comprehensive regulatory framework currently in effect, requiring specific measurement and documentation for high-risk AI systems. It is not optional for organizations operating in or selling into the European market, and its requirements are driving measurement adoption globally.

These frameworks tell you what to measure and why. The challenge is translating their requirements into the specific quantitative metrics I described above, and doing so continuously rather than at a single point in time.

The Challenges That Make This Hard

If measuring AI governance were easy, every organization would already be doing it. Several factors make it genuinely difficult.

Concepts like fairness and transparency are contextually dependent. What counts as fair in a lending model may differ from what counts as fair in a hiring algorithm. There is no single universal formula, and measurement requires thoughtful interpretation alongside the numbers.

Many complex AI models, particularly large language models, are inherently difficult to explain. This makes transparency measurement challenging not because the metric is wrong but because the underlying system resists the measurement.

Standardization is still evolving. While frameworks exist, universally accepted methods for calculating specific metrics like bias are not yet settled. Different tools and approaches can produce different results for the same system.

Organizations have historically incentivized performance over responsibility. Accuracy and speed get rewarded. Governance measurement introduces a different set of priorities, and that cultural shift is often harder than the technical implementation.

And finally, data quality and lineage remain fundamental obstacles. You cannot measure governance properly if you do not understand the data your AI systems are trained on, and many organizations have complex or undocumented data flows that make this difficult.

Where This Is Heading

Every one of these challenges is real, and none of them are reasons to avoid measurement. They are reasons to invest in building the measurement infrastructure now, before regulators require it and before the gap between what your organization claims about its AI governance and what it can actually prove becomes a liability.

The organizations that solve the measurement problem first will not just be compliant. They will set the standard that others measure against. They will have the data to report to boards, the benchmarks to negotiate with partners, and the scores to prove what checklists never could.

AI governance measurement is not a nice-to-have. It is the infrastructure that makes governance real.