Two Towns and One Bridge

A true number is not the same as a safe one, and the difference is a person standing in the gap between them

Image created using AI tools

There was once a cement company that learned to build bridges a new way.

They were faster than anyone believed a bridge could be. A town that had waited a generation for a proper crossing, that had made do with a ferry and a long detour and the river taking someone every few winters, could now have a bridge in a single season. And they were not flimsy for being fast. They were strong. They lasted. The company had found something real, and the something real was the kind of thing that changes how a place lives.

Two towns on the same river signed the same contract. They took the same design and the same crews, and in the same season they each got the same bridge. And for a long time, in both towns, the story was simply a good one. The detour was gone. The ferry was retired. Children who had grown up being told not to go near the water crossed the river twice a day without thinking about it at all, which is the truest sign that a bridge is working. You stop seeing it. It becomes part of the ground.

I want to stay in that period for a moment, because it lasted years and it deserves its due. Nothing was wrong. Nothing was being hidden. The bridge was good and the towns were right to love it and there was no dark secret waiting in the concrete. If you had gone to either town in those years and asked whether the bridge was safe, people would have looked at you the way you look at someone asking whether the floor is safe. You do not think about the floor. You stand on it. That is what the floor is for.

And then, on a day that had nothing to distinguish it, in both towns, the bridge came down.

Here is the part that matters, and I want to move through it carefully, because the easy version of this story is a lie.

The easy version is that the company cut a corner, or the concrete was bad, or someone was negligent, and the collapse was a crime with a culprit. That is the story we prefer, because it comes with a villain and a lesson and the comfortable sense that if we simply find the bad man and remove him, the floor is safe again.

That is not what happened.

What happened is that the bridge did exactly what it was built to do, for exactly as long as it was built to do it, and then it reached the edge of what it could do, and physics collected. When people went looking, afterward, for the thing that had gone wrong, they found something worse than a mistake. They found that nothing had gone wrong. The bridge had been, from the first day, a very good bridge that would stand almost always. And almost always is a phrase that contains a day.

The company had known this. Of course they had known it; they were serious people and they had measured their work honestly. They could have told you, from the beginning, precisely how reliable the bridge was, and the figure was high enough that no reasonable person would have hesitated to sign. It was the kind of number that ends a conversation rather than starting one. It sounded like a yes. And so both towns, hearing it, did the most natural thing a person can do with a reassuring number. They filed it, and they exhaled, and they stopped thinking about the bridge, because that is what the number seemed to permit.

The number was true. It had always been true. And true, it turns out, is not the same word as safe. A number tells you the odds. It does not stand on the bridge. Somewhere inside that reassuring figure was a day, and the day was real, and no amount of the number being honest made the day less real. It only made it easy not to look at.

So both bridges fell. That part was the same.

But the two towns were not the same, and the difference did not show up in the concrete. It showed up in a person.

In the first town, there had been someone whose actual job was the day inside the number. Not the bridge on its good days, which needed no one. The day. He walked the structure when there was no reason to. He watched the river pull at the footings in the spring. He knew, in the particular way that only comes from standing on a thing in the cold when everyone else has gone home, that a very reliable bridge is not the same as a bridge you may stop watching, and that the whole meaning of almost always is that someone has to be there for the remainder. When the river came up wrong that year, he was the one who saw it, and he closed the bridge the night before, and in the morning it fell into an empty river and hurt no one at all.

In the second town, there was also a name. There was a signature on the contract and a line of responsibility and a person who, if you had asked, was accountable for the bridge. On paper the two towns were identical. Both had measured. Both had a name in the box. But in the second town the name was there for a different purpose. It was there so that afterward, when something happened, there would be someone to answer for it. The accountability was real, and it was the wrong kind. It faced the wrong direction. It was pointed at the aftermath instead of the river, and a bridge cannot tell the difference between a watchman and a name in a drawer until the day it needs one, and by then the difference is the only thing that matters.

Same bridge. Same true number. Same collapse. In one town a story people tell about the year the bridge fell and no one was hurt, thank God, because someone was paying attention. In the other, a stone with names on it.

Now I will do the thing these stories are for, and turn it toward the systems I actually spend my days on, and I will try to do it without pretending your situation is simpler than it is.

Every organization running artificial intelligence has been handed a number, or will be. It is a good number. The systems are, in the main, remarkably reliable, and the people who build them are serious and measure honestly, and the figure they produce is high enough to end the conversation. It sounds like a yes. And the most natural thing in the world, the thing that is not foolish or negligent but simply human, is to hear that number and exhale and stop watching the bridge.

But the number is the odds. It is not the watchman. Inside every reassuring figure about a deployed system is a day, and the day is real, and the honesty of the number does nothing to keep the day from coming. The only thing that meets the day is a person who was watching for it before it arrived, who understood that almost always is a sentence with a remainder in it, and whose job was the remainder.

And here is the quiet, uncomfortable thing, the one I cannot resolve for you and would not insult you by trying to. On paper, the two towns look the same. A name in a box looks like a name in a box. You cannot tell, by reading the org chart, whether the person accountable for your AI is a watchman or a signature. You cannot tell whether they are out on the structure in the cold, or whether they are simply the one who will answer when it comes down. Both are called accountability. They face opposite directions. And they look identical right up until the day they don’t.

So I will not ask whether you have measured your systems, because you have, or you will, and the number will be good. That was never the question. The question is the one the two towns answer differently, and answer on the same terrible day, too late to change:

When the day inside the number arrives, is there someone standing on the bridge? Or is there a name in a drawer, waiting to answer for it?

And if the honest answer is that you would rather have the watchman, then the real question is quieter still, and it is the only one worth acting on while there is still time to act. Not do we have a number. You have a number.

Is anyone actually watching the river?

I build instruments for the day inside the number. But an instrument is only ever half of it. A gauge with no one watching it is a number in a drawer, and a watchman with no gauge is a person in the cold with nothing to read. It takes both. It always took both.

Suneeta Modekurty