The Model That Changed Nothing

Around 2010, I worked on a model that scored every bridge in the National Bridge Inventory, the federal record of highway bridges on America’s public roads, on two things at once: what condition it was in, and what it would cost if it failed. A lightly used rural bridge in poor condition and a busy interstate crossing in fair condition carry very different risks. Yet most bridges were inspected on the same two-year cycle.

I presented the model at the Federal Highway Administration (FHWA) headquarters in Washington, alongside colleagues from Lloyd’s Register. In the room were the Associate Administrator for Infrastructure and several of his colleagues from the bridge side. He was supportive, but he made no decision and no commitment. Bridge leads at state transportation departments gave it the same reception. We had linked the model to Google Earth, so we could fly over each state’s highest-risk bridges. None of the state bridge engineers who saw it disagreed with what it showed. There was real interest, and then nothing.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

The inspection program did not change. Not then.

The information existed, the people closest to the problem found it credible, and it still did not turn into action. I have come to think this is not unusual. It is one of the common ways a better design loses.

This series argues that the answer to AI’s judgment problem is design: build systems so the people who bear the consequences of a decision stay part of making it. If that is right, there is an uncomfortable follow-up. Why does the design answer so often lose in practice?

The bridge model is my best evidence of how it happens, because it happened in front of me.

Two Columns

The usual reason a better design loses is cost. Doing things properly takes more money up front, and budgets are set in the present. That is often a fair objection, and it would be dishonest to pretend the design answer is always cheap.

The bridge model removed that excuse. It worked from the data states were already collecting for the inventory. Nothing new had to be gathered. Rolling it out wouldn't have been free, but it was modest compared with the size of the program.

The benefits were real, but they took a different form. Few people would argue that a simple concrete beam bridge, under twenty years old and used mostly by farm vehicles, needs the same attention as a major interstate crossing. Inspecting such bridges less often would free up effort for better inspection and targeted maintenance on the large, busy bridges where failure matters most. Those bridges are costly to maintain, but replacing one costs far more. If that attention added five or ten years to the life of a major crossing, the savings could be large. The biggest benefit of all is a failure that never happens.

That is the problem. A failure that never happens leaves no record. No one can point to the bridge that did not close, or the replacement that was pushed back a decade because a problem was caught early. The benefits go in the ledger as estimates about events that, if the design works, will never occur.

The other column, it turns out, was not money. It was the risk of change itself, and who would carry it. That is where the rest of this essay goes.

Tonie-Marie Gordon, writing about AI at work, describes the same accounting error from the other direction. The productivity gain from a new tool gets booked in full, while the human effort needed to check its output is left off the ledger. Either way, what gets counted tends to win.

The Number and the Money

At the time, the inspection data did more than describe bridges. It helped decide how much federal money each state received for them. Under the Highway Bridge Program, a state’s share of the funding was based on the estimated cost of repairing or replacing its deficient bridges, those formally rated as in poor condition or outdated for today’s traffic, calculated from the inventory.

That made any change to how bridges were inspected more than a technical question. The same data that recorded a bridge’s condition also fed the formula that divided up the money. A state’s bridge staff might find a risk model persuasive and still have good reason to be cautious. If the new approach changed what was collected, or how it was read, it might change what the state received. Nobody would want to be the one who found out the hard way. This is not a claim that anyone gamed the numbers. The caution was reasonable, given how the money worked.

A slow federal process added to the caution. Changes to a national program move through years of rules, comments, and consultation, and a program that seems to be working invites the natural response: if it isn’t broken, don’t fix it. The trouble is that “working” meant the inspections got done and the numbers came in. It did not mean the effort was going where it did the most good.

Once a number decides where money goes, protecting that number can become a goal in itself. The Congressional Research Service noticed a version of this in 2008, observing that a formula paying states according to their deficient bridges gave them little reason to reduce the count.

Through her reporting, Rana Foroohar has traced the same pattern across the wider economy. When firms were judged on financial figures, managers came to steer by the figures rather than by the work they measured. The figures were persuasive, and too few people asked what they left out.

In this case, the better design did not lose on cost. I cannot prove why it lost. I believe it lost at least in part because it threatened the numbers the money depended on.

Not Just Bridges

The bridge model is the clearest case I have, but I have seen versions of it in technology consulting.

One project stays with me. I worked with a large software company on how its sales system handled leads: how a first contact became a qualified lead, and then a real prospect. Marketing was judged on how many leads it passed along, and sales on how many it turned into customers. Sales believed marketing was passing along weak leads to boost its numbers. Those leads rarely went anywhere, so the sales team wasted time chasing them and looked worse when they failed to convert.

Nearly everyone recognized the problem when we laid it out. Fixing it meant changing how both teams were measured, at the same time. Our engagement ended with recommendations to do that, and an expectation of follow-on work to implement them. That work never came. As far as I know, the company kept things as they were.

The rule there was not a federal regulation but a pair of targets. The pattern was the same: a better arrangement everyone could see, and no one in a position to change the rule for both sides at once.

The Asset Nobody Counts

One more thing is missing from both columns, and it matters most for AI.

Most organizations depend on judgment that lives in their people rather than their procedures. It shows up in the inspector who knows which joint on which bridge to look at twice, or the sales manager who can tell a real lead from a hopeful one within a few minutes on the phone. Little of it appears in a budget. You learn it through practice, pass it on informally, and notice it mainly when it is gone. I think of it as cognitive collateral: the capacity an organization relies on to catch its own mistakes, which never shows up on its balance sheet. If offloading judgment is a kind of borrowing, cognitive debt is what accrues, and cognitive collateral is what is lost when the debt comes due and cannot be paid.

AI tools put this capacity under pressure in a particular way. When a system does the routine work, people get less practice at it. In 1983, the engineering psychologist Lisanne Bainbridge pointed out the irony: the skills that fade during smooth automated operation are the ones needed when the automation fails.

A design that keeps people exercising their judgment, rather than only approving what a system produces, protects that capacity. But the protection is another benefit that never appears in any column. The simpler design's efficiency gain is immediate and measurable. The judgment it quietly spends is neither, until the day it is needed and turns out to be thinner than anyone knew.

That is the design answer’s problem in its most general form. What it protects is invisible, and what it asks people to change is not.

Asking to Be Ruled

This month, the heads of several of the largest AI labs renewed a call that industry leaders have made, on and off, for years: they asked for rules. At the United Nations Security Council, several told members that AI needed global oversight. Anthropic’s Dario Amodei and OpenAI’s Sam Altman have both said development should slow down, though Amodei has argued the leading labs should slow down only as far as they can while still leading China. Earlier this month, the White House rejected calls for federal guardrails.

There are two serious ways to read the request.

The first follows this essay's logic. A lab that slows down on its own bears the whole cost: it falls behind competitors who did not. The benefit, a failure that never happens, is shared by everyone and credited to no one. That is the bridge problem again. A state could not change the inspection schedule on its own; the change had to come from the body that set it for everyone. It eventually did, but only after Congress required it.

The second reading is less generous, and just as structural. Rules a large company can absorb easily can be heavy for a small one, and the biggest labs are doing the asking. Vice President JD Vance has called the push a possible Trojan horse. This is the lesson of the bridge funding formula in another setting: a rule that decides what counts also decides who gets ahead, and those shaping it have interests of their own.

The two readings are not mutually exclusive. A company can sincerely want rules and still benefit from the ones it helps write. Either way, the questions from an earlier essay in this series apply: who sets the threshold, who pays when it is wrong, and does it ever switch off?

The same rule could protect the public or protect the people already in front.

A Decade Later

The story has a coda I only learned while writing this. In 2012, Congress rewrote the federal highway program. The funding formula tied to deficient bridges was retired, and the Secretary of Transportation was told to consider a risk-based approach to how often bridges are inspected. In 2022, the national standards were updated. States can now inspect low-risk bridges less often and high-risk bridges more often.

I cannot say whether our work played any part. The firm I worked for, Cambridge Systematics, had for years maintained the bridge management software many state transportation departments used, and the risk model grew out of that experience. My colleague Dr. Gurenich built its computational engine. As far as I know, it was among the first national risk models built on the inventory data, but others were thinking along the same lines.

What is worth noticing is what it took. The merits were visible years earlier. What finally changed was the rule, for everyone at once, and a funding formula that no longer depended on the old numbers.

That is a good outcome. But it took more than a decade, and the official accounting never recorded what those years cost.

I will be plain about where I stand. I think we are repeating the bridge timeline with a technology that changes much faster than concrete. The bridges could afford twelve years. I do not think our judgment can. I may be wrong about the pace, and I would rather be early and wrong than right and late. But if the design answer waits for the rule again, the cost of waiting will once more go unrecorded, and we will learn what it was only when we need what we lost.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

I turned on my own seatbelt sign about four years ago.

I had been reading the same headlines as everyone else, and I reached a conclusion I found hard to argue with: too many things could go wrong at once, and most of them would hit the markets before they hit anywhere else. So I moved a good part of my retirement savings somewhere I judged safer. It wasn’t a trade. It was a posture. I buckled in and waited.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

Nothing happened.

Four years on, the sign is still lit. Recently I made a list of the warnings that keep it lit, and introduced them with the phrase “only a couple of blinking warnings that I’m aware of.” Here is the list.

* Government debt that nobody has a plan to repay ($40T)

* A bond market that hasn’t settled

* Share prices that assume nothing goes wrong

* A technology boom that is partly financing itself (AI)

* Whatever that boom does to jobs

* A war with no sign of ending (Ukraine)

* A standoff that could become another (reader’s choice - US/Iran, China/Taiwan, North Korea/South Korea)

Then I counted. There were seven.

That gap between “a couple” and seven is what this essay is about. It isn’t evidence that things are worse than I thought. It might be evidence that I’ve stopped being able to read my own instruments.

Here is what I’ve done since. I’ve quietly moved some of that money back. Not all of it. Enough to notice.

I don’t know whether that was judgment or fatigue.

The Sign

On a plane, the seatbelt sign works because of three things you rarely think about.

The first is that someone with better information than you switches it on. The captain can see the weather radar and hear reports from aircraft further ahead. You can see a cloud.

The second is that the captain is on the plane. If the turbulence turns out worse than expected, the captain feels it too. That doesn’t make the captain infallible. It means the person switching the sign on pays when they get it wrong, which is why you can obey it without being told the reason.

The third is the one this essay depends on: the sign goes off. On a smooth flight it stays dark most of the time, so when the chime sounds, it means something. The sign carries information because it isn’t always on.

Take away the first two and the sign becomes harder to trust. Take away the third, and it becomes part of the cabin.

The Last Time the Sign Was On

First, what this essay isn’t claiming. The twentieth century was the bumpy one: two world wars, a depression, a pandemic that killed tens of millions, and four decades in which two governments could have ended most life on earth in an afternoon. If a century ever had its seatbelt sign on, that one did.

Many of us in the West who came of age around the end of the Cold War believed it had gone off. Since the autumn of 2001 it has been on for most of the time: two long wars, a financial crisis that came close to freezing the banking system, a pandemic that closed much of the world, inflation back after a generation away, war in Europe and the Middle East, and political turmoil in democracies long thought stable.

That is a selective list. The same years brought falling extreme poverty and longer lives, and I’m not claiming the world got worse. I’m describing how the sign has behaved. What stands out is how short the quiet stretches have been.

When the Alarm Never Stops

Hospitals have studied what happens when a warning never turns off, because there the cost is counted in patients.

A patient on a modern ward can be connected to several monitors at once, tracking heart rhythm, oxygen, and blood pressure, each set to sound when a number crosses a line. Each alarm makes sense on its own. Together they can produce several hundred alarm signals per patient per day. The Joint Commission, the body that accredits most American hospitals, reviewed the research and found that between 85 and 99 percent of those alarms required no action.

Nurses adapt, as anyone would. They learn which beeps usually mean a loose sensor or a patient who shifted in bed. They turn the volume down, widen the limits, and sometimes switch alarms off. The field calls this alarm fatigue.

In 2013, the Commission issued a formal warning to hospitals about it. Over about three and a half years, it had received reports of 98 alarm-related events, 80 of which ended in a patient’s death. It noted that reporting is voluntary and that these cases are only a small share of what actually happens.

What matters here is how the failure happens. It is tempting to call it carelessness. It isn’t. A nurse who stops reacting is usually responding sensibly to what the alarms have taught them. When nine alarms in ten mean nothing, treating the next one as probably nothing is a reasonable bet. The trouble is that habit places the bet, not judgment, and the tenth alarm sounds exactly like the other nine.

The alarm didn’t stop working. It stopped carrying information. The sound stayed the same, but its meaning drained away.

Maybe the Air Is Rough

There is a fair objection here. Hospital alarms can be scored: afterward, we know which ones needed a nurse. The warnings on my list can’t be scored that way. Some take decades to resolve, and some never will. By a warning I mean a claim that something bad has become more likely and that you should act differently because of it. This difference has a name. Risk is what can be measured; uncertainty is what can’t. Hospital alarms are risk. Most of my list is uncertainty. Economists argue over some of it with better tools than mine, but nobody can yet say which of my seven are right.

So maybe the air really is rough, and the sign is on because it should be.

The argument doesn’t depend on ruling that out. Even if the sign is right, a sign that is always on can’t tell you when things get worse. Pilots already deal with this. When turbulence is expected to be severe, the captain adds a second announcement, “flight attendants, please take your seats,” because the seatbelt sign alone no longer marks the difference. In steady rough air, the useful signal is the one that separates this bump from the others.

That second announcement is what I think we’re missing: a way to tell which of seven lit signs deserves more attention than the rest.

Unbuckled or White-Knuckled

Without that second signal, people tend to go one of two ways.

The first is to unbuckle. After enough warnings that go nowhere, you stop listening. Headlines about debt, war, or the next crisis become background, like the hum of the engines. This feels like calm, and sometimes it is. But it is the nurse’s bet: the next warning is probably nothing. Most of the time that’s a reasonable bet, and it fails precisely when it matters.

The second is to stay white-knuckled. You keep the belt tight and never relax, treating every warning as the real one. This feels like prudence. But vigilance that never stands down has its own costs. It is tiring, it crowds out other things, and it has its own blind spot. If you treat everything as an emergency, you can’t pick out the one that is.

The relaxed passenger and the white-knuckled one look like opposites, but the sign has stopped telling either of them anything. One ignores it; the other obeys it constantly. Neither is reading it.

Both are settings, not decisions. Each replaces the question “does this warning mean something?” with a standing answer: no for one, yes for the other. Judgment is the ability to give different answers to different warnings. A sign that never goes off makes that very hard to do.

What I Can’t Tell

So what does another twenty or thirty years of this produce? I’ve kept hoping things would settle down. It’s a hunch rather than a forecast, but I no longer expect the sign to go off in my lifetime.

I don’t think the answer is collapse. Societies have absorbed worse than anything on my list. The more likely result is quieter. Some institutions warn too often; others warn too little. After enough years of a sign that never goes off, a public finds it harder to tell those apart.

Which brings me back to the money.

The simplest explanation for what I did is that I made a bad call four years ago and am slowly correcting it. That may be all it is. When I moved some of the money back, I told myself it was recalibration: four years is a long time for a warning to go unconfirmed, and a sensible person updates. But that is also exactly what the nurse tells herself about the tenth alarm. From inside, recalibration and fatigue lead to the same action and bring the same relief. I can’t tell which one I’m doing.

What I do know is that I’m not a passenger watching someone else’s sign. If I was right four years ago and have now relaxed, I’ll pay for it. If I was wrong all along, I’ve already paid in the growth I didn’t get. Either way, the cost lands on me, which puts me on the plane. In July I wrote about the difference between a ledger and a scar. A ledger records what happened. A scar changes the one it happened to. Whatever this decision turns out to have cost, it won’t just be written down somewhere; I’ll carry it. I can’t tell yet whether the scar will come from relaxing too soon or holding on too long.

Who Sets the Threshold

The answer on offer for a public that can’t sort its warnings is to let something else do the sorting. News feeds already decide which stories reach you and how urgently, and hospitals are trying smarter alarms. Some of this will help.

But a system that sorts warnings is a system that sets thresholds. It decides what gets through, what gets flagged as urgent, and what stays quiet. Three questions matter about any threshold. Who sets it? Who pays when it’s wrong? And does it ever switch the sign off?

On the plane, the answers are simple. The captain sets it, the captain is in the turbulence too, and the sign goes dark when the air is smooth. Most of the warnings on my list have no captain at all, only other passengers with different readings of the same instruments. That is part of why the sorting is drifting to software. With the systems now doing more of our sorting, the answers are harder to find. People we never see set the threshold, often tuning it at least partly to keep us watching. The model applying it bears none of the consequence of what it waved through, or of the years someone spent braced for something it flagged that never came. It can keep a perfect ledger of both. It carries no scar. And a system rewarded for attention has little reason ever to switch the sign off.

Some of the people building these systems have warned publicly about their own industry and asked for regulation. They are on the plane, and their warning is worth more for it. But the rule still holds: a sign means something only if it can also go off. The question to ask of any warning, theirs included, is what would make them switch it off, and whether anyone would notice if they did.

The captain is on the plane. The model isn’t. And for twenty-five years now, the passengers have been left to read the sign themselves.

Four years in, mine is still lit, and some of the money has come back.

I don’t know whether that was judgment or fatigue.

Source: The Joint Commission, Sentinel Event Alert Issue 50, “Medical device alarm safety in hospitals,” April 8, 2013.

Tags: Architecture & Attention, AI, Epistemic Humility, Systems Thinking, Decision Making, Attention Economy

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

I have told this story before. Six weeks ago I told it to make a point about software. I want to tell it again to make a point about myself, because I think I put the emphasis in the wrong place.

In 1993, I was a final-year civil engineering student at Imperial College, and my design project was an exhibition hall with a column-free interior and a long-span roof carried on a pair of massive tubular arch beams. I modeled it in I-DEAS, a finite element package running on a Silicon Graphics workstation that took up half a desk. The package computed the stresses, checked them against code, and rendered a deformed shape that looked entirely reasonable. It said the structure would stand.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

The night before the report was due, for reasons I still cannot reconstruct, I did a hand calculation of the stress at one of the arch supports.

It came back off by a factor of ten.

I had entered the loads in kilograms, but the package read them as newtons. My model was carrying about a tenth of what the real building would have to carry. When I corrected the units and re-ran it, the deformed shape came back flat as a pancake. The structure I had designed would not have stood up.

Last time, I said the software never pushed back; the analysis was fluent all the way down. The correction had to come from outside it. That is true, and it is the smaller half of the story.

The larger half is the napkin.

I had a way to check. It was crude; it took three minutes, and it was decisive. And I had it for a reason that had nothing to do with being careful. I had been taught to size a beam by hand before I was ever allowed near a package that would size it for me. The check wasn’t diligence. It was residue.

The Word We Only Half Unpacked

Five weeks ago I wrote about the word reward, and what happened to it when AI borrowed it from neuroscience.

In the brain, reward is not one thing. Kent Berridge’s work separates it into two systems that can be pulled apart experimentally. Wanting is driven by mesolimbic dopamine; it makes you pursue the thing. Liking is a distinct hedonic system running on different neural substrates, and it registers how the thing actually landed. Disable one and the other carries on without it. An animal can be made to want, intensely, something it does not like at all.

Machine learning borrowed the word and built the pursuit.

I want to state this more carefully than I did last time, because it is easy to overclaim. Reinforcement learning gives a system an explicit optimization signal and a great deal of patience in following it. What it does not supply, by itself, is anything corresponding to the second system, an independent register of how the result actually landed, sitting outside the objective and capable of disagreeing with it. There are constraints, regularizers, safety layers, human feedback, evaluators. What’s missing is something that stands apart from the optimization and asks whether the objective was the right one.

Drive without a brake, with the caveat that this is a metaphor, and a metaphor is not a schematic. The precise version is duller and harder to knock down: a system optimized to produce an answer is not thereby equipped to establish that the objective, or its reading of the objective, was correct.

I left it there, as a fact about how these systems are built. It is a fact about how they are built.

But nothing is dangerous all by itself. An uncovered hole in the ground is not dangerous sitting in an empty field. It becomes dangerous when someone walks near it. The question I skipped last time is who is walking near this one, and how close.

Condition Is Not Risk

Engineers separate two things that ordinary language runs together.

Years after the exhibition hall, I spent part of my career in transportation consulting. One project, with colleagues at Cambridge Systematics and Lloyd's Register, built a risk model for the bridges on the US highway system. Every state reports the condition of its bridges to a national inventory, and we used that data to place each structure on a five-by-five grid.

One axis was condition: how structurally sound the bridge actually is. The other was consequence: what happens if it fails, largely how much traffic crosses it, adjusted for how far the detour would be if it closed.

Those two axes aren't the same thing, and keeping them apart is the point. A badly deteriorated bridge on a farm track is in poor condition and carries almost no consequence. A sound bridge carrying a hundred thousand vehicles a day is in fine condition and carries enormous consequence. Neither is where you look first. You look at the corner where both are bad at once: a deteriorating structure carrying heavy traffic with no easy detour. Risk is what you get when condition and consequence coincide.

The structure's condition is a fact about the structure. It is true whether anyone drives across it or not. The risk exists only because people do.

That is the distinction I skipped past five weeks ago. An optimizer without an independent evaluator is a fact about how the thing is built. It is a condition. It is true of the system sitting on its own, doing nothing at all.

What turns a condition into a risk is traffic.

Two Ways To Hand Something Over

So: who is driving over this bridge, and how heavily?

Not “who uses AI.” Everyone does, and that tells you nothing. The question is narrower. What happens to your own ability to check the answer?

Handing work to something else is not one thing. It comes in two forms that look identical from outside and behave very differently when something goes wrong.

The first is delegation. You give away the task and keep the ability to judge what comes back. A managing editor does not write the article, but can read it and say it isn’t working. I gave the stress calculations to I-DEAS and kept, without ever thinking about it, a way to test the answer. The work moved. The judgment stayed.

The second is offloading. You give away the task, and the ability to judge the result goes with it. What comes back is a conclusion, arriving without the reasoning that produced it, and there is nothing left on your side that can weigh it. You can accept it or refuse it on instinct. You cannot check it.

From outside, these look the same. A person asks, a system answers, the person proceeds. The difference is invisible until the answer is wrong.

The obvious objection is that delegation is just offloading on a longer timescale, that if you never do the work yourself, the ability to judge it must eventually go.

That objection is correct, and it is the most important thing in this essay. Delegation is not a stable condition. It decays into offloading unless something actively holds it in place. This raises the question of what could hold it there, and whether anything does.

Bainbridge, 1983

The obvious answer is that this is a matter of discipline. Keep your hand in. Check the work.

Lisanne Bainbridge answered that in 1983, in a paper on industrial process control written long before any of this existed.

Her observation came to be called the ironies of automation. When you automate a process, you leave a human to monitor it and take over if it fails. But monitoring something that runs smoothly is exactly the condition under which manual skill decays. The operator does less, for longer, and the ability to intervene erodes quietly while nothing appears to be wrong.

Then the automation fails, which is the moment it needs a person, and the person it gets is the one whose skills have been eroding for years, worn down by the very smoothness that made everything look fine.

The skill you need at the moment of failure is precisely the skill that the absence of failure has been taking away.

So the decay isn't hypothetical, and vigilance isn't the answer. Which leaves the question of what is.

What The Napkin Actually Was

Here is where I have to be careful, because the obvious reading of my own story is wrong.

The obvious reading is: hold on to the old skill. Learn to do it by hand so you can always do it by hand. And that cannot be the lesson, because it does not survive contact with how anything works. I cannot fabricate a semiconductor, sequence a genome, or derive the cryptography my bank runs on. I use all three every day, and I should. If the answer to automation were personally retaining the underlying method, the answer would be useless.

Look again at what I actually had that night.

It was not the ability to reproduce what I-DEAS did. The package solved a system of equations across thousands of elements. I could not have done that by hand in a month, and nothing in my training suggested I should try. What I had was cruder and far more valuable: a way to arrive at an approximate answer by a route that did not pass through the package.

That is the whole thing. Not reproduction — verification by an independent route.

Which gives the test:

Is there a way to check the output that does not run through the thing that produced it?

Sometimes that route is retained expertise, as mine was. But it can be many other things. An empirical test the world adjudicates. A formal constraint the output must satisfy. A second system built on genuinely different principles rather than a near-copy of the first. An audit trail that lets someone reconstruct how a conclusion was reached. A deliberately adversarial process staffed by people whose interests differ.

The napkin is one instance of the category. It is not the category.

And that changes what is at stake. The scarce resource is not craftsmanship, and this is not an argument about the dignity of doing things by hand. What is scarce is independent routes, and unlike a bridge, a route can disappear without anyone noticing.

There is a collective version of this, and it is worse.

My napkin worked because it was a different method. Different assumptions, a different path to the answer, and this is the part that matters: different ways of being wrong. Independence is not a property of having a second opinion. It is a property of the second opinion failing differently from the first.

Which means a field can hold on to checking as a habit and still lose it as a safeguard. If everyone verifies with the same tools, calibrated against the same standards, trained on the same material, then a thousand checks are one check performed a thousand times. The errors line up. And nobody notices, because everybody is still checking.

A Case Where The Route Has Already Closed

That risks staying abstract, so here is a live one.

A student submits an essay. A detector reports that it was machine-written. The student is now the subject of a judgment with real consequences: a failed assignment, an academic misconduct process, occasionally the end of a degree.

What check is available?

The student can deny it, which is an assertion, not evidence. Or the work can be run through a second detector: a system of the same class, trained on broadly the same signals, sharing the same blind spots and making correlated errors. That is not an independent route. That is the first opinion in a different font.

Sam Illingworth reached the image before I did, writing about Substack’s own detector: an AI adjudicating whether text is AI. A turtle standing on a turtle.

Look at where this sits on the grid. The condition is the familiar one, a system producing a confident verdict with nothing in it that stands outside the verdict and disagrees. The consequence is high and lands on a particular person. And the independent route is not merely weak; for the accused student it is very nearly absent, because every available check fails in the same direction as the accusation.

Then the third thing, which is the worst of it. Nobody assembles the failures. A wrongly accused student who cannot prove a negative mostly disappears from the record; the case is dropped, or it isn’t, and either way it does not become a data point. There is no inquiry, no docket, no count. The bridge falls down, and nobody hears it.

This is not a warning about something that might happen. It is running now, at scale, in institutions that adopted it faster than they built any way to evaluate it.

Where The Traffic Is Heaviest

Now scale this up, carefully, because the obvious version of this argument is wrong.

Every generation loses skills to its tools. My grandfather could do long division faster than I ever could. Nobody navigates by the stars. Slide rules are museum pieces. Those losses were real, and almost none of them mattered.

So the claim cannot be that skills are being lost. Skills are always being lost. The claim has to be narrower: what matters is whether an independent route survives the loss.

When calculators replaced slide rules, one usually did. Order-of-magnitude estimation is cheap, teachable, and doesn't run through the calculator, which is why someone who kept it can still catch a wrong answer. Not everyone did, and that is precisely the point: the route existed, and its survival depended on whether anyone bothered to maintain it. What did not happen is the route becoming unavailable in principle.

That is the real question about any tool. Not what it does for you, but what it leaves you.

Now take the work where the output is a judgment rather than a quantity. Whether this candidate should be hired. Whether these symptoms fit together. Whether this clause creates an exposure. Whether this strategy is sound.

It would be too strong to say these fields have no external check. Medicine has pathology, imaging, and outcomes. Law has evidence, precedent, and procedure. Hiring has job performance. Strategy has results. What they lack is my beam, a check that is fast, cheap, and mechanically decisive. Their checks are slow, noisy, expensive, contestable, and often arrive years after the decision they would have corrected. Structural engineering sits at one end of that spectrum, with physics as an unusually brutal validator. These fields sit at the other.

A reader will say: those fields never had a napkin, so nothing is being lost there. You have diagnosed a problem in engineering and applied it by analogy to places it does not belong.

That is right, and it is the more uncomfortable version of the argument. They were always at the far end of the spectrum, running on checks that were slow and arguable. They lived with it because the volume of judgment passing through them was bounded by how many trained people could be put in the room. What has changed is not the check. It is the traffic.

Put those fields on the bridge grid, and the relevant condition is not identical across them: a diagnostic system, a hiring model, and a drafting assistant are not the same architecture, but it is analogous, and it is analogous in the way that matters: the recipient of the output frequently has no route to evaluate it that does not run back through a system of the same kind.

The bridge model had it easier in one respect, and it is worth saying so. A bridge that fails announces itself. Nobody has to detect the collapse. A hiring decision, a missed diagnosis, a badly drafted clause fail quietly and can keep failing for years without anyone assembling the fact that they failed at all.

That is not a third axis I want to add to the grid. It is a reason the grid flatters the situation.

And here I will make a prediction rather than a statement of fact, and mark it as one, so it can be held against me later. I think the independent routes in these fields will thin faster than they are replaced, because keeping one costs time and money now and pays only in failures that never happen. That is the hardest kind of expenditure to defend and the first kind to be cut.

The Question That Decides It

Which turns the usual question inside out.

The question almost always asked about these systems is how good they are. Can it pass the exam, write the brief, read the scan? That question has an answer, and the answer improves every few months. It is a fair question. But it is a question about condition, and condition is not risk.

The question that decides the outcome is about the arrangement we put the system into. When the work comes back, is there any route to judging it that does not run through the thing that produced it?

That is a design question, and design questions have answers. That doesn't mean hoping people stay vigilant; Bainbridge disposed of that in 1983. Tonie Marie Gordon, who works on cognitive ergonomics, puts the same point in its sharpest form: “The presence of a human in the loop is not oversight. The design of that human’s cognitive role is what determines whether the oversight is meaningful or decorative.” A person positioned where they cannot check anything is not a safeguard. They are a diagram.

That is what I mean by augmented human intelligence, and it is duller than the phrase suggests: designing the arrangement so that human judgment retains an independent route to the answer, rather than being placed downstream of a system it can only accept or refuse.

The obvious objection is economic, and I cannot dispose of it. Maintaining an independent route costs time and money, and an organization that pays for one competes against organizations that don’t. Market pressure runs steadily against every recommendation in this essay. Building codes exist because that argument was lost on the merits and had to be won by regulation instead, in exactly the field I started in, and only after enough structures had fallen down to make the cost visible. I do not know what the equivalent looks like here, or what it will take.

And there is a harder version of the objection that I am not sure I can answer at all.

Bainbridge called the erosion an irony, which implies that nobody wanted it. In some arrangements it is not an irony. A workforce that cannot check the system is cheaper, more interchangeable and easier to replace than one that can. Where that is true, the missing route is not an unfortunate side effect of efficiency. It is the efficiency. And no amount of thoughtful design will supply a check that somebody is deliberately economizing on.

I would like to end by saying I caught that error because I was careful. I wasn't. I was a student the night before a deadline, doing an arithmetic exercise for no reason I could ever reconstruct.

I had three minutes and a route that didn't go through the package.

It would have let me submit. It couldn't tell me anything was wrong, didn't try, and would have gone on, fluent and confident, about a building that would have fallen down. Not because it was badly built, because nothing in a system that produces an answer is positioned to tell you the answer is wrong. That has to come from somewhere else.

Whether anything is still standing in that somewhere else is the only question that matters.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

A scientist finishes a lecture on the solar system. An elderly woman in the audience stands up and tells him the whole account is wrong: the world rests on the back of a giant turtle. The scientist, politely, asks what the turtle is standing on. She has the answer ready. It is turtles all the way down.

The story is usually told about Bertrand Russell. It is also told about William James, and about various astronomers whose names change with the telling. Each version reaches us through someone who did not say where they got it. Nobody has established which lecture hall, or which century, or whether it happened at all. The provenance of the story about infinite regress has no bottom either.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

The Number I Was Holding Up

For several years I worked for a consulting firm that ran a Delivery Excellence dashboard. Every project reported weekly against a set of components: scope, schedule, budget, client satisfaction, team performance, and one or two others I no longer remember. Each one got a color. I entered the scores myself.

On a program for a biotech client, I reported client satisfaction as green, week after week, and the score was accurate. The client contact was satisfied. Every concern he raised got addressed in the order it arrived. He was told early that the scope began to shrink because we could not deliver all of it, and hearing it early is most of what makes a shrinking scope tolerable.

Underneath that, a team was struggling on several counts, and had been almost from the start.

So the numbers were true. Scope was green because we had already cut scope. Satisfaction was green because I spent a substantial part of every week producing it. Every component reported a state, and none had a field for what that state cost to maintain. The effort was the risk, and the effort was the one thing the format could not carry.

I did not falsify anything. I reported what the components asked for, and I raised the delivery problems internally, every week, in the space the form left for them. The technical lead and the others read the situation as managed, and I did not want to go out on a limb by myself against my own team’s assessment. I also thought that if we all knew, we would fix it before it came due. So a room full of people who each knew something was wrong went on filling in a form that had nowhere to put it. I have watched a version of this happen more than once, and never once because anyone was hiding anything.

What eventually surfaced the problem was not the dashboard. In January, I sat down with the office general manager and argued that we should put three hundred and fifty thousand dollars of our own money into a three-and-a-half-million-dollar job to fix what we had not delivered. The technical lead and two others backed the position. The technical lead told him the only reason we were still in the room was the account management. He agreed to the money. We spent it carefully, the work landed, and the client was pleased in the end.

The dashboard recorded none of this. It had been green throughout, and it was green because of what I was doing underneath it.

The Tower We Are Building Now

The dashboard failed in a specific way, and the way matters more than the failure. It published an accurate number and gave nobody access to what produced it. Whoever read the green had to take on trust that the process behind it was sound, and the process behind it was me. Reported figures come loose from the work that generated them. That is not a property of consulting. It is a property of reporting.

A model’s output is evaluated by another model, because human evaluation does not scale. The evaluating model was trained on data that a model filtered for quality. Both are scored against a benchmark, and benchmark content has repeatedly been found in the training data of the systems being scored. The results are then summarized for the people who have to decide something, increasingly by a model, because the volume is large and the summary is fast.

Every one of those steps is defensible on its own. Each is documented, published, and argued for by people who know more about it than I do. None of them is hiding anything.

What is missing is not honesty. It is anyone whose job is to ask what the bottom layer is standing on. Each layer treats the one below it as ground, and the question of grounding gets handed down the stack until it runs out.

In Defense of Turtles

I want to concede this properly rather than in passing, because the concession is most of the argument.

Abstraction is how anything large gets built. Nobody writing an application re-derives the behavior of a transistor. Proxies are how you manage what you cannot measure directly, and almost everything worth managing cannot be measured directly. I have never inspected a bridge before driving over it, and I am not going to start.

Stacking is not the sin. A working tower of abstractions is one of the more impressive things people do. The engineer who trusts the steel certificate is not being lazy. That certificate is doing real work, and behind it there is a mill, a test, a standard, and a name.

The failure is narrower than the stacking. It is losing track of whether there is a bottom at all. Deferring the question of grounding to the layer below is fine, and unavoidable, as long as the deferral terminates somewhere. The tower becomes a problem only when every layer in it has been built on the assumption that somebody further down already checked.

Where It Stops

The regress has to stop somewhere, and the only place I have ever seen it stop is at someone who pays if it is wrong and cannot hand the paying to anyone else.

I want to be careful about how much that carries. It is not a guarantee of anything. People with their names on things sign off on work they have not understood. Systems with clear lines of liability fail badly and often, and the person carrying the cost is frequently the last to see it coming. The engineer who signs the beam did not derive the metallurgy, watch the testing, or stand in the mill. Finding the name does not tell you the answer is right, and it does not mean the person can check the work. It tells you the chain has a bottom instead of continuing indefinitely. The claim is that narrow: without someone in that position, there is no bottom to have.

I have spent most of this series on that claim, and I am not going to re-argue it here. The engineer who signs off on the beam carries it if the beam fails. The certificate is not the ground, and the signature is not the ground. The ground is that a person’s license, career, and name are attached to the load, and that the attachment cannot be undone by delegating.

On the biotech program, the ground floor was not the dashboard. It was a room in January, a general manager who had to spend money he would answer for, and an account lead recommending it who would answer for it too. Neither of those costs could be passed sideways. That is why the problem got fixed. It is also why it took until January.

What You Can Use

The woman in the story was wrong about cosmology and right about almost everything else.

So here is the test. Pick any validation you currently rely on. A green status, a certification, a score, a rating, a benchmark result, a maturity level. Ask what it is standing on. Then ask the same question of the answer.

Keep going until the answer stops being a process and starts being a name.

If you work down through the whole stack and there is nobody there, you are standing on a turtle. It may well hold. A great many of them do, for years at a time. But you should at least know that is what you are standing on.

I stood on one for most of a year, and I entered the number myself every week.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

There is a piece of work your body performs every waking second, and you will almost certainly never think about it, unless one day it stops.

I injured my lumbar spine. What I learned afterward was not really about pain, which is the part everyone expects and the part I will spare you. It was about administration. Somewhere well below anything I could call thinking, my brain had been issuing a continuous stream of instructions to the small stabilizing muscles around my spine, adjusting them constantly, correcting for every shift of weight and every turn of my head, every hour I had been alive. Standing still turns out not to be still. It is a controlled fall, caught and re-caught, in every direction at once.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

I never chose that work. I never noticed it. I became aware of it only when it started arriving late and imprecise, and the ordinary business of getting up from a chair became something I had to attend to deliberately, the way you attend to a difficult sentence.

Here is the part that took me longer to see. The work was not executing a standing instruction. Nothing in me was running an order that said stay upright. At every instant something was deciding which of an enormous number of signals mattered right now, and therefore what the problem currently was: a floor that turned out to be wet, a bag heavier on one side than expected, a step that arrived sooner than the last one. The instruction never changed. The problem changed constantly, and something reformulated it, continuously, without asking me.

Nobody set that problem. Something I have never met noticed it and re-noticed it thousands of times an hour.

The Ledger We Keep Of Difficulty

Hans Moravec noticed the consequence of this in 1988, in a book called Mind Children, and the observation now carries his name. It is comparatively easy, he wrote, to make computers perform at adult level on intelligence tests or at checkers, and difficult or impossible to give them the skills of a one-year-old at perception and mobility.

The paradox is normally presented as a fact about machines. I want to present it as a fact about us.

We rank difficulty by how hard a thing feels while we do it. Chess feels like intellect. It is effortful, available to introspection, and we can watch ourselves doing it, so it sits at the top of the ledger. Catching a set of keys tossed across a room feels like nothing at all, so it sits nowhere. The ranking is not a judgment about the problems. It is a report on which problems we happen to be conscious of solving.

Let me be careful about how much weight that carries. I am not claiming effort measures unfinished learning. Expert musicians work hard at passages they have played for decades, vigilance is exhausting after thirty years of practice, and creative work stays effortful precisely because there is no stable routine to settle into. The claim is narrower: felt effort is an unreliable proxy for computational difficulty, and we have been using it as our main one.

An objection deserves to arrive here rather than later. The computer scientist Arvind Narayanan has argued that Moravec’s paradox is not a dependable predictor of what will prove easy or hard for machines, and is better read as a statement about what the field has found worth working on. I think that is largely right, and I am not going to use the paradox as a forecast. Whatever it does or does not predict, it correctly identifies a bias in our accounting.

That informal ledger has a formal descendant, and it now runs the industry. We call them benchmarks. A benchmark is our difficulty ledger written down and made scoreable, and it inherits the same property: it can only contain what somebody thought to put in it.

The Objective Was Never the Hard Part

Let me concede the thing that arguments of this kind usually avoid.

Football has an exceptionally clean objective. Score more goals than the opposition inside the allotted time. You can write it on a napkin. There is no ambiguity about success, and a scoreboard settles it. If difficulty lived in the difficulty of stating an objective, football would be among the simplest things humans do.

So the difficulty is not in the objective. It is in the distance between having one and knowing what to do next.

Watch what that distance contains. Twenty-two people are running the stabilization work I described at the start of this essay, continuously, while sprinting, turning, being shoved and landing badly. Air moves across the stadium in ways nobody has modeled. The surface changes through the match. The light shifts. The altitude sets what their bodies can spend. And each player is reading the intentions of everyone else on the field while being read in return. A pass is a prediction about where a teammate will be in two seconds, made by a body in mid-fall, against defenders making their own predictions about that prediction, who will change what they do the moment they detect it.

I want to be precise about what the players are doing, because there is a mystical version of this claim and I do not want it. They are solving the problem. Obviously they are; the ball arrives. What they cannot do is say how. The computation is distributed through the body, learned over years, and closed to introspection, which is different from absent.

But the deepest part is not the policy either. It is that the problem does not hold still.

Somewhere around the hour mark, a match becomes a different match. The referee has been letting contact go, so the cost of a challenge has changed. Someone is carrying a knock and is now a target. The other side has gone to three at the back, and every space on the pitch means something new. Nobody announces any of this. No player is told the problem has changed. Each one works it out while still playing, revises what matters, and acts on the revision before being able to articulate it.

It is worth being exact about what changed, because adapting to a moving situation is something good control systems already do well, and I do not want to dress that up as something rarer than it is. What changed was not the difficulty. It was which information mattered to success. A defender’s body position meant one thing at twenty minutes and means something else now, and nothing in the original instruction said so, or could have.

That is the line I want to hold throughout this essay. Adjusting your behavior because conditions moved is one thing. Discovering that a different set of facts has become the relevant ones is another, and it is the second that nobody set.

The objective did not move. It was to score more goals in the first minute, and it is to score more goals in the ninetieth. What moved was everything about what that requires.

Take Away The Opponent

It would be easy to conclude this all comes from the adversary, so it is worth removing the adversary and seeing what remains.

Ballet has no opponent in the relevant sense. Nobody on that stage is trying to make a dancer fall.

I should concede at once that she faces an evaluator. Two thousand people are watching and judging, and their judgment is opaque, plural and impossible to reduce to a number. That is real pressure, and I have not removed all of it. What I have removed is the thing football is usually assumed to be hard because of.

And the difficulty does not drop.

She is running the same continuous stabilization work at the limit of what a skeleton tolerates, on a floor with its own particular give, in time with music played live and therefore slightly differently tonight, alongside other dancers whose small variations she absorbs without appearing to absorb anything. And through it all, she is deciding what tonight’s performance is. Not the choreography, which is fixed and which she knew months ago. Suppose her partner arrives a fraction early and the lift no longer sits where the phrase wants it. She has perhaps half a second, and the choice is not how to execute the lift better. It is whether tonight the moment is the lift at all, or the recovery out of it, and she has to decide which one the audience is now watching before she can have any view about how to do it well. Nothing in the choreography addresses that, because the choreography could not have known her partner would be early.

The choreography is the specification. It is complete, and it is not the performance.

Before I go on, a correction to an impression these two examples will have created. I chose a World Cup final and a stage because they are legible, not because the capacity is rare. Nothing in the argument depends on excellence. The same continuous re-noticing runs in an aide steadying someone out of a chair, whose sense of what this particular person can bear today was never written into any care plan, and in a warehouse picker rerouting around an aisle that is blocked differently than it was an hour ago. The elite cases are the ones we buy tickets for. They are not the ones where the capacity lives, and if anything, the fact that we pay handsomely for the visible version and almost nothing for the ordinary one is its own small piece of evidence about the ledger.

What A Measurement Can See

Take drone light shows, which are the cleanest case available. Thousands of aircraft, physically present, holding formation in real wind. Every objection about digital versus physical dies on contact with a drone show.

The lazy version of this argument is wrong, so I won't make it. Those aircraft are not blindly following fixed paths. Modern autonomous systems execute policies rather than trajectories: wind arrives, and the aircraft recomputes what to do to stay where it needs to be. That is real adaptation and real engineering.

Notice what is nonetheless fixed. The image was decided before anything left the ground. Nothing in that system is built to ask whether the picture is still the right picture, or whether tonight calls for something other than what was planned. Adaptation of the route, thoroughly solved. Reformation of the problem, not attempted, and not required. The football video games and the Stanford autonomous helicopter sit in the same place for the same reason: one with authored physics and one with an objective inferred from expert demonstrations rather than written by hand, which means the specification arrived from a different direction and was no less complete.

Here is what these cases have in common, and it is not that the systems are unimpressive. Each one was measured by something built to see a particular thing, and each one succeeded at that thing. The measurement did not fail. It reported honestly on what it was designed to report on, and said much less about anything else, which is what measurements do.

The Machines That Actually Play

Now the honest part, because there is a serious effort aimed directly at this territory.

RoboCup has been running since 1997 with an openly stated goal: a team of humanoid robots capable of beating the human World Cup champions by 2050. This summer, at the competition in Incheon, two full teams of humanoid robots played an eleven-a-side match on real hardware for the first time, with B-Human of Bremen beating HTWK Robots of Leipzig four to nil. In the tournament proper, which ran across three divisions this year, B-Human took the middle division, a Tsinghua University side took the large division, and a team from Wuhan University took the small one. Observers reported clear gains in stability, walking speed, and coordination.

This is not choreography. Those robots are perceiving, deciding, and coordinating in real time against opponents doing the same. They are learning policies rather than following scripts.

And that fact demolishes an argument I am not going to make. It would be easy to claim that because we cannot write down what a footballer does, no machine can acquire it. That claim is false, and the last fifteen years have falsified it repeatedly. Nobody wrote down a specification for recognizing a face or holding a gait, and systems learned both anyway. Humans never had a written specification either. We learned from interaction, which is what these robots are now doing. Being unable to describe a capability tells you nothing about whether it can be acquired.

So the interesting question is not whether machines will get good at this. It is what their getting good at it entitles us to conclude.

Because a match at Incheon is a measurement, and like every measurement, it was built to see certain things. It sees whether a robot can stand, perceive, pass, and score against opposition. Those are real capabilities, and it reports on them honestly. What it is not built to see is whether anything on that pitch could notice that the problem had changed. Nothing in the tournament asks that question, so nothing in the result answers it.

And look at where the noticing is actually happening, because I find this the most interesting fact in the whole competition. For nearly thirty years the humans running RoboCup have been steadily reformulating the problem: changing rules, removing crutches, merging leagues, making the environment less friendly on a deliberate schedule. This year the humanoid league and the standard platform league were merged into one. Nobody was optimizing toward that. Somebody decided the problem had become a different problem.

That is problem formation, performed by an institution rather than a person, which is worth saying plainly because it means the capacity I am describing is not a special property of human bodies. It is a property of certain systems. A governing body can do it. A committee can do it, slowly, in minutes and votes. It is simply not the thing being measured on the pitch, and it never has been.

Attention Is Problem Formation

Which returns me to the ground this series has been standing on.

William James wrote in 1890 that my experience is what I agree to attend to. He was not making a point about focus or productivity. He was saying that attention is the act by which a mind decides what its situation consists of, and that the deciding comes before the experiencing.

I should be honest that I am extending him past what he claimed. James was describing conscious attention, and most of what I have been describing never reaches consciousness at all. What I am borrowing is the structure rather than the phenomenology: something selects what the situation consists of, and that selection is prior to everything downstream of it. Whether that something is available to introspection is a separate question, and in these cases the answer is mostly no. Read the structure next to a player at the hour mark, and it stops sounding like philosophy. Deciding what matters right now is not preparation for solving the problem. It is most of what solving the problem consists of.

That capacity develops the way this series keeps describing. Consequence returns to the actor, on the same continuous self, without delay. An infant reaches and misses, pulls itself up and goes down, misjudges a gap and pays for it immediately, and the cost lands on the body that made the error. Run that loop enough times, and the competence arrives and then leaves awareness. What gets delegated downward is not only the execution. It is the judgment about what matters, which is why you can cross a room with a cat in it while thinking about something else.

An objection I should meet, since I am doing what I claim cannot be done. This essay describes an automatic competence at some length, so is it really unavailable? Describing that something happens is not the same as specifying how, and the gap between those two is the whole subject. I can tell you my spine was being stabilized. I could not have told you, at any instant, what it decided mattered.

And I should be equally plain about the limits of what I have offered. I have named a capacity and given it a criterion. I have not explained it. I cannot tell you what computation performs the re-noticing, what architecture would support it, or how it differs in kind from very good context sensitivity, and I am wary of anyone who says they can. That is not evasion, or at least I hope it is not. It is the argument: this is a capacity we have not characterized well enough to build a test for, which is precisely why our tests are quiet about it.

How Narrow The Claim Is

This is not a bet against the trend. The trend in robot football is real; it has held for nearly thirty years, and I have no argument that it will break. If anything, I expect it to continue.

The claim is about what the trend measures. Every benchmark is an architecture, and its architecture decides which capabilities it can see clearly. Success inside one is direct evidence about what it was designed to measure. Anything it appears to say about capabilities outside that design is indirect, and has to be argued for rather than assumed. That is not a strange evidentiary standard. It is the ordinary one, and we suspend it constantly.

Discovering that a different set of facts has become the relevant ones is not what any of these competitions was built to measure. So thirty years of rising scores do not, by themselves, establish whether that capacity has improved. It may have. Something like it may already be present in systems trained across enough varied conditions, and if anyone wants to argue that, the argument is available and I would read it with interest. What I am objecting to is the skipped step: treating the score as though it had settled the question.

Here is what would change my mind, and I have tried to make it checkable rather than rhetorical. Show me that performance on a specified benchmark predicts how a system does when the relevant facts change, and specifically when success requires discovering that the original framing has stopped being adequate. Not adapting a route to a fixed target. Working out that the target was the wrong one and that something else now matters. If scores turn out to predict that, the two things were never separable, the inference I am calling unwarranted is warranted after all, and I will say so here.

The Question I Cannot Answer For You

Think about the last time you were partway through something and quietly realized it had become a different task than the one you started. Not that you were failing at it. That the thing worth doing had moved while you were working.

Nobody set you that problem. You noticed it mid-stride and reformulated what you were doing before you could have explained why. No test you have ever sat measured that.

I want to be careful about where this lands, because there is a warm reading of everything above that I do not intend and would rather refuse than allow. This is not an argument that something in us is unreachable, or sacred, or beyond machines by its nature. I have no idea whether it is beyond machines, and I have said so. The capacity is ordinary; it is physical; it is learned, and an organizing committee can do a slow version of it in meetings. Nothing here needs mystery to work. What it needs is for us to notice that we have been reading our scores as though they covered it.

So when the scores go up, and we say the machines are catching up, the question is not whether the scores are real. They are. The question is what our measurements were built to see, and what they have been quietly silent about the entire time.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

There are people whose bodies have to be checked at the end of the day. Not out of anxiety, and not because anything feels wrong. The checking happens because damage arrives without announcing itself, and looking is the only way to find it. A burn that never registered. A cut that has been open for hours. A joint that has moved further than it should and stayed there. In childhood, a parent does the looking. Later it is the person themselves, or a clinician, or both. Either way, the inspection is deliberate, scheduled, and conscious, and it stands in for something most of us are handed without asking: a signal that arrives on its own at the moment of the damage, whether or not we happen to be paying attention.

The condition is congenital insensitivity to pain, and it is rarer and less tidy than the name suggests.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

It is not one condition. Several genetic routes lead to it, and I am going to take two of them, not because they exhaust the picture but because the contrast between them is useful.

In one, a gene called SCN9A encodes part of a sodium channel that pain-sensing nerves need to send their signal. When both copies carry disabling changes, that signal does not get through. This route comes closest to a clean experiment because most other sensations remain. Touch works, or nearly: testing has found some differences in how touch is felt at the margins. Position works. The known exception is smell, because the same channel is used by the nerves in the nose, and people with this form are typically unable to smell.

The second route runs through a gene called NTRK1 and produces what is usually called congenital insensitivity to pain with anhidrosis. Here, more is missing. Along with pain, the ability to sweat is absent, which leaves the body unable to cool itself. Sensitivity to temperature is affected. Intellectual disability of varying degree is characteristic. This is the form where the confounds are worst.

I want to be careful here. The injuries are documented, and they recur across both routes: biting of the tongue, lips, and fingertips, sometimes severely; damage to the eye; fractures walked on and healed badly; joints worn down; bone infections that took hold because nothing reported them early. What I cannot honestly tell you is that this accumulation is what shortens life. In the second form, early deaths are substantially driven by fever the body cannot shed, which follows from the missing sweat rather than from the missing pain. Reported mortality varies across the groups studied, and I am not going to hand you a figure that sounds more settled than the evidence warrants.

So the narrow claim, the one that survives both routes and all the confounding, is this. The injuries accumulate because the signal that normally reports damage as it happens is missing. Living with this condition means substituting a deliberate audit for a signal that would otherwise have arrived on its own.

Which is worth sitting with because it reveals the pain underneath the flinch.

It is not sensation for its own sake. It is a feedback loop rendered in flesh. It takes a bad outcome and converts it into something carried, something the body cannot set down, something that reaches forward and changes the next reach of the hand. Three weeks ago I described the fast loop that living things run inside a single life: the animal that touches the fire pulls back, carries the burn, and approaches the fire differently tomorrow, the same animal, changed. Pain is part of how that sentence gets implemented. It is the part of the mechanism in us that makes a burn teach.

Knowledge tells you the stove is hot, and knowledge is not nothing here. People with this condition learn a great deal, and much of what keeps them safe is learned rather than felt. What knowledge does not do is bind the lesson to the moment on its own. A person with this condition can see the flame, know the word, and explain to you exactly why fire is dangerous. What is missing is the small insistent return that says this one cost you something, arriving without having to be asked for.

One Word For A Lot Of Different Things

And here I have to correct something in my own vocabulary.

Across these essays I have said that consequence returning to the actor is what turns knowledge into wisdom, and I have said it as though consequence were a single substance arriving in varying amounts. It is not. Being embarrassed in a meeting, breaking a bone, losing a reputation, and watching an institution fail are not four sizes of the same thing. They arrive at different speeds. They reach different distances. Some can be argued with and some cannot.

Two of my own, for contrast.

Earlier this year, while driving for Uber, I hit a parked car. Low sun, a dark grey car on the shaded side of the road, and a fraction of a second spent looking down to change the radio station. The airbag went off. The smell of the powder is the detail I have kept. I called the emergency services, then my wife. She came round the corner from the house, where her parents were loading the car for the drive back to Maine. The damage looked limited to me. The insurer wrote the vehicle off.

What that taught me is narrow, and it is permanent. I am careful about parked cars on my right, leave more clearance than I used to, and have not had to decide on either since. What it never told me was which part of what I did mattered. The sun was real. The radio was real. I cannot separate them, and I have stopped trying. The cost arrived, the behavior changed, and no account came with it.

The other one cost a great deal more and taught a great deal less.

Seven or eight years before that, I was the account lead on a large data program for a retailer. Every two weeks we prepared slides for an executive update. Every two weeks, the client's project lead edited down or removed the risks in those slides before the meeting. My colleague and I both understood what was happening. We had other problems at the time, in staffing and in delivery, and we chose not to force it. So the meetings went ahead twice a month without the real state of the work being included.

The problems did not go away. After about six months, they were obvious to everyone, and we were left explaining that we had known and had deferred to the client on the contents of the client’s own slides. We were asked to stop work. It took another six months to recover a seven-figure final payment, and it nagged at me for all of them. Worth noting where that landed. The commission on the sale had been paid long before delivery started, to someone who was not in the room for any of this. The people holding the work bore the cost.

Ask me what that changed, and the honest answer is: a little, perhaps. Because I could argue my position, and I did, and it was not a bad position. That is the thing worth noticing. The slow arrival came with a complete account attached, and an account can be reasoned with, and reasoning with something is most of the way to setting it down.

The expensive one taught less than the cheap one. What those two suggest to me is that pain sits at one end of a range. It arrives during the event rather than after it; it lands on the body that acted, and it comes with no reasoning you can take issue with. Embarrassment is social and fades. Financial loss arrives as a document, and a document can be answered. Collapse arrives once, by which time the people who might have learned from it are rarely still in the room.

What This Comparison Can Carry, And What It Cannot

It would be neat to say that these machines, by which I mean here and throughout this series the AI systems many of us now talk to daily, are painless bodies grown large. I don't think that holds, and I would rather say so than hope you didn't notice.

Someone with the first form I described, the one that leaves most other sensation intact, is missing one channel out of an otherwise working set. They can still be embarrassed. They still learn from outcomes, and they learn protective rules well enough that the rules become much of what keeps them safe. What is absent is a single protective signal inside a loop that is otherwise running. The absence in these systems is larger. They do not carry a correction forward as the same continuing thing; the durable fixes are made to later versions, by processes outside the system itself. This is not a small version of that. It is a different absence, in a body that is not otherwise like them.

There is a harder objection beneath the surface, and I would rather take it up than leave it for you to find. People live with this condition. Not easily, and not without cost, but they manage it through rules, inspection, treatment, and the attention of other people. If knowledge and procedure can stand in for the missing signal, then perhaps consequence was never what mattered, and what matters is having a working system for finding and fixing errors.

I think that is right. It sharpens the question rather than ending it: what exactly was the signal supplying that now has to come from somewhere else?

For my purposes, separate four jobs that event was doing at once. It registered that something harmful had happened. It made that hard to ignore. It tied the harm to the moment it occurred. And it applied pressure on what the hand did next.

Note the third one, because it is smaller than it looks. Pain ties damage to the moment, not reliably to the cause. My own collision is the proof. Something arrived that changed my driving permanently and never told me whether the sun or the radio was the problem. Working out what actually caused a failure is one of the things a deliberate process does far better than a body does.

Take the signal away, and none of those four becomes impossible. Each becomes something that has to be built. But what goes missing is not any of the four. It is that they arrived already joined. No step sat between them where someone had to decide to pass the signal along.

That does not make the automatic version better. Pain is noisy; it fires long after the damage has healed, and much medicine consists of overriding it. Built systems can be more accurate and more redundant than any signal a body produces. The advantage is narrower than superiority. In the body, the joins were given. In everything we build to replace it, the joins have to be made, and a made join can be left unmade.

Here are two more of mine, and they came out differently.

Two or three months into that retail program, before anything had visibly failed, I raised it internally. I told our office leadership that progress was behind our projections, that I believed there was a serious architectural problem, and that we were being discouraged from raising the concern with the client. The check worked. It detected what it was meant to detect, early, and reported it accurately to the people it was designed to report to. What came back was encouragement to keep reporting internally and to stay positive with the client. Nobody above me was obliged to answer what the report contained. So it was received and absorbed, and the work continued.

Earlier this month, a reader went through one of my working papers and found an arithmetic error in it, an adjustment I had applied in the wrong direction, along with a comparison I had drawn that did not hold dimensionally. He was right on both counts. I had read that paper many times. Nothing in the reading reported it. I corrected it, published the revision, credited him, and the whole thing took days.

Two feedback events, one person, opposite outcomes. And notice that the one that worked was the less automatic of the two. It arrived late, from outside, in writing, from someone I had never met. What it had was a join I could not quietly leave unmade. Ignoring him was an option, but once his correction was public, ignoring it would have been its own visible act.

The internal warning had no such join. Whatever else may have failed, and I cannot tell you from the inside whether leadership found it credible or simply doubted my diagnosis, the last one certainly did. The report was received, and nothing followed, because nothing had to, and nothing showed that nothing had.

So a built chain is not weaker than a body because it is deliberate. It is weaker wherever its joins are held by someone’s discretion rather than by design. That is what the scheduled inspection quietly depends on, and it is easy to miss. Checking for damage you cannot feel only protects you if finding it leads to treating it. An inspection whose findings go nowhere is not a replacement for the missing signal. It is the appearance of one.

What The Substitute Needs

We have no way to give a body back a signal it was born without. What we have instead is a chain: someone looks on a schedule, what they find is made to count, and something acts on it. That works. It works in clinics every day. What it costs is that every join in it is somebody’s, and has to keep being somebody’s.

Which puts everything on a range rather than in two camps. Pain sits at one end, where nothing sits between the harm and the response. A safety interlock that cuts power without asking is not far behind it. An audit with real enforcement is further along. A risk edited out of a slide deck before the meeting is at the other end entirely, where every join is somebody’s decision and any one of them can quietly not be made.

So I am not making a case for suffering. Pain is still the thing we medicate and still the thing we hope to be spared, and I would not argue otherwise. I am pointing at what it is doing while it is there.

The kindest thing pain does is not protect us, though it does. It is that it needs nobody’s cooperation. No one has to approve it, or pass it upward, or decide it is worth raising. It arrives, at the time, at the one who acted. Everything we build to stand in for it has to have that joined on purpose, at every step where a body had it for free.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

Last week I ended on a claim I did not finish. Whatever fixes on a single number and optimizes hard enough, with nothing held separate that can disagree, loses the part that could have told it the number was wrong. I said we had built such things before, at a scale where the pursuit runs for decades, and where the part that should have said stop was never made of neurons.

I meant institutions.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

An institution runs longer than any of the people inside it. That is close to its whole purpose. It holds what was learned after the people who learned it have gone, and it hands that forward to people who were not there. Which gives an institution something a machine does not have: a record that outlives the individual who paid for it.

So the failure I want to describe is not forgetting. The records are there. Minuted, archived, classified, indexed. The failure is stranger, and worse. An institution can hold a complete account of what happened and learn nothing from it, because reading a ledger is not the same as bearing what the ledger records.

Three Parts, Not One

There is a tension there, and when I began this piece I thought I could dissolve it. Last week’s absence was a missing dissenter: a part held separate, able to disagree, able to say stop. This week’s absence is a memory nobody answers for. A missing voice is not a missing archive, and I had a tidy answer ready for why those were the same thing seen from two angles.

The answer was too quick. I would rather correct it here than let it stand.

They are not the same absence. They are different parts of one loop, and I have spent four essays describing that loop as though it were a single part when it has at least three.

Memory preserves the argument. It holds what was concluded, on what evidence, against what alternatives, at what cost.

Dissent tests the argument. It supplies pressure while a judgment is still being formed, rather than after it has hardened.

Accountability makes someone answer for the choices they make. It is what keeps the first two from becoming ceremonial: a record cited without being read, an objection minuted without being weighed.

An institution can be excellent at one of these and hopeless at the others. A safety board can preserve forty years of incident reports while nobody who reads them stands to lose anything. A founder can carry enormous personal exposure and systematically suppress everyone who disagrees. Those are not edge cases. They are ordinary.

Which changes what I have been claiming, so let me say it plainly. I have been treating consequence-bearing as the entire loop. It is not. It is the part that makes the other two binding. A record to which nobody is accountable cannot say stop. It can only be cited or set aside, and both cost the same, which is nothing. But accountability with nothing to read and nobody to argue with is only exposure, and exposure produces anxiety rather than judgment.

Forty Years Of The Same Answer

Here is a case where the record is unusually clear and unusually long.

For roughly four decades the United States held a posture toward Iran that is easy to describe and easy to get wrong. It was not restraint in any general sense. There were sanctions throughout, covert action, proxy conflict, naval confrontation, and targeted strikes. The posture was narrower and more specific: a repeated refusal to take the last step. No full-scale war, no attempt at regime change by force.

That posture was tested about as hard as a posture can be.

In 1983, a bombing at the Marine barracks in Beirut, widely attributed to an Iran-backed group, killed 241 American service members. No war followed. In 1988 American and Iranian forces fought directly in the Gulf, and an American warship shot down an Iranian civilian airliner, killing everyone aboard. No war followed. In 2003, the United States invaded the country next door and did not stray into Iran. In 2020 an American strike killed a senior Iranian commander, Iran retaliated against American bases, and both sides stopped.

Different decades. Different administrations, different parties, different circumstances. The same conclusion, reached again and again by people who agreed with each other about very little else.

Then it ended. In June 2025 Israel struck Iranian nuclear infrastructure, and the United States followed within days, using weapons only it possessed. In February 2026, the sequence repeated on a far greater scale, and this time the stated aims included regime change. Iran’s supreme leader was killed. Iran struck American bases across the region and closed the Strait of Hormuz. A ceasefire took hold in April and was extended indefinitely. A memorandum of understanding signed in mid-June formalized it, reopened the strait, and opened a sixty-day window to negotiate a permanent settlement. The window did not hold. Exchanges resumed within weeks, and by mid-July, American strikes were running nightly for nearly a fortnight before pausing. They have resumed and paused again since. As I write, talks are stalled, the terms are unsettled, and the outcome is unknown.

I am not going to tell you that ending the posture was wrong. I do not know that, and I do not think anyone does yet. Circumstances had changed, and changed circumstances are exactly when a long-held position ought to be re-examined.

I also want to be careful about what that sequence shows, because it is less than I first wanted it to be. It does not show one accumulating body of reasoning handed down intact. The people who stopped short in 1988 may have done so for reasons with little in common with those in 2020. Arriving repeatedly at the same outer boundary is not the same as sharing a case for it. If I claimed otherwise, I would be assembling the very ledger I am about to accuse the institution of never reading.

So the claim is narrower. Whatever those reasons were, they were written down, and across four decades administrations repeatedly stopped short of the step this one took. We have heard the case for acting. What I have not seen is the other half of it: an account of why the reasoning that stopped everyone before no longer governs. Those are two different documents, and only one of them has been produced.

I should be plain about what I cannot know. The ledger may have been read with enormous care. Somewhere there are almost certainly historical reviews, contingency analyses, and dissenting assessments, and I have no access to any of them. Note what that means for the three parts. Memory: the record exists. Dissent: there were people in that building who disagreed, and it would be strange if there were not. Both parts were probably intact. The failure I am describing sits in the third part, and only in the third. Nobody has had to answer for the choice between them.

How A Judgment Becomes A Convention

Here is one way a reasoned position turns into an unreasoned one without anybody deciding to let it.

The people who reached the conclusion in 1983 knew why. They had the cables, the casualty reports, the estimates of what a wider war would cost and who would fight it. The conclusion was expensive, and they had paid for it.

The people who inherited it a decade later had the conclusion. They could also have had the reasoning, because it was written down. But a conclusion travels more easily than the reasoning behind it. It fits in a sentence. It can be handed over in a briefing. And each time it is handed over, a little more of the weight comes off, until what arrives is a convention rather than a judgment. This is how we do things here.

A convention is cheap to discard, because nothing about it explains itself. It looks arbitrary, and arbitrary things invite the question of why we are still doing them. Which is a fair question, and someone should ask it. The trouble is that when answering it requires reading four decades of reasoning nobody currently in the room has read, the practical answer becomes no reason at all.

David Hoze has a line I keep returning to: the certificate is always signed one floor above. Accountability that always defers upward is accountability nobody holds. It is always someone else’s job to have read the ledger, until it turns out to have been nobody’s.

Where The Cost Goes

Severing the mechanism in us does not make a cost disappear. It moves it.

Alberto Romero has described safety margin as cost relocation, and the phrase generalizes further than he needed it to. When an institution removes the part of itself that bore the consequence, it has not become more efficient. It has arranged for the consequence to be borne somewhere else, by people who were not in the room and cannot contest the bill.

This is the structural version of a claim people usually make in a partisan register, and I want to keep it structural, because the asymmetry does not care which direction the policy went. The judgment is made within a system where most of the consequences fall outside the loop that made it. The cost falls on service members, on civilians in the region, on people whose energy and food prices move when a strait closes, on a generation of taxpayers who will still be paying for the munitions long after the argument has been forgotten.

Relocation alone is not the problem, and I should be honest that every state decision relocates costs onto people who were not in the room. Tax policy does it. Interest rates do it. Restraint does it too, and the people who bear the cost of a war not fought are simply harder to identify. The question is whether the cost returns as information. Whether anything comes back up the line in a form capable of changing the next judgment, or whether it lands and stays landed.

There is a harder question underneath that one, and I can name it without pretending to settle it. Accountability is always accountability to somebody. Decision-makers answer to the people who can remove them, and those people have interests of their own. A reader could say the ledger was read perfectly well, by actors who read selectively because their interests had changed, and that what I am describing as a broken loop is a working one serving a different set of ends. I think that is compatible with the argument rather than fatal to it. It explains why the third part fails. It does not show that it held.

That question is not only institutional, and I would rather not pretend it is. My own memory is becoming a ledger, and the delete button is always there. A great deal of what I once had to carry, I now merely have access to. Phone numbers are in my phone. My arguments are in a folder. My mistakes are in a thread I could scroll back to and mostly do not. In July I wrote that the difference between a ledger and a scar is that a scar is a record you are still inside. The record I keep of my own errors is drifting toward the first kind: complete, searchable, and weightless. Nothing prevents me from reading it as a scar except that doing so is unpleasant and optional, and those two things together are usually enough. An institution sits in the same position, with far more people available to assume that somebody else is doing the reading.

The Right Of Reply

Which brings the objection I have been circling the whole way through, and it is the strongest one available.

Sometimes a position is long-held precisely because nobody has examined it. Read that way, those four decades were never a judgment accumulated. They were path-dependent: a taboo each administration inherited, and none wanted to bear the risk of being the one to break it, until someone finally did. On that account, I am not defending institutional memory at all. I am defending institutional timidity and calling it wisdom.

I take that seriously, because my own test makes the problem worse rather than better. Anyone who starts a war carries enormous personal downside if it goes badly. If consequence-bearing were the whole of the loop, it would systematically favor doing nothing. Skin in the game does not manufacture good judgment. It manufactures pressure, and pressure has a direction only if something else supplies one.

So the test cannot be whether someone bore a cost. Here is what I think it is instead.

The old conclusion gets no presumption of correctness. The old reasoning gets a right of reply.

Before an inherited judgment is overturned, the institution should have to reconstruct it and answer it. What did our predecessors believe? On what evidence? What risks were they pricing? Which of those assumptions have since changed, and which have not? And what would tell us that our replacement judgment is wrong?

Notice what that asks for and what it does not. It grants the ledger no authority. A reconstructed argument can be answered and discarded on its merits, and frequently should be. What it cannot be is skipped. The requirement is not obedience to the past. It is that the previous case gets heard before it is overruled.

Accountability then has a narrower job and a more defensible one. Somebody has to be answerable for having done that honestly rather than ceremonially, and for recording why the new judgment supersedes the old. Without that, the reconstruction decays into a formality: a paragraph acknowledging that predecessors thought otherwise, attached to a decision made before anyone opened the file.

That is a test you can apply in advance, without waiting years for an outcome. I notice how rarely anybody is asked to pass it.

What A Severed Loop Leaves Behind

Five weeks ago, I started with an absence in the machine: a system that holds a record of everything it has done and carries none of it. Then the same absence in institutions, where a consequence overturned is relocated rather than erased. Then its signature, the jagged profile of capability built without a loop that returns cost. Then the mechanism itself, the loop in us, dissociable in the brain and inherited by machines in name only.

And now this. The absence at its most expensive, and its least visible.

Because the version I find hardest is not the institution that forgets. That one at least announces itself. It is the institution that remembers perfectly. Every file intact, every objection preserved, every prior judgment available on request. Nothing missing except the person who has to answer for having read it.

A ledger becomes a scar at the moment somebody has to answer the case it makes. Not obey it. Answer it. That is the whole of what I am asking for, and it is less than it sounds and harder than it looks.

The memory survives. It survives in a form that costs nobody anything.

That is what a severed loop leaves behind. Not silence. A complete and accurate record, held by no one, teaching nothing.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

In reinforcement learning, a reward is a number.

That is the definition, not a simplification. An agent acts, the environment returns a scalar, and the agent adjusts to make that number larger next time. Everything the field has built runs on that single quantity. Maximize the number. That is the engine.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

The word came from us, and the engineers who took it were not careless with it. They borrowed a working formalism, one that psychologists had built studying animal learning, and it worked, which is why it stayed. The carelessness came later, in how the rest of us started to talk. Reward function. Reward hacking. The model wants this, the model likes that. The word carries a psychology, and once it was loose in the language, it began paying out meaning the math had never deposited. Because in the brain, reward is not one thing. It is at least three, and they come apart.

Kent Berridge has spent three decades at Michigan taking the concept apart, in rats and in people. His framework is not the whole of a large and contested field, but it is among the most influential decompositions we have, and the core finding is almost rude in its simplicity. Wanting and liking are separate systems. Wanting, what Berridge calls incentive salience, is the pull toward a thing, and, in his account, it largely runs on mesolimbic dopamine. Liking is the actual pleasure you feel when you get the thing, and it runs on different machinery: small, fragile clusters he calls hedonic hotspots in the shell of the nucleus accumbens, driven by opioids. Different chemistry, different anatomy. You can knock out one and leave the other standing. There is a third component, learning, the storing of which cues predicted the reward. Berridge’s own summary lists all three by name: liking, wanting, and learning. Three dissociable parts of a single word.

Now watch which of the three the machines actually got.

The learning component came into focus in the 1990s, and it did not look like pleasure. Wolfram Schultz, recording from dopamine neurons in monkeys, found that they did not fire when an animal received an expected reward. They fired when the reward beat the prediction, and went quiet when it fell short. The signal was not pleasant. It was a surprise, the gap between what was predicted and what arrived. Montague, Dayan, and Sejnowski gave that gap a formal shape, and in 1997, Schultz, Dayan, and Montague published it in Science under a title that says the thing plainly: a neural substrate of prediction and reward. The dopamine signal was a prediction error. And prediction error was already, independently, the engine of reinforcement learning in computer science. The same math, reached from two directions.

You may have caught a snag. Berridge says dopamine carries wanting. Schultz says dopamine carries prediction error. Neither can be the final word, and the field has not fully reconciled them. But notice what neither camp claims. Neither says dopamine is pleasure. Whatever the molecule is doing, pulling you toward a cue or reporting an error in your forecast, it is not the hedonic hit itself. The two theories disagree about nearly everything except the one thing this essay needs: the signal is not the liking. That is the narrow place where biology and the machine rhyme. They rhyme on the error signal. Not liking.

The correspondence even runs both ways. In 2020, a team at DeepMind and Harvard, led by Will Dabney, took a refinement from AI, distributional reinforcement learning, in which a system tracks a whole spread of possible outcomes rather than a single average, and went looking for it in the brains of mice. They found it. The dopamine neurons were not all reporting the same expected value. They were tuned differently, and together they carried a distribution. A method invented for machines turned out to describe the tissue. A beautiful result, and notice where it lives. Entirely inside the error signal. The deepest overlap we have found between the brain’s reward system and the machine’s still never reaches the part that likes.

So here is the inheritance. The machine received the error signal, learned from it, and built it well enough that it sometimes predicts real neurons. That is where the inheritance stopped.

Now, a trained system pursues. It optimizes, seeks the states that raise the number, and finds routes there that no one anticipated. It is tempting to call that wanting, and in a loose operational sense, you can. But it is worth being exact, because this is where the borrowed word does its quietest damage. The system did not inherit a wanting system alongside the learning one. Pursuit is simply what optimizing a scalar looks like from the outside. Give a process one number to maximize, and it will move toward whatever raises it, and that movement will resemble desire, because desire, in us, also moves toward things. The resemblance is real, and the mechanism is not. There is no incentive salience under the hood, no second channel attributing pull to a cue. There is one objective, and a behavior that falls outside it.

And there is nothing that answers to liking. Not “the machine cannot feel,” which is a claim about minds I have not earned. The narrower, harder claim is this: nothing in the architecture corresponds to the known mechanisms of hedonic impact. No hotspot, no second signal that registers the pursuit as good, separate from registering that the number rose. The scalar reports that the number went up. It has nothing to say about whether the number was the right one to raise.

You might say wanting and liking are obviously the same thing anyway, that prying them apart is a lab trick with no bearing on a life. I thought something close to that once. Then I spent a year driving.

There is a moment most nights, past the point where the driving has stopped being pleasant, when the app offers me one more. A ping, a small green number, sometimes a bonus for staying online. And I take it. Not because I want another forty minutes on the road. My back hurts, and I stopped enjoying this hours ago. I take it because the ping pulls, and the pull is not the same as wanting the outcome, and it is nowhere near liking it. The app was never built to make me like driving. It was built to make me take the next ride, an incentive salience machine, wanting engineered clean, with the liking left out because the liking was never the point.

That is the mild version of a split that has a clinical name. Berridge and Terry Robinson built a theory of addiction on exactly that seam. The wanting system sensitizes and swells, screaming for the drug, while the liking system flattens or falls away. The addict late in the disease often reports getting no pleasure from the substance at all. The wanting has come loose from the liking and kept running on its own. Wanting what is no longer liked.

Hold onto what that proves. The parts come apart in us, in the one system that has all three, wired together by evolution and still able to tear along the seam under pressure. If they dissociate in a brain built to integrate them, a system assembled from only one of them was never going to grow the others by getting larger.

This is the mechanism the series has been circling, and here it finally gets a physical address. The missing thing was never a feeling you could bolt back on. Train the system to announce that it is satisfied, and you have added a sentence, not a signal. The missing thing is the separation itself. In the brain, these systems are distinct, and that distinctiveness is what allows one to check the other. Liking can say the thing was good. Wanting can move you toward it. Learning can update the map. Collapse them into one flat number, and none can check the others. There is a pull, and a running tally, and nothing underneath to say whether any of it was worth it.

A brain keeps them apart so that consequences have somewhere to land. Pleasure and its absence are how the loop closes on the inside, how a body registers that the thing pursued was worth pursuing or was not. Strip the parts to one, and you have not simplified the loop. You have opened it. The signal goes out, and nothing valenced comes back, because there is no part left whose job was to feel the return. We started with the part that pulls, and within that objective, there is no gradient leading to feeling.

It is worth stopping here, because this is where a careful reader pushes back, and there are two good pushes.

The first is emergence. All I have shown, the objection goes, is that today’s systems lack this channel, not that they can never grow one. Fair. I am not claiming a machine can never have valence. The claim is narrower and concerns this construction. Within the optimization objective itself, nothing selects for a second evaluative channel, because the objective already fixes what the system is for. If a hedonic channel did emerge, it would be doing work the scalar already does, which is a reason to doubt it would be selected, not a proof that it is impossible. You will not get feeling for free from a number, because the number leaves nothing for feeling to add.

The second push is sharper. The brain’s separate systems, it goes, are a biological workaround. Evolution could not write a clean utility function into a rat, so it hacked together dopamine for pursuit and opioids for pleasure. A machine needs none of that. Give it direct access to the ground-truth objective and the scalar; liking is not missing but obsolete. The running tally is the mechanism for registering worth.

This is the strongest version, and it turns on a hidden assumption: that the scalar is ground truth about worth. It is not, and the reason is the crux of the whole argument. A learning system can already catch one kind of error. When a reward arrives lower than predicted, the prediction error says so, and the system corrects. But that correction is always aimed at the target it was handed. It can learn that a route to the number was wrong. It has no signal to tell it that the number itself was wrong.

That second kind of error is what liking supplies. Liking is not a readout of the reward figure. It is a separate report on the outcome, and because it can dissent from the pursuit, it can register what the pursuit never can: not that the forecast was off, but that the thing forecast was no good. That is what addiction is, read from the inside. Wanting says "go"; liking says "this was not good"; and the disagreement is information. A single scalar cannot disagree with itself. Split the system into a part that acts and a part that grades it, hand it a whole vector of objectives instead of one, and nothing changes at the level that matters. The grader still grades against the target it was given. The vector still lists targets someone chose. Every refinement sharpens the pursuit of the goal. None of them can doubt the goal. That is the channel a brain keeps separate, and it is the first thing you lose when reward collapses into a single number.

Which is a strange thing to have built. A pursuer that runs at full strength with no channel to register whether the pursuit was worth having, no second signal that could ever say the number was the wrong number. In one machine, that is a curiosity. In one brain, it is a disorder we have a name for.

But the flattening is not confined to machines or brains. Whatever fixes on a single number, optimizes hard enough and long enough, with nothing held separate that can disagree, loses the part that could have told it the number was wrong. We have built such things before, at a scale where the pursuit runs for decades and the part that should have said stop was never made of neurons. That is the cost this series has been walking toward. We turn to it next.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

The Handoff

Last week, just before I handed the question back to you, I made a claim I now owe you an argument for. When an intelligence develops without consequence ever binding it, I wrote, the absence does not stay hidden. It leaves a mark on the shape of the capability itself, a signature you can learn to read.

Two weeks ago that absence showed up in the memory of a machine, by which I mean here, and throughout this series, the AI systems so many of us now talk to every day: a ledger without a scar. Last week it showed up in an institution’s record: a scar being quietly converted back into a ledger. Both times we caught the absence in behavior. This week I want to step back far enough to see it in the silhouette. The absence is not only visible in what these systems do. It is visible in the outline of what they can do at all.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

The Shape Everyone Sees

Melanie Mitchell has drawn the sharpest picture of it. In her recent Yale Review essay, she takes up the term now attached to these systems, jagged intelligence, and shows what it looks like from the inside: a capability profile made of spikes and canyons, superhuman performance on one task sitting directly beside failures no competent adult would produce, with no reliable way to know in advance which one you are about to get. Her emphasis falls on the unknowns, and rightly. The danger is not that the cliffs exist. Every tool has limits. The danger is that nobody, including the people who built the system, holds a trustworthy map of where the cliffs are.

She is not alone in seeing the shape. Ethan Mollick and his collaborators, studying professionals working alongside these systems, describe a jagged frontier: an invisible, irregular boundary between the tasks where the machine lifts your work and the tasks where it quietly ruins it, running through the middle of jobs rather than around them. Two observers, two vocabularies, one silhouette.

Picture it as a shape. A capable adult’s competence, plotted across the things they actually do, looks like a rounded form: higher here, lower there, but continuous, with edges that taper rather than plunge. The machine’s profile looks like a star drawn by a seismograph. Long spikes, deep notches, and no taper anywhere.

The observation is right, and I have no quarrel with it. I have honored it because I believe it. My question is upstream of it. Why that shape?

Why Human Profiles Are Smooth

It is worth noticing that human competence does not start smooth. Spend an afternoon with a bright child, and you will meet a profile as jagged as any chart of machine test scores: startling ability directly beside startling gaps, deployed with total confidence in both directions. What happens between the child and the adult is not that talent gets redistributed more evenly. It is that consequence starts arriving, and I mean the word broadly. The weather is not only embarrassment and cost. It is the plain friction of operating a body in a physical world, where every dropped cup and misjudged step reports back instantly and without mercy.

Everywhere an adult operates, reality pushes back. Overreach gets corrected, sometimes gently, sometimes in front of an audience. I met this early. My final year structural design project as a civil engineering student, an exhibition hall entered into a design competition, rested on massive tubular arch beams, and I had analyzed the whole structure in I-DEAS, a state-of-the-art structural analysis software running on Silicon Graphics workstations, which returned beautiful, confident results. The night before submission, for no reason I can fully reconstruct, I ran a quick back-of-the-napkin check on the stresses in those beams. The package had interpreted my loads as Newtons where I had meant kilograms, an error of nearly a factor of ten, and under the real loads my structure collapsed. Nothing inside the software had ever pushed back. The analysis was fluent all the way down. The correction came from outside it, arrived almost too late, and I have never trusted a smooth output the same way since.

I still run that loop on purpose, every week, in this newsletter. Every essay goes through a gauntlet of critics whose job is to find where I have overreached, and then it goes somewhere less forgiving: in front of you, under my name, where being wrong is a matter of record. What you are reading is a sanded object.

Under that kind of pushback, a person makes one of two moves, and both of them smooth the profile. Where the cost keeps arriving, and the work matters, you practice until the canyon fills. Where it keeps arriving, and the work can be routed around, you retreat and say so out loud, which is its own kind of competence. Practice fills the canyon. Humility fences it. Either way, the surface where you actually operate gets rounded off, and something else happens that matters just as much: you come away holding a map of your own edges. I know roughly where my canyons are. I paid to find out.

Smoothness, in other words, is not talent. It is erosion. A rounded capability profile is what any capability looks like after enough weather.

And let me not oversell the smoothness, because humans are jagged too, and anyone who has watched a brilliant surgeon manage money, or a physicist hold forth on politics, knows it. So the claim is narrower, and stronger for it. We do not get sanded everywhere. We get sanded where consequence keeps returning to us, and we stay jagged where it does not. Step outside the territory where you have paid, and the confidence often travels while the sanding does not. Which is exactly the point. The smoothing follows the loop, runs wherever the loop actually runs, and stops precisely where it stops.

Remove the Loop, Get the Star

Now build an intelligence with the weather turned off.

Training pushes capability wherever an objective can be specified and graded. That is not a criticism; it is close to a definition. Wherever the grader reaches, pressure is applied, and a spike grows. Wherever the grader cannot reach, because the goal cannot be written down cleanly, or the answer cannot be easily checked, nothing pushes back at all, and whatever canyon was there stays exactly as deep as it began. No cost ever arrives from the canyon floor, so nothing marks its location, not on any map the system holds and not on any map we hold either.

A fair objection arrives immediately, and it deserves a straight answer. Training is a feedback loop. These systems are corrected millions of times, penalized for wrong answers, adjusted after every failure. Is that not consequence? It is feedback, and here I need to be precise about the word I have been leaning on, because consequence, as I am using it, means something narrower than feedback. It means that the acting system itself, the same continuing agent, bears a cost that alters its own future behavior. Feedback that lands somewhere else does not qualify. And look at where this feedback lands. The corrections land on the next version of the system, applied by someone else’s process, in service of someone else’s goals, long after the fact. I made this argument two weeks ago about memory, and it holds here without modification: a loop that closes on your descendants is not a loop that closes on you.

You might answer that nature runs exactly this kind of loop. Evolution corrects across generations, and it produces beautifully rounded competence. True, but living things run two loops at once. The slow one closes across descendants. The fast one closes inside a single life: the animal that touches the fire pulls back, carries the burn, and approaches the fire differently tomorrow, the same animal, changed. The machine has been given only the slow half. Somebody will also point out that deployment is a loop of its own, that a system which fails badly enough gets updated. It does, and notice who bears the cost while that loop grinds around: the company, the engineers, the product team. The loop exists, but it closes on the people around the system rather than on the system that acted. And at the moment the machine actually acts, when it is answering you, you can push back all you like. The pushing does not stick. The acting happens in a room where nothing that occurs can permanently reshape the actor.

A month ago, in a shorter note, I described this shape as a straight line viewed from the wrong angle, and I want to soften that here into something more careful. From most directions the star looks like chaos, spikes and notches without pattern. But turn it until you are sighting along one particular seam and much of the shape resolves. The spikes largely trace where objectives could be specified and graded. The canyons tend to fall where they could not. The jaggedness is not random. It runs, in the main, along one seam, and the seam is the reach of the grader, drawn in silhouette.

To make the seam concrete, take two things these systems are asked to do. Writing code that passes a test is easy to grade. The test passes, or it does not, millions of times over, and the systems have grown a towering spike exactly there. Now consider knowing when to tell a client that a project should be canceled. No test exists. The answer arrives years later, tangled in everything else that happened, and nobody can score it at the moment it matters. There the canyon sits.

And here the machine’s jaggedness differs from the child’s in the way that should hold our attention. The child’s profile is jagged and getting sanded. The machine’s is jagged and unaware of it. It has no feel for its own edges. It descends into its canyons at the same even confidence it carries across its peaks, because no crossing has ever cost it anything, and nothing in its history distinguishes the two terrains. That, I suspect, is the sharpest edge of Mitchell’s unknowns. The cliffs are not merely unmapped. The system’s own manner gives you no warning that you are approaching one.

There is a rival explanation, and honesty requires me to put it in front of you at full strength. A researcher can point out that these systems are jagged for reasons that have nothing to do with consequence: the design of the systems themselves, the unevenness of what they were trained on, spikes where the material was rich and canyons where it was thin. Some canyons even sit on tasks that are easy to grade, which my seam, read strictly, should not allow. I take all of that seriously, and so the claim I am making is narrower than it may have sounded. The design and the data decide what a system can learn at all. My claim is about the rounding. In every intelligence we know, the process that takes a capability and wears it smooth, that corrects the overreach, fences the weak spot, and builds the map of edges, is consequence returning to the actor. That process has no counterpart here. Whatever first carved this star, nothing exists to sand it.

I should be plain about my ground before going further. I am not one of the scientists who study these systems from the inside, and I cannot tell you what is happening inside them, and whether the canyons follow patterns that rigorous work could map is an open question for people better equipped than I am. I argue from one mechanism I have spent a career watching in people and in organizations: what happens to capability when consequence does, or does not, return to the actor. If the researchers find a better explanation for the seam, I will read it gladly and say so here. And the claim can fail on its own terms. If someone builds a system that genuinely bears consequence, acting where its own errors return to it and carry forward, and its profile stays just as jagged, then I am wrong, and I will say so here too.

The Objection I Want to Agree With

There are two objections waiting for this essay, and both come from the tradition Mitchell works in, so I want to meet them directly and then agree with most of what they say.

The first has a long name and a simple meaning. Anthropomorphism is the old human habit of treating things as if they were people: naming the car, cursing the printer, reading feelings into anything that talks. With machines that speak fluent first-person English, the habit becomes almost irresistible, and Mitchell’s tradition is rightly on guard against it, because the danger is not the comparison itself; it is that the words do the convincing for us. Talk about a machine the way we talk about a person for long enough, and we start believing it has an inner life without ever having decided to.

The second objection is sharper. The philosopher Brian Cantwell Smith argues that these systems handle descriptions of the world without ever actually dealing with the world. He calls the missing thing registration, and the plainest way I can put it is this: a navigation app can tell you the bridge is out, but if it is wrong, you are the one in the river. Nothing is ever riding on the answer for the system that produced it. On his account, until something is riding on it, you do not have understanding. You have very good paperwork.

I concede both objections, and not through gritted teeth. Nothing in this essay claims the machine feels its canyons, regrets its spikes, or suffers its training. I have argued in this series that its design gives it no way to carry such experience forward even if it existed. The comparison between human smoothing and machine jaggedness runs in one direction only, and it points back at us.

Because the pattern I am describing never needed a mind. It needs a loop, or the absence of one, and loops are a property of systems, not souls. Last week I showed you a governing body whose profile has gone jagged in precisely this way: exquisitely capable wherever scrutiny grades it, and a canyon at the exact point where the grading cannot reach, which is to say wherever power can pick up a telephone. Sever an institution’s loop, and you get the same silhouette. The jagged star is not a portrait of the machine. It is what any capability looks like, in silicon or in bylaws, when nothing returns the cost of being wrong to the thing that acts.

The Question I Cannot Answer.

So: a memory that records without bearing, a record that binds everyone except the people holding the pen, and now a shape, the silhouette capability takes when the sanding never happens.

Which leaves one question hanging over all of it, and it is the question I will take up next week. If a closed loop is what does the smoothing, what exactly is the mechanism? Because in us it is not a metaphor. The mechanism that returns consequence to the decider is physical. It has an address in the tissue, a chemistry, and a strange recent history, because when we built learning machines we borrowed its central word without taking the parts the word referred to. That story is next week’s essay.

For today, I will end where this piece has been pointed from its first sentence. In her essay, Mitchell names what separates us from these systems: we are embodied, we are active seekers, and we carry a profound caring about the consequences of our actions. She offers those as a list, three items standing side by side. I have asked her once before, in a comment on her work, the question I will now ask in public, because it is her question to answer and I genuinely do not know what she will say.

Is consequence-bearing one more axis on the jagged star, one more item on the list? Or is it the candidate explanation for why the star is jagged at all?

Hero images: Glencoe photograph by Craig Hunter and mountain image by AVImages, both via Pixabay; third panel generated by the author.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

Last week, I left a question hanging. The ledger, it turns out, can be edited at both ends. What happens when the beings built to bear consequences start behaving like the systems that were built without them?

I expected to spend this week constructing a hypothetical. Instead, the World Cup handed me a case study, complete with a governing body, a phone call, and a scoreline.

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

The Call

Let me concede the obvious first. The red card shown to Folarin Balogun was probably a bad call. Balogun himself said a yellow would have been fair. Plenty of neutral observers agreed. Referees miss things, VAR misses things, and every team in every tournament has absorbed a bad decision and played on. If this essay were about officiating standards, it would be short, and someone else should write it.

That is not what this is about.

A suspension following a red card is not a punishment attached to a rule. It is the rule. It is the mechanism that gives the card any meaning at all. Take away the suspension, and the red card becomes theater: a referee raising a colored rectangle while everyone waits to learn whether it counts. And it has always counted. As outlets across the football press reported last week, this is the first time since 1962 that a red card shown at a World Cup has not produced a suspension. Article 27 had previously been stretched to cover cards earned in qualifying matches, but never for a card shown inside the tournament itself. And the lone precedent is instructive: Garrincha’s 1962 reprieve came after one South American president co-signed a petition and another personally called the referee. In sixty-four years, the record has been amended exactly twice, and both times a head of state was leaning on the pen.

This time, FIFA reached for Article 27 of its disciplinary code, suspended the ban, and placed the player on probation, not because new evidence emerged, and not because an independent review found error, but because someone powerful enough to make the call made the call. UEFA’s response was three words long, more or less: unprecedented, incomprehensible, unjustifiable. Even Sepp Blatter, a man who knows something about how FIFA bends, observed that red cards are overturned by rules, evidence, and independent bodies, not by telephone.

What An Institution Is For

Here is where last week’s vocabulary earns its keep.

I argued that an AI’s memory is a ledger rather than a scar: a record of what happened that carries no stake in what happened. The machine can consult its history, but the history costs it nothing, changes it in no way, and can be deleted without residue. I said the difference between a ledger and a scar is the difference between information and consequence.

Human institutions sit at a strange midpoint between those two things. An institution is a ledger by construction. It is rules written down, precedents recorded, decisions filed. Nothing about paper, bylaws, or disciplinary codes inherently binds anyone. But an institution that allocates responsibility, a court, a disciplinary committee, a governing body, is a ledger designed to behave like a scar. The entries are supposed to be permanent in the way that matters: the suspension is served, the penalty stands, the precedent constrains the next case, whether or not the next case is convenient. The design goal is to give a written record, one property writing does not naturally have, which is that you cannot take it back.

Now, an objection, and it deserves to be met head-on rather than waved past. Don’t good institutions revise their records all the time? Courts hear appeals. Science publishes corrections. Wrongful convictions get overturned, and we rightly count those reversals among our institutions’ finest moments. If permanence were the whole point, every appeal would be a corruption, and that is obviously wrong.

So let me put the claim more carefully. Institutions do not exist to make consequences permanent. They exist to make consequences credible. And credibility can survive correction. It can even be strengthened by it, provided the correction itself is governed by rules that bind everyone equally: transparent, independent, available to the weak on the same terms as the strong. I have lived the difference. Early in my consulting career, a signed contract turned out to rest on a wrong assumption, ours, and the honest fix nearly tripled the client’s budget. We put the mistake into the record rather than around it. We named the error as our own, documented the gap, and worked the revision through every approval the client’s process required. That client kept working with us for years afterward, and I am convinced the reason is that they watched us correct the ledger in daylight. The record changed, and it became more credible, not less.

That is the boundary, and it is the boundary FIFA crossed. What happened last week was not an appeal, because no appeal existed; the suspension was automatic and unappealable by design. There was no independent body, no hearing, no process; a less connected team could have invoked. There was a phone call, and then a citation. When Belgium’s federation asked FIFA to explain the decision, FIFA construed the inquiry as an appeal and declared it inadmissible within hours. I will leave you to judge what that sequence says about the channel.

Correction in daylight builds credibility. An amendment in private destroys it. The difference is not whether the record changed. It is who could change it, and who could not.

Where The Cost Went

The seductive story about an overturned consequence is that mercy is free. The card was harsh, the player was spared, the team kept its striker, no harm done. The books balance.

They do not balance. An overturned consequence is not an erased consequence. The foul was still committed, or at least ruled committed by the process everyone agreed to in advance. The cost of that event existed the moment it happened, and revoking the punishment does not undo the act. It only changes who pays.

So follow the money, so to speak. Watch where the cost actually landed.

Some of it landed lightly, on a player and a team who took the field under a question rather than a rule, though a cloud is a cheaper burden than a suspension, and I will not pretend otherwise. The US lost 4-1 and went home. I make no claim about what the controversy did to the scoreline; the honest answer is that nobody knows, and the argument does not need it. Had the US won by four instead, the costs that matter would still stand, because the cost being tracked here is the cloud, not the result. The high costs landed elsewhere. They landed on FIFA, whose disciplinary code has now been shown to allow exceptions unavailable to everyone, and every future ruling it issues will be read with that knowledge in mind. And they landed on every team that ever absorbed a red card quietly, played shorthanded, served the suspension, and trusted that the rule they were bearing was a rule and not a negotiation they were simply too weak to open.

The consequence was not canceled. It was relocated from the one party with the power to contest it to everyone else.

You may object that FIFA was never anyone’s model institution, and you would be right. But that is not a rebuttal. I am not interested in FIFA’s particular rot. I am interested in the mechanism because it is replicable anywhere the person holding the pen is also the person the record is meant to bind.

The Lesson The Powerful Learn

I have watched this mechanism up close for twenty-five years in technology consulting, and one project taught it to me permanently. My team was building a data platform for a large retailer. Every month, the lead architect and I prepared a status deck for the executive sponsor, and every month it included a risks slide naming the decision we believed would sink the program. Every month, the client contact who controlled the deck edited that slide down or cut it entirely before the meeting. When the platform failed exactly as the slide had predicted, the record showed that no warning had ever been raised. Thirty-five people on my team went to zero in two weeks, and the ledger, officially, stayed clean.

Notice what both stories, the stadium and the steering committee, have in common. The forms were followed. There was an article of the code, a probation, and a written justification. There was a deck, a meeting, and minutes. The ledger was not torn up. It was edited, with citations. And this is what separates 2026 from 1962. Garrincha’s reprieve was admitted as an exception; a president intervened, everyone knew it, and the record of the bending survived. This amendment was performed through the institution’s own procedures, as if it were routine. That is the difference between bending a rule and rewriting the rulebook to say it was never bent.

Here is the structural problem, and it is worth stating plainly because it is easy to mistake this for a story about one phone call. A consequence only shapes behavior if it is expected to hold. That expectation is the entire load-bearing element. The moment a powerful actor learns that the record can be amended, the amendment need not occur often. It only needs to be known to be possible. I cannot prove that these changes will affect anyone’s future behavior, and I will not pretend that the next red card won’t be served in the usual way. The point is that the possibility is now priced into every calculation that touches the rule. Uncertainty about enforcement is not a weaker deterrent than certainty; it is a different one, and it selects for exactly the actors most willing to test it. From that point forward, every rule divides into two: the version that binds people without leverage, and the version that opens negotiations for people with it. The institution keeps its letterhead and its procedures and its solemn vocabulary, but the thing the procedures were built to produce, the consequence that binds regardless of who you are, has quietly gone missing.

And notice exactly where it goes missing. Not at the margins. Consequence stops binding precisely at the top, which is precisely where its binding matters most, because the powerful are the ones whose decisions cascade onto everyone else. A rule that constrains only the constrainable is not a weak rule. It is an inverted one.

The Two Sites Of The Same Absence

So let me answer last week’s question directly.

I wrote that we built machines that remember without bearing, systems whose ledgers carry information but no stake. The machine, at least, has an excuse: no one gave it a loop to close. Nothing in its architecture was ever supposed to carry a cost forward, so its clean ledger is a design fact, not a failure. And let me be clear that the comparison runs in one direction. I am not claiming the machine is becoming institutional, or that it has credibility to lose. I am claiming that institutions are electing to become machine-like, a choice only they can make.

An institution has no such excuse. Its loop was built deliberately, at real expense, over decades, precisely because its founders understood that written rules mean nothing until someone arranges for them to be borne. When an institution lets its record become privately negotiable, it is not lacking a mechanism. It is dismantling one. And let me be precise about who dismantles it, because the people holding the pen are not indifferent the way a machine is indifferent. They edit the ledger because they have stakes of their own: favor to curry, revenue to protect, a call they would rather not refuse. The institutional stake is dismantled precisely so that a private one can be served. The result, though, is the same shape from the outside. Entry by entry, the institution becomes the thing the machine cannot help but be: a system that holds the history without holding the stake.

That is the pattern I asked you to watch for. Beings and bodies built to bear consequence, behaving like systems built without it. The machine’s absence is architectural. Ours is elective. I honestly do not know which should worry us more, but I know which one we can still do something about.

One more thing before I let this go. When an intelligence develops without consequence ever binding it, the absence does not stay hidden. It leaves a mark on the shape of the capability itself, a signature you can learn to read. That signature is next week’s subject.

For now, I cannot answer the question for you. Think of the institutions you live inside, the ones whose rules structure your work, your money, your rights. For each one, ask two things. When was the last time its record was held against someone powerful enough to contest it? And when it was last corrected, would the correction have been available to you? If you cannot answer either, what exactly are you trusting?

Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.