James Maconochie | Architecture & Attention Podcast

James Maconochie | Architecture & Attention Podcast

By James MaconochieSociety & CultureTechnology
Download on the App Store

All James Maconochie | Architecture & Attention Podcast episodes

  • You Have Already Run This Experiment

    Think Back To the Last Time You Caught One Out

    You asked an AI something. It answered immediately, fluently, and with complete confidence. And on at least one occasion, it told you something you happened to know was wrong.

    Not obviously wrong. Wrong in a way that would have sailed past almost anybody. Wrong about a detail in your own field, or your own town, or the thing you have been doing for twenty years.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

    You caught it. Most people file that away as a funny story about the technology. It is actually a result, and the interesting part is not about the machine. It is about you.

    Two Groups, One Tool

    Researchers gave students an AI assistant to research a scientific question and compared their reasoning with what they produced without it.

    Both groups were given the same task. Only one group had been trained in the subject.

    Same task, same tool, opposite direction. The rule underneath is the finding. The boxes are only its evidence.

    Stadler, Bannert and Sailer, 2024, with a 2026 preprint adding domain expertise as the moderating variable. The follow-up is a preprint and has not yet been peer-reviewed. The study reports which way each group moved rather than the size of the effect, so neither do I.

    And It Felt Easier

    The earlier study in the same line of work found something more uncomfortable than a drop in quality. The students using the assistant reported a lower mental workload while producing worse reasoning.

    It felt easier at exactly the moment it was going wrong. Which is why nobody catches this from the inside.

    So What Was the One Group Carrying?

    Not more intelligence. Not more effort. They already knew the subject well enough to argue with what came back. They could tell which claims to be suspicious of, which numbers smelled wrong, what had quietly been left out.

    That capacity has a name. It is judgment, and it is built the only way it can be built: by making calls with something riding on them, and then finding out. The wrong ones show you where you personally tend to go wrong. The right ones tell you what your judgment is actually worth. Neither lesson arrives if nothing was at stake, which is why it takes a career and never quite finishes.

    The debugging at eleven at night. The hundredth presentation of the same condition. The case that collapsed on the one witness you had not prepared.

    Nothing A Model Does Costs It Anything

    A model has read every account of being right and every account of being wrong that anyone has written down. That is not a small thing. It is why the answers are as good as they are.

    What it has not done is live through any of it. When it is wrong, nothing happens to it. No consequence arrives, so nothing in it updates.

    That is not a complaint about the technology. It is a description of what it is, and it has one straightforward implication.

    The judging has to stay with people.

    Which Would Be Fine, Except For Two Things

    Nobody can see whether it happened. Two people review the same document and reach the same conclusion. One thought about it. The other agreed with it. Every record of those two decisions is identical, and nothing any organization currently inspects can tell them apart. Not because anyone hid anything. Because there is no field for it.

    A great deal of money assumes the problem solves itself. Hundreds of billions of dollars are committed to an argument whose first step is the assumption that scaling handles this. If it doesn't, the cost doesn't stay with the people who made the bet. It arrives in retirement accounts whose contents nobody chose, in towns that borrowed against the building, and in jobs that quietly aren't refilled.

    So What Would You Build Instead?

    Here is the distinction that changes the question. A tool that replaces you has to do the whole job. Every part of it, at least as well as you did, or what you end up with is worse than what you had.

    A tool that amplifies you only has to do one part: the part where you were actually stuck. The rest stays with you, because you are still there doing it.

    The same five functions, two designs. The left column has to be built from nothing. The right column is mostly already there.

    Replacement is five problems, and all five are unsolved. Amplification is one, and it happens to be the one that was slow anyway.

    One Question To Take Away

    Whatever you do next, this is the one worth asking about any tool you are using seriously:

    After a year of this, what can I do that I could not do before?

    If the honest answer is “produce more of what I used to produce”, the tool widened your throughput. If it is “notice things I used to miss”, it widened you.

    Those are different products. Nearly everything on the market is sold on the first. That is a choice somebody made, not a fact about the technology.

    Where To Go From Here

    Everything above is asserted. None of it is argued out here. Each claim gets made properly in one of six short pieces, with the evidence, sources, and unanswered objections. They are at jamesmaconochie.com/companions.

    * Why Does It All Feel Like Too Much? Language got cheap to make. Attention did not.

    * Why Can’t You Just Be Told? The study above, and why judgment cannot be transferred in text.

    * What Holds a Claim Up? Four checks for whether a claim can bear weight.

    * How Would You Know? The two reviewers, and why no inspection anybody runs can separate them.

    * Who Pays If It’s Wrong? The argument carrying the money, and where the cost lands if it fails.

    * What Would You Build Instead? The constructive half: five principles and a working prototype.

    Each one is short, stands entirely on its own, and needs no technical background. They are the plain-language editions of six working papers, which are on the same site if you want the full argument with its footnotes and its limitations.

    This newsletter comes out every other week, on one subject approached from different angles: what judgment actually is, why it has to be carried by people who pay for being wrong, and what that means for building with machines that pay for nothing.

    Working papers are revised when they are shown to be wrong, and what changed is kept on the record. If something here is wrong, I would rather know than be told I am right.

    Humilitas, Humanitas, et Curiositas · SemperHumility, humanity, and curiosity. Always.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com
    9 min
  • The Hydraulic Jump

    In 1991, in the hydraulics lab at Imperial College London, Dr. Hardwick sent water down a long, clear-sided flume and told us to watch.

    The water came in fast and shallow, three or four inches deep. Then, partway down the flume, it stood up. Over a short stretch it rose to at least twice its depth, churned white in a standing roll, and carried on slowly, deep and calm, as if nothing much had happened.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

    It’s called a hydraulic jump. I have forgotten nearly all the equations that describe it. I have not forgotten what it was like to watch. If I were of a religious disposition, I might have called it a small miracle: the same water in the same channel, yet something quite different on the other side of a line you could point to.

    More than thirty years later, I think about that channel more than I expected to, because it describes where I think AI is now. For years, we have widened the channel and pushed more water down it. The flow gets faster and more impressive. What I keep looking for is the jump, and I don’t think it comes from upstream.

    What Actually Happens

    Water in an open channel can flow in two quite different ways.

    In the first, it is fast and shallow. In the second, it is slow and deep. The dividing line between them has to do with ripples. Drop a pebble into still water and the ripples spread in every direction at a speed set by the depth. In slow, deep flow, a ripple can still travel upstream against the current. In fast, shallow flow, it can’t: the water moves faster than any ripple can travel, so everything gets swept downstream.

    That has an odd consequence. In the fast section, nothing upstream can sense what is happening farther downstream. The water has no way of knowing what is waiting for it.

    When fast, shallow water meets slow, deep water, it can’t transition gradually. So it changes all at once. That is the jump.

    If you’d like to see one, here is a short clip.

    Three things are worth noticing about it.

    First, the amount of water stays the same. The same volume passes every second before and after the jump; it simply trades speed for depth.

    Second, the jump costs energy. The churning roll at its face turns much of the water’s speed into turbulence and, in the end, a little heat. That is why engineers build hydraulic jumps on purpose at the foot of dam spillways: fast water would scour away the riverbed, and the jump calms it before it gets there.

    Third, the jump depends on what is downstream. Upstream changes alter its strength and where it sits, but without enough depth ahead of it, the fast water simply runs on.

    The Upstream Argument

    For most of the past decade, AI progress has come mainly from upstream: bigger models, more data, more computing power. Each round has made the flow faster and more impressive, and I don’t want to understate that. The systems we have now write, summarize, translate and code at a speed and fluency that would have seemed far-fetched not long ago.

    But in the flume, more flow doesn’t change the kind of flow you have. A wider channel carrying more fast, shallow water is still fast, shallow water. It still can’t hear what is waiting for it downstream, because that is a property of how it is moving, not of how much there is.

    I should be careful about what I’m claiming. Few serious people think judgment is simply more fluency, and many see scale as one ingredient alongside tools, memory, and evaluation. My point is narrower. Better answers, measured at the moment they are produced, don’t by themselves show that a system has learned from what happened after it produced them. Those are different capabilities, and progress in the first is not evidence of the second.

    None of this means scaling has been wasted, or that it will stop producing gains. More speed, more reach, more fluency: those are real and useful. They are just not the same thing as depth.

    Where the Jump Comes From

    So where does the jump come from? In the flume, the answer is downstream. The fast water meets deeper water ahead of it, and at that point it has no choice but to change.

    My argument across this series is that judgment works similarly, and it’s worth being precise about what I mean, because three different things tend to blur together.

    The first is feedback: information about how a decision turned out. The second is learning: that information changing the next decision. The third is cost: the outcome landing on the one who made the decision. Feedback is easy to come by. Learning is harder. My claim is that the third is what makes the second reliable. When a mistake costs you something, the lesson is hard to set aside, hard to explain away, and it competes with convenience on equal terms. When the cost lands on someone else, the same information arrives, but it’s easier to discount, delay, or forget.

    A surgeon who has lost a patient, an engineer whose design has failed, a manager whose confident forecast fell apart in front of the board: each has the information, and each also carries the cost, and that is why the lesson tends to stay. That combination is what I mean by consequence-bearing, and it is the downstream water.

    Current AI systems have a downstream. Their outputs go out into the world and have effects, some good, some bad. What they mostly lack is contact with it. When a confident answer turns out to be wrong, the cost lands on the person who relied on it, the organization that deployed it, or someone further along who never knew a machine was involved. The system that produced the answer rarely carries that particular mistake into its next decision. The information exists, but like a ripple in fast water, it never travels back to the source.

    One partial exception is worth naming. Some systems now work in loops: write code, run it, see it fail, fix it. That is real contact with a downstream, and it does change what happens next. But the downstream is a test someone specified in advance. It tells the system whether the code ran, not whether it should have been written. It resembles the flume more than the river.

    There is broader feedback, of course. Users rate answers, developers retrain models, companies face reputational and legal costs. But that feedback is gathered in aggregate, applied later, by people who didn’t make the decision, to a system that didn’t pay for it. It fails both tests at once: the lesson isn’t tied to the decision, and the decider doesn’t bear the cost. It is closer to someone reshaping the upstream channel from the bank than to the fast water meeting the deep water ahead of it.

    In other words, the jump isn’t waiting further upstream. It needs something the current design routes around, and has little commercial reason not to. See “The Better Design Waits for the Rule.”

    The Energy Loss Is the Point

    The flume teaches one more thing, and it runs against the way we usually talk about progress.

    The jump is wasteful by any measure of speed, and engineers build it anyway, because the slow, deep water is what the riverbed can live with.

    Judgment has a similar cost, though not because slowness is a virtue. Experienced people are often faster than novices, because they know what to ignore. The cost is in the checks they won’t skip: stopping before signing off, doubting a confident answer, asking what happens if this is wrong. Friction in the wrong place is just delay. Friction in the right place, against the failures that matter, is much of what experience buys. From the outside, it can look like hesitation; often it is the point.

    In September I argued that AI has made producing judgments cheap without making checking them cheap. The flume puts that another way. Speed is easy to add upstream. Depth costs something, and the cost has to be paid where the consequences are.

    That is also why I’m wary of measuring progress in AI mainly by how fast and how fluently it answers. Those measures reward the flow before the jump. They can…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com
    15 min
  • Fifty-Two Weeks

    When I started this series, I made myself a promise: an essay every week. I have kept it every week but one. That one, I wrote and thought I had posted from a few days at the beach, but when I got home, I realized I'd missed the last step. This is the fifty-third essay. The first fifty-two went out in just under eleven months.

    From here, I am going to publish every two weeks. The next essay will arrive on October 27, and every second Tuesday after that.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

    I have plenty to say, and I am not stopping. But I wrote one recent essay the day before it was due, and the next on the day itself. That told me something I had been avoiding.

    When I began, weekly was a decision. I made it on purpose, for reasons I could have explained to you. At some point it stopped being a decision and became simply what I do each week. I hadn't asked whether it still fitted my life since the day I made it.

    This essay is me asking. It turns out the weekly essay isn't the only routine I stopped examining, and it isn't the one that cost me most.

    Why Weekly

    When I began, writing was the whole job. I had spent twenty-five years in technology consulting, and I had already written several long working papers on how I think artificial intelligence should be built and used. The material was there. What I lacked was a track record as a writer.

    So I set a rule: one essay a week, each one good enough that I was happy to stand behind it. My thinking was simple. A new writer has no record, and the only way to build one is to turn up on time, again and again, until people can see what you are about. A weekly essay was how I would lay a base.

    I still think that was the right call for the conditions. I had the time, and I had plenty of material. The discipline turned a stack of papers into fifty-two essays that someone can read from the start. It also pushed me, week by week, to write more plainly for readers who do not work in technology. I am still working at that.

    But the rule was made for those conditions. The conditions changed, and I left the rule as it was.

    What Went, and in What Order

    Start with the walking.

    In 2013, my doctor told me I was pre-diabetic. I started measuring my blood sugar, and I started walking every day. Within a year, I had lost 60 lbs. By then, the walking was no longer a treatment. It was my time: anywhere from 4 to 14 miles a day, in any weather, with music or a book in my ears. For ten years I was up at four or half past to fit it in before work. By my count I have walked once round the Earth and started a second lap.

    I kept it up for most of twelve years, through the year I was out of work and looking, and into the months after, when I began my research and writing. Then, in the autumn of 2025, I slowed down. Some months later I had more or less stopped. The slowing began before I had published an essay here, and well before the driving.

    No single thing ended it, and time wasn't the reason. I began to question a lot I had taken for granted, and the walking got caught up in that. I had lost sight of why I started. The benefits had been there so long that I doubted walking caused them. And, if I am honest, part of it became about how I looked and what other people thought of me. So I let it go, and I am worse off for it. The books went too, because I listened while I walked. I am still 30 lbs lighter than at my worst, but I am not where I was.

    Five months ago, I started driving for Uber, 20 to 25 hours a week. I had promised my wife that I would bring in income and keep writing, and this was how. For a while, the essays didn't suffer, because I had plenty stored up. But something had to give. A month or two ago it was the time I spent reading and replying to other writers here. In the last few weeks it has been the essay itself, squeezed into the final day.

    The plain account of that is not very interesting. I took on more than twenty hours of driving a week and expected everything else to fit around it. It did not. What interests me is that it took me about five months to ask whether the weekly essay still made sense.

    Two Standing Answers

    In The Fasten Your Seatbelts Century — Again, I wrote about living with a seatbelt sign that never goes off. You can ignore it, or you can stay white-knuckled. Either way, you have stopped asking whether this particular warning means anything. I called those settings, not decisions. A decision looks at the conditions. A setting was once a decision, and now runs without anyone checking it.

    I have been running two of them.

    The walking was a setting that was good for me. It began as a decision, driven by a doctor’s warning and a blood sugar reading. Over the years, the reason faded, and only the habit remained. When I finally questioned it, I had lost sight of what it was for, so I had nothing to defend it with. I dropped a good routine because I had lost its reason.

    The weekly essay was a setting that no longer fit. It also began as a decision, for the reasons I gave above. Then the driving filled my weeks, and I did not ask whether a rule made for open weeks still suited full ones. I kept a routine because I never went back to its reason.

    Those look like opposite mistakes, and the causes were different. The walking faded through apathy. The weekly essay stopped fitting because my time changed. But I think the mistake underneath is the same one. In both cases, I had the routine and had lost track of why.

    I do not think routines are the problem. Twelve years of walking happened because I did not have to decide each morning. The trouble starts when the reason gets obscured by the mist of life, because then I cannot tell a routine worth keeping from one that needs changing.

    Is this adjusting, or giving up? Fifty-two essays show that I can keep going. They do not show that slowing down is right. What tells me this is an adjustment is that the conditions changed. The weeks I made the rule for are not the weeks I have now.

    What Changes for You

    An essay will arrive every second Tuesday, starting October 27.

    In the weeks between, I will go back to reading and replying to other writers here. I stopped a month or two ago, when the driving squeezed it out. Many of the ideas in these essays began as a comment on someone else’s work, or as an objection someone raised to mine. I do not want to write without listening.

    The walking is a separate matter. It is not for the weeks between. It is for every week. I want to be careful here. A lighter schedule will not put me back on the road, because time was never the reason I stopped. The change gives me room. The rest is up to me. I am saying so here, in front of you, because a promise made in public is harder to let slide quietly. I know I am saying that in the same essay in which I change an earlier public promise. The difference, I would claim, is that I am changing that one in the open and telling you why. With the walking come the books, and the books feed the writing.

    I cannot promise the essays will be better for being fewer. I do intend that I will not write them on the day they are due.

    What I Do Not Know

    I do not know that every two weeks is right either. It is my best reading of the conditions I am in now: the driving, the walking I have started again, and the conversations I want to get back to. Those will change again.

    So this time I am writing the reason down. That is partly what this essay is. When I next wonder why I publish every other Tuesday, the answer will be here, dated, and I can check whether it still holds. I mean to do that in the spring, after a dozen essays at the new pace.

    Fifty-two essays showed me I can hold to a routine. The walking showed me that holding to one is not the same as remembering what it is for. I would like to manage both.

    Thank you for reading the first fifty-two. The next one is on Octo…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com
    10 min
  • The Batting Average

    Some years ago, while consulting, I spent many hours in a car with a younger colleague. Our client was three or four hours away, and for most of two projects we made the drive nearly every week. That is a great deal of time to talk.

    She was early in her career, bright and ambitious, and good at the work. From time to time the conversation came round to careers, and it was plain she was frustrated with how slowly hers was moving.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

    I should say at once that I knew the feeling. I had it many times in my own career, and I am not sure I ever entirely lost it.

    So I tried to encourage her. I pointed to what I had watched happen over those two projects: the progress she had made, and the trust and respect she had earned from the client and from our own team. I told her that the frustration was common, and that measuring yourself against people you believe are at your level is a reliable way to stay unhappy.

    I said something like this more than once. If you have ten goals on an engagement, and you achieve six of them, you have not failed. You have six wins.

    I did not think it landed. She heard me, she was gracious about it, and as far as I could see, the frustration stayed where it was. In her accounting, as I read it then, anything short of ten was a loss.

    I have thought about those drives often since, because over a consulting career that ran from 1998 to 2023 I met that accounting more and more often, and found it harder and harder to argue with.

    No Running Total

    Baseball has a number for this. A hitter’s batting average is simply how often he gets a hit. One who manages it three times in ten is among the best in the game, which means the best hitters fail far more often than they succeed. Nobody watching thinks of them as failures, because nobody judges a hitter on a single trip to the plate. The average only exists because someone keeps the record.

    That is what I was trying to offer my colleague on those drives, though I did not have the words for it then. I was asking her to keep a ledger: a running record of what she had attempted and how it turned out, read as a whole. Six of ten, on a ledger, is a good season.

    What I was arguing against has no ledger. Each goal, each review, each argument is scored as a win or a loss on its own, and then the score is thrown away. Nothing accumulates. With no running total, the only result that feels safe is a perfect one, and anything less reads as defeat.

    I think this is one way absolutism gets made. We usually describe it as stubbornness or certainty, and sometimes that is all it is. There are also matters of principle on which people are right not to settle for six of ten. But much of the all-or-nothing I met was neither. I have come to see that part as a bookkeeping problem. It is judgment with no record behind it.

    In August, in The Ledger We Refused to Read, I wrote about a government that had a record of its own past judgments and, as far as we were shown, did not consult it. This is the same failure at the scale of one person. The government had a ledger and left it closed. The absolutist never started one.

    What Gets Counted

    I said I met this accounting more often as the years went on. I should be careful about what that is worth. It is one consultant’s impression, formed over twenty-five years of working alongside people younger than me, and I have no study to back it up. But I do not think I imagined it, and I think I know one place it is taught.

    It is taught at work. In several companies I worked for, the review process ran the same way. Each year started more or less from zero. What counted was this year, and often the last few months of it. The result that mattered was a yes or a no on promotion, so six strong achievements and no promotion registered as a no. And where places were limited, the firm itself measured you against the people at your level. I told my colleague not to compare herself with her peers. The system she worked in may have been doing it for her.

    None of these firms lacked records. The reviews were on file, year after year. I am not aware that anyone read them as a whole. It would have taken too long. Whether that was oversight or convenience, I cannot say.

    The same accounting runs well outside the office. In politics, on television, online, a deal that secures most of what a side wanted is reported as a surrender, and conceding a point is treated as losing the exchange. I cannot prove what caused this. I can point to something that would reinforce it.

    Much of public conversation now happens on platforms that keep score, and what they count matters. They count reactions: how many people responded to this post, this clip, this reply, today. Nobody’s page shows how their judgments held up over five years. These platforms do keep a running total. It is a total of attention.

    So the scoreboards most of us look at every day, at work and online, reward the recent and the visible. Neither has a place to show how your judgment held up over the years. If you wanted to teach people to score each exchange alone and discard the result, I am not sure you could design a better classroom.

    Where The Frame Could Break

    This argument is weaker in at least three places than I have made it sound.

    First, a batting average treats every attempt as equal, and decisions are not equal. I trained as a civil engineer. Nobody wants a bridge designer who gets six out of ten right.

    In September, in Who Is Left Holding the Napkin, I described a model that scored every bridge on two things: its condition, and the consequence if it failed. A ledger of judgment needs that second column. Counting six of ten as a win is sound advice for a career, an argument, or a negotiation. It is dangerous advice anywhere one error is permanent, because the people who pay for it do not get another turn.

    It is also the first question I would ask of any AI system offered to me on the strength of its average: is this use a negotiation, or a bridge?

    The second concerns my colleague, and I owe her this. I told her to temper her expectations. Her expectations may have been reasonable, and the pace may have been the problem. Capable people early in their careers are sometimes held back for reasons that have nothing to do with their outlook, and “be patient, count your wins” is convenient advice for an employer to have in circulation. I believed what I told her, and I still think it was true. But I was a senior colleague describing the view from further along, and I cannot rule out that some of her frustration was an accurate reading of her situation.

    The third is the hardest. A ledger you keep for yourself does not change the one your employer keeps. If the firm counts only this year, and only yes or no, your private record of steady progress settles nothing at review time. I do not have a full answer to that. Companies are the way they are, and short of starting your own, there is little you can do about how they count. I wrote last week, in The Better Design Waits for the Rule, about why a better arrangement everyone can see still doesn't happen until the rule changes for everyone at once. Review systems are no different. This is the individual’s side of that problem.

    What you can change is how much their count weighs with you. I am not the person to preach this. For most of my life, I have wanted other people to confirm my worth, and I have been disappointed by promotions, raises, bonuses, and awards more often than I would like to admit. It is less than it was. It has not gone. But I have slowly learned to give my own opinion, and my family’s, at least as much weight as an employer’s. That only works if my own opinion is worth trusting, and an honest record is what makes it so.

    None of these undoes the argument. All three narrow it. Keeping a ledger is no reason to accept whatever you are given. It lets you tell the difference between a bad seas…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com
    12 min
  • The Better Design Waits for the Rule

    The Model That Changed Nothing

    Around 2010, I worked on a model that scored every bridge in the National Bridge Inventory, the federal record of highway bridges on America’s public roads, on two things at once: what condition it was in, and what it would cost if it failed. A lightly used rural bridge in poor condition and a busy interstate crossing in fair condition carry very different risks. Yet most bridges were inspected on the same two-year cycle.

    I presented the model at the Federal Highway Administration (FHWA) headquarters in Washington, alongside colleagues from Lloyd’s Register. In the room were the Associate Administrator for Infrastructure and several of his colleagues from the bridge side. He was supportive, but he made no decision and no commitment. Bridge leads at state transportation departments gave it the same reception. We had linked the model to Google Earth, so we could fly over each state’s highest-risk bridges. None of the state bridge engineers who saw it disagreed with what it showed. There was real interest, and then nothing.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

    The inspection program did not change. Not then.

    The information existed, the people closest to the problem found it credible, and it still did not turn into action. I have come to think this is not unusual. It is one of the common ways a better design loses.

    This series argues that the answer to AI’s judgment problem is design: build systems so the people who bear the consequences of a decision stay part of making it. If that is right, there is an uncomfortable follow-up. Why does the design answer so often lose in practice?

    The bridge model is my best evidence of how it happens, because it happened in front of me.

    Two Columns

    The usual reason a better design loses is cost. Doing things properly takes more money up front, and budgets are set in the present. That is often a fair objection, and it would be dishonest to pretend the design answer is always cheap.

    The bridge model removed that excuse. It worked from the data states were already collecting for the inventory. Nothing new had to be gathered. Rolling it out wouldn't have been free, but it was modest compared with the size of the program.

    The benefits were real, but they took a different form. Few people would argue that a simple concrete beam bridge, under twenty years old and used mostly by farm vehicles, needs the same attention as a major interstate crossing. Inspecting such bridges less often would free up effort for better inspection and targeted maintenance on the large, busy bridges where failure matters most. Those bridges are costly to maintain, but replacing one costs far more. If that attention added five or ten years to the life of a major crossing, the savings could be large. The biggest benefit of all is a failure that never happens.

    That is the problem. A failure that never happens leaves no record. No one can point to the bridge that did not close, or the replacement that was pushed back a decade because a problem was caught early. The benefits go in the ledger as estimates about events that, if the design works, will never occur.

    The other column, it turns out, was not money. It was the risk of change itself, and who would carry it. That is where the rest of this essay goes.

    Tonie-Marie Gordon, writing about AI at work, describes the same accounting error from the other direction. The productivity gain from a new tool gets booked in full, while the human effort needed to check its output is left off the ledger. Either way, what gets counted tends to win.

    The Number and the Money

    At the time, the inspection data did more than describe bridges. It helped decide how much federal money each state received for them. Under the Highway Bridge Program, a state’s share of the funding was based on the estimated cost of repairing or replacing its deficient bridges, those formally rated as in poor condition or outdated for today’s traffic, calculated from the inventory.

    That made any change to how bridges were inspected more than a technical question. The same data that recorded a bridge’s condition also fed the formula that divided up the money. A state’s bridge staff might find a risk model persuasive and still have good reason to be cautious. If the new approach changed what was collected, or how it was read, it might change what the state received. Nobody would want to be the one who found out the hard way. This is not a claim that anyone gamed the numbers. The caution was reasonable, given how the money worked.

    A slow federal process added to the caution. Changes to a national program move through years of rules, comments, and consultation, and a program that seems to be working invites the natural response: if it isn’t broken, don’t fix it. The trouble is that “working” meant the inspections got done and the numbers came in. It did not mean the effort was going where it did the most good.

    Once a number decides where money goes, protecting that number can become a goal in itself. The Congressional Research Service noticed a version of this in 2008, observing that a formula paying states according to their deficient bridges gave them little reason to reduce the count.

    Through her reporting, Rana Foroohar has traced the same pattern across the wider economy. When firms were judged on financial figures, managers came to steer by the figures rather than by the work they measured. The figures were persuasive, and too few people asked what they left out.

    In this case, the better design did not lose on cost. I cannot prove why it lost. I believe it lost at least in part because it threatened the numbers the money depended on.

    Not Just Bridges

    The bridge model is the clearest case I have, but I have seen versions of it in technology consulting.

    One project stays with me. I worked with a large software company on how its sales system handled leads: how a first contact became a qualified lead, and then a real prospect. Marketing was judged on how many leads it passed along, and sales on how many it turned into customers. Sales believed marketing was passing along weak leads to boost its numbers. Those leads rarely went anywhere, so the sales team wasted time chasing them and looked worse when they failed to convert.

    Nearly everyone recognized the problem when we laid it out. Fixing it meant changing how both teams were measured, at the same time. Our engagement ended with recommendations to do that, and an expectation of follow-on work to implement them. That work never came. As far as I know, the company kept things as they were.

    The rule there was not a federal regulation but a pair of targets. The pattern was the same: a better arrangement everyone could see, and no one in a position to change the rule for both sides at once.

    The Asset Nobody Counts

    One more thing is missing from both columns, and it matters most for AI.

    Most organizations depend on judgment that lives in their people rather than their procedures. It shows up in the inspector who knows which joint on which bridge to look at twice, or the sales manager who can tell a real lead from a hopeful one within a few minutes on the phone. Little of it appears in a budget. You learn it through practice, pass it on informally, and notice it mainly when it is gone. I think of it as cognitive collateral: the capacity an organization relies on to catch its own mistakes, which never shows up on its balance sheet. If offloading judgment is a kind of borrowing, cognitive debt is what accrues, and cognitive collateral is what is lost when the debt comes due and cannot be paid.

    AI tools put this capacity under pressure in a particular way. When a system does the routine work, people get less practice at it. In 1983, the engineering psychologist Lisanne Bainbridge pointed out the irony: the skills that fade during smooth automated operation are the ones needed when the automation fails.

    A design that…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com
    15 min
  • The Fasten Your Seatbelts Century - Again

    I turned on my own seatbelt sign about four years ago.

    I had been reading the same headlines as everyone else, and I reached a conclusion I found hard to argue with: too many things could go wrong at once, and most of them would hit the markets before they hit anywhere else. So I moved a good part of my retirement savings somewhere I judged safer. It wasn’t a trade. It was a posture. I buckled in and waited.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

    Nothing happened.

    Four years on, the sign is still lit. Recently I made a list of the warnings that keep it lit, and introduced them with the phrase “only a couple of blinking warnings that I’m aware of.” Here is the list.

    * Government debt that nobody has a plan to repay ($40T)

    * A bond market that hasn’t settled

    * Share prices that assume nothing goes wrong

    * A technology boom that is partly financing itself (AI)

    * Whatever that boom does to jobs

    * A war with no sign of ending (Ukraine)

    * A standoff that could become another (reader’s choice - US/Iran, China/Taiwan, North Korea/South Korea)

    Then I counted. There were seven.

    That gap between “a couple” and seven is what this essay is about. It isn’t evidence that things are worse than I thought. It might be evidence that I’ve stopped being able to read my own instruments.

    Here is what I’ve done since. I’ve quietly moved some of that money back. Not all of it. Enough to notice.

    I don’t know whether that was judgment or fatigue.

    The Sign

    On a plane, the seatbelt sign works because of three things you rarely think about.

    The first is that someone with better information than you switches it on. The captain can see the weather radar and hear reports from aircraft further ahead. You can see a cloud.

    The second is that the captain is on the plane. If the turbulence turns out worse than expected, the captain feels it too. That doesn’t make the captain infallible. It means the person switching the sign on pays when they get it wrong, which is why you can obey it without being told the reason.

    The third is the one this essay depends on: the sign goes off. On a smooth flight it stays dark most of the time, so when the chime sounds, it means something. The sign carries information because it isn’t always on.

    Take away the first two and the sign becomes harder to trust. Take away the third, and it becomes part of the cabin.

    The Last Time the Sign Was On

    First, what this essay isn’t claiming. The twentieth century was the bumpy one: two world wars, a depression, a pandemic that killed tens of millions, and four decades in which two governments could have ended most life on earth in an afternoon. If a century ever had its seatbelt sign on, that one did.

    Many of us in the West who came of age around the end of the Cold War believed it had gone off. Since the autumn of 2001 it has been on for most of the time: two long wars, a financial crisis that came close to freezing the banking system, a pandemic that closed much of the world, inflation back after a generation away, war in Europe and the Middle East, and political turmoil in democracies long thought stable.

    That is a selective list. The same years brought falling extreme poverty and longer lives, and I’m not claiming the world got worse. I’m describing how the sign has behaved. What stands out is how short the quiet stretches have been.

    When the Alarm Never Stops

    Hospitals have studied what happens when a warning never turns off, because there the cost is counted in patients.

    A patient on a modern ward can be connected to several monitors at once, tracking heart rhythm, oxygen, and blood pressure, each set to sound when a number crosses a line. Each alarm makes sense on its own. Together they can produce several hundred alarm signals per patient per day. The Joint Commission, the body that accredits most American hospitals, reviewed the research and found that between 85 and 99 percent of those alarms required no action.

    Nurses adapt, as anyone would. They learn which beeps usually mean a loose sensor or a patient who shifted in bed. They turn the volume down, widen the limits, and sometimes switch alarms off. The field calls this alarm fatigue.

    In 2013, the Commission issued a formal warning to hospitals about it. Over about three and a half years, it had received reports of 98 alarm-related events, 80 of which ended in a patient’s death. It noted that reporting is voluntary and that these cases are only a small share of what actually happens.

    What matters here is how the failure happens. It is tempting to call it carelessness. It isn’t. A nurse who stops reacting is usually responding sensibly to what the alarms have taught them. When nine alarms in ten mean nothing, treating the next one as probably nothing is a reasonable bet. The trouble is that habit places the bet, not judgment, and the tenth alarm sounds exactly like the other nine.

    The alarm didn’t stop working. It stopped carrying information. The sound stayed the same, but its meaning drained away.

    Maybe the Air Is Rough

    There is a fair objection here. Hospital alarms can be scored: afterward, we know which ones needed a nurse. The warnings on my list can’t be scored that way. Some take decades to resolve, and some never will. By a warning I mean a claim that something bad has become more likely and that you should act differently because of it. This difference has a name. Risk is what can be measured; uncertainty is what can’t. Hospital alarms are risk. Most of my list is uncertainty. Economists argue over some of it with better tools than mine, but nobody can yet say which of my seven are right.

    So maybe the air really is rough, and the sign is on because it should be.

    The argument doesn’t depend on ruling that out. Even if the sign is right, a sign that is always on can’t tell you when things get worse. Pilots already deal with this. When turbulence is expected to be severe, the captain adds a second announcement, “flight attendants, please take your seats,” because the seatbelt sign alone no longer marks the difference. In steady rough air, the useful signal is the one that separates this bump from the others.

    That second announcement is what I think we’re missing: a way to tell which of seven lit signs deserves more attention than the rest.

    Unbuckled or White-Knuckled

    Without that second signal, people tend to go one of two ways.

    The first is to unbuckle. After enough warnings that go nowhere, you stop listening. Headlines about debt, war, or the next crisis become background, like the hum of the engines. This feels like calm, and sometimes it is. But it is the nurse’s bet: the next warning is probably nothing. Most of the time that’s a reasonable bet, and it fails precisely when it matters.

    The second is to stay white-knuckled. You keep the belt tight and never relax, treating every warning as the real one. This feels like prudence. But vigilance that never stands down has its own costs. It is tiring, it crowds out other things, and it has its own blind spot. If you treat everything as an emergency, you can’t pick out the one that is.

    The relaxed passenger and the white-knuckled one look like opposites, but the sign has stopped telling either of them anything. One ignores it; the other obeys it constantly. Neither is reading it.

    Both are settings, not decisions. Each replaces the question “does this warning mean something?” with a standing answer: no for one, yes for the other. Judgment is the ability to give different answers to different warnings. A sign that never goes off makes that very hard to do.

    What I Can’t Tell

    So what does another twenty or thirty years of this produce? I’ve kept hoping things would settle down. It’s a hunch rather than a forecast, but I no longer expect the sign to go off in my lifetime.

    I don’t think the answer is collapse. Societies have absorbed worse than anything on my list. The more likely result is quieter. Some institutions warn too often; others warn too little. After enough year…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com
    14 min
  • Who Is Left Holding The Napkin

    I have told this story before. Six weeks ago I told it to make a point about software. I want to tell it again to make a point about myself, because I think I put the emphasis in the wrong place.

    In 1993, I was a final-year civil engineering student at Imperial College, and my design project was an exhibition hall with a column-free interior and a long-span roof carried on a pair of massive tubular arch beams. I modeled it in I-DEAS, a finite element package running on a Silicon Graphics workstation that took up half a desk. The package computed the stresses, checked them against code, and rendered a deformed shape that looked entirely reasonable. It said the structure would stand.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

    The night before the report was due, for reasons I still cannot reconstruct, I did a hand calculation of the stress at one of the arch supports.

    It came back off by a factor of ten.

    I had entered the loads in kilograms, but the package read them as newtons. My model was carrying about a tenth of what the real building would have to carry. When I corrected the units and re-ran it, the deformed shape came back flat as a pancake. The structure I had designed would not have stood up.

    Last time, I said the software never pushed back; the analysis was fluent all the way down. The correction had to come from outside it. That is true, and it is the smaller half of the story.

    The larger half is the napkin.

    I had a way to check. It was crude; it took three minutes, and it was decisive. And I had it for a reason that had nothing to do with being careful. I had been taught to size a beam by hand before I was ever allowed near a package that would size it for me. The check wasn’t diligence. It was residue.

    The Word We Only Half Unpacked

    Five weeks ago I wrote about the word reward, and what happened to it when AI borrowed it from neuroscience.

    In the brain, reward is not one thing. Kent Berridge’s work separates it into two systems that can be pulled apart experimentally. Wanting is driven by mesolimbic dopamine; it makes you pursue the thing. Liking is a distinct hedonic system running on different neural substrates, and it registers how the thing actually landed. Disable one and the other carries on without it. An animal can be made to want, intensely, something it does not like at all.

    Machine learning borrowed the word and built the pursuit.

    I want to state this more carefully than I did last time, because it is easy to overclaim. Reinforcement learning gives a system an explicit optimization signal and a great deal of patience in following it. What it does not supply, by itself, is anything corresponding to the second system, an independent register of how the result actually landed, sitting outside the objective and capable of disagreeing with it. There are constraints, regularizers, safety layers, human feedback, evaluators. What’s missing is something that stands apart from the optimization and asks whether the objective was the right one.

    Drive without a brake, with the caveat that this is a metaphor, and a metaphor is not a schematic. The precise version is duller and harder to knock down: a system optimized to produce an answer is not thereby equipped to establish that the objective, or its reading of the objective, was correct.

    I left it there, as a fact about how these systems are built. It is a fact about how they are built.

    But nothing is dangerous all by itself. An uncovered hole in the ground is not dangerous sitting in an empty field. It becomes dangerous when someone walks near it. The question I skipped last time is who is walking near this one, and how close.

    Condition Is Not Risk

    Engineers separate two things that ordinary language runs together.

    Years after the exhibition hall, I spent part of my career in transportation consulting. One project, with colleagues at Cambridge Systematics and Lloyd's Register, built a risk model for the bridges on the US highway system. Every state reports the condition of its bridges to a national inventory, and we used that data to place each structure on a five-by-five grid.

    One axis was condition: how structurally sound the bridge actually is. The other was consequence: what happens if it fails, largely how much traffic crosses it, adjusted for how far the detour would be if it closed.

    Those two axes aren't the same thing, and keeping them apart is the point. A badly deteriorated bridge on a farm track is in poor condition and carries almost no consequence. A sound bridge carrying a hundred thousand vehicles a day is in fine condition and carries enormous consequence. Neither is where you look first. You look at the corner where both are bad at once: a deteriorating structure carrying heavy traffic with no easy detour. Risk is what you get when condition and consequence coincide.

    The structure's condition is a fact about the structure. It is true whether anyone drives across it or not. The risk exists only because people do.

    That is the distinction I skipped past five weeks ago. An optimizer without an independent evaluator is a fact about how the thing is built. It is a condition. It is true of the system sitting on its own, doing nothing at all.

    What turns a condition into a risk is traffic.

    Two Ways To Hand Something Over

    So: who is driving over this bridge, and how heavily?

    Not “who uses AI.” Everyone does, and that tells you nothing. The question is narrower. What happens to your own ability to check the answer?

    Handing work to something else is not one thing. It comes in two forms that look identical from outside and behave very differently when something goes wrong.

    The first is delegation. You give away the task and keep the ability to judge what comes back. A managing editor does not write the article, but can read it and say it isn’t working. I gave the stress calculations to I-DEAS and kept, without ever thinking about it, a way to test the answer. The work moved. The judgment stayed.

    The second is offloading. You give away the task, and the ability to judge the result goes with it. What comes back is a conclusion, arriving without the reasoning that produced it, and there is nothing left on your side that can weigh it. You can accept it or refuse it on instinct. You cannot check it.

    From outside, these look the same. A person asks, a system answers, the person proceeds. The difference is invisible until the answer is wrong.

    The obvious objection is that delegation is just offloading on a longer timescale, that if you never do the work yourself, the ability to judge it must eventually go.

    That objection is correct, and it is the most important thing in this essay. Delegation is not a stable condition. It decays into offloading unless something actively holds it in place. This raises the question of what could hold it there, and whether anything does.

    Bainbridge, 1983

    The obvious answer is that this is a matter of discipline. Keep your hand in. Check the work.

    Lisanne Bainbridge answered that in 1983, in a paper on industrial process control written long before any of this existed.

    Her observation came to be called the ironies of automation. When you automate a process, you leave a human to monitor it and take over if it fails. But monitoring something that runs smoothly is exactly the condition under which manual skill decays. The operator does less, for longer, and the ability to intervene erodes quietly while nothing appears to be wrong.

    Then the automation fails, which is the moment it needs a person, and the person it gets is the one whose skills have been eroding for years, worn down by the very smoothness that made everything look fine.

    The skill you need at the moment of failure is precisely the skill that the absence of failure has been taking away.

    So the decay isn't hypothetical, and vigilance isn't the answer. Which leaves the question of what is.

    What The Napkin Actually Was

    Here is where I have to be careful, because the…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com
    22 min
  • Turtles All The Way Down

    A scientist finishes a lecture on the solar system. An elderly woman in the audience stands up and tells him the whole account is wrong: the world rests on the back of a giant turtle. The scientist, politely, asks what the turtle is standing on. She has the answer ready. It is turtles all the way down.

    The story is usually told about Bertrand Russell. It is also told about William James, and about various astronomers whose names change with the telling. Each version reaches us through someone who did not say where they got it. Nobody has established which lecture hall, or which century, or whether it happened at all. The provenance of the story about infinite regress has no bottom either.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

    The Number I Was Holding Up

    For several years I worked for a consulting firm that ran a Delivery Excellence dashboard. Every project reported weekly against a set of components: scope, schedule, budget, client satisfaction, team performance, and one or two others I no longer remember. Each one got a color. I entered the scores myself.

    On a program for a biotech client, I reported client satisfaction as green, week after week, and the score was accurate. The client contact was satisfied. Every concern he raised got addressed in the order it arrived. He was told early that the scope began to shrink because we could not deliver all of it, and hearing it early is most of what makes a shrinking scope tolerable.

    Underneath that, a team was struggling on several counts, and had been almost from the start.

    So the numbers were true. Scope was green because we had already cut scope. Satisfaction was green because I spent a substantial part of every week producing it. Every component reported a state, and none had a field for what that state cost to maintain. The effort was the risk, and the effort was the one thing the format could not carry.

    I did not falsify anything. I reported what the components asked for, and I raised the delivery problems internally, every week, in the space the form left for them. The technical lead and the others read the situation as managed, and I did not want to go out on a limb by myself against my own team’s assessment. I also thought that if we all knew, we would fix it before it came due. So a room full of people who each knew something was wrong went on filling in a form that had nowhere to put it. I have watched a version of this happen more than once, and never once because anyone was hiding anything.

    What eventually surfaced the problem was not the dashboard. In January, I sat down with the office general manager and argued that we should put three hundred and fifty thousand dollars of our own money into a three-and-a-half-million-dollar job to fix what we had not delivered. The technical lead and two others backed the position. The technical lead told him the only reason we were still in the room was the account management. He agreed to the money. We spent it carefully, the work landed, and the client was pleased in the end.

    The dashboard recorded none of this. It had been green throughout, and it was green because of what I was doing underneath it.

    The Tower We Are Building Now

    The dashboard failed in a specific way, and the way matters more than the failure. It published an accurate number and gave nobody access to what produced it. Whoever read the green had to take on trust that the process behind it was sound, and the process behind it was me. Reported figures come loose from the work that generated them. That is not a property of consulting. It is a property of reporting.

    A model’s output is evaluated by another model, because human evaluation does not scale. The evaluating model was trained on data that a model filtered for quality. Both are scored against a benchmark, and benchmark content has repeatedly been found in the training data of the systems being scored. The results are then summarized for the people who have to decide something, increasingly by a model, because the volume is large and the summary is fast.

    Every one of those steps is defensible on its own. Each is documented, published, and argued for by people who know more about it than I do. None of them is hiding anything.

    What is missing is not honesty. It is anyone whose job is to ask what the bottom layer is standing on. Each layer treats the one below it as ground, and the question of grounding gets handed down the stack until it runs out.

    In Defense of Turtles

    I want to concede this properly rather than in passing, because the concession is most of the argument.

    Abstraction is how anything large gets built. Nobody writing an application re-derives the behavior of a transistor. Proxies are how you manage what you cannot measure directly, and almost everything worth managing cannot be measured directly. I have never inspected a bridge before driving over it, and I am not going to start.

    Stacking is not the sin. A working tower of abstractions is one of the more impressive things people do. The engineer who trusts the steel certificate is not being lazy. That certificate is doing real work, and behind it there is a mill, a test, a standard, and a name.

    The failure is narrower than the stacking. It is losing track of whether there is a bottom at all. Deferring the question of grounding to the layer below is fine, and unavoidable, as long as the deferral terminates somewhere. The tower becomes a problem only when every layer in it has been built on the assumption that somebody further down already checked.

    Where It Stops

    The regress has to stop somewhere, and the only place I have ever seen it stop is at someone who pays if it is wrong and cannot hand the paying to anyone else.

    I want to be careful about how much that carries. It is not a guarantee of anything. People with their names on things sign off on work they have not understood. Systems with clear lines of liability fail badly and often, and the person carrying the cost is frequently the last to see it coming. The engineer who signs the beam did not derive the metallurgy, watch the testing, or stand in the mill. Finding the name does not tell you the answer is right, and it does not mean the person can check the work. It tells you the chain has a bottom instead of continuing indefinitely. The claim is that narrow: without someone in that position, there is no bottom to have.

    I have spent most of this series on that claim, and I am not going to re-argue it here. The engineer who signs off on the beam carries it if the beam fails. The certificate is not the ground, and the signature is not the ground. The ground is that a person’s license, career, and name are attached to the load, and that the attachment cannot be undone by delegating.

    On the biotech program, the ground floor was not the dashboard. It was a room in January, a general manager who had to spend money he would answer for, and an account lead recommending it who would answer for it too. Neither of those costs could be passed sideways. That is why the problem got fixed. It is also why it took until January.

    What You Can Use

    The woman in the story was wrong about cosmology and right about almost everything else.

    So here is the test. Pick any validation you currently rely on. A green status, a certification, a score, a rating, a benchmark result, a maturity level. Ask what it is standing on. Then ask the same question of the answer.

    Keep going until the answer stops being a process and starts being a name.

    If you work down through the whole stack and there is nobody there, you are standing on a turtle. It may well hold. A great many of them do, for years at a time. But you should at least know that is what you are standing on.

    I stood on one for most of a year, and I entered the number myself every week.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com
    10 min
  • The Problem Nobody Set

    There is a piece of work your body performs every waking second, and you will almost certainly never think about it, unless one day it stops.

    I injured my lumbar spine. What I learned afterward was not really about pain, which is the part everyone expects and the part I will spare you. It was about administration. Somewhere well below anything I could call thinking, my brain had been issuing a continuous stream of instructions to the small stabilizing muscles around my spine, adjusting them constantly, correcting for every shift of weight and every turn of my head, every hour I had been alive. Standing still turns out not to be still. It is a controlled fall, caught and re-caught, in every direction at once.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

    I never chose that work. I never noticed it. I became aware of it only when it started arriving late and imprecise, and the ordinary business of getting up from a chair became something I had to attend to deliberately, the way you attend to a difficult sentence.

    Here is the part that took me longer to see. The work was not executing a standing instruction. Nothing in me was running an order that said stay upright. At every instant something was deciding which of an enormous number of signals mattered right now, and therefore what the problem currently was: a floor that turned out to be wet, a bag heavier on one side than expected, a step that arrived sooner than the last one. The instruction never changed. The problem changed constantly, and something reformulated it, continuously, without asking me.

    Nobody set that problem. Something I have never met noticed it and re-noticed it thousands of times an hour.

    The Ledger We Keep Of Difficulty

    Hans Moravec noticed the consequence of this in 1988, in a book called Mind Children, and the observation now carries his name. It is comparatively easy, he wrote, to make computers perform at adult level on intelligence tests or at checkers, and difficult or impossible to give them the skills of a one-year-old at perception and mobility.

    The paradox is normally presented as a fact about machines. I want to present it as a fact about us.

    We rank difficulty by how hard a thing feels while we do it. Chess feels like intellect. It is effortful, available to introspection, and we can watch ourselves doing it, so it sits at the top of the ledger. Catching a set of keys tossed across a room feels like nothing at all, so it sits nowhere. The ranking is not a judgment about the problems. It is a report on which problems we happen to be conscious of solving.

    Let me be careful about how much weight that carries. I am not claiming effort measures unfinished learning. Expert musicians work hard at passages they have played for decades, vigilance is exhausting after thirty years of practice, and creative work stays effortful precisely because there is no stable routine to settle into. The claim is narrower: felt effort is an unreliable proxy for computational difficulty, and we have been using it as our main one.

    An objection deserves to arrive here rather than later. The computer scientist Arvind Narayanan has argued that Moravec’s paradox is not a dependable predictor of what will prove easy or hard for machines, and is better read as a statement about what the field has found worth working on. I think that is largely right, and I am not going to use the paradox as a forecast. Whatever it does or does not predict, it correctly identifies a bias in our accounting.

    That informal ledger has a formal descendant, and it now runs the industry. We call them benchmarks. A benchmark is our difficulty ledger written down and made scoreable, and it inherits the same property: it can only contain what somebody thought to put in it.

    The Objective Was Never the Hard Part

    Let me concede the thing that arguments of this kind usually avoid.

    Football has an exceptionally clean objective. Score more goals than the opposition inside the allotted time. You can write it on a napkin. There is no ambiguity about success, and a scoreboard settles it. If difficulty lived in the difficulty of stating an objective, football would be among the simplest things humans do.

    So the difficulty is not in the objective. It is in the distance between having one and knowing what to do next.

    Watch what that distance contains. Twenty-two people are running the stabilization work I described at the start of this essay, continuously, while sprinting, turning, being shoved and landing badly. Air moves across the stadium in ways nobody has modeled. The surface changes through the match. The light shifts. The altitude sets what their bodies can spend. And each player is reading the intentions of everyone else on the field while being read in return. A pass is a prediction about where a teammate will be in two seconds, made by a body in mid-fall, against defenders making their own predictions about that prediction, who will change what they do the moment they detect it.

    I want to be precise about what the players are doing, because there is a mystical version of this claim and I do not want it. They are solving the problem. Obviously they are; the ball arrives. What they cannot do is say how. The computation is distributed through the body, learned over years, and closed to introspection, which is different from absent.

    But the deepest part is not the policy either. It is that the problem does not hold still.

    Somewhere around the hour mark, a match becomes a different match. The referee has been letting contact go, so the cost of a challenge has changed. Someone is carrying a knock and is now a target. The other side has gone to three at the back, and every space on the pitch means something new. Nobody announces any of this. No player is told the problem has changed. Each one works it out while still playing, revises what matters, and acts on the revision before being able to articulate it.

    It is worth being exact about what changed, because adapting to a moving situation is something good control systems already do well, and I do not want to dress that up as something rarer than it is. What changed was not the difficulty. It was which information mattered to success. A defender’s body position meant one thing at twenty minutes and means something else now, and nothing in the original instruction said so, or could have.

    That is the line I want to hold throughout this essay. Adjusting your behavior because conditions moved is one thing. Discovering that a different set of facts has become the relevant ones is another, and it is the second that nobody set.

    The objective did not move. It was to score more goals in the first minute, and it is to score more goals in the ninetieth. What moved was everything about what that requires.

    Take Away The Opponent

    It would be easy to conclude this all comes from the adversary, so it is worth removing the adversary and seeing what remains.

    Ballet has no opponent in the relevant sense. Nobody on that stage is trying to make a dancer fall.

    I should concede at once that she faces an evaluator. Two thousand people are watching and judging, and their judgment is opaque, plural and impossible to reduce to a number. That is real pressure, and I have not removed all of it. What I have removed is the thing football is usually assumed to be hard because of.

    And the difficulty does not drop.

    She is running the same continuous stabilization work at the limit of what a skeleton tolerates, on a floor with its own particular give, in time with music played live and therefore slightly differently tonight, alongside other dancers whose small variations she absorbs without appearing to absorb anything. And through it all, she is deciding what tonight’s performance is. Not the choreography, which is fixed and which she knew months ago. Suppose her partner arrives a fraction early and the lift no longer sits where the phrase wants it. She has perhaps half…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com
    23 min
  • The Kindest Thing Pain Does

    There are people whose bodies have to be checked at the end of the day. Not out of anxiety, and not because anything feels wrong. The checking happens because damage arrives without announcing itself, and looking is the only way to find it. A burn that never registered. A cut that has been open for hours. A joint that has moved further than it should and stayed there. In childhood, a parent does the looking. Later it is the person themselves, or a clinician, or both. Either way, the inspection is deliberate, scheduled, and conscious, and it stands in for something most of us are handed without asking: a signal that arrives on its own at the moment of the damage, whether or not we happen to be paying attention.

    The condition is congenital insensitivity to pain, and it is rarer and less tidy than the name suggests.

    Thanks for reading James Maconochie | Architecture & Attention! Subscribe for free to receive new posts and support my work.

    It is not one condition. Several genetic routes lead to it, and I am going to take two of them, not because they exhaust the picture but because the contrast between them is useful.

    In one, a gene called SCN9A encodes part of a sodium channel that pain-sensing nerves need to send their signal. When both copies carry disabling changes, that signal does not get through. This route comes closest to a clean experiment because most other sensations remain. Touch works, or nearly: testing has found some differences in how touch is felt at the margins. Position works. The known exception is smell, because the same channel is used by the nerves in the nose, and people with this form are typically unable to smell.

    The second route runs through a gene called NTRK1 and produces what is usually called congenital insensitivity to pain with anhidrosis. Here, more is missing. Along with pain, the ability to sweat is absent, which leaves the body unable to cool itself. Sensitivity to temperature is affected. Intellectual disability of varying degree is characteristic. This is the form where the confounds are worst.

    I want to be careful here. The injuries are documented, and they recur across both routes: biting of the tongue, lips, and fingertips, sometimes severely; damage to the eye; fractures walked on and healed badly; joints worn down; bone infections that took hold because nothing reported them early. What I cannot honestly tell you is that this accumulation is what shortens life. In the second form, early deaths are substantially driven by fever the body cannot shed, which follows from the missing sweat rather than from the missing pain. Reported mortality varies across the groups studied, and I am not going to hand you a figure that sounds more settled than the evidence warrants.

    So the narrow claim, the one that survives both routes and all the confounding, is this. The injuries accumulate because the signal that normally reports damage as it happens is missing. Living with this condition means substituting a deliberate audit for a signal that would otherwise have arrived on its own.

    Which is worth sitting with because it reveals the pain underneath the flinch.

    It is not sensation for its own sake. It is a feedback loop rendered in flesh. It takes a bad outcome and converts it into something carried, something the body cannot set down, something that reaches forward and changes the next reach of the hand. Three weeks ago I described the fast loop that living things run inside a single life: the animal that touches the fire pulls back, carries the burn, and approaches the fire differently tomorrow, the same animal, changed. Pain is part of how that sentence gets implemented. It is the part of the mechanism in us that makes a burn teach.

    Knowledge tells you the stove is hot, and knowledge is not nothing here. People with this condition learn a great deal, and much of what keeps them safe is learned rather than felt. What knowledge does not do is bind the lesson to the moment on its own. A person with this condition can see the flame, know the word, and explain to you exactly why fire is dangerous. What is missing is the small insistent return that says this one cost you something, arriving without having to be asked for.

    One Word For A Lot Of Different Things

    And here I have to correct something in my own vocabulary.

    Across these essays I have said that consequence returning to the actor is what turns knowledge into wisdom, and I have said it as though consequence were a single substance arriving in varying amounts. It is not. Being embarrassed in a meeting, breaking a bone, losing a reputation, and watching an institution fail are not four sizes of the same thing. They arrive at different speeds. They reach different distances. Some can be argued with and some cannot.

    Two of my own, for contrast.

    Earlier this year, while driving for Uber, I hit a parked car. Low sun, a dark grey car on the shaded side of the road, and a fraction of a second spent looking down to change the radio station. The airbag went off. The smell of the powder is the detail I have kept. I called the emergency services, then my wife. She came round the corner from the house, where her parents were loading the car for the drive back to Maine. The damage looked limited to me. The insurer wrote the vehicle off.

    What that taught me is narrow, and it is permanent. I am careful about parked cars on my right, leave more clearance than I used to, and have not had to decide on either since. What it never told me was which part of what I did mattered. The sun was real. The radio was real. I cannot separate them, and I have stopped trying. The cost arrived, the behavior changed, and no account came with it.

    The other one cost a great deal more and taught a great deal less.

    Seven or eight years before that, I was the account lead on a large data program for a retailer. Every two weeks we prepared slides for an executive update. Every two weeks, the client's project lead edited down or removed the risks in those slides before the meeting. My colleague and I both understood what was happening. We had other problems at the time, in staffing and in delivery, and we chose not to force it. So the meetings went ahead twice a month without the real state of the work being included.

    The problems did not go away. After about six months, they were obvious to everyone, and we were left explaining that we had known and had deferred to the client on the contents of the client’s own slides. We were asked to stop work. It took another six months to recover a seven-figure final payment, and it nagged at me for all of them. Worth noting where that landed. The commission on the sale had been paid long before delivery started, to someone who was not in the room for any of this. The people holding the work bore the cost.

    Ask me what that changed, and the honest answer is: a little, perhaps. Because I could argue my position, and I did, and it was not a bad position. That is the thing worth noticing. The slow arrival came with a complete account attached, and an account can be reasoned with, and reasoning with something is most of the way to setting it down.

    The expensive one taught less than the cheap one. What those two suggest to me is that pain sits at one end of a range. It arrives during the event rather than after it; it lands on the body that acted, and it comes with no reasoning you can take issue with. Embarrassment is social and fades. Financial loss arrives as a document, and a document can be answered. Collapse arrives once, by which time the people who might have learned from it are rarely still in the room.

    What This Comparison Can Carry, And What It Cannot

    It would be neat to say that these machines, by which I mean here and throughout this series the AI systems many of us now talk to daily, are painless bodies grown large. I don't think that holds, and I would rather say so than hope you didn't notice.

    Someone with the first form I described, the one that leaves most other sensation intact, is…



    This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit jamesmaconochie.substack.com
    17 min