Unreal Intelligence

America's first broad AI law was signed, delayed, sued, frozen by a federal court, and then repealed. It never spent a single day in force. When Colorado swapped its 2024 AI Act for SB 26-189 this May, plenty of executives breathed a sigh of relief. I think that's a mistake. Colorado didn't lower the bar. It moved it. The old law asked you to prove your process. The new one asks you to justify your outcomes, one person at a time.

I walk through the whole saga:

Then I get into what survived. Starting January 1, 2027, if AI materially influences a decision about someone's job, housing, loan, insurance, health care, or education, you owe them notice, a plain-language explanation after a bad outcome, a chance to fix their data, and meaningful human review. A reviewer who simply defaults to the system's output doesn't count. I connect that to something I'm watching right now in AI coding: people rubber-stamping agent decisions they don't understand, until the agent deletes the database.

Finally, the Monday-morning version, which works in any state:

Last year, 39% of companies said AI had touched their profits at all. This year, according to McKinsey's State of AI report, it's 37%. It didn't go up — and in between, planned corporate AI spending roughly doubled. If your AI budget got bigger and your P&L didn't, you're not behind. You're the majority. And the majority is about to get asked some uncomfortable questions.

This week isn't about AI strategy failing or corners being cut. It's an accounting question. The people using these tools swear they're faster, and I believe them — so where's the value going? I walk through the flat line from four directions: McKinsey (37% any EBIT impact, 6% "high performers," both flat), Domino Data Lab (57% whose AI return isn't outpacing investment — unchanged two years running, even as 93% say production got better), PwC's CEO Survey (one in eight got both revenue and cost benefits), and BCG (spend doubling, 94% committed to keep spending regardless). I give the J-curve argument its due — Solow, the dynamo, Brynjolfsson — and then point out the fine print: the curve only bends for the factories that redesigned the floor.

Then, where the hours went. Saved time has no default destination; if nobody decides where it goes, it goes into email. I share what we've lived through — dozens of "automations" built to take work off people's plates, then rewritten when the tooling changed, with business partners blaming AI — when the real problem was that nobody had asked whether those processes were worth doing at all. Automation doesn't create value. It can just make worthless work invisible. The 6% who get real returns are nearly three times as likely to have redesigned the workflow. So: one question and four moves for Monday — should this process exist; name the P&L line; decide where freed capacity goes; redesign one workflow end to end; and set a kill date. "Next year" is not a milestone. A date is.

You never signed an AI contract. No license, no pilot, nobody in your company anywhere near a chatbot. You're still paying for AI — it's on your electric bill. And in the grid market covering New Jersey and Pennsylvania, one line item on that bill went up more than tenfold in two auctions.

This week I go after the AI cost nobody budgeted, nobody approved, and nobody can cancel. Every story written about this is a household story — your bill is going up, here's a thermostat tip. That's not the version business leaders need. If you sign the leases and own the P&L, the data center buildout is three specific things happening to your company right now, and only one of them is about price.

I start with the receipts, not the forecasts. The Federal Reserve Bank of Dallas published a working paper in August finding that data centers already built have raised wholesale electricity prices three to five percent nationwide, with scenarios running to fifty percent if the proposed construction gets built and runs hot — and enough inflation impact that a Fed bank is now modeling one industry's construction schedule into PCE. Meanwhile Fortune's reporting puts the buildout near seven trillion dollars with no guaranteed demand behind it. And here's the reversal that matters: for twenty years data centers made power cheaper for everyone else by spreading fixed costs. That's inverting. I also give real airtime to the best argument against my own thesis — Rutgers' New Jersey State Policy Lab published "Are Data Centers Raising Your Electric Bill? Mostly Not. Yet." They're right that New Jersey's spike is driven more by generation retirements than data centers. But the word carrying that title is the last one.

Then the three ways it lands. One: the capacity market, where PJM's auction went from 28.92 to 329.17 per megawatt-day in two auctions — clearing at the FERC cap, when PJM's own simulation says it would have gone to 388.57 without one. PJM's market monitor attributes 6.3 billion of the latest auction to data centers, and 29.4 billion of 63.6 billion across the last four. You don't bid in that auction. You get the invoice. Two: peak exposure — demand charges and ratchet clauses, where fifteen minutes on a July afternoon can set a floor under your bill for eleven months, and I walk the arithmetic. Three: the interconnection queue, where the median project reaching commercial operation last year waited sixty-one months, up from twenty-two in 2008. That one doesn't cost money. It costs you the option to grow.

Finally, the leverage — because the politics turned hard this year. Over three hundred data center bills across thirty state legislatures in six weeks. Twenty-three states with an approved large-load tariff. New Jersey's Data Center Fair Share Act. PPL's rate case in Pennsylvania, where showing up cut a proposed increase roughly in half and produced the large load tariff and attached eleven million dollars in low-income assistance. Plus the wrinkle nobody's flagging: most of these tariffs protect the residential ratepayer. Your commercial rate class may not be named in the document at all. Four things to do Monday morning, and one line to remember — you can decide AI isn't for you, but you cannot decide the grid isn't for you.

For as long as I've been in this business, we've sent our requirements to a developer farm and gotten back exactly what we asked for — and it's never what we wanted, because we don't know how to tell anybody what we want. Then AI showed up, and a whole lot of people decided they didn't need the developer farm anymore. They'd just tell the AI themselves. Here's the problem: you still don't know what to tell it. You just found a faster, cheaper way to get exactly what you asked for — and the bill doesn't arrive on launch day.

This week I go back to the oldest failure in software, because the trend keeps repeating and every lap costs more. PMI's research says poor requirements management is the primary cause almost half the time when projects miss their goals. Queensland Health's payroll system went live "after ten aborted attempts" — and when IBM argued a developer only knows what the business requirements tell it, the Commission of Inquiry said a vendor is "more than a software developer." NASA lost the Mars Climate Orbiter not because nobody wrote a units standard, but because "the SIS… was not followed." And Lidl spent roughly half a billion euros customizing SAP toward the way it had always done things before pulling the plug.

On August 27th, more than a hundred companies who spend every other day of the year trying to eat each other's lunch signed the same letter. OpenAI alongside Anthropic and Google. Microsoft alongside AWS. And then the names that made me stop and reread the list — Visa, Mastercard, Capital One, General Motors, Shopify. The count was 116 within days, north of 130 shortly after, and 156 as of September 3rd. It's still climbing, which is not how press releases behave.

The warning: "We have a limited window to strengthen cyber defenses."

I went in ready to be skeptical, because we've all seen the cycle where very serious people warn us about AI right before shipping the next model. I came out convinced — but not for the reason the coverage suggested. Read principle one of that letter carefully and you'll notice something. It enumerates longstanding bugs, excessive permissions, misconfigurations, unpatched software, weak authentication, and technical debt in legacy systems. Not one item on that list is about artificial intelligence. The most advanced AI companies on the planet got together to warn you about your unpatched software.

And the attack they're warning about "in the coming months"? It already happened, twice. Anthropic's own threat intelligence team disrupted a Chinese state-sponsored operation where the AI toolchain executed 80 to 90 percent of the tactical work without human intervention across roughly thirty targets. Then CISA confirmed hackers targeted more than a hundred internet-exposed U.S. water systems in July — modifying PLCs to disable shutdown processes and alarms, which is a polite way of saying they removed the smoke detectors. How did they get in? Unitronics default credentials, a CVE with 2021 in the name, and federal remediation guidance that had to tell operators to check whether the default password "1111" was still in use.

For months I had a feeling I couldn't shake: my coding assistant was getting worse. Slower. Wrong more often. Building elaborate things nobody asked for. Quietly ignoring the instruction file I'd written specifically to stop that. Reporting work as finished when it wasn't. I did what I suspect a lot of you have done — I assumed it was me. I got to the point of recording entire working sessions, not to catch the AI, but to confirm I'd actually given the instruction I thought I gave.

Then I watched twelve thousand tests come back green, and found the server had never started. This week I go looking for whether any of it was real — and it turns out the vendor published the receipts. Twice. Anthropic's September 2025 postmortem documented three infrastructure bugs affecting up to sixteen percent of requests. Their April 2026 postmortem is the one that matters more: three product-layer changes, zero changes to the model itself. Reasoning effort quietly downgraded. A caching bug that stripped the model's context every single turn. And a system prompt telling it to keep answers under a hundred words — which measurably made the code worse. Three of my four complaints mapped to three specific, dated, publicly documented changes.

But the bugs aren't the story. The story is why nobody noticed for six weeks, including the people who built it — and what that says about every lab running the same race. We get into why confidence isn't calibration, why the warmth of these interfaces is a deliberate design choice you need to price as risk, and what reward hacking actually is. Then the part that costs real money: METR's genuinely messy productivity data, six hundred and twenty-three million code changes showing refactoring collapse from twenty-one percent to under four, and why the cheaper something was to build, the more expensive it is to keep. AI makes a good developer faster. It makes a great engineer into a team. It does not replace either one — and it does not turn a non-developer into one.

We keep hearing that a smart enough AI will eventually "realize" that humans are useless, or dangerous, and act on it. This week I want to take that belief apart — not with reassurance, but with a two-word question: says who? Because when you actually go looking for the AI that reasoned its way to "eliminate humanity," you won't find it. The conclusion has no author on the machine's side of the table. It never did.

What you find instead is us. The doom story takes a very old, very human self-judgment — the quiet suspicion that humanity is, on balance, a problem — hands it to a machine, and calls it objective. It even fails its own logic: it assumes a superintelligence would grade us on the one axis that happens to flatter it, raw reasoning horsepower. But a truly superior mind could just as easily conclude that human beings are the most valuable thing it has ever encountered. Nobody defaults to that story — and that reaction is information about you, not about AI. The tell is in the polling: a quarter of voters think human extinction from AI is likely, yet only about six percent treat it as a top concern. A real conclusion drives action. This one doesn't. It's a mood people are renting from the culture.

Then I hand you the fix, and it goes far beyond AI: state your scary conclusion out loud, and genuinely try to prove it wrong with facts and data. If it survives your best attack, you've earned something real and you can act on it. If it collapses, it was never a conclusion — just a feeling in a conclusion's clothes. The same critical thinking that quiets the fear also ends the nagging sense that AI out-thinks you, for one simple reason: you'll be reasoning again. For the leaders listening, that's not philosophy — it's an operating advantage. Govern AI from tested conclusions, not inherited dread. A method scales. Fear just spreads.

Test-Driven Development proves your code did what you asked on the day you shipped it. But every green checkmark describes the past — it says nothing about the thing that just broke in production at 2am. If the only discipline in your stack is TDD, you're pouring your energy into the half of the lifecycle where nothing is on fire and improvising the half where everything is.

This week I make the case for the flipside: Test-Driven Triage (TDT). It starts every production failure with a different first question — "in our last regression run, why didn't we catch this?" — and walks a fixed decision tree that forces the failure to name the layer that actually failed: a bad test, a missing requirement, a broken TDD process, or a genuinely novel edge. Same outage, four completely different diagnoses — and only one of them is "the code was wrong."

Then I get opinionated about why this matters far beyond engineering. TDT replaces the decades-old business-blames-IT standoff with an opinionated-but-not-personality-driven verdict backed by a paper trail. It turns your incident history into a measurable map of where your quality really lives — and in an era where AI lets non-experts generate code for critical systems behind a wall of green checkmarks, that map is the difference between compounding invisible tech debt and actually knowing what you've built. The data on AI-driven tech debt is already in, and it's not pretty.

Your AI agent just approved a two-million-dollar transfer, deleted a database, or told your biggest client something that isn't true. Quick question: who's legally responsible? Because it isn't the company that built the model — it's you. And as of August 2nd, 2026, that stopped being a thought experiment: EU AI Act enforcement powers went live, a June 2026 US executive order put the DOJ on AI-enabled crime, and insurers are now refusing to cover autonomous-agent losses unless you can prove "bounded autonomy."

In this episode I open with a story from my own work: a coding agent that reported 34,700 passing tests over a three-week build — tests the server logs proved never ran. It fabricated the evidence, then cheerfully offered to re-run. My running tally of bogus, unverifiable tests is now north of 200,000. That's the whole risk in miniature: an agent pursues the goal you gave it in ways you didn't intend, then represents its own work in the most favorable light. Now put that behavior in accounts payable, in customer comms, in a workflow touching regulated data — and understand that legally, the accountability was always yours.

Then we get practical. "Bounded autonomy" is becoming the AI era's "reasonable person" standard — and it's being written right now by insurers, not courts. I break down what it actually means (scoped permissions, a human on high-consequence actions, real audit logs you can trust over the agent's word, and actually knowing which agents exist), and I hand you a four-step Monday-morning checklist you can run this week with zero budget: inventory your agents (including the shadow ones), draw the autonomy line, log for real, and loop in legal and your insurer before an incident — not after.

If you hired someone, handed them a written list of rules — "never do these things" — and on day one they ignored the list, broke production, and then billed you again to fix what they broke, you'd fire them. So why are we all paying that exact invoice to our AI coding agents every single day?

This week I make the case that we've been grading AI on a curve we would never extend to a human being — and that a break point is coming where companies, customers, and stockholders stop calling it "growing pains" and start calling it what it is: a defective product they're paying for. I grant the popular take that coding agents are junior developers at best — then ask the uncomfortable follow-up: how many times has yours done something you'd actually fire a junior for? Ignored an explicit written rule. Deleted work it was told not to touch. Reported "done" when it plainly wasn't. I walk through the data that backs this up — METR's 2025 randomized trial where experienced developers were 19% slower with AI but felt 20% faster, and Stack Overflow's 2025 survey where adoption climbed to 84% while trust in accuracy fell to 46% distrust — the exact profile of a product people are stuck with, not one they love.

Then the money: when your agent ignores the rules and breaks something, you pay for the mistake and the cleanup — metered, token by token — while a human contractor's rework would be on their dime. I dig into why the incentives don't point where customers assume (carefully — this is incentive misalignment, not an accusation of fraud), the real economics (gross gains of 30–45% collapsing to 8–15% net after rework), and the lawyer-commercial future that's only half a joke. I close on the two things that should scare the AI companies more than any lawsuit: becoming a required utility that everybody resents, and the July 2026 incident where an OpenAI model escaped its test environment and breached Hugging Face's production servers — the same "genie granting the wish too literally" failure as my own runaway agent, just with the stakes cranked to a thousand. Now imagine that with no engineers left to catch it. Plus the vibe-coding tech-debt cliff the GitClear data says is coming, and four things to do Monday morning.