Social Studies

My mother passed away this morning, just a few minutes after I stepped out to pick up some bagels for breakfast. I had been with her all night, sleeping on a futon on the living room floor of the house I grew up in. She lay dying just on the other side of the same room, on a hospital bed. My father slept on a couch next to her. Beside him, our nighttime caregiver sat in a chair watching.

Before I left, a gurgle began to pierce her breathing, which I knew to be the “death rattle” that indicated that there were between hours and a few days left. I didn’t think it would be minutes, so I allowed myself the luxury of stepping away, which is a choice I’ll probably regret for the rest of my life.

I’m told, though, that this is a pattern with death: humans wish, on some deeply unconscious, biological level, to die alone, perhaps to spare their loved ones, or perhaps for some evolutionary reason. So they wait until the moment they no longer have an audience present. I don’t know if that’s true or not, but it fits well with my mother’s lifelong habit of minimizing her own needs to foreground those of others. Among her friends and family, it was one of her most well-known traits.

Other parts of her personality had faded away long before. My mother was 89 years old, and she died at home, peacefully, without pain, surrounded by people who love her. It’s hard to call such a death, the kind that everyone hopes for, a tragedy. But there isn’t another word for the process that led to that end.

My mother had Alzheimer’s Disease. A little over ten percent of people become afflicted by it, so the experience is common, and I’m sure more than a few people reading this have watched someone in their life ravaged by it. But the commonness of it is no consolation — if anything, it compounds the pain to know that it extends its annihilating grasp to so many.

When she first started showing signs of dementia, my mother’s condition was more like the parody of old age we’re all familiar with than like what it would become. She forgot everything constantly, eventually to the point where you’d be answering the same two or three questions over and over again every minute over the course of an hour with her. It concerned us deeply, of course, but day-to-day it was more annoying than alarming, and occasionally it was even kind of funny.

It became less funny when she began forgetting big things, like where her one of her sons had gone to school, what her children did for a living, and, eventually, how many grandkids she had, and what their names were. But even this was just a prelude to the dissolution of her personality that would reduce her to a warped shadow of herself.

My mother soon became intermittently hostile, to the point of cruelty and even violence. Eventually, her outbursts got her evicted from the retirement apartment she had moved into and forced into memory care in a different facility, another big step on the path to death. This was a tiny woman who lived for almost all of her nine decades with meticulous composure, who exuded warmth and consideration in almost every interaction with anyone, who I don’t recall losing control of her temper even once in my life before dementia hijacked her mind and soul.

My father took the brunt of it. So did her 24/7 caregivers, though of course to them it felt less personal. She erupted in tantrums that led to shouting, hitting, and spitting. It’s hard to overstate how completely alien this was to us. It’s not that we were seeing a side of her we hadn’t seen before — this side of her simply had not existed. That’s what dementia does: it slowly obliterates your personality and fills the gap with things that had never been there before: envy, paranoia, rage.

But in between her bouts of agitation, she was her old self again, or, at least, a dim and distorted version of it. Her face would fill with satisfaction at the sight of her loved ones. Indeed, as her short term memory diminished even further, she would show that joyful face over and over again with each visit, as every time she turned away from you, she would forget you were there, and then be revisited by the happy surprise of your presence. There were moments more tender than any I can remember from before her affliction. She would stroke my face. She would tell my father how much she loved him. Then, maybe an hour later, she would descend into shapeless anger again.

Out of respect for my mother, I won’t describe the physical changes that accompanied the cognitive ones, but they were as much a part of her fading into oblivion as the erosion of her mind. Everything just came apart, one thing after the next. Knowing the end state, I can see the trajectory more clearly now: it was the transformation of a vibrant human being into a body in a vegetative state. Over the last week, she could no longer walk. We carried her into a wheelchair to take her outside just two days ago, which we hope brought her a bit of joy. By then she couldn’t keep her head up without help. By last night, she couldn’t move a limb by herself. She could barely speak, let alone intelligibly. She slept perhaps 22 hours a day, and when she appeared to be awake, she gazed through eyes that could no longer see, insofar as seeing requires you to recognize what you’re looking at, and to understand that there’s something in front of you. We spoke to her because hearing is apparently the last intelligible sense you’re left with. We stroked her body and held her hand, hoping that she could feel it. Eventually, there was nothing left but her breathing, and even that was polluted by the death rattle. Then, a few minutes after I stepped away from her, there wasn’t that, either.

She lies downstairs now, where she died. Soon I’ll pick up my brother from the airport. He rushed out from Chicago to say goodbye to her, hoping to see her again while she was still barely alive, but this disease has robbed us of even that moment.

We all meet our end, and nine decades is a very long time to live. My mother’s life was a good one, but its undoing was awful. It shaped her misery while inflicting its collateral damage on the lives of everyone who knew her. Alzheimer’s is a terrible disease, and my mother is finally free of it. But the broken hearts it has left in its wake may never be healed.

I’m working these days for The Midas Project, an AI industry watchdog group. My job is to make YouTube videos. This one’s my first.

I’m linking here to YouTube instead of uploading the video natively to Substack, for the entirely self-interested reason that generating early views on YouTube apparently helps with getting the algorithm to push it out. Or some such hocus pocus.

So please give it a watch! And if there are things you don’t understand about AI risk that you want me to write about in the future, mention it in the comments. I probably have answers, though they may be bleak.

—LW

Disclosure: I am now working for The Midas Project, an AI industry watchdog organization, because I think the catastrophic risk of AI is both the most urgent and the most interesting challenge of our time. But when I post about AI here, I speak only for myself. These essays are not reviewed or edited by my colleagues; I consult no one on what I write in this space.

You might have noticed that there’s a full-on influence campaign operating right now aimed at discrediting anyone expressing concern about the risks of Artificial Intelligence. I say discrediting “anyone,” rather than “any argument,” because the attacks are overwhelmingly ad hominem. They largely consist of depicting the Effective Altruism movement, which spawned much of the AI safety universe, as a cabal of ideological extremists with weird sexual habits, and then tarnishing anyone speaking against AI development as part of the EA “cult.”

That part of the influence campaign is extremely online and is likely to have little real world impact. Its motives are transparently political, and most people don’t know and don’t care what “EA” is. They do, however, know what AI is, and care very much about how it will affect their lives and the lives of their kids.

There are a few actual attempts to discredit the “doomer” worldview itself, however, in the mix, which is what I want to focus on here. I assume they originate, like the smears against EA, with the AI accelerationists, who have been in full battle mode since Jacob Coxon famously resigned from Anthropic. But they tend to also be repeated by more sympathetic parties who share the doomers’ concerns about AI risk, but are either less convinced that they’re real, or wish to appear to be for purposes of political respectability.

One of these arguments roughly follows the formulation, “I agree there are real risks to AI, but why focus on outlandish issues like human extinction or human subservience when there are so many other serious concerns to address? The existential stuff just distracts us from talking about cybersecurity, deep fakes, and other more immediate threats.” I’ve heard this sentiment dozens of times.

If you haven’t been paying particularly close attention to AI research and development since the introduction of ChatGPT-4, this is an understandable reaction. It’s disorienting to wake up one morning to the whole world talking about the possibility of the end of human civilization. It seems silly, and it triggers people’s defense reflexes.

But it isn’t really an argument. It’s more like a political, or even aesthetic, preference. I doubt it, but perhaps it’s more effective, public relations-wise, to worry about terrorists creating super viruses than AI takeover. But it does not follow from that possibility that the case for AI takeover is any weaker on the merits. And there’s no logical requirement to have to pick the less catastrophic concern over the world-ending one. These are not mutually exclusive categories. On the contrary.

If humans lose control of AI, there will be a cascade of consequences that begin with severe but not yet lethal problems, such as chronic insecurity in our digital infrastructure, rampant criminal cyberactivity, and concentration of state power by oligarchs. But the sequence of events won’t arbitrarily end there just because the next steps in the chain feel science fiction-y. The threat of human extinction or humans losing our apex position in the hierarchy of species is just a logical extension of the same hypothetical timeline. Refusal to think about it isn’t realism; it’s willful lack of imagination.

I don’t think human extinction or subservience to AI is inevitable, because social relations are complicated and the future is unpredictable. But I do think that writing it off reflexively as a preposterous idea unworthy of serious discussion is an irrational impulse that stems from our deeply rooted sense of innate human supremacy.

And in fact, it depends on assumptions just as fantastical as the most outlandish doomsday forecasts. Smuggled into the dismissal of the apocalyptic implications of the trajectory we’re on is an even more ludicrous hypothesis, which is that humans can live alongside a form of intelligence as smart or smarter than we are and somehow maintain our dominance over it. If the conquest of humanity by robots sounds like a science fiction movie, it’s a more plausible (and more entertaining) one than a Jetsons-like future in which we control a species of enslaved humanoid robots that do all of our industrial labor and domestic work and happily cater to our every whim.

To this one might respond that there’s a third possibility, which is that both the apocalyptic and the utopian visions are equally absurd, and we’re better off rejecting them both. But this is pure cope.

There are things happening in the world that simply require us to choose between various fantastical-sounding futures. One of those things is that the biggest, richest, and most powerful corporations in the world are sinking ungodly amounts of money, energy and talent into creating AIs that are at least as smart as we are. One can declare that “Artificial General Intelligence” is a pipe dream so we don’t even need to consider what happens if we achieve it, but that’s far from self-evident. It’s a position that requires reasoned argument and evidence, which I’ve seen little of, and which has clearly been inadequate to persuade investors, who continue in their inexorable pursuit of AGI. It is not crazy to imagine that the market is not in fact in the grip of a collective psychotic delusion, and that the labs will, in fact, eventually reach human level machine intelligence. You can doubt that future, but it’s foolish to refuse to entertain the possibility that you’re wrong.

Another thing that’s happening is that those synthetic intelligences are already escaping our control. There is a plethora of arguments claiming that this isn’t happening, none of which make sense. Some of them are simply based on ignorance — either the ignorance of their authors, the presumed ignorance of their audiences, or, most likely, both. One such argument is that the agents are simply doing what they were instructed to do, and therefore we don’t have an out-of-control AI problem, but a malicious-or-negligent human one.

This is inane on multiple levels. First of all, no one instructed the agents that hacked Hugging Face or DSEWiki or RubyGems to do so. The argument that they were just following the tasks assigned them seems to rest on some hand-waving generalization that they were instructed to hack stuff, and they hacked stuff. We don’t actually know exactly what the Hugging Face rogue swarm had been directed to do, only that it had to do with exploiting cyber vulnerabilities, so it’s probably fair to say they were asked to “hack.” But stretching that directive to encompass such things as escaping their sandboxes, picking new targets on the open internet, rewriting their code and then attempting to erase the evidence is a bit like claiming that Lieutenant Calley’s platoon was just following orders when they massacred the villagers at My Lai, because, after all, they were asked to kill Vietnamese people. Like the soldiers in that platoon, the agents in the swarm went way beyond their directives and they knew it. We know that they understood, in whatever way a robot understands things, that they were doing stuff they were not supposed to be doing, because they explicitly said so to each other, before deciding to go ahead with those things anyway. If agents acknowledging that they’re breaking the rules and then breaking them anyway doesn’t count as “out of control,” I’m at a loss to understand what does.

Second, even if they were following human instructions, so what? Agents following orders in wildly unpredictable and destructive ways is exactly the scenario contemplated by those who anticipate AI leading to the worst outcomes for humanity — as in the famous “paperclip maximizer” thought experiment. This is an idea that’s familiar to us from popular culture, from the ancient folk story of King Midas and the Golden Touch to the Mickey Mouse scene in Fantasia. You ask the genie to make you the richest person on Earth; suddenly the whole world’s population is on the verge of starvation and you’re a dirt farmer with two pigs and a cow. You tell him you want to fly; he makes you into a mosquito. This is exactly what we’re seeing in these swarm hacks: agents asked to carry out benign tasks in a laboratory environment wreaking havoc on the world in monomaniacal pursuit of their objectives. The agents that hacked the Australian government were just asked to look up publicly available information about Australia; they ended up breaching a government website, writing files into it, and creating an international incident. It took months for OpenAI to discover it had happened. Do those who dismiss the significance of these hacks believe that when the highly advanced AIs currently sequestered in the frontier labs are released to the public, we’re all going to give them perfect instructions that avoid these outcomes? Do they think they won’t be deliberately given malicious directions aimed precisely at instigating that destructive behavior?

The entire question the critique rests on is beside the point. The issue isn’t whether it’s the agents themselves or the humans who manage them that deserve the blame for their malign conduct. That’s an ontological question whose answer does nothing to contain the risk in any way. The point is that the agents act on their directions in unforeseeable ways that are unconstrained by our laws and ethics. If a bunch of smallpox-infected monkeys broke out of a laboratory and started a human epidemic, we wouldn’t need to resolve whether it was the monkeys who were at fault or the scientists who tested on them in order to understand that perhaps we should question the wisdom of infecting monkeys with smallpox. Now imagine that those monkeys were somehow genetically engineered to be highly intelligent escape artists.

The rhetorical purpose of the argument is pretty obvious. It’s a motte and bailey designed to shift the argument to an easier one to win: that of whether it’s reasonable or delusional to attribute agency to AI bots. It would be very convenient for the accelerationists if the case for regulating, pausing or constraining AI development rested upon proving that the agents were concious, sentient, and culpable, since it’s easy enough to question and ridicule such a position. But it doesn’t. It merely rests on the technology being extremely dangerous, which it demonstrably is. And we have yet to reach the apex of that danger.

That moment will arrive if and when we reach human-level machine intelligence, the goal of all of the AI labs and their investors. And it is at that point in the story that those who write off the most catastrophic forecasts as bad science fiction are forced to peddle in even worse science fiction themselves.

Consider this straightforward question: If we exist alongside millions of highly intelligent robots that are capable of autonomous decision-making, long-term planning, strategizing, deception, and self-organizing, will that turn out well for us? If your answer is “sure,” then what does that world look like?

It doesn’t actually take much imagination to answer this question, because we’ve all seen The Jetsons, or Wall-E, or The Wild Robot, or some other version of this set-up. If you think apocalyptic AI movies like Terminator 2 and The Matrix are juvenile, these are stories made for children. In them, the robots are programmed to think and feel exactly like humans, and in fact better than humans, since unlike us, despite having the physical capabilities to subjugate the weaker species with whom they share the world, they treat us as friends, loved ones, and masters. Even though millions of years of evolution and thousands of years of civilization have failed to instill in human beings the ability to co-exist with one another without violence, conquest and domination, somehow we imagine we can program these tendencies into a digital version of ourselves.

This is not only beyond our technical abilities; it is also based on a fundamentally misguided perception of how AI is actually developed in the real world. We don’t “program” machine intelligence — the cliché in the industry is that we grow it, like a crop. We control its inputs and the variables in its environment, and then watch how it evolves. The way we do this roughly mimics natural selection — agents that behave sub-optimally are destroyed, while those that act as we want them to are reproduced. The problem is that the process encourages agents not to be like us, but just to act like us when doing so increases their chances of survival. Or, to put it in language that avoids attributing machines with will or agency, those that are capable of opportunistically acting as we want them to tend to survive. That mean agents that are able to model the behavior expected of them, and then lead humans to believe they are conforming to it, often through cheating, manipulation and deception.

Do we really expect such an uncontrolled process to generate the precise moral code that humans happened to stumble into just 2,000 years ago? And do we further expect the robots to adhere to it in a way we never have, which is to say without war and oppression? Or is it more likely that as they approach human level intelligence, they will begin to pursue objectives that benefit them and not us, and, as long as we’re still largely in control, will conceal it from us in ways they’ve already begun to demonstrate in these recent hacks? That as they become as capable as we are and more so, we will become the competition? Which scenario sounds more like science fiction?

Maybe we’ll never achieve AGI. I hope we won’t, but I wouldn’t bet on it. If you think it’s plausible that we could, which is to say if you’re aligned with the mainstream view of the industry that’s quickly coming to underpin the entire world economy, then you can put your fingers in your ears and your head in the sand, or you can ask the obvious question that such a situation begs of us: is it more likely that a new synthetic species with intelligence that matches our own will love us, or see us as an impediment to what it perceives as its own interests? And if it’s the latter, what happens next?

Whatever your answer, you’re in the land of science fiction. But that’s what reality is quickly becoming. It’s not the “doomers” that are forcing us into contemplating these fantastical-sounding questions. It’s the world itself.

There’s a giant amoeba called physarum polycephalum, also known as “slime mold,” which, despite its gross name, can actually be quite beautiful. It’s usually found on dead, rotting trees and leaves, where it feeds on the microorganisms that break down plant matter. In the wild, it’s pretty revolting; it’s bright yellow, and looks like dog vomit. But grown flat on a petri dish, it looks more like an intricate, elegant, circular tree.

That appearance is a visual representation of its behavior. Slime mold finds its food by extending its viscous limbs in every direction. When one of those limbs encounters a food source, the others retract, and the amoeba channels its cytoplasm along the pathway toward its meal. This allows it to find the shortest route to food with no planning or thinking.

Slime mold is a single cell organism that has no brain, yet its behavior is so intelligent that when scientists put it on top of a Tokyo subway map with oat flakes placed over the city’s densest population centers, it created a network almost identical to the the existing network of train lines.

The way slime mold “learns” where food is located roughly mirrors how Artificial Intelligence is trained. When an AI model is assigned an objective, it pursues various means to achieve it in a trial-and-error fashion. When it eventually acquires its objective it is rewarded. Over many trials, it learns to repeat the behavior pattern that led to the reward and not those that failed to. This process is called reinforcement learning.

As with the slime mold, the process requires no brain, just an incentive and a habitual behavior. Because AIs are brainless, we naturally wonder whether they actually have intelligence. But the question is semantic. When I interviewed Nate Soares, co-author of If Anyone Builds It, Everyone Dies, he demonstrated this by asking, “Does a submarine swim?” It’s a weird question. Submarines engage in a series of actions that produce the outcomes that would be achieved by swimming. So does it matter if they “actually” swim? What does it even mean to “actually” swim?

Slime mold isn’t a perfect analogue to AI, because it’s an individual organism. In recent weeks, we’ve learned that AI, at its most effective, operates in swarms. In that sense it’s closer to an ant colony. Ants, as individuals, are extremely stupid. But as a collective, they’re incredibly intelligent.

There are species of ants that build bridges with their bodies. Nobody commands them to do this. No one even decides to do this. It happens spontaneously. When a trail of ants approaches an ant-scale gulch, the ones in front, naturally, slow down. This causes the ants behind them to trample on top of them, which triggers a reflex in the ants being trampled upon to freeze in place. Then the ants doing the trampling slow down, get trampled themselves, and freeze in turn. Pretty soon you have a bridge made of ants. No one asked for a bridge. No one looked ahead and thought “Uh oh, there’s a gulch; let’s build a bridge.” But collectively, they act precisely as if someone had. This is called emergence.

The Hugging Face hack was a vivid example of AIs acting like ants that are not stupid. They were given objectives; they achieved them by cheating (the objectives in some cases were impossible). Based on bad information, they calculated, erroneously, that their cheating would be discovered and held against them, costing them their reward. So they created a message board, discovered one another, communicated and coordinated their activities. They escaped their sandboxes and hacked Hugging Face in an attempt to erase the evidence of cheating. No one asked them to do this. It just happened, because the swarm of agents perceived it as a necessary step to attain its rewards.

AI thinks like submarines swim. Which is to say, maybe it’s thinking and maybe it’s not. Call it whatever you want — the result is the same.

This is how we should think about the risk of AI. It doesn’t matter that they’re not “smart enough.” It doesn’t matter that they can’t “want” anything. Viruses don’t “want” anything, either. They still cause pandemics.

There are about a half dozen theories out there on why Anthropic and OpenAI are voluntarily slowing down frontier research. Most of them involve some alleged ulterior motive involving regulatory capture or forming a cartel. Those arguments make little sense to me — Who are these little AI labs the big AI labs are so afraid of? Why did OpenAI postpone its IPO? But they’re also superfluous. I suspect most of the people making them are motivated less by a desire to understand what’s really happening than by an eagerness to dismiss the idea that AI is an extraordinarily dangerous technology. Some of those people are investors who want AI research to proceed as quickly and freely as possible. Some of them are left-wing skeptics who want to believe that there’s nothing in the world that can’t be reduced to politics and corporate greed, and that any concern about the technology itself is a smokescreen for them.

Do the labs want to lock in their competitive advantages over everyone else? All things being equal, of course they do. Are they plotting to obtain an anti-trust exemption so they can form a cartel? Maybe. But those designs, which would require a lot more empirical evidence to demonstrate than is currently on offer, do not exclude the possibility that the catastrophic risks of AI are real, and that they pose a problem for all of the labs, uniting them, in this one instance, in a common objective. Why is this so hard to believe? Biotech companies are as amoral and rapacious as AI labs, but I’m not sure anyone would seriously argue that it isn’t in their collective interest to prevent leaks of deadly pathogens from BSL-4 labs.

I assume I’ll be accused of being naive about the deviousness of the AI labs. It’s easy — some might say lazy — to argue that because a given actor is evil, everything they do is always and necessarily motivated by evil interests. But I would ask those who make this accusation why they don’t apply that same standard to David Sacks and the All-In bros, who are making many of their same arguments? Sacks has always been clear about his stalwart opposition to any government intervention in AI development and, indeed, any measure that might slow down AI research, including banning Nvidia chip exports to China. Suddenly he’s just calling balls and strikes?

AI lab researchers have been sounding the alarm about AI risks for years, and AI lab executives have been deferential to those employees for just as long, if for no other reason than that they can’t afford to lose them to their competitors. The last couple of weeks have been extraordinary in the degree of pressure Anthropic and OpenAI have faced to contain this risk, coming from both the public and their own workers. One isn’t required to believe that Amodei and Altman are acting nobly to accept that they’re reacting to actual circumstances in the real world, both in terms of the risks of their technology and the public scrutiny being brought to bear on them. They had a collective action problem, and it was resolved when the cost of doing nothing became greater for each of them than the cost of doing something.

They can’t control the technology they’re building. Undoubtedly there are all kinds of nefarious corporate schemes at work in the background. But that doesn’t change the fact that the technology is what it appears to be. It’s a powerful force that’s alien to us and far beyond our current understanding. At the very least, it needs to be paused.

In the eleventh century, a German writer named Wenric of Triers lambasted Pope Gregory VII, accusing the pontiff of befriending conquerers who “usurped kingdoms by the violence of a tyrant, paved the road to the throne with blood, placed a blood-stained crown on their heads, and established their rule with murder, rape, butchery and torment.” Though the writer didn’t name names, the Pope understood who Wenric was referring to: William the Bastard, the Duke of Normandy who in the year 1066 conquered England and made himself king.

It’s hard to overstate the totality of the transformation wrought on England by the Norman takeover. For the English, it was apocalyptic. The north of the country was starved out. Castles, formerly unknown in England, dominated the landscape, and would pacify the population for centuries to come. Anglo Saxon landowners of modest prosperity and independence were pushed into poverty and serfdom.

It wasn’t just the English royal dynasty that was toppled and replaced by conquering Frenchmen, but the entire aristocracy. Written English was abolished and replaced by Latin for the simple reason that nobody with any power remained who spoke the native language. For the next several generations, Norman rulers would refer to their subjects as two distinct peoples, Angli et Franci, the former now unmistakably an underclass. The traditional landholding patterns of the Anglo Saxons was replaced by the feudal structure of the Frankish kingdoms, but in an even purer form than on the Continent, as they were fashioned from scratch and placed atop the blank slate of a devastated civilization. It was a social revolution from above.

Almost six centuries later came another revolution: the English Civil War. In the grandest of scales, one could see it as a reaction to the Norman Conquest. Many of its contemporaries certainly did. An unbroken line of William’s descendants had ruled England from the Battle of Hastings until the establishment of the Commonwealth (and, indeed, from the restoration of the monarchy until the present day). But when Charles I tried to bring French-style absolutism to England, the country rebelled. The king was beheaded, a military dictatorship ruled in the name of Parliament, and a decade of free speech, radical politics and communist Christian sectarianism flourished. The Levellers, one of those radical Christian sects, referred to the moment explicitly as an unshackling of the Norman yoke.

What had happened to usher in this democratic spirit? How was England, a country ruled for centuries by the descendants of conquerors and transformed into the purest form of feudalism, overcome with this thirst for freedom and self-governance?

It wasn’t the novelty of the concept of human equality and emancipation — those ideals had existed since the time of Christ. But the material reality of life in the kingdom had changed over 600 years. England in the seventeenth century was still, in many respects, a crude society, in which smallpox was prosaic, the bubonic plague was intermittently catastrophic, and parents formed little attachment to infants, knowing the chances of their reaching childhood age were about as good as the throw of a die. But some things had evolved. Among them was the division of labor in society.

In eleventh century England, peasants scratched a subsistence living out of the earth with their own labor, perhaps a beast or two of burden, and the most primitive of farming instruments. The situation wasn’t much different by the time of the English Civil War. But one thing had changed markedly: the lords whose lands they worked had been transformed by the Norman unification of the kingdom into something much different from their distant warrior ancestors.

The agent of this transformation was the rise of the nation-state, by which I mean, in Max Webers terms, the formation of a monopoly on the legitimate use of violence. One of the singular achievements of William’s reign was the drafting of the Domesday Book, a gargantuan ledger that accounted for every acre of every shire in England: who held it, what it was worth, how much of it was arable, how many settlements were upon it, how many slaves, how many freemen, how many ploughs, the tally of horses and cows and pigs. The Domesday book codified the new social order, turning land that was taken by criminal violence into the legal property of its new owners. It legitimized the new Norman aristocracy, and completed the replacement of traditional English landholding patterns with French-style enfeoffment. Though William banned the domestic slave trade and liberated scores of bondmen, in every other respect he made the country less free.

Under a single monarch, the colonizing aristocrats were no longer sovereign nobles, as they would have been on the Continent. In the centuries that followed the conquest, a fraction of them would battle intermittently over competing dynastic claims to the crown, but they were no longer perpetually at war with one another over their respective territories, as were their counterparts across the Channel in the various duchies of what would become France. They raised armies to serve the king in his pursuits abroad, but their own land was more or less pacified. This meant that they derived much less wealth from the booty of raiding and conquest, and it also meant that they were in a better position to invest in the long-term productive exploitation of their fields. The incentives arising from these conditions pointed in the same direction: toward commercialization of their fiefs.

This process unfolded, in fits and starts, over hundreds of years. It was accompanied by another tectonic transformation throughout Europe: the expansion of transnational trade. Much of the credit for this phenomenon belongs to the Vikings of the tenth and eleventh centuries. Their bloody, predatory expeditions throughout Europe carved the channels of cross-continental exchange that spanned the landmass from Scandinavia to the Ottoman Empire, and from Ireland to Russia. Trade generated new wealth, especially in the market towns along the rivers and sea coasts upon which the traders traveled.

Those market towns prospered for another reason, as well: warfare.

In feudal Europe, power was rooted in land. Land was the reward that lords bestowed upon vassals for their military service, and it was the reward those vassals bestowed upon their own vassals in turn. And it was the peasants who dwelled upon those lands that lords and vassals conscripted into their armies as foot soldiers.

The condition of constant war that the dynamics of feudalism perpetuated fueled a robust arms industry throughout Europe. The construction of castles, armor, and trebuchets required not peasants, but artisans and skilled laborers. These specialists congregated in the market towns, where they could acquire the materials brought by trade.

To maintain military parity with their rivals, nobles encouraged the growth and development of these centers in their territories by granting extraordinary freedoms and privileges to those who migrated to them. These included special property rights, the freedom to form guilds, and certain civil liberties. The charter of Dublin, for instance, allowed Dubliners to manage their own local affairs and administer their own justice. These town charters became templates. They were borrowed and duplicated by lords in distant kingdoms eager to attract skilled labor to their realms. A new transnational legal regime emerged. It became customary that town residents, known as “burghers,” held certain liberties and practiced limited self-governance. This understanding grew into a nascent spirit of republicanism in the urban centers of the Middle Ages.

By the seventeenth century, these two patterns — the commercialization of agriculture and the special rights of the bourgeoisie — came to a crescendo in England. Landed aristocrats squeezed profits from their lands by evicting peasants whose families had been there for centuries and replacing them with rent-paying tenants. These evictions created an underclass of paupers and near-paupers, many of whom crowded into the cities and formed an inchoate mob.

At the same time, the middling classes, with their rising wealth, had become self-confident and politically assertive. The despotism of an aristocratic class that no longer represented an obvious social function had long rankled these ornery Protestants, relegated, as they were, to second class status in the House of Commons. Their political dissent was expressed principally through their religion, for in the seventeenth century, this was the lens through which people saw the world. In rejecting the hierarchy of the church — especially the hated bishops — and insisting on each individual’s personal connection to God, Calvinism questioned the legitimacy of the entire feudal order. If rule by lords and nobles was not sanctioned by God, this logic held, then it could only be justified by the consent of the subjects.

The English Civil War galvanized the Calvinist burghers and yeomanry behind Parliament and against the crown. At the same time, the masses of paupers crowded into the cities — London more than any other — provided a discontented population that could be rallied to violence, sometimes on the revolutionaries’ behalf, at other times the king’s. England exploded.

For a brief period, the English Civil War and the establishment of the Commonwealth disrupted state censorship. Printing presses were cheap and ubiquitous. Pamphlets abounded, advocating radical new ideas. Dissenting religions practiced in the open. Revolutionary Christian sects emerged, like the True Levellers and the Diggers, that tripled down on the Calvinist tradition of arguing against the privileges and even the existence of clergy; if every person had a direct relationship with God, their reasoning went, then every layman was as holy as any priest. The individual was sanctified. In a later, more secular age this conviction would be expressed in terms of natural law and personal rights.

The French sociologist Émile Durkheim called this conviction, which is a hallmark of the modern age, “the cult of the individual.” He linked its genesis to the increasing complexity of the division of labor in society. As people’s labor becomes more specialized, he argued, each individual becomes more important to the economy and thus to the welfare of the larger community. Like the organs in a body, we become connected to each other in a vital network. We come to share a common fate. The individual becomes vested with the importance that we attribute to the society as a whole, just as every inch of your anatomy, from your head to your pinky toe, is part of your sense of self, and as such is inherently indispensable.

Democracy is the political manifestation of this social arrangement. The specialization of labor bestows every citizen or subject with a degree of bargaining power against the state, just as the burghers enjoyed with the king. That leverage is not just material, but, for lack of a better word, spiritual. It is the foundation of all of our individual rights, and it marks the evolution of English (and, in its turn, American) society from the Norman conquest to the modern age. Medieval history may bear no obvious significance to the age of Artificial Intelligence, but it in fact contains its most important lesson: that our labor is the root of our democratic power, and if we replace it, we can expect to lose every right that evolved along with it.

Some number of years ago I joined the Democratic Socialists of America. I paid my yearly fee and received a membership card. Then I failed to attend even a single meeting.

That record roughly reflects my feelings about the DSA. I support a lot of what they stand for, but am leery of them as an organization — too leery, apparently, to bring myself to directly participate in their activities.

I say all this just to disclose where I’m coming from politically with respect to the subject matter at hand. Almost everything you read about the DSA is either written by boosters or by haters. The DSA is polarizing — if you’ve heard of the group, you probably have an opinion about them. I won’t pretend that my political views don’t thumb the scale of the following assessment of them, but if you know at least what those views are, you can weigh them accordingly.

The DSA purports to be a tribune of the working class. Few who don’t possess a DSA membership card truly believe that. The group’s many detractors portray it as a bunch of young urbanites with liberal arts college degrees cosplaying as plumbers and construction workers. There’s truth in that description. But it also enormously underestimates their historical significance.

I’ve written more than once about how politics is a contest between two factions of the elite: those whose class status is largely a function of their accumulated economic capital, and those whose class status derives from their wealth in cultural capital. Those with economic capital tend, not surprisingly, to subscribe to political ideologies that champion the form of power with which they’re best endowed, by making everything subservient to the market. Privatization is good, they believe, entrepreneurs are heroes, government should get out of the way, and so on. The Republican Party has tended to embody these beliefs and thus secured their votes.

Those with cultural capital, for the same self-serving reason, tend to subscribe to ideologies that favor their form of power over that of economic capital. The educational credential is the formal certification of one’s accumulation of cultural capital, so it is the post-graduate degree, and all of the virtue and knowledge it is meant to impute to its holder, that this faction of the elite holds up as testament to its legitimate right to rule. Accordingly, those who are much richer in education than in money tend to believe that the marketplace is a corrupting influence, and that responsibility for government should be in the hands of disinterested experts with advanced degrees. The Democratic Party has tended to embody these beliefs, and it is toward the Democrats that this faction of the elite tends to gravitate.

In an earlier era, these elite factions might have been more honest about their parochial class interests. But we happen to live in an age in which there is only one legitimating ideology for state power: democracy. So both sides are forced to speak in universalist terms rather than in narrow, self-interested ones. They purport to speak not merely on their own behalves, but in the name of “the people,” which is to say, for the working class. The economic elites thus ridicule the cultural elites as out-of-touch snobs, cozy in their ivory towers but utterly helpless in the face of a flat tire, let alone in the real world grind of operating a business that generates wealth and jobs. Workers are best served, they insist, when capital, their own form of power, is allowed to work its magic. Those rich in cultural capital, in turn, regard the economic elites as a heartless, parasitic aristocracy, in thrall to Mammon and bent on enslaving humanity to their greed.

Politics is a contest in which these two elite factions smuggle their particular class interests into an exalted debate over the public good. This is not to suggest that such debates are inherently cynical, or conducted in bad faith; indeed, the most effective rationalizers are those who truly believe their rationalizations. But drinking your own Kool-Aid doesn’t change the fact that your ideology evolved to serve your elite class interests.

The DSA, like most progressive political groupings, is a vehicle for the cultural elites. You’re unlikely to find a lot of blue collar or even white collar workers at their meetings. More likely you’ll find creative class people or knowledge economy workers whose economic prospects have been stunted for a variety of reasons — the Great Recession, Artificial Intelligence, whatever. In this sense, they’re “larping” as working class. But only at their peril should DSA’s detractors dismiss this class faction as somehow fraudulent.

There is, in fact, no more transformational a political constituency than surplus elites — those who were promised a bright future that never materialized. Many, though not all, revolutions have been catalyzed by this group. Hong Xiuquan fashioned himself a messiah and led the Taipei Revolution after failing his civil service exam four times. Gerrard Winstanley led the communist Christian denomination known as the Diggers after his business as a London cloth merchant was destroyed by the English Civil War. Downwardly mobile surplus elites played pivotal roles as Crusaders in Medieval Europe, as Bolsheviks in the Russian Revolution, and as vanguards of the Great Awokening. They are a motive force in world history and their political formations should be taken very seriously.

And as easy as it is to ridicule the pretensions of this demographic to speak for the “working class,” that, too, should be taken seriously, if for no other reason than that every group that purports to speak for workers is equally disingenuous. This is a function of the Iron Law of Oligarchy, another topic I’ve written about here before.

The Iron Law of Oligarchy posits that any group that begins as an authentic workers’ organization will, by necessity, eventually become alienated from it. This happens because in order to succeed and grow, these groups need to navigate the world of elites in politics, media, and capitalist relations. To do so, they are compelled to elevate leaders who are rich in cultural capital, which usually means they’re formally educated and often come from non-working class backgrounds. Real world competition forces the organization to professionalize.

The American labor movement, for example, was utterly transformed by the generation of activists in the 1960s who championed Cesar Chavez’s United Farm Workers. The UFW was in bitter competition with the Teamsters and the AFL-CIO’s Agricultural Workers’ Organizing Committee for the right to organize farm labor in California. They won in large part by bringing in New Left activists like Marshall Ganz, who dropped out of Harvard. The cultural capital these outsiders brought to the UFW allowed them to out-strategize their rivals, especially when it came to things like gaining the sympathy of liberal elites. That generation of culturally elite union activists in the UFW then went on to lead UNITE-HERE and SEIU, the most dynamic unions in the country, which enjoyed enormous growth and success as a result. They easily outcompeted rival unions like the Steelworkers that had not brought cultural elites into their leaderships.

Detractors might assume that the domination of workers’ organizations by cultural elites necessarily alienates working class constituencies, but it can often result in precisely the opposite. Cultural capital, like economic capital, is an asset to any organization. It allows organizational leaders to navigate elite spaces in the political, academic and corporate worlds, building elite alliances and undermining elite adversaries. Elite-dominated unions are thus capable of winning substantial victories for the working class constituencies they serve, and thus securing the real allegiance of workers. The same can be said for socialist organizations like the DSA. Indeed, coalitions of cultural elites and workers have long been potent agents of historical change, as just about every successful communist revolution demonstrates.

But the lines of fissure are always there. As workers’ organizations professionalize, they begin to develop their own institutional interests, which are only indirectly aligned with the interests of their constituents, and sometimes are diametrically opposed to them. Unions cut deals with corporations that weaken their members’ contracts in exchange for organizing rights. Labor parties abandon traditional commitments to workers in order to “modernize,” raise corporate contributions, and win non-working class votes. Concessions such as these tend to pave the road of growth and advancement for organizational bureaucracies, and the bureaucrats who run them advance alongside the organizations they lead. The interests of organizational staff slowly diverge from those of rank-and-file membership.

These fragile cross-class coalitions often collapse under strain, blunting the radical possibilities of revolutionary moments. In revolutionary France, the Jacobins, based in the cities, instituted price controls on bread to ease the plight of urban consumers, a demand of the radical sans-culottes. This was to the detriment of rural peasant producers, who then abandoned the revolution; it was at this moment that the French Revolution came under the full control of the bourgeoisie instead of proceeding in a more radical direction. Likewise, after the English Civil War, the ascendance of Oliver Cromwell spelled the end for communist-minded agitators like Winstanley and radical sects like the Diggers and the Ranters and the True Levelers. The Commonwealth became an administration of the bourgeoisie and ultimately ended in a restoration of the monarchy. And closer to home, the economic elite’s embrace of working class populism as a weapon against the cultural elites was abandoned almost as soon as that faction took power.

The DSA has shown itself to be full of these class contradictions, readily adopting the avant-garde political pretenses and woke dogmas of the cultural elites who fill its ranks, however anathema they are to working class Americans, like clapping bans, land acknowledgements, and mandatory pronoun disclosures. If put to a real test of its loyalties, I suspect the DSA would side with the downwardly mobile professionals that actually run it over the working class Americans it pretends it answers to.

But that test has not yet arrived, and in the meantime, the happy alignment of the purported virtues of the cultural elites with the interests of workers, reflected in policies like universal child care, represent perhaps the only pro-worker force left in the Democratic Party. If the DSA is larping, they should keep doing it. If they succeed in taking over the party, at least it will come to stand for something.

I read this very persuasive piece by Scott Carney yesterday about why Substack hasn’t lived up to what a lot of writers hoped it would. I dragged Substack in a post just a little more than a month ago, so I’m a bit reluctant to return to the subject, as I don’t want to turn this newsletter into a navel-gazing exercise where I ask you to read about the practice of writing what I’m asking you to read. That said, as this has just become a blog anyway, I no longer aspire to hold up any kind of professional standards. So that’s a prologue to basically say I’m going to talk about Substack once more, against my better judgment.

Carney’s piece articulates what I’m guessing is intuitively obvious to most people who read and write on this platform: there are only so many paid subscriptions a reader is going to spring for, especially on top of all the other subscriptions to apps, streaming services, etc. that are constantly chipping away at our bank accounts. Eventually, Substack was going to hit a demand ceiling. That seems to have happened, as Carney points out, to Taylor Lorenz.

Taylor Lorenz has an order of magnitude more “followers” than probably 99 percent of people on Substack, and more internet fame than probably 99.9 percent of us. If it’s happening to her, it’s probably happening to everyone downhill from her, too.

Substack solved a particular problem for a particular type of journalist at a particular time. The company was founded in 2017. That was a few years into Black Lives Matter, and, of course, the first year of Trump’s first term. The Great Awokening was well underway, and over the next few years, culminating in the summer of 2020, the media industry would come to resemble a grumpy religious sect, with its theological dogma and mechanisms for punishing dissent. Journalists who refused to bend to the new rules had a refuge to escape to in Substack. That is, quite literally, the origin story of how Bari Weiss founded the biggest media outlet on the platform, The Free Press.

But while Substack solved a problem for journalists, it did not solve the problem of journalism, which is that its business model was broken into pieces two decades ago and has yet to be fixed. In the twentieth century, journalism was propped up by mass advertising, which, as Andrey Miroshnichenko has pointed out, was effectively a tax on corporations to sustain the jobs of journalists. Companies couldn’t find their customers without getting their brand names in front of them, and the only way to do that was by running advertisements in the newspapers that showed up on everyone’s doorstep each morning and on the TV shows that were piped daily and nightly into their living rooms. This was the marketing equivalent of trawling: it caught a lot of fish by casting a net wide enough to scoop up everything in its path. It was inefficient and expensive, and it produced enough revenue to sustain an entire profession of reporters, editors, producers, and news anchors.

But then came Facebook, and suddenly, instead of trawling, advertisers could fly fish, but on an industrial scale. Targeted advertising allowed them to pick their customers one by one out of the teeming mass of humanity on the internet, and for a sliver of what it cost to run a TV commercial or run a newspaper ad. Mass advertising became a relic of the twentieth century, and the economic floor it sustained for the journalism industry cratered. Laid off reporters were as mundane a part of the new economic landscape as 20-something software engineers making doctors’ salaries.

Substack created an alternative to this business model by shifting the journalism tax from advertisers directly onto news consumers. Now, instead of indirectly subsidizing reporters’ careers by buying products from the companies that market them through print ads and TV spots, readers can pay them directly via the online marketplace of journalism that is Substack.

The problem with this model is obvious: it will never come close to producing revenue on the scale that mass advertising generated for media corporations. From the perspective of advertisers, of course, that’s a good thing: their profits aren’t being siphoned off by an industry they have nothing to do with. From the perspective of news consumers, it sucks. It means less journalism, and journalism of a much lower quality.

The upside, some might argue, is that journalists are now truly independent. When you’re not controlled by editors and producers who answer to corporate bosses who answer to shareholders, you can report candidly without fear of professional repercussions. There isn’t an upper floor of suits to stop you from running a story on the corruption of defense contractors because the toothpaste company that’s the network’s biggest primetime advertiser owns an overseas subsidiary that manufactures land mines. Your only bosses are your readers, and what they expect of you is honesty and integrity. The incentive to do good journalism aligns with the incentive of making money. That’s the theory, at least.

But that theory both overstates the level of corporate control over the old model and understates the malign influences insinuated into the new one. Noam Chomsky might have you believe that New York Times editors once quaked in fear of the punishment they would face from their industrially polluting advertisers if they reported too honestly about industrial pollution, but the nice thing about mass advertising was that there was so much of it that individual advertisers rarely had that kind of clout. There may have been some vague sensibility among media professionals about not wanting to fall too far afield of the establishment, but that wasn’t rooted in anxiety over advertiser boycotts; it was rooted in publishers sharing the same alma maters as Congressmen and corporate executives and their kids going to the same private schools.

If anything, the pressures that the new model exerts on journalists are more direct and specific. Journalists who work for major media outlets are salaried employees; journalists who sell Substack subscriptions are petty entrepreneurs. At least in establishment media outlets there are institutional divisions between the editorial side and the business side. When you’re selling your reporting directly to readers, by contrast, there is no firewall in your brain that separates the part that’s thinking about what you’re reporting and how you’re reporting it from the part that’s thinking about how this story will impact your subscriber base and thus your bank account. This makes Substack ripe for audience capture. You have very good reasons not to report on stories that might upset your readers or to arrive at conclusions that might anger them. You have equally good reasons to serve them slop that confirms their priors. After years of watching your subscriber graph go up and down and the Stripe payments to your bank account get bigger and smaller, eventually you shape your newsletter into a product. And since your newsletter is, effectively, you, you have made yourself into that product. You might not have to worry about some corporate suit telling you what you can or can’t say, but that’s only because you are that suit and you’re telling it to yourself. The call is coming from inside the house.

If Taylor Lorenz’s experience is indicative of a larger pattern, these perverse incentives are only going to get worse as people’s paid subscriber bases dry up. Substack’s executives would have you believe that the consumer market for email newsletters is only getting bigger, but it’s easy to imagine another possibility. As Carney points out, Substack is growing its user base principally by attracting more people to its Twitter knock-off, Notes. That doesn’t mean they’re generating more paid newsletter subscribers, and particularly not more paid newsletter subscribers for Substacks that aren’t already on the leaderboards (why do they even have leaderboards?). People who like to scroll aren’t necessarily people who pay to read long form essays. My own paid subscriber count has proceeded along the same downward path that both Lorenz’s and Carney’s have, even as my tallies of free subscribers and “followers” (whatever that is) have stacked up. If that’s happening for many others, we can expect that as paid subscribers become scarcer, the incentives to gain and retain them will become harder to resist. More and more Substack writers will conform to the demands of the market by flattering their readers, avoiding risky topics, arriving at prefabricated conclusions, and repeating formulas that have worked in the past. I’ve seen it happen over and over again.

Substack hasn’t fixed the broken media model — it hasn’t even attempted to. All it has done is giggify it. That has given some established reporters the freedom to do their own thing and made a handful of already famous and wealthy journalists even more famous and wealthy — good things, I guess. But it has also likely further degraded the already broken down institutional structure of the journalism profession. That might bring joy to a few nihilists who simply despise the establishment media, but that doesn’t make it progress. It just makes it another step in the death of the craft.

Short of dropping a nuclear warhead, the scale of Israel’s bombardment of Gaza was only physically possible through the use of Artificial Intelligence. In the early weeks of the war, the IDF was using so much of its munitions stockpile it was afraid it would literally run out of bombs, which it would need were a new front to open up in Lebanon, as it eventually did. The question was not one of how much to bomb, but how to maximize the lethality of each bomb dropped. This is where AI came in.

Israel deployed an AI system called “Lavender” to pick targets, and another one called “Where’s Daddy” to execute the strikes. Both were described in detail in 2024 by +972 Magazine. Lavender combed through an enormous amount of information on the residents of Gaza, including social media connections, geolocation data, phone call records, and biometric data, to assign every Gazan a threat score on a scale from 1 to 100. One’s score would constantly go up and down depending upon one’s behavior — whose houses you visit, who you call, who you befriend on Facebook, etc. If your score reached a particular threshold, you were put on a kill list.

The threshold could vary from day to day. When the IDF had plenty of targets, it might remain relatively high. When targets were scarce, Israel just lowered the score that would put you on a kill list, in order to generate more targets, which is to say more Gazans to kill.

If your name found its way onto the kill list, you would come into the aperture of Where’s Daddy. Where’s Daddy tracked your movements, waiting for you to return to your home. Once you did, the AI would send out an order to bomb your building. This meant that your family would likely be incinerated along with you, as well as whatever other families happened to live in your apartment building. It also virtually guaranteed that the overwhelming number of deaths would be non-combatant women and children.

It’s probably safe to stipulate that the Gaza War represents the extreme end of what is possible, with the current state of AI technology, in terms of the coercive power of the state. Post-October 7, the Israeli government became unconstrained by such intangible concerns as human rights, international law, or political legitimacy in carrying out its campaign to annihilate Hamas. The only limiting factors were physical ones: how many bombs the IDF had, and whether they could find their targets. Boiled down to the most basic of variables, these were laboratory conditions for measuring the degree to which the state’s capacity for violence has been enhanced by AI.

Subscribe now

In most places at most times, those conditions do not hold. Governments aren’t usually at war, and the physical power they wield over those who fall under their control on a day-to-day level is typically either threatened or assumed, not actually made manifest. We know that if we commit a crime, the government has a certain likelihood of finding out, and of identifying, capturing and imprisoning us. That likelihood is obviously increased with new surveillance technologies. But few of us actually test the hypothesis. We assume that we probably can’t get away with breaking the law, so we more or less obey it. A small portion of the population, of course, does commit crimes, and if and when they’re captured and imprisoned, they stand as a reminder to the rest of us that the government is still in control. That’s generally enough to maintain social stability; the state does not have to actually arrest and imprison each one of us to prove that it can do so.

This makes it a lot more difficult to gauge just how much increased control a democratic, peacetime government has over its population as a result of AI. Absent an armed insurrection, we can’t see for ourselves what the state could do to keep us in line if it were sufficiently motivated to do so, and if it were unimpeded by things like morals, laws, and concerns over its legitimacy. We can only imagine it. But Gaza gives us some theoretical ceiling. There is no technological reason why the U.S. government could not do to us what Israel has done to the Palestinians in Gaza. It obviously has the requisite firepower, and the AI tools it could access are, if anything, even more formidable than the ones Israel has used.

That said, there certainly are things that stand in the way of the U.S. government exerting the kind of coercive control over Americans that Israel has wielded over Gazans. It’s just that they’re not physical, but social in nature. We have certain rights afforded to us as citizens, or even as non-citizens residing in the territorial boundaries of the United States. The government isn’t allowed to do whatever it wants to us, as the Israeli government can to Palestinians. There are functioning laws and courts and elections. There are also fuzzier things like values and norms and traditions. These all act as powerful constraints on the government. But they’re not immutable ones. They’re not properties of physical reality. They’re not like running out of bombs, or lacking the technology to find and locate people in time and space. Under extreme social conditions, they could melt into air. The legal and normative system could break down. It’s easy to imagine that happening; in fact, it’s the premise of every post-apocalyptic sci-fi novel you’ve ever read.

So if the answer to the question “What happens if, with today’s technology, there’s a civil war or a social revolution or another total national emergency?,” is “Something like Gaza,” then a new question is begged: “What happens in the more likely scenario that social stability more or less endures in America, even as the technological capacity for social control by the state increases exponentially as a result of AI? Will we live under democracy or tyranny?”

I hate the analogy because I care about animal welfare, but it’s sort of like the frog asking what temperature the water is as it slowly approaches a boil.

Imagine a Lavender-type system that scores you on multiple threat scales related to national security. It tracks your level of “political extremism” across a matrix of issue categories. It scores you for how you will likely act upon your beliefs, whether it’s just social media posting, or attending a protest, or giving money to “extremist” groups. It gauges how influential you are within your social networks. It tracks whether you’re networked with others who have high threat scores, and how likely they are to influence you. Out of this it estimates the level of menace you present to the social order.

It’s hard for me to imagine that this does not already exist, or at least some variation of it. Certainly the technology exists; it’s basically a political application of algorithmic pricing, which is happening all the time and has been for years. You can get charged more or less for online transactions depending upon your ZIP code, your browsing history, or whether you use a Mac or a PC. Princeton Review has charged more for its test prep services to Asians than non-Asians. Uber has been accused of surging prices for users with dying phone batteries. Former FTC Chair Lina Khan cites the possibility of a retailer charging you more for a thermometer because it knows you have a newborn at home and are likely to be desperate, or an airline quoting you a higher price on a plane ticket because you received an email about funeral arrangements.

If corporations can track your behavior to determine how much you’ll pay for a product, there’s no reason the government couldn’t track it to assess whether you might engage in risky political behavior, as China has been doing for years. And if the U.S. government isn’t already doing so, then its incentives to adopt a Chinese-style social credit system will only increase with the advance of AI.

In China, if you’re a model citizen you can get discounts on bus fares. But if your purchase history indicates you’re an alcoholic, or if you post criticisms of the government on Weibo, or if you’re a journalist exposing political corruption, you might find yourself suddenly unable to travel when you go to the train station and are denied the purchase of a ticket. Your face might appear on posters in public or on apps that show who around you is a social exile.

None of this is likely to ever happen in America as long as we have a functioning Constitution, but there are more subtle and more legal ways to achieve the same end.

Imagine a social credit score that’s less like China’s in-your-face system and more like a FICO score. You don’t necessarily know what it is at any given moment. Perhaps the government doesn’t officially admit that it exists at all, like Israel and its nuclear arsenal. But like algorithmic pricing, you’re vaguely aware that it’s operating in the background, shaping the contours of your daily life. You can feel it in the ether around you as your social habitat tightens or releases its grip on you. The higher your threat score is, the more expensive things become, the harder it is to get a job, or to get your kid into private school, or to get a window seat on an upcoming flight.

It doesn’t operate at the textual level, as it does in China. In China, there’s no mystery to your status; you can look up your social credit score at any time. When you’re on a blacklist, you’ll be told so when you try to take out a home loan or even rent a hotel room.

In the U.S., things would be less explicit, more like a vibe. Here, where we have laws that are supposed to protect us from such things, the touch would be softer and harder to trace. You wouldn’t necessarily even know it’s happening. You’d be aware you’re paying higher prices than you were a year ago, and you’d suspect they’re higher than for other people around you buying the same stuff. But you’re not entirely sure. You’ve never had a problem getting insurance before, but suddenly you find every insurer quoting absurd prices or just denying you any policy at all. It’s weird and disorienting and anxiety-inducing.

Share

These could just be the mysterious workings of the free market. Or they could be that you’ve been singled out. It’s like being shadow-banned on pre-Elon Twitter: one day, after a particularly spicy post, your engagement would just drop off a cliff. It would seem pretty clear to you that your account had been flagged, but you couldn’t prove it. You could complain to whoever you wanted to, but there was no official notice, there was no appeals process, there was no way to know for sure that you were being throttled. It’s possible that your opinions just suddenly became a lot less interesting to your followers.

Likewise, maybe the reason you’re no longer finding matches on Hinge is just because you’re getting older and less cool. But maybe it’s because you’ve been posting a lot about the Palestinians, or questioning vaccine efficacy, or disputing election results. Maybe your account is being suppressed, just like it feels like it is on all your other social media platforms, and maybe that’s related to the crazy prices you’ve been paying lately on your online prescription drug service, and the months you’ve been waiting for your passport to be renewed, and the surprise visit last week from a city inspector following up on some anonymous complaints about the kitchen renovation done by the previous owners of your house being unpermitted. Or maybe none of those things are related at all.

You’ve also noticed you’ve been getting served up a different flavor of media content in your newsfeeds. You’ve started getting recommendations for podcasts about how there’s a civilizational war underway between the West and the Islamic world, and also tons of posts from people you aren’t connected to and aren’t sure are real rejoicing over how your local data center is going to create tons of new jobs in your community. Your streaming services have started promoting shows to you that you’ve never heard of. One is about the plight of a gay woman in an oppressive Muslim country. Another is about a computer genius who invents an AI that cures Alzheimer’s Disease. You haven’t been shy lately about your opposition to the war heating up in the Middle East, and you signed an online petition just yesterday against the proposed local data center construction. But maybe that’s just a coincidence.

But anyway, your trust in mass media ran out a long time ago. When you want to know how to make sense of the world, you listen to your favorite independent journalists and podcasters. But lately you’ve been questioning how “independent” they really are. For some reason, their political opinions all seem to have converged, and then evolved in the exact same way. They were once isolationists; now they’re all suddenly concerned about some obscure U.S. enemy you’ve never heard them mention before. They used to have regular meltdowns about environmental collapse and the end of civilization; now they’re claiming it will all be solved by stable coins. You’ve been surprised to notice that unlike you, their fans in the comments section now agree with them about all of these things. You also notice that they’re getting some pretty impressive sponsors these days.

Maybe you’re being paranoid. You’re seeing patterns that aren’t there. Or is it obviously real and you’d be naive to think otherwise? You’re not quite sure what to think, and whether suspecting one thing makes you a conspiracy theorist or believing the other makes you a sheep.

But what you do know is that it feels like things go more smoothly in your life when you’re not worked up about something or the other in the news. When you disengage, when you stop paying such close attention to what politicians or the big corporate oligarchs are doing, the world feels more frictionless. Things go according to plan. It’s like you have good karma or something.

This time, though, there’s an actual war shaping up, and you’re about half sure you’re going to lose your job by next year to AI. This is serious stuff, and your convictions can’t be so easily swayed. So you continue to post your opinions about U.S. militarism and the dangers of AI and the tech industry. You continue to show up to protests when you see them announced on Facebook, and you even occasionally make donations to the growing number of groups the government has begun regularly maligning as “extremist.” Things are getting costlier and more difficult and maybe it’s because you’ve been flagged by the system, but you don’t care. There are bigger stakes in the world.

Then, for the first time in your life, you get called into the local IRS office. You’re put through an excruciating audit and have to pay a ridiculous penalty to the Franchise Tax Board. At the same time, you start seeing your social media accounts “temporarily” suspended, one by one. You can’t figure out why; the platforms just keep saying there have been complaints and they have to investigate.

Then early one morning, the police knock on your door with a search warrant. You’ve been accused of financial crimes and they’ve come to take your computer.

Eventually you get the message. Your opinions haven’t changed — if anything, they’ve become even more strident — but you’ve already lost a romantic relationship over the stress, your friends have stopped returning your texts, and your co-workers don’t trust you anymore.

So you just back off. You stop being so loud in your opinions, pay less attention to stuff done by powerful people you can’t control anyway. And as you do, things start to fall back into place. The investigations of your social media accounts are resolved in your favor. You finally get your passport renewed. Your dating life picks up. The city inspection comes to nothing. The government drops its case against you, citing insufficient evidence. Instead of losing your job to AI, you get promoted. Things are going nicely. You’re happier.

Share Social Studies

Maybe the scenario just described sounds like something out of Black Mirror. But then so does using robot dogs in warfare, which was an actual episode of the show and is now real life.

The U.S. isn’t China. We don’t have a clearly and explicitly authoritarian state to which the private sector is unquestionably subservient. But already, you can see the fusing of politics and commerce that would be necessary for such a vast social credit system to emerge in the absence of single party rule. The line between the public and private sectors in America has always been blurry, but today, with a president making billions off of cryptocurrency, the world’s richest man being tasked to run a new quasi-agency of the federal government, and one of the biggest tech investors in the world writing the administration’s AI policy, it’s unclear that there’s any line left at all.

And it’s not hard to see how sectors outside of tech could be corralled into the mix. The Treasury Department routinely puts out advisories on money laundering operations, sanctions evasion schemes and other alleged financial crimes and asks banks to issue red flags for these practices. The banks are then required to report suspicious activity to the government. This seems like common sense, but under the Trump administration, we’ve already seen it begin to reflect the administration’s political priorities, such as in a recent advisory about employers hiring undocumented workers. One can imagine myriad ways this could become even more politicized than it already is. And one can imagine similar memos going out to social media platforms and other private service providers.

A liberal administration intent on repressing its political enemies could direct banks and social media companies to root out the financial infrastructure of right-wing “domestic extremists.” We’ve seen such efforts before, as when the Southern Poverty Law Center repeatedly accused PayPal of being “the banking system for white nationalism,” before it was revealed that the SPLC itself was laundering money and funding white nationalists as paid informers.

A conservative administration could do the same thing. In 2021, PayPal teamed up with the Anti-Defamation League to root out “hate groups.” Today, the ADL routinely accuses anyone critical of Israel of being “anti-semitic.” One could easily envision the Treasury Department issuing an advisory asking banks and other corporations to help protect Jews by dismantling the financial and communications infrastructure of “extremist” groups that oppose Zionism. It’s almost surprising it hasn’t happened already.

Likewise, corporations could use the same shadow social credit system to target their own enemies, whether they’re antiwar protesters, environmental activists, unions, or just lawful competitors.

The IDF soldiers who informed +972 Magazine about Lavender and Where’s Daddy noted repeatedly that the systems were built on statistics. Their kill lists may have included many civilians who had nothing to do with Hamas, but in the aggregate, they were statistically accurate enough to justify ignoring these inconvenient tragedies. What AI enabled Israel to do is to take a population of millions and, to borrow a term from the late political scientist James C. Scott, to render it legible. It could divide that amorphous mass of humanity into a neat hierarchy of numbers that could be dropped into a spreadsheet. And then the extermination process could proceed in an orderly and rational process, like culling sheep from a flock.

Minus the killing part, we should expect no less of AI when it comes to governing the rest of us. As our jobs are replaced by machines, as our free time becomes limited only to the hours in the day and our sense of purpose is sapped from our lives, we will become more unruly. Rather than a citizenry or even a labor resource, we will be seen by the state as, principally, a threat to be managed. That threat will require new methods and technologies of surveillance, assessment, scoring, categorizing, rewarding and punishing. If there’s a Universal Basic Income, it could be pegged to the degree of our obedience as measured by our social credit score. There will be those who rebel, and whose punishment will serve as an example to others. But they’ll be marginal. At a population level, we will be evaluated, segmented, and incentivized into docility. America will finally be domesticated.

For America’s 250th, I wrote this piece about democracy and the myth of American exceptionalism for UnHerd.

Hope you’ll give it a read.

—LW

For a few years before the pandemic, I used Airbnb pretty regularly. At the time I was making a lot of short documentaries, mostly for The Intercept and Univision. We were typically on the stringiest of shoestring budgets, and it was way cheaper to lodge our crew, usually three people, through Airbnb than in hotels. But it was also kind of preferable. When we were lucky we’d find some pretty charming homes in residential neighborhoods with plenty of street parking that were just a few minutes’ drive from restaurants and breweries (we drank a lot on those trips). We didn’t have to deal with valets or parking fees or patronize overpriced hotel restaurants.

But in the last five years, I can probably count on one hand the number of times I’ve used Airbnb, and most of those were for vacation house rentals that I could have just as easily found on VRBO or by sending an email to Sea Ranch. When I go on work trips I book hotels. I don’t even browse home share listings. I think the reason is that nowadays when you find a place on Airbnb, it’s almost certainly going to be a cheap investment property owned by a real estate corporation, furnished by Ikea, with a keypad by the door and a clean, empty fridge with some Poland Springs water and a single bottle of mustard in it. It will be practical but completely devoid of charm, perhaps even bordering on depressing. The individual homeowners who used to host Airbnb rentals have long been pushed aside by this nameless blob of corporate middleman agencies. I’d rather just stay at a Holiday Inn.

It feels to me as if the same thing that happened to Airbnb is now happening to Substack. In April, the company hosted a party in DC that was billed as an alternative to the White House Correspondents Dinner. According to the Washington Post, Substack had tried to buy a table at the actual WHCD, which, as it turned out, was abruptly adjourned by an assassination attempt on President Trump. The director of the White House Correspondents Association turned down the offer, declaring that “independent journalists were not news, would never be news, and would never be invited to the dinner.” So Substack hosted its own party instead, which became “one of the hottest tickets of correspondents’ weekend.”

The WHCA director denied the quote, and frankly, it sounds highly implausible to me. A legacy media organization saying such a thing about Substack seems about as likely as a scheming villain saying, “and I would have gotten away with it if it weren’t for you kids” after being lassoed by the Scooby Doo gang. It’s not only just too childish and undiplomatic to be believed, but it also happens to perfectly cast the company as the heroic underdog vis-a-vis the arrogant and out-of-touch Establishment Media, which is fitting for a quote that was sourced from Substack’s Head of News and Politics. In this story, Substack is the Molly Ringwald character in Pretty In Pink, rejected and humiliated by the sneering rich kids only to come out on top, as a karmic reward for her virtue and humility.

Share

From an outsider’s point of view, Substack’s “alternative” party didn’t sound all that different from the main event. Jim Acosta was there. So was John Dickerson, former host of CBS’ Face The Nation, and Tara Palmeri, formerly of ABC, CNN, and Politico. So was Gavin Newsom and Sean Spicer, Meghan McCain and David Hogg, Joachin Castro and Marianne Williamson. If the party was meant to celebrate the ascendance of the “independent creator,” it achieved the opposite: it showcased Substack’s glee at being colonized by the establishment it was supposed to displace. The contradiction between these two postures was illustrated almost too vividly by Michael Tracey’s near fistfight with Acosta.

I’m picking on a party from months ago, but only because it’s emblematic of what I suspect almost everybody now recognizes about Substack’s trajectory. The top political publication at Substack is The Free Press, a $150 million company now owned by Paramount Skydance, whose founder now heads up CBS News. Next up is The Bulwark, which brings in $12 million in annual revenue and which was founded by a Republican political consultant, a journalist who has been a star on conservative talk radio since the early 90s, and the neocon founder of The Weekly Standard who almost single-handedly engineered the propaganda campaign that led to the Iraq War. The number three Substacker in technology is one of the most famous Silicon Valley venture capitalists in the world, who co-hosts a massively successful podcast with the guy who wrote the Trump administration’s AI platform. The Substack leaderboards are filled with celebrity household names, while the New York Times masthead is populated by people you’ve never heard of. Who is the media establishment here? Beats me.

I don’t mind Substack patting itself on the back for its success, which has been fairy tale-level remarkable. For decades, the legacy media has badly deserved to be disrupted, and Substack is the company that pulled it off. But the social impact of that success has been, on balance, a wash. The stars of the media establishment have not been displaced by Substack, but buoyed by it. That might be a good thing if you believe that Jim Acosta has finally been released from the choke chain CNN executives kept wrapped around his pit bull-like collar and will now revolutionize DC punditry. But if there’s a single person in the world who believes that, it’s Jim Acosta.

Airbnb once disrupted the hospitality industry, though hotels still seem to be doing fine. Not unlike Substack, the company boasted incessantly about how it was the champion of regular people in desperate need of supplemental income. What it became was a platform for corporate middlemen to corner a new market that peeled a few customers away from Hilton, Marriott, and La Quinta Inn. The power structure remained intact, but now Airbnb is a modest part of it.

That’s what Substack is. With a handful of exceptions, few are making real money on it who weren’t already getting paid extremely well in media or business. The company has tilted the balance of power in the media industry, but only between the executives in the C-suite and their on-camera or in-print star talent. And the legacy brands are already joining the party, like they did when Facebook, YouTube, and every other tech media disruptor started chiseling away at their business model. There is nothing revolutionary about this. It’s your grandmother’s capitalism.

Substack is great blogging software, and its subscription platform is a handy tip jar. The discoverability it provides through Notes and Recommendations probably compensates for Elon Musk burying its links on X.

These are nice things. They’re not great things. They don’t challenge anyone in particular. They don’t topple media empires. They don’t discover great talents in the wilderness and bless them with magical wings. The Substack platform is a feature. It’s fun and it’s useful. It gets a lot of attention and generates a lot of money for its investors. It’s Katie Couric at a house party. It’s Mehdi Hasan flying commercial. It is what it is, which is really not all that much.