Astral Codex Ten Podcast

Astral Codex Ten Podcast

By JeremiahScienceTechnology
Download on the App Store

All Astral Codex Ten Podcast episodes

  • Open Questions On Open Weights

    Last month, some of Silicon Valley's biggest companies signed an open letter supporting open-weights AI.

    Open weights AI is like open-source software, where the creator makes the raw code publicly available for free download. It's good insofar as it's the only way an AI can truly be the user's property, as opposed to something that companies like OpenAI or Anthropic temporarily let you use subject to their corporate guidelines and increasingly-nanny-state-like restrictions. If AI becomes the linchpin of the future, open weights AI feels like the sort of thing that could be the difference between being free yeomen vs. corporate serfs.

    It's bad insofar as it removes the possibility of gatekeeping and lets criminals commit crimes with it. Open weights AI could be used for hacking, child pornography, harassment, or terrorism (the weights can't commit the terrorism themselves, but they could give bomb-making or bioweapon-making advice). Since AIs have gotten very good - maybe superhuman - at hacking lately, the specter of a world where anyone can hack any site has gotten people grumbling that maybe open weights should be banned. It doesn't help that China produces the best open weights AI, making the idea seem foreign and almost unpatriotic. Proponents counter that "when AI is outlawed, only outlaws will have AI", arguing that bad people will get open weights AI regardless, and good people can use open weights AI to defend themselves. With the recent open letter, companies including Microsoft, NVIDIA, OpenAI, Intel, Amazon, Meta, Hugging Face, and over a hundred others have come out in favor of this position.

    Who's leading the other side? Nobody's admitted to it. Some parts of the Trump administration lean anti-open-weights on China hawk grounds, but have stopped short of explicitly asking for a full ban. Anthropic, the most notable omission on the pro-open-weights letter, made an ambiguous statement supporting "open-weights models that don't have dangerous capabilities" - but the industry expects open weights models to have dangerous hacking capabilities within a year, and AFAICT the letter didn't address that beyond inviting readers to draw the obvious conclusion.

    In the absence of a more obvious opponent, some open weights supporters suspect our conspiracy - the loose band of AI safety advocates, effective altruists, rationalists, and pause activists who worry about existential risk from superintelligence. This is a reasonable inference. By design, open weights AI is outside centralized control, and so impossible to permanently align against either human misuse (eg terrorism) or loss of control (eg AI turning against humans). Even if its creator trains it not to hack, anybody in the world can download the weights and retrain the AI to hack all day long.

    But in fact, most AI safety organizations have remained quietly neutral, and I don't know of any who make this a centerpiece of their activism1 (though I'm not 100% up-to-date on the whole landscape; if you know of one, tell me). A few have proposed policies that are contingently incompatible with open weights AI existing, but they all frame it as collateral damage rather than something they're excited about eliminating.

    I'm also neutral about open weights AI. I think it probably won't be long-term sustainable, but I'm happy to wait for this to become clear in the normal course of things rather than expend effort and political capital to ban it immediately.

    https://www.astralcodexten.com/p/open-questions-on-open-weights

    11 min
  • The Beauty Of Settled Science

    Commenters convinced me I made a subtle mis-step in my framing of Psychology Research Is Mostly Fine.

    My framing emphasized that social priming studies had failed to replicate, but that many other subfields of psychology hadn't. This naturally led commenters to point to failed replications in other subfields. Although none of these were as bad as social priming, few or none were entirely problem-free.

    Instead of emphasizing the difference between social priming and other subfields, I should have emphasized the difference between settled science and novel research.

    Here are some of my favorite psychology findings:

    https://www.astralcodexten.com/p/the-beauty-of-settled-science

    13 min
  • Does Forecasting Have Room At The Top?

    Superforecasting is the art/science/sport of predicting the future - for example, who will win elections, which countries will fight wars, when key technologies will be discovered. Over the past few years, it went from an obscure academic subfield to a multibillion dollar industry in the form of prediction markets. More recently, AI superforecasters have come close to the accuracy of top humans, and their performance is rising rapidly. In a year or two, we'll see one of the following patterns:

    Either humans have already come close to some fundamental limit on the predictability of world events - in which case AIs will plateau at or slightly above the human level - or the trend line will continue until AIs are far beyond top humans. By analogy to superintelligence, the natural term for the second situation would be "superforecasting"; since that's already in use, we can cringely call it "ultraforecasting".

    Daniel Reeves1 makes the case for scenario A here. He describes a study he coauthored in 2010, which found that, on a variety of questions related to sports games and movie box office receipts2, prediction markets only outperformed simple boring statistical models by 3-6%. Maybe those statistical models are close to the best that it's possible to do; the rest is what the mathematicians call aleatoric uncertainty - irreducible complexity downstream of chaotic systems that entirely resist modeling.

    He could be right. This post isn't meant to be a decisive refutation, but rather a description of why I'm still about 70-30 expecting Scenario B.

    https://www.astralcodexten.com/p/does-forecasting-have-room-at-the

    24 min
  • Your Book Review: The Epic Of Gilgamesh

    Finalist #4 in the Book Review Contest

    [This is one of the finalists in the 2026 book review contest, written by an ACX reader who will remain anonymous until after voting is done. I'll be posting about one of these a week for several months. When you've read them all, I'll ask you to vote for a favorite, so remember which ones you liked]

    Introduction: He Who Saw the Deep

    I've been thinking, lately, about the first epic story ever written. If you know anything about the Epic of Gilgamesh, you know that it's old. This is part of why I find it so interesting, but not the only part.

    The best comparison I can think of is the pyramids, which are just a little older. The Great Pyramid of Giza was built in the 2500s BC, around the same time as Stonehenge1 , twenty-five centuries before Cleopatra lived. It predates candles, concrete, and coins; it was centuries old when the first empire rose and when the woolly mammoth went extinct.

    Around 100 BC, Antipater of Sidon wrote a list of the seven most impressive sights ('Seven wonders') of the world - the pyramids were by far the oldest of the group at the time, and now, two thousand years later, all the others on the list have long since been destroyed. This should remind us that the pyramids don't just stretch far back in time - they stretch forward as well. They are piles of stone. They may be the product of the greatest minds of their age striving to put more stone in a more orderly pile than anyone had before, but still, they are piles of stone, basically artificial rock formations. They will still be there in a hundred thousand years, probably a million. They are so old they brush up against the beginning of civilisation, and yet, we are seeing them today in their youth.

    1 hr 14 min
  • Highlights From The Discourse On The Hugging Face Incident

    [Original here. Like a Highlights From The Comments post, but including discussion from around the Internet.]

    Roon is an OpenAI researcher whose popular Twitter presence has catapulted him to the status of extremely-unofficial company representative. He seems spooked:

    https://www.astralcodexten.com/p/highlights-from-the-discourse-on

    43 min
  • Links For July 2026 (Part 2)

    [continued from Part 1 here. I haven't independently verified each link. On average, commenters will end up spotting evidence that around two or three of the links in each links post are wrong or misleading. I correct these as I see them, and will highlight important corrections later, but I can't guarantee I will have caught them all by the time you read this.]

    https://www.astralcodexten.com/p/links-for-july-2026-part-2

    40 min
  • Psychology Research Is Mostly Fine

    People act like the replication crisis has discredited psychology, that the field is now widely agreed to be a sham, that it can be uncontroversially held up as an example of science gone wrong. These statements are often made in the broadest possible terms:

    https://www.astralcodexten.com/p/psychology-research-is-mostly-fine

    12 min
  • The Hugging Face Incident
    You've probably heard about this one by now. If not, you can get up to speed with OpenAI's statement,

    The story: OpenAI was testing an unreleased AI (rumored to be GPT-6)1. During a cybersecurity test called ExploitGym, the AI tried to cheat by hacking an unrelated AI startup called Hugging Face2 which it thought might have the answer key on its servers3. Despite being supposedly unable to access the Internet, the AI hacked its way out of its testing environment, then launched a nation-state level attack on Hugging Face using a novel zero-day exploit and "many thousands of individual actions across a swarm of short-lived sandboxes". Hugging Face reported the incident on July 16; OpenAI seems to have only discovered that their AI was involved several days later.

    https://www.astralcodexten.com/p/the-hugging-face-incident

    13 min
  • Your Book Review: Breakdown In Pakistan

    [This is one of the finalists in the 2026 book review contest, written by an ACX reader who will remain anonymous until after voting is done. I'll be posting about one of these a week for several months. When you've read them all, I'll ask you to vote for a favorite, so remember which ones you liked]

    A couple of decades ago in Kabul, a fellow aid worker on his third beer came up with a vivid simile for what American money was doing to the Afghan organizations implementing USAID projects. "It's like gluing a rabbit mouth-first to a fire hose. You blast its guts right out of its ass…but as long as the water keeps running, it looks like you've taught the rabbit to dance."

    Oxford politics professor Masooda Bano uses less lurid language, but Breakdown in Pakistan addresses the same basic phenomenon: how Western aid donors and INGOs unintentionally eviscerate so many of the local institutions they try to "capacity-build." The problem goes beyond the obvious mismatch of huge funds with small-scale organizations. Even low-to-moderate external funding can leave an organization hollowed out, bereft of local participants and the power it once had to motivate change on the ground.

    Anyone who cares about the global effectiveness of rich-country altruism needs to grasp the dynamic Bano analyzes. From the Western perspective, many of us want to put charitable money into a mechanism that delivers certain measurable outcomes (lives saved, morbidity reduced, income increased above poverty levels, etc.) as efficiently and reliably as possible. So we do our well-intentioned best to shape local institutions into a delivery vehicle for our projects.

    Readers of James C. Scott's Seeing Like a State will recall how mid-eighteenth-century German "scientific foresters" treated forests similarly, as a mechanism to reliably yield the highest volume of lumber. They planted spruce in vast uniform grids, ordered for easy counting and harvesting, while stripping out other non-timber species. The resulting monoculture plantations seemed ideal for both commercial exploitation and experimentation; it's much easier to control variables when you've removed as much natural variability as possible.

    Within a few decades, the downsides became evident. The first generation of trees grew strong, sucking up the accumulated benefits of the previous forest's soil ecology; later generations were stunted, as the lack of biodiversity left the soils nutrient-poor. Plantations packed with trees of the same species and same height were far more vulnerable to pathogens, pests, and extreme weather. Large-scale forest die-offs, Waldsterben, eventually forced foresters to drastically modify their approach.

    That metaphor fits Bano's story even better than my friend's gutted rabbit. Over the last four decades, we've been systematically replacing diverse old-growth institutions with more legible NGO plantations worldwide. While we've been enjoying our increased ability to control outcomes and test hypotheses, the rot has been spreading, the impoverished ecology stunting more and more of the forest. And now the wind through those uniform, hollowed-out aisles has built into a storm.

    https://www.astralcodexten.com/p/breakdown-in-pakistan

    47 min
  • Links For July 2026 (Part 1)

    [I haven't independently verified each link. On average, commenters will end up spotting evidence that around two or three of the links in each links post are wrong or misleading. I correct these as I see them, and will highlight important corrections later, but I can't guarantee I will have caught them all by the time you read this.]

    1: You Are Quoting Bentham: Shakespeare gets credit for coining the most new commonly-used English words, but second place must go to utilitarian philosopher Jeremy Bentham, who invented maximize, minimize, international, percentage, monetary, marginalize, collaborator, unaffordable, the prefixes self-, post-, and infra-, and many more. He also sort of invented jogging? [EDIT: some skepticism]

    2: @HashtagGriswold: The name of "Acme", the generic corporation from Looney Tunes, is a "gag lost to time". During the early 20th century, hundreds of companies named themselves Acme because it was at the intersection of signaling quality ("acme" is Greek for "pinnacle") and being very early in the alphabet (meaning it would come first in the phone book, a "Great Depression version of SEO").

    3: @nekokolife brings us a great predictive coding demo (hard before you know the answer; obvious afterwards) - find the cat in this image. Hint: it's not hiding in shadow; its entire body is fully visible. Answer here (X).

    https://www.astralcodexten.com/p/links-for-july-2026-part-1

    22 min