Why aren’t there dojos that teach rationality?

―The Martial Art of Rationality by Eliezer Yudkowsky

For the last 6 months, I've been running a dojo that teaches rationality.

Why

I was at an ACX meetup and met an acolyte who grew up in an evangelical Christian community. He had recently discovered the Sequences and was really excited about this whole Rationality thing. He was very confident in Yudkowsky's teachings.

I asked him a couple questions and he realized his beliefs were full of holes. He wondered how he could understood so little. After all, he had read all of Yudkowsky's Sequences.

"I have read 100 books about chess," I said, "Surely I must be a grandmaster by now."

At that moment, he was enlightened.

The problem

The objective of rationality is to become right instead of wrong. Being wrong feels exactly like being right. We are not aware of [...]

---

Outline:

(00:20) Why

(00:57) The problem

(01:36) The solution

(02:51) The right conditions

(04:02) The right topics

(04:44) Want to try it out?

(05:26) What to expect if you participate

---

First published:

Source:

Narrated by TYPE III AUDIO.

I am seeking crowdsourced wisdom.

Suppose I[1] want to influence public opinion on a complicated, nuanced topic.[2]

And further suppose that one of the ways I want to do that is by participating actively in public discourse: posting short-form content, reacting to other people's short-form content, signal boosting good takes, thoughtfully rebutting bad takes, etc.

Even further suppose that the people I most want to reach are those who make and influence policy in the most technologically advanced countries in the world, the general voting public (because they vote for legislators), and the intelligentsia in fields related to my topic (because some of them advise policymakers).

On what platform(s)/in what outlet(s) should I be doing this in 2024?

And more specifically, does Twitter/X still matter, and how much? I am aware that many people have moved to mastodon or bluesky or whatever. Is there critical mass anywhere?

1. The original text contained 2 footnotes which were omitted from this narration.

   ---

First published:

Source:

Narrated by TYPE III AUDIO.

Google cofounder Larry Page thinks superintelligent AI is “just the next step in evolution.” In fact, Page, who's worth about $120 billion, has reportedly argued that efforts to prevent AI-driven extinction and protect human consciousness are “speciesist” and “sentimental nonsense.”

In July, former Google DeepMind senior scientist Richard Sutton — one of the pioneers of reinforcement learning, a major subfield of AI — said that the technology “could displace us from existence,” and that “we should not resist succession.” In a 2015 talk, Sutton said, suppose “everything fails” and AI “kill[s] us all”; he asked, “Is it so bad that humans are not the final form of intelligent life in the universe?”

This is how I begin the cover story for Jacobin's winter issue on AI. Some very influential people openly welcome an AI-driven future, even if humans aren’t part of it.

Whether [...]

---

First published:

Source:

Linkpost URL:
https://jacobin.com/2024/01/can-humanity-survive-ai

Narrated by TYPE III AUDIO.

TLDR: Legal liability could substantially mitigate AI risk, but current law falls short in two key ways: (1) it requires provable negligence, and (2) it greatly limits the availability of punitive damages. Applying strict liability (a form of liability that does not require provable negligence) and expanding the availability and flexibility of punitive damages is feasible, but will require action by courts or legislatures. Legislatures should also consider acting in advance to create a clear ex ante expectation of liability and imposing liability insurance requirements for the training and deployment of advanced AI systems. The following post is a summary of a law review article. Here is the full draft paper. Dylan Matthews also did an excellent write-up of the core proposal for Vox's Future Perfect vertical.

AI alignment is primarily a technical problem that will require technical solutions. But it is also a policy problem. Training and [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

California Senator Scott Wiener of San Francisco introduces SB 1047 to regulate AI. I have put up a market on how likely it is to become law.

“If Congress at some point is able to pass a strong pro-innovation, pro-safety AI law, I’ll be the first to cheer that, but I’m not holding my breath,” Wiener said in an interview. “We need to get ahead of this so we maintain public trust in AI.”

Congress is certainly highly dysfunctional. I am still generally against California trying to act like it is the federal government, even when the cause is good, but I understand.

Can California effectively impose its will here?

On the biggest players, for now, presumably yes.

In the longer run, when things get actively dangerous, then my presumption is no.

There is a potential trap here. If we put our rules [...]

---

Outline:

(02:48) Close Reading of the Bill

(10:50) My High Level Takeaways From the Close Reading

(11:37) Another More Skeptical Reaction to the Same Bill

(13:01) What is a Covered Model Here?

(15:39) Precautionary Principle and Covered Guidance

(18:07) Non-Derivative

(18:28) So What Would This Law Actually Do?

(20:44) Crying Wolf

---

First published:

Source:

Narrated by TYPE III AUDIO.

The introduction post is here.

This post illustrates the first of three techniques which, when used in conjunction, should whack you out of Akrasia.

The technique is called "stopping".

I must mention what this post definitely isn't. The wierdos of the world have built entire industries around the practice of deliberately doing ostensibly nothing. Monks in the Himalayas encourage you to sit still cross-legged while you perform their particular brand of vague mental hocus pocus. Grander fraudsters and/or wierdos will implore you to freeze suddenly in the middle of dancing in order to achieve congruence with the fourth way enneagram or something, a practice unceremoniously robbed from bored toddlers or Sufis. You don't really need to get weird or dramatic or mystical about it - your weightlifting instructor might ask you to painfully pause for a few moments at the bottom of a squat in order to make your [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Personal Views Only

I thought Maxwell Tabarrok's The Offense-Defense Balance Rarely Changes was a good contribution to the discourse, and appreciated his attempt to use various lines of data to “kick the tires” on the case that an AI could in principle defeat all humans by inventing technologies that would allow them to dominate humans. I think Tabarrok is correct that this is a crux in the debate about how pessimistic to be about the possibility of humanity defending ourselves against a misaligned AGI,[1] and so in turn a crux about the probability of extinction or similarly bad outcomes from misaligned AI.

One thing I think Tabarrok is importantly correct about is that the idea of an “offense–defense balance” in hypothetical future human–AI conflicts is important but often undertheorized. Anyone interested in this question should be humbled by the fact that offense–defense theory and its [...]

---

Outline:

(01:26) Why I am not as reassured by the historical trends in conflict fatality rates as Tabarrok

(02:17) Reason 1: Deaths in conflict is not a measure of the offense–defense balance

(04:03) Reason 2: Only a temporary period of offense-dominance may be sufficient for AI takeover.

(05:29) Reason 3: AI might not need to defend itself against humans wielding AI-made offensive weapons.

(07:30) Reason 4: The destructiveness of the most destructive conflicts is increasing

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

First, addressing misconceptions that could come from the title or from the paper's framing as relevant to LLM scaling:

1. The model didn't learn from observing many move traces from high-level games. Instead, they trained a 270M-parameter model to map _(text{board state}, text{legal move})_ pairs to Stockfish 16's predicted win probability after playing the move. This can be described as imitating the play of an oracle that reflects Stockfish's ability at 50 milliseconds per move. Then they evaluated a system that made the model's highest-value move for a given position.
2. The system resulted in some quirks that required workarounds during play. 
   1. The board states were encoded in FEN notation, which doesn't provide information about which previous board states have occurred; this is relevant in a small number of situations because players can claim an immediate draw when a board state is repeated three times.
   2. The model is a classifier, so [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

I just read the article https://arbital.com/p/likelihoods_not_pvalues/, but this left me with one big question: what is the actual sequence of actions a researcher would take to report a likelihood function based on their research? For background, I'm an undergraduate student who has participated in multiple undergraduate research positions, and who has so far been uncertain about how to compute and use likelihood functions based on the data (mostly econometric data) involved in this research.

In particular, I want to find out:

1. How do you compute a likelihood function in Stata, R, Python, etc. based on data you have, for a given hypothesis class
2. How do you obtain a data file for this likelihood function that can be distributed in a way that other researchers can download and use in e.g. a meta-analysis
3. What kinds of graphs are good to visually represent these likelihood functions, particularly if the hypothesis [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

[Setting: a suburban house. The interior of the house takes up most of the stage; on the audience's right, we see a wall in cross-section, and a front porch. Simplicia enters stage left and rings the doorbell.]

Doomimir: [opening the door] Well? What do you want?

Simplicia: I can't stop thinking about our last conversation. It was kind of all over the place. If you're willing, I'd like to continue, but focusing in narrower detail on a couple points I'm still confused about.

Doomimir: And why should I bother tutoring an Earthling in alignment theory? If you didn't get it from the empty string, and you didn't get it from our last discussion, why should I have any hope of you learning this time? And even if you did, what good would it do?

Simplicia: [serenely] If the world is ending either way, I think [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.