The rate at which China is able to advance towards TAI is a crucial consideration in for many policy questions. My current take is that, without significant political reforms which seem very unlikely while Xi is alive (although considerably more likely after his death,) it's very unlikely that China will be able to mount a meaningful challenge to AI firms in US and allies in the race for TAI. I don’t think it requires democratic reforms are required to China to be competitive with the US and allies, but I do think rule of law reforms are likely to be required.

The first post is going to be me forecasting Chinese growth on the theory that, if China reaches rich country status, it's likely that it will be able to compete with the US and allies for leadership in AI. I’ll write a second post looking at Chinese AI [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TLDR: It's often said that Bayesian updating is unbiased and converges to the truth—and, therefore, that biases must emerge from non-Bayesian sources. That's not quite right. The convergence results require updating on your total evidence—but for agents at all like us, that's impossible—instead, we must selectively attend to certain questions, ignoring others. Yet correlations between what we see and what questions we ask—“ideological” Bayesian updating—can lead to predictable biases and polarization.

Professor Polder is a polarizing figure.

His fans praise him for his insight; his critics denounce him for his aggression.

Ask his fans, and they’ll supply you with a bunch of instances when he made an insightful comment during discussions. They’ll admit that he's sometimes aggressive, but they can’t remember too many cases—he certainly doesn’t seem any more aggressive than the average professor.

Ask his critics, and they’ll supply you with a bunch [...]

---

Outline:

(03:26) Limited-Attention Bayesians

(06:55) Ideological Bayesians

(12:36) What happens?

(17:12) What to make of this?

(20:00) What next?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Linkpost URL:
https://kevindorst.substack.com/p/ideological-bayesians

Narrated by TYPE III AUDIO.

I see people use "in-context learning" in different ways.

Take the opening to "In-Context Learning Creates Task Vectors":

In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood. In particular, it is challenging to map it to the “standard” machine learning framework, where one uses a training set _S_ to find a best-fitting function _f(x)_ in some hypothesis class.

In one Bayesian sense, training data and prompts are both just evidence. From a given model, prior (random weights), and evidence (training data), you get new model weights. From the new model weights and some more evidence (prompt input), and a distribution of output text. But the "training step" _(text{prior}, text{data}) rightarrow text{weights}_ and "inference step" _(text{weights}, text{input}) rightarrow text{output}_ could be simplified to a single function:_(text{prior}, text{data}, text{input}) rightarrow text{output}_. An LLM trained on [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Evidential cooperation in large worlds (ECL) is a proposed way of reaping gains—that is, getting more of what we value instantiated—through cooperating with agents across the universe/multiverse. Such cooperation does not involve physical, or causal, interaction. ECL is potentially a crucial consideration, because we may be able to do more good this way compared to the “standard” (i.e., causal) way of optimizing for our values.

The core idea of ECL can be summarized as:

• According to non-causal decision theories, my decisions relevantly “influence” what others who are similar to me do, even if they never observe my behavior (or the causal consequences of my behavior). (More.)
• In particular, if I behave cooperatively towards other value systems, then other agents across the multiverse are more likely to do the same. Hence, at least some fraction of agents can be (acausally) influenced into behaving cooperatively towards my value system. This [...]

---

Outline:

(02:53) The twin prisoners’ dilemma

(02:57) Exact copies

(04:57) General case

(06:58) Why is this type of acausal cooperation important in practice? Introducing ECL

(07:06) Our situation as a generalized twin prisoners’ dilemma-type situation

(14:41) Many agents

(17:19) Acausal influence on agents with other values

(24:12) Net gains from cooperation

(27:42) Putting it all together

(29:23) How to get involved

(29:27) Short-term research fellowships

(30:16) Long-term employment

(30:55) Reach out to Chi

(31:46) Funding

(32:02) How to skill up to work on ECL

(32:27) Recommended readings

(36:31) Acknowledgements

The original text contained 28 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is the second post in a sequence that demonstrates a complete naturalist study, specifically a study of "Hug the Query", as described in The Nuts and Bolts of Naturalism. This one demos phase zero, all the preparation that's often needed before you can really get to work. It corresponds to the how-to posts "Getting Started With Naturalism" and "Catching the Spark". For context on this sequence, see the intro post.

The Dead Words Of Others

At the outset of any naturalist study, original seeing and curiosity are paramount. If they’re already present—and they aren’t crowded out by other concerns, such as a desperation to solve your problem as quickly as possible—then you can dive right in. Otherwise, some deliberate cultivation is needed.

Where did I stand with original seeing and curiosity, at the beginning of this study? I was pretty low on both.

---

Outline:

(00:33) The Dead Words Of Others

(10:37) Choosing My Quest

---

First published:

Source:

Narrated by TYPE III AUDIO.

6 months ago I wrote Feedbackloop-first Rationality. I didn't followup on it for awhile (except for sporadic Deliberate (“Purposeful?”) Practice Club).

I just spent 6 weeks actually exploring "how would I build my own cognition training program?". In the process of doing so, I've iterated a bunch. I'm still in an orienting phase, but it seemed worth writing down the current stage of my thoughts.

What's my goal?

A rough overview:

• I want to get more, higher quality "X-risk thinker hours" hours. 
  • This includes AI alignment technical research, AI macrostrategy research, policy, governance, as well as people (such as Lightcone team) deciding which infrastructure to build, 
• I'm particularly interested in getting more "serial research", as opposed to more "parallel research." We can throw more researchers at a problem, but if there are some problems that require one person to synthesize 10+ years of experience, all the parallel [...]

---

Outline:

(00:32) Whats my goal?

(01:28) Rationality for the sake of existential risk

(03:18) The Story So Far

(03:22) Feedback-loops and deliberate practice, vs Just Clicking

(05:00) What About CFAR? Didnt they teach just click skills?

(06:12) Hamming-nature, 10x plans, OODA Loops

(08:23) Planning vs OODA Loops

(09:29) My Process: Test Driven Development

(10:50) Alternate Strategies and/or Theories of Change

(11:38) #1: Help senior researchers with specific targeted problems.

(13:14) #2: Build a Thinking Assistant Pipeline

(15:23) #3. Learning Generalized Research Taste

(18:16) #4. Filtering/enculturation for Overall Community Epistemic Health

(19:17) #5. Investigating s factor?

(20:58) Itd be cool if a second group also worked towards rationality skill assessment.

(22:31) What Have I Actually Done?

(25:02) Whats Next?

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Fellow progress blogger Alex Telford and I have had a friendly back-and-forth going over FDA reform. Alex suggests incremental reforms to the FDA, which I strongly support, but these don’t go far enough. The FDA's failures merit a complete overhaul: Remove efficacy requirements and keep only basic safety testing and ingredient verification. Any drug that doesn’t go through efficacy trials gets a big red warning label, but is otherwise legal.

Before getting into Alex's points let me quickly make the positive case for my position.

The FDA is punished for errors of commission: drugs they approve which turn out not to work or to be harmful. They don’t take responsibility for errors of omission: drugs they could have approved earlier but delayed, or drugs that would have been developed but were abandoned due to the cost of approval. This asymmetry predictably leads to overcaution.

---

Outline:

(02:46) Patent medicines and snake oil

(04:53) Chesterton's fence

(06:38) Does the FDA actually help innovation?

(11:59) Addendum

---

First published:

Source:

Linkpost URL:
https://www.maximum-progress.com/p/we-need-major-but-not-radical-fda

Narrated by TYPE III AUDIO.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Representation engineering (RepEng) has emerged as a promising research avenue for model interpretability and control. Recent papers have proposed methods for discovering truth in models with unlabeled data, guiding generation by modifying representations, and building LLM lie detectors. RepEng asks the question: If we treat representations as the central unit, how much power do we have over a model's behaviour?

Most techniques use linear probes to monitor and control representations. An important question is whether the probes generalise. If we train a probe on the truths and lies about the locations of cities, will it generalise to truths and lies about Amazon review sentiment? This report focuses on truth due to its relevance to safety, and to help narrow the work.

Generalisation is important. Humans typically have one generalised notion of “truth”, and it would be enormously convenient if language models also had just one[1]. This would result in [...]

---

Outline:

(01:44) Methods

(02:02) What makes a probe?

(03:44) Probe algorithms

(04:51) Datasets

(05:51) Measuring generalisation

(06:17) Recovered accuracy

(07:25) Finding the best generalising probe

(08:06) Results

(09:24) Examining the best probe

(10:22) Examining algorithm performance

(11:03) Examining dataset performance

(13:27) How do we know we’re detecting truth, and not just likely statements?

(14:48) Conclusion and future work

(16:00) Appendix

(16:03) Validating implementations

(16:47) Validating LDA implementation

(17:25) Thresholding

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Note on genre: This sequence is a demonstration of a complete naturalist study, as described in Intro to Naturalism and The Nuts and Bolts Of Naturalism. I think of naturalism demos as reference material. I’ve tried to make it readable, but like a dictionary or a user manual, I only expect it to be of interest to people who already have a reason to consult it.

Epistemic status: The explicit concepts I’m building around what I’ve learned are still under construction. I think the framing emphasized in this demo is askew, or incomplete, or some-other-how flawed. Perhaps I will come back in a future year to describe how my concepts have evolved. However, I stand pretty firmly behind the broad strokes of the process-level stuff.

Goals

“Hug the Query” is an essay by Eliezer Yudkowsky advocating a certain discipline of rationality that he calls closeness to the issue [...]

---

Outline:

(00:51) Goals

(01:42) Motivation

(07:06) Structure

(08:52) Is this for you?

---

First published:

Source:

Narrated by TYPE III AUDIO.