Background: “Learning” vs “Learning About”

Adaptive systems, reinforcement “learners”, etc, “learn” in the sense that their behavior adapts to their environment.

Bayesian reasoners, human scientists, etc, “learn” in the sense that they have some symbolic representation of the environment, and they update those symbols over time to (hopefully) better match the environment (i.e. make the map better match the territory).

These two kinds of “learning” are not synonymous[1]. Adaptive systems “learn” things, but they don’t necessarily “learn about” things; they don’t necessarily have an internal map of the external territory. (Yes, the active inference folks will bullshit about how any adaptive system must have a map of the territory, but their math does not substantively support that interpretation.) The internal heuristics or behaviors “learned” by an adaptive system are not necessarily “about” any particular external thing, and don’t necessarily represent any particular external thing[2].

We Humans Learn About Our Values

---

Outline:

(00:06) Background: “Learning” vs “Learning About”

(01:09) We Humans Learn About Our Values

(03:25) Two Puzzles

(03:28) Puzzle 1: Learning About Our Own Values vs The Is-Ought Gap

(03:53) Puzzle 2: The Role of Reward/Reinforcement

(04:35) Using Each Puzzle To Solve The Other

(05:11) What This Looks Like In Practice

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The big news of the week was of course OpenAI releasing their new model o1. If you read one post this week, read that one. Everything else is a relative sideshow.

Meanwhile, we await Newsom's decision on SB 1047. The smart money was always that Gavin Newsom would make us wait before offering his verdict on SB 1047. It's a big decision. Don’t rush him. In the meantime, what hints he has offered suggest he's buying into some of the anti-1047 talking points. I’m offering a letter to him here based on his comments, if you have any way to help convince him now would be the time to use that. But mostly, it's up to him now.

Table of Contents

---

Outline:

(00:49) Language Models Offer Mundane Utility

(02:11) Language Models Don’t Offer Mundane Utility

(03:34) Deepfaketown and Botpocalypse Soon

(05:59) They Took Our Jobs

(07:34) Get Involved

(08:09) Introducing

(10:11) In Other AI News

(13:20) Quiet Speculations

(15:15) Intelligent Design

(19:15) SB 1047: The Governor Ponders

(27:13) Letter to Newsom

(31:36) The Quest for Sane Regulations

(34:19) Rhetorical Innovation

(42:13) Claude Writes Short Stories

(45:54) Questions of Sentience

(48:22) People Are Worried About AI Killing Everyone

(49:56) The Lighter Side

The original text contained 6 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Would you like to be tutored in infrabayesianism, natural latents, Agent Foundations, decision theory, anthropics, or some other topics?

I'm thinking about prototyping some topic-specific LLM tutor bots, and would like to prioritize topics that multiple people are interested in.

Topic-specific LLM tutors would be customized with things like pre-loaded relevant context, helpful system prompts, and more focused testing to ensure they work.

I'm going to use the same poll infrastructure that Ben Pace pioneered recently. There is a thread below where you add and vote on topics/domains/areas where you might like tutoring.

1. Karma: upvote/downvote to express enthusiasm about there being tutoring for a topic.
2. Reacts: click on the agree react to indicate you personally would like tutoring on a topic.
3. New Poll Option. Add a new topic for people express interest in being tutored on.

For the sake of this poll, I'm more interested in whether you'd [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

1.1 Summary & Table of Contents

This is the first of a series of eight blog posts, which I’ll be serializing over the next month or two. (Or email or DM me if you want to read the whole thing right now.) Here's an overview of the whole series, and then we’ll jump right into the first post!

1.1.1 Summary & Table of Contents—for the whole series

This is a rather ambitious series of blog posts, in that I’ll attempt to explain what's the deal with consciousness, free will, hypnotism, enlightenment, hallucinations, flow states, dissociation, akrasia, delusions, and more.

The starting point for this whole journey is very simple:

• The brain has a predictive (a.k.a. self-supervised) learning algorithm.
• This algorithm builds generative models (a.k.a. “intuitive models”) that can predict incoming data.
• It turns out that, in order to predict incoming data, the algorithm winds up not only building [...]

---

Outline:

(00:05) 1.1 Summary and Table of Contents

(00:27) 1.1.1 Summary and Table of Contents—for the whole series

(04:50) 1.1.2 Summary and Table of Contents—for this first post in particular

(06:56) 1.2 Generative models and probabilistic inference

(07:02) 1.2.1 Example: bistable perception

(08:08) 1.2.2 Probabilistic inference

(09:59) 1.2.3 The thing you “experience” is the generative model (a.k.a. “intuitive model”)

(11:24) 1.2.4 Explanation of bistable perception

(12:19) 1.2.5 Teaser: Unusual states of consciousness as a version of bistable perception

(13:52) 1.3 Casting judgment upon intuitive models

(14:33) 1.3.1 “Is the intuitive model real, or is it fake?”

(15:22) 1.3.2 “Is the intuitive model veridical, or is it non-veridical?”

(18:08) 1.3.2.1 Non-veridical intuitive models are extremely common and unremarkable

(19:48) 1.3.2.2 …But of course it's good if you’re intellectually aware of how veridical your various intuitive models are

(21:32) 1.3.3 “Is the intuitive model healthy, or is it pathological?”

(23:23) 1.4 Why does the predictive learning algorithm build generative models / concepts related to what's happening in your own mind?

(26:37) 1.4.1 Further notes on the path from predictive learning algorithms to intuitive self-models

(28:56) 1.5 Appendix: Some terminology I’ll be using in this series

(29:18) Learning algorithms and trained models

(31:21) Concepts, models, thoughts, subagents

(33:53) 1.6 Appendix: How does this series fit into Philosophy Of Mind?

(34:00) 1.6.1 Introspective self-reports as a “straightforward” scientific question

(35:52) 1.6.2 Are explanations-of-self-reports a first step towards understanding the “true nature” of consciousness, free will, etc.?

(39:46) 1.6.3 Related work

(40:47) 1.7 Conclusion

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In my Xenosystems review, I discussed the Orthogonality Thesis, concluding that it was a bad metaphor. It's a long post, though, and the comments on orthogonality build on other Xenosystems content. Therefore, I think it may be helpful to present a more concentrated discussion on Orthogonality, contrasting Orthogonality with my own view, without introducing dependencies on Land's views. (Land gets credit for inspiring many of these thoughts, of course, but I'm presenting my views as my own here.)

First, let's define the Orthogonality Thesis. Quoting Superintelligence for Bostrom's formulation:

Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal.

To me, the main ambiguity about what this is saying is the "could in principle" part; maybe, for any level of intelligence and any final goal, there exists (in the mathematical sense) [...]

---

Outline:

(04:28) Bayes/VNM point against Orthogonality

(07:18) Belief/value duality

(10:35) Logical uncertainty as a model for bounded rationality

(14:42) Naive belief/value factorizations lead to optimization daemons

(16:40) Intelligence changes the ontology values are expressed in

(20:18) Intelligence leads to recognizing value-relevant symmetries

(22:11) Human brains dont seem to neatly factorize

(25:04) Models of ASI should start with realism

(27:58) On Yudkowskys arguments

(31:28) Conclusion

---

First published:

Source:

Narrated by TYPE III AUDIO.

Disclaimer: I am a computational physicist's and this investigation is outside of my immediate area of expertise. Feel free to peruse the experiments and take everything I say with appropriate levels of skepticism.

Introduction:

The centre for AI safety is a prominent AI safety research group doing technical AI research as well as regulatory activism. It's headed by Dan Hendrycks, who has a PHD in computer science from Berkeley and some notable contributions to AI research.

Last week CAIS released a blog post, entitled “superhuman automated forecasting”, announcing a forecasting bot developed by a team including Hendrycks, along with a technical report and a website “five thirty nine”, where users can try out the bot for themselves. The blog post makes several grandiose claims, claiming to rebut Nate silvers claims that superhuman forecasting is 15-20 years away, and that:

Our bot performs better than experienced human forecasters and performs [...]

The original text contained 7 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's that time again for all the sufficiently interesting news that isn’t otherwise fit to print, also known as the Monthly Roundup.

Bad News

Beware the failure mode in strategy and decisions that implicitly assumes competence, or wishes away difficulties, and remember to reverse all advice you hear.

Stefan Schubert (quoting Tyler Cowen on raising people's ambitions often being very high value): I think lowering others’ aspirations can also be high-return. I know of people who would have had a better life by now if someone could have persuaded them to pursue more realistic plans.

Rob Miles: There's a specific failure mode which I don’t have a name for, which is similar to “be too ambitious” but is closer to “have an unrealistic plan”. The illustrative example I use is:

Suppose by some strange circumstance you have to represent your country at olympic gymnastics [...]

---

Outline:

(00:14) Bad News

(03:45) Anti-Social Media

(07:20) Technology Advances

(08:56) High Seas Piracy is Bad

(12:17) The Michelin Curse

(15:21) What's the Rush?

(17:48) Good News, Everyone

(19:49) Let it Go

(22:09) Yay Air Conditioning

(23:27) Beast of a Memo

(36:58) For Science!

(37:18) For Your Entertainment

(39:47) Properly Rated

(45:26) Government Working

(49:21) Grapefruit Diet

(58:38) Gamers Gonna Game Game Game Game Game

(01:06:09) Gamers Winning At Life

(01:10:52) I Was Promised Flying Self-Driving Cars

(01:11:21) While I Cannot Condone This

(01:17:21) Nostalgia

(01:23:51) The Lighter Side

The original text contained 12 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Introduction

Every single time I design a protein — using ML or otherwise — I am confident that it is capable of being manufactured. I simply reach out to Twist Biosciences, have them create a plasmid that encodes for the amino acids that make up my proteins, push that plasmid into a cell, and the cell will pump out the protein I created.

Maybe the cell cannot efficiently create the protein. Maybe the protein sucks. Maybe it will fold in weird ways, isn’t thermostable, or has some other undesirable characteristic.

But the way the protein is created is simple, close-ended, cheap, and almost always possible to do.

The same is not true of the rest of chemistry. For now, let's focus purely on small molecules, but this thesis applies even more-so across all of chemistry.

Of the 1060 small molecules that are theorized to exist, most are likely extremely [...]

---

Outline:

(00:16) Introduction

(02:23) What is a small molecule anyway?

(05:31) Why is synthesis hard?

(11:45) How the synthesis bottleneck manifests in ML

(17:36) Potential fixes

(17:39) Synthesis-aware generative chemistry

(24:36) Improvements in synthesis

(26:48) Steelman

The original text contained 7 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A year ago, I started trying to deliberate practice skills that would "help people figure out the answers to confusing, important questions." I experimented with Thinking Physics questions, GPQA questions, Puzzle Games , Strategy Games, and a stupid twitchy reflex game I had struggled to beat for 8 years[1]. Then I went back to my day job and tried figuring stuff out there too.

The most important skill I was trying to learn was Metastrategic Brainstorming[2] – the skill of looking at a confusing, hopeless situation, and nonetheless brainstorming useful ways to get traction or avoid wasted motion.

Normally, when you want to get good at something, it's great to stand on the shoulders of giants and copy all the existing techniques. But this is challenging if you're trying to solve important, confusing problems because there probably isn't (much) established wisdom on how to solve it. You may [...]

---

Outline:

(02:33) Taking breaks, or naps

(03:25) Working Memory facility

(04:56) Patience

(06:17) Know what deconfusion, or having a crisp understanding feels like

(07:50) Actually Fucking Backchain

(10:00) Ask Whats My Goal?

(11:09) Always have at least 3 hypotheses

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Terrible name (with a terrible reason, that this ‘resets the counter’ on AI capability to 1, and ‘o’ as in OpenAI when they previously used o for Omni, very confusing). Impressive new capabilities in many ways. Less impressive in many others, at least relative to its hype.

Clearly this is an important capabilities improvement. However, it is not a 5-level model, and in important senses the ‘raw G’ underlying the system hasn’t improved.

GPT-o1 seems to get its new capabilities by taking (effectively) GPT-4o, and then using extensive Chain of Thought (CoT) and quite a lot of tokens. Thus that unlocks (a lot of) what that can unlock. We did not previously know how to usefully do that. Now we do. It gets much better at formal logic and reasoning, things in the ‘system 2’ bucket. That matters a lot for many tasks, if not as much [...]

---

Outline:

(01:26) Introducing GPT-o1

(05:05) Evals

(07:55) Chain of Thought

(08:57) Coding

(11:08) Human Preference Evaluation

(11:37) What Is It?

(20:24) Doing Math Without Terrance Tao

(25:02) Doing Real Math with Terence Tao

(30:04) Positive Examples

(38:51) Skeptical Reactions

(42:32) Report from Janus World

(45:30) Same Old Silly Examples

(53:47) Latency

(55:14) Paths Forward Unrelated to Safety

(59:17) Safety Last

(01:07:06) Deception

(01:10:50) External Red Teaming

(01:11:23) Apollo's Red Teaming Finds Deceptive Alignment

(01:22:17) Preparedness Testing Finds Reward Hacking

(01:26:43) METR's Red Teaming

(01:29:52) What Are the Safety and Policy Implications?

The original text contained 37 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.