I originally wrote this in 2021, when Lightcone was exploring a vision of building "a rationalist campus" or maybe "a rationalist society."

Right now, neither I, nor Lightcone as a whole, are focused on this in the same way. You *might* make a guild that taught these skills on purpose, or a barrier-to-entry that filtered for them.

But, I mostly am publishing it now because I still roughly like it as an aspirational world I'd like to be a part of some day. I like these skills, and I'd like my collaborators to have them, however that ends up being mediated. I haven't edited the post since 2021.

At the LW Team, we’ve been tossing around the idea of founding some kind of rationalist/x-risk guild, society, or other closed barrier social/professional network.

There's a bunch of reasons I think a guild might be good. But one key [...]

---

Outline:

(03:11) Rationality Skills

(09:26) Collaboration Skills

(10:43) “The Basics”

---

First published:

Source:

Narrated by TYPE III AUDIO.

Tl/Dr:

• Generalisation is a capability in its own right, and in our experiments it scales less quickly than specific capabilities. 
• Our experiments show that on a range of tasks, finetuning Babbage on task A transfers more to task B than finetuning Davinci on task A transfers to task B. This is relevant for identifying the nature of the alignment problem we might encounter (for things in this vague cluster see here, here, and here).
• Of independent interest is the fact that a transfer learning metric can give us a fairly non-arbitrary way to measure how similar any two tasks are. 

Epistemic status: written quickly, but oh look we did 4×8^2 experiments and we have a lovely graphic

This post makes two contributions. First, we compare results of transfer learning experiments from Babbage and Davinci and suggest that, at least in these experiments, the ability to generalize scales less quickly than do [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

First published:

Source:

Narrated by TYPE III AUDIO.

In some discussions (especially about acausal trade and multi-polar conflict), I’ve heard the motto “X will/won’t be a problem because superintelligences will just be Updateless”. Here I’ll explain (in layman's terms) why, as far as we know, it's not looking likely that a super satisfactory implementation of Updatelessness exists, nor that superintelligences automatically implement it, nor that this would drastically improve multi-agentic bargaining.

Epistemic status: These insights seem like the most robust update from my work with Demski on Logical Updatelessness and discussions with CLR employees about Open-Minded Updatelessness. To my understanding, most researchers involved agree with them and the message of this post.

What is Updatelessness?

This is skippable if you’re already familiar with the concept.

It's easier to illustrate with the following example: Counterfactual Mugging.

I will throw a fair coin.

• If it lands Heads, you will be able to freely choose whether to pay me [...]

---

Outline:

(00:48) What is Updatelessness?

(05:03) A fundamental trade-off

(07:11) Multi-agentic interactions

(11:36) Is the trade-off avoidable?

(14:45) Superintelligences

(18:06) Other lenses

(20:57) Conclusion

(21:47) Acknowledgements

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

In a week with two podcasts I covered extensively, I was happy that there was little other news.

That is, until right before press time, when Google rebranded Bard to Gemini, released an app for that, and offered a premium subscription ($20/month) for Gemini Ultra.

Gemini Ultra is Here

I have had the honor and opportunity to check out Gemini Advanced before its release.

The base model seems to be better than GPT-4. It seems excellent for code, for explanations and answering questions about facts or how things work, for generic displays of intelligence, for telling you how to do something. Hitting the Google icon to have it look for sources is great.

In general, if you want to be a power user, if you want to push the envelope in various ways, Gemini is not going to make it easy on you. However [...]

---

Outline:

(00:21) Gemini Ultra is Here

(03:16) Language Models Offer Mundane Utility

(05:38) Language Models Don’t Offer Mundane Utility

(07:12) GPT-4 Real This Time

(09:03) Fun with Image Generation

(09:20) Deepfaketown and Botpocalypse Soon

(14:09) They Took Our Jobs

(15:16) Get Involved

(16:00) Introducing

(16:12) In Other AI News

(20:42) Quiet Speculations

(24:44) Vitalik on the Intersection AI and Crypto

(30:10) The Quest for Sane Regulations

(30:23) The Week in Audio

(34:25) Rhetorical Innovation

(35:23) Aligning a Dumber Than Human Intelligence is Still Difficult

(35:54) People Are Worried About AI, Many People

(38:04) Other People Are Not As Worried About AI Killing Everyone

(39:54) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

“In America, we believe in driving on the right hand side of the road.”

Tl;dr: Beliefs are like bets (on outcomes the belief doesn’t affect). “Believing in”s are more like kickstarters (for outcomes the believing-in does affect).

Epistemic status: New model; could use critique.

In one early CFAR test session, we asked volunteers to each write down something they believed. My plan was that we would then think together about what we would see in a world where each belief was true, compared to a world where it was false.

I was a bit flummoxed when, instead of the beliefs-aka-predictions I had been expecting, they wrote down such “beliefs” as “the environment,” “kindness,” or “respecting people.” At the time, I thought this meant that the state of ambient rationality was so low that people didn’t know “beliefs” were supposed to be predictions, as opposed to group affiliations.

I’ve [...]

---

Outline:

(01:29) Examples of “believing in”

(03:45) A model of “believing in”

(06:50) Some pieces of the lawlike shape of “believing in”s:

(07:17) Some differences between how to “believe in” well, and how to predict well:

(09:51) Some commonalities between how to “believe in” well, and how to predict well:

(11:32) Example: “believed in” task completion times

(14:17) “Believing in X” is not predicting “X will work and be good.”

(15:50) Compatibility with prediction

(17:25) Distinguishing “believing in” from deception / self-deception

(19:22) Getting Bayes-habits out of the way of “believing in”s

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

A Chess-GPT Linear Emergent World Representation

Among the many recent developments in ML, there were two I found interesting and wanted to dig into further. The first was gpt-3.5-turbo-instruct's ability to play chess at 1800 Elo. The fact that an LLM could learn to play chess well from random text scraped off the internet seemed almost magical. The second was Kenneth Li's Emergent World Representations paper. There is an excellent summary on The Gradient and a follow-up from Neel Nanda. In it, they trained a 25 million parameter GPT to predict the next character in an Othello game. It learns to accurately make moves in games unseen in its training dataset, and using both non-linear and linear probes it was found that the model accurately tracks the state of the board.

However, this only worked for a model trained on a synthetic [...]

---

Outline:

(00:07) A Chess-GPT Linear Emergent World Representation

(02:35) Training Chess GPT

(05:12) Chess-GPTs Internal World Model

(08:46) Probing for latent variables

(12:41) \- I fine-tuned GPT-2 on a 50 / 50 mix of OpenWebText and chess games, and it learned to play chess and continued to output plausible looking text. Maybe theres something interesting to look at there?

---

First published:

Source:

Linkpost URL:
https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html

Narrated by TYPE III AUDIO.

Epistemic status: I spent about 5 hours looking into this, and the next day developed covid myself. I did a bit more research plus all of the writing while sick. So in addition to my normal warning that I have no medical credentials, you should keep in mind that this knowledge may be cursed.

Nitric Oxide Nasal Spray, sold under the brand name Enovid, is a reactive compound that kills viruses (and I suspect taxes your nasal tissue). It has recently been tested and marketed for treatment of covid. The protocol I found in papers was 2 sprays per nostril every 2-3 hours, after you develop symptoms. Enovid's instructional pamphlets say twice per day, also after you get sick. This seems a little late to me.

I suspect the real power of NONS lies in use before you develop symptoms, ideally as close to [...]

---

Outline:

(01:54) How does it work?

(02:52) How well does it work?

(04:57) Effect size on nasal viral load

(05:42) Effect on symptoms/transmission, as measured by poorly designed studies

(06:39) Other Diseases

(06:55) My wild ass guess of impact

(08:30) Can I get this effect for free, without the painful stinging or logistical hassle of a nasal spray?

(09:24) Should I just use this all the time?

(09:58) Where can I buy Enovid?

---

First published:

Source:

Narrated by TYPE III AUDIO.