One day a physics professor presents the standard physics 101 material on gravity and Newtonian mechanics: g = 9.8 m/s^2, sled on a ramp, pendulum, yada yada.

Later that week, the class has a lab session. Based on the standard physics 101 material, they calculate that a certain pendulum will have a period of approximately 3.6 seconds.

They then run the experiment: they set up the pendulum, draw it back to the appropriate starting position, and release. Result: the stand holding the pendulum tips over, and the whole thing falls on the floor. Stopwatch in hand, they watch the pendulum sit still on the floor, and time how often it returns to the same position. They conclude that the pendulum has a period of approximately 0.0 seconds.

Being avid LessWrong readers, the students reason: “This Newtonian mechanics theory predicted a period of approximately 3.6 seconds. Various factors we ignored [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TL;DR: We present a retrieval-augmented LM system that nears the human crowd performance on judgemental forecasting.

Paper: https://arxiv.org/abs/2402.18563 (Danny Halawi*, Fred Zhang*, Chen Yueh-Han*, and Jacob Steinhardt)

Forecasting future events is important for policy and decision-making. In this work, we study whether language models (LMs) can forecast at the level of competitive human forecasters. Towards this goal, we develop a retrieval-augmented LM system designed to automatically search for relevant information, generate forecasts, and aggregate predictions. To facilitate our study, we collect a large dataset of questions from competitive forecasting platforms. Under a test set published after the knowledge cut-offs of our LMs, we evaluate the end-to-end performance of our system against the aggregates of human forecasts. On average, the system nears the crowd aggregate of competitive forecasters and in some settings, surpasses it. Our work suggests that using LMs to forecast the future could provide accurate predictions [...]

---

Outline:

(01:29) Current LMs are not naturally good at forecasting

(02:19) System building

(03:24) Data and models

(04:14) Evaluation results

(04:37) Unconditional setting

(05:03) Selective setting

(06:16) Calibration

(06:54) Future work

---

First published:

Source:

Narrated by TYPE III AUDIO.

The main event continues to be the fallout from The Gemini Incident. Everyone is focusing there now, and few are liking what they see.

That does not mean other things stop. There were two interviews with Demis Hassabis, with Dwarkesh Patel's being predictably excellent. We got introduced to another set of potentially highly useful AI products. Mistral partnered up with Microsoft the moment Mistral got France to pressure the EU to agree to cripple the regulations that Microsoft wanted crippled. You know. The usual stuff.

Table of Contents

---

Outline:

(00:39) Language Models Offer Mundane Utility

(05:00) Language Models Don’t Offer Mundane Utility

(06:15) OpenAI Has a Sales Pitch

(10:16) The Gemini Incident

(19:19) Political Preference Tests for LLMs

(22:13) GPT-4 Real This Time

(23:13) Fun with Image Generation

(23:57) Deepfaketown and Botpocalypse Soon

(33:00) They Took Our Jobs

(36:34) Get Involved

(36:46) Introducing

(40:08) In Other AI News

(41:46) Quiet Speculations

(45:13) Mistral Shows Its True Colors

(51:13) The Week in Audio

(51:47) Rhetorical Innovation

(54:43) Open Model Weights Are Unsafe and Nothing Can Fix This

(01:02:54) Aligning a Smarter Than Human Intelligence is Difficult

(01:05:03) Other People Are Not As Worried About AI Killing Everyone

(01:06:36) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

Yoshua Bengio recently posted a high-level overview of his alignment research agenda. I'm pasting the full text below since it's fairly short.

What can’t we afford with a future superintelligent AI? Among others, confidently wrong predictions about the harm that some actions could yield. Especially catastrophic harm. Especially if these actions could spell the end of humanity.

How can we design an AI that will be highly capable and will not harm humans? In my opinion, we need to figure out this question – of controlling AI so that it behaves in really safe ways – before we reach human-level AI, aka AGI; and to be successful, we need all hands on deck. Economic and military pressures to accelerate advances in AI capabilities will continue to push forward even if we have not figured out how to make superintelligent AI safe. And even if [...]

---

First published:

Source:

Linkpost URL:
https://yoshuabengio.org/2024/02/26/towards-a-cautious-scientist-ai-with-convergent-safety-bounds/

Narrated by TYPE III AUDIO.

TLDR: I’ve collected some tips for research that I’ve given to other people and/or used myself, which have sped things up and helped put people in the right general mindset for empirical AI alignment research. Some of these are opinionated takes, also around what has helped me. Researchers can be successful in different ways, but I still stand by the tips here as a reasonable default.

What success generally looks like

Here, I’ve included specific criteria that strong collaborators of mine tend to meet, with rough weightings on the importance, as a rough north star for people who collaborate with me (especially if you’re new to research). These criteria are for the specific kind of research I do (highly experimental LLM alignment research, excluding interpretability); some examples of research areas where this applies are e.g. scalable oversight [...]

---

Outline:

(00:34) What success generally looks like

(08:04) Tactical Research Tips and Approach

(08:58) Workflow

(14:26) Reading research papers:

(16:19) General Mindset Tips

(18:23) Three modes of research

(24:07) Work Habits

(26:01) Machine Learning/Engineering Footguns

(28:08) Default Norms for Projects with Me

---

First published:

Source:

Narrated by TYPE III AUDIO.

Gabriel Weil (Assistant Professor of Law, Touro University Law Center) wrote this post series on the role of Liability Law for reducing Existential Risk from AI. I think this may well be of interest to some people here, so wanted for a linkpost to exist.

The first post argues that Tort Law Can Play an Important Role in Mitigating AI Risk.

The second post addressed directly How Technical AI Safety Researchers Can Help Implement Punitive Damages to Mitigate Catastrophic AI Risk

Here is the full paper.

TLDR (from the first post)

Legal liability could substantially mitigate AI risk, but current law falls short in two key ways: (1) it requires provable negligence, and (2) it greatly limits the availability of punitive damages. Applying strict liability (a form of liability that [...]

---

First published:

Source:

Linkpost URL:
https://forum.effectivealtruism.org/s/nYJ26tgc2JkFQRG2w

Narrated by TYPE III AUDIO.

This is the third post in a sequence that demonstrates a complete naturalist study, specifically a study of query hugging (sort of), as described in The Nuts and Bolts of Naturalism. This one demos phases one and two: Locating Fulcrum Experiences and Getting Your Eyes On. For context on this sequence, see the intro post. If you're intimidated by the length of this post, remember that this is meant as reference material. Feel free to start with "My Goals For This Post", and consider what more you want from there.

Having chosen a quest—”What's going on with distraction?"—my naturalist study began in earnest.

In "Nuts and Bolts of Naturalism", the first two phases of study that I discussed after "Getting Started" were "Locating Fulcrum Experiences" and "Getting Your Eyes On". In practice, though, I often do a combination of these phases, which is what happened this time. For [...]

---

Outline:

(01:14) My Goals For This Post

(03:37) Distraction

(13:56) Concentration

(26:09) Crucial

(33:42) The Study So Far

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

What is this post?

This post is a companion piece to recent posts on evidential cooperation in large worlds (ECL). We’ve noticed that in conversations about ECL, the same few initial confusions and objections tend to be brought up. We hope that this post will be useful as the place that lists and discusses these common objections. We invite the reader to advance additional questions and objections of their own.

This FAQ does not need to be read in order. The reader is encouraged to look through the section headings and jump to those they find most interesting.

ECL seems very weird. Are you sure you haven’t, like, taken a wrong turn somewhere?

We don’t think so.

ECL, at its core, takes two reasonable ideas that by themselves are considered quite plausible by many—albeit not completely uncontroversial—and notices that when you combine them, you get something quite interesting [...]

---

Outline:

(00:06) What is this post?

(00:43) ECL seems very weird. Are you sure you haven’t, like, taken a wrong turn somewhere?

(01:43) Do I need to buy evidential decision theory for this to work?

(04:19) Okay, I see the intuitive appeal of cooperating in the near twin prisoners’ dilemma, but I am still skeptical of noncausal decision theories. Is there any theoretical reason to take noncausal decision theories seriously?

(06:19) Why aincha rich?

(07:01) CDT is exploitable.

(08:03) CDT offers counterintuitive advice.

(10:38) EDT, medical Newcomb problems and the tickle defense.

(12:45) Okay, so maybe I am not 100% confident in causal decision theory, but I am also not fully convinced by noncausal decision theory. What now?

(13:46) Is this a Pascal's mugging?

(16:43) Should we be clueless about our acausal effects?

(18:47) Does infinite ethics mess with ECL?

(21:06) Is ECL the same thing as acausal trade?

(21:41) Is ECL basically a multiverse-wide moral trade?

(23:00) Okay, I’ll grant that your toy setups work. But in the real world, we don’t have two neat actions with one labeled “cooperate.” We have lots of different available actions and compromise points. How is that supposed to work?

(25:22) You keep talking about “agents” for us to “do ECL” with… Who exactly are these agents?

(28:43) Is ECL action-relevant? Time-sensitive?

(31:02) How big a deal are the implications of ECL? How much does it change the impact of my actions?

(34:21) Acknowledgements

The original text contained 21 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Timaeus was announced in late October 2023, with the mission of making fundamental breakthroughs in technical AI alignment using deep ideas from mathematics and the sciences. This is our first progress update.

In service of the mission, our first priority has been to support and contribute to ongoing work in Singular Learning Theory (SLT) and developmental interpretability, with the aim of laying theoretical and empirical foundations for a science of deep learning and neural network interpretability.

Our main uncertainties in this research were:

• Is SLT useful in deep learning? While SLT is mathematically established, it was not clear whether the central quantities of SLT could be estimated at sufficient scale, and whether SLT's predictions actually held for realistic models (esp. language models). 
• Does structure in neural networks form in phase transitions? The idea of developmental interpretability was to view phase transitions as a core primitive in the [...]

---

Outline:

(01:57) Contributions

(02:17) Research Contributions

(02:21) What we learned

(04:33) Papers

(05:45) Outreach Contributions

(05:49) Posts

(06:43) Code

(06:57) Talks

(07:19) Events

(08:11) Organization

(08:27) What we need

(09:08) Whats next

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

(Also announcing: annual review prediction markets & full-height table of contents)

The top 50 posts of each of LessWrong's annual reviews have a new home: The LeastWrong.

(If you're looking for this year's review results, you can find them here.)

What will I see when I click that link?

By default, you will find the posts organised into six “books”: Rationality, Optimization, World, Practical, AI Strategy, and Technical AI Safety. Each square on the grid is a post that made the top 50 of the review in some year.

If you're logged-in, then essays will be dark until you've read them, to help guide you to essays you've not read before.

How can I see more of a book?

If you click on the name of a book, like “Rationality”, you’ll get a full width view of the content.

And clicking “Show All” will let you see all posts [...]

---

Outline:

(00:21) What will I see when I click that link?

(00:52) How can I see more of a book?

(01:10) How are the collections ordered?

(01:19) What about grouping by years?

(01:28) What happens when I click on one of the posts?

(01:39) What are the little gold buttons above the title?

(01:47) Any other goodies?

(02:11) Why did you make this?

(02:50) Thats it

---

First published:

Source:

Narrated by TYPE III AUDIO.