Hours after Google announced Gemini 1.5, OpenAI announced their new video generation model Sora. Its outputs look damn impressive.

How Sora Works

How does it work? There is a technical report. Mostly it seems like OpenAI did standard OpenAI things, meaning they fed in tons of data, used lots of compute, and pressed the scaling button super hard. The innovations they are willing to talk about seem to be things like ‘do not crop the videos into a standard size.’

That does not mean there are not important other innovations. I presume that there are. They simply are not talking about the other improvements.

We should not underestimate the value of throwing in massively more compute and getting a lot of the fiddly details right. That has been the formula for some time now.

Some people think that OpenAI was using a game engine [...]

---

Outline:

(00:12) How Sora Works

(02:07) Sora Is Technically Impressive

(06:42) Sora What's it Good For?

(09:43) Until we can say exactly what we want, and get it, mostly I expect no dice. When you go looking for something specific, your chances of finding it are very bad.

(15:19) Sora Comes Next?

---

First published:

Source:

Narrated by TYPE III AUDIO.

In this post I'll discuss an apparent limitation of sparse autoencoders (SAEs) in their current formulation as they are applied to discovering the latent features within AI models such as transformer-based LLMs. In brief, I'll cover the following:

• I'll argue that the L1 regularisation used to promote sparsity when training SAEs may cause neurons in the sparse layer to learn to represent common combinations of features rather than the individual features that we want them to discover
• As well as making it more difficult to understand what the actual latent features are, I'll also argue that this limitation may result in some less common latent features not being discovered at all, not even within combinations
• I'll then explain why I think that the phenomenon of feature splitting observed in Anthropic's SAE paper appears to demonstrate that this limitation does indeed have a large impact on the features discovered [...]

---

Outline:

(01:07) Rough definition of true features

(02:17) Why SAEs are incentivised to discover combinations of features rather than individual features

(08:33) Relation to feature splitting

(14:57) Proposed solution

---

First published:

Source:

Narrated by TYPE III AUDIO.

Premises

Here I tackle some bold claims made by Roko in three separate posts about SARS-CoV-2 being an engineered virus (Brute Force Manufactured Consensus is Hiding the Crime of the Century, The Math of Suspicious Coincidences, and A Back-Of-The-Envelope Calculation On How Unlikely The Circumstantial Evidence Around Covid-19 Is).
A first rebuttal to Roko's post is already out, and since I have been preparing this post for some time, certain arguments are going to be similar.

The purpose of this post is not to incontrovertibly demonstrate that SARS-CoV-2 was of zoonotic origin, as I myself believe a laboratory leak to be somewhat plausible, but rather that the degree of Roko's confidence in the engineered-pandemic belief is terribly overstated in light of the presented evidence.

I believe that the RootClaim debate is a great way to familiarise oneself with core arguments from all sides, it is exceedingly [...]

---

Outline:

(00:04) Premises

(01:16) A Set of Explainable Coincidences

(01:51) One-in-a-century pandemic

(06:13) The bat coronavirus

(07:29) The BSL-4 lab

(08:20) From Yunnan to Wuhan

(10:12) Probabilities and Confounding Factors

(15:06) Questionable Use of Sources

(15:10) Misrepresentation of evidence

(27:47) DaszakandFriends outsized influence

(31:03) Conclusion

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Previously: I hit send on The Third Gemini, and within half an hour DeepMind announced Gemini 1.5.

So this covers Gemini 1.5. One million tokens, and we are promised overall Gemini Advanced or GPT-4 levels of performance on Gemini Pro levels of compute.

This post does not cover the issues with Gemini's image generation, and what it is and is not willing to generate. I am on top of that situation and will get to it soon.

One Million Tokens

Our teams continue pushing the frontiers of our latest models with safety at the core. They are making rapid progress. In fact, we’re ready to introduce the next generation: Gemini 1.5. It shows dramatic improvements across a number of dimensions and 1.5 Pro achieves comparable quality to 1.0 Ultra, while using less compute.

It is truly bizarre to launch Gemini Advanced as a paid [...]

---

Outline:

(00:32) One Million Tokens

(04:29) Mixture of Experts

(07:39) Quality Not Quantity

(11:52) What To Do With It?

---

First published:

Source:

Narrated by TYPE III AUDIO.

Why do we root for the antiheroes?

Walter White, Light Yagami, Lucifer Morningstar, Doctor Frankenstein... It seems to be natural to root for the bad guys, and perhaps even more natural for rationalists.

“I must confess I have always had some sympathy with villains. Heroism makes fine entertainment but sooner or later someone has to get things done.” - Bayaz, Joe Abercrombie's First Law Trilogy.

I think this is the reason, at least for me. Villains (particularly the most beloved villains) are agentic. They know what they want and they attempt to obtain it through their own actions. Sometimes those actions are even reasonable.

This is notable because it's usually untrue of heroes. Heroes tend to be called to adventure, accepting a goal thrust upon them by circumstances or a bearded stranger or both. Then they pursue it, but often alongside a handful of other eclectic goals [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

MIRI is hiring a managing editor and one or more writers. Here's the job listing. We're accepting applications until March 13th.

I am the Communications Manager at MIRI and I'm the hiring manager for these positions. I'm happy to answer questions.

---

First published:

Source:

Narrated by TYPE III AUDIO.

1.

Prerequisite reading: Being the (Pareto) Best in the World.

A summary of Being the (Pareto) Best in the World: Being the world's best mathematician is hard. Being the world's best musician is hard. Being the world's best mathematician/musician is much easier, especially since there are multiple slots; an amazing mathematician who is also a competent musician, someone who is good at both, and a competent mathematician who is also an amazing musician can all find a niche.

I like this concept, and have kept it in my back pocket ever since I read it. I have sometimes described myself as a software engineer who was competent at public speaking and project management. That particular overlapping skillset is, it turns out, fairly valuable. While I was attempting to become a better software engineer, I was also trying to add competence at corporate budgets and accounting to that skillset.

These [...]

---

Outline:

(00:03) 1.

(01:49) 2.

(05:44) 3.

(08:16) 4.

(10:01) 5.

(12:56) 6.

---

First published:

Source:

Narrated by TYPE III AUDIO.

TL;DR: Scaling labs have their own alignment problem analogous to AI systems, and there are some similarities between the labs and misaligned/unsafe AI.

Major AI scaling labs (OpenAI/Microsoft, Anthropic, Google/DeepMind, and Meta) are very influential in the AI safety and alignment community. They put out cutting-edge research because of their talent, money, and institutional knowledge. A significant subset of the community works for one of these labs. This level of influence is beneficial in some aspects. In many ways, these labs have strong safety cultures, and these values are present in their high-level approaches to developing AI – it's easy to imagine a world in which things are much worse. But the amount of influence that these labs have is also something to be cautious about.

The alignment community is defined by a [...]

---

Outline:

(02:02) Misaligned Incentives

(02:43) Power Seeking

(03:30) Lack of Transparency

(04:30) Deception

(05:18) Incorrigibility

(06:03) Subverting Constraints

(06:21) Concluding Thoughts

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is an informal post intended to describe a workflow / setup that I found very useful, so that others might consider adopting or experimenting with facets of it that they find useful.

In August 2023, I was a part of MATS 4.0 and begun learning the skill of deconfusion, with an aim of disentangling my conflicting intuitions between my belief that shard theory seemed to be at least directionally pointing at some issues with the MIRI model of AGI takeoff and alignment difficulty, and my belief that Nate Soares was obviously correct that reflection will break Alex Turner's diamond alignment scheme. A friend lent me his Kindle Scribe to try out as part of my workflow. I started using it for note-taking, and found it incredibly useful and bought it from him. A month later, I bought a second Kindle Scribe to add to my [...]

---

Outline:

(01:10) The Display

(04:06) Single-purpose devices enhance focus

(05:34) One to read, another to write on

(07:59) Moving on

---

First published:

Source:

Narrated by TYPE III AUDIO.

Just a short post to highlight an issue with debate on LW; I have recently been involved with some interest in the debate on covid-19 origins on here. User viking_math posted a response which I was keen to respond to, but it is not possible for me to respond to that debate (or any) because the LW site has rate-limited me to one comment per 24 hours because my recent comments are on -5 karma or less.

So, I feel that I should highlight that one side of the debate (my side) is simply not going to be here. I can't prosecute a debate like this.

This is funnily enough an example of brute-force manufactured consensus - there will be a debate, people will make points on their side and the side I am arguing for will be missing, so observers will conclude that there [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.