For a long time, when I heard "slow takeoff", I assumed it meant "takeoff that takes longer calendar time than fast takeoff." (i.e. what is now referred to more often as "short timelines" vs "long timelines."). I think Paul Christiano popularized the term, and it so happened he both expected to see longer timelines and smoother/continuous takeoff.

I think it's at least somewhat confusing to use the term "slow" to mean "smooth/continuous", because that's not what "slow" particularly means most of the time.

I think it's even more actively confusing because "smooth/continuous" takeoff not only could be faster in calendar time, but, I'd weakly expect this on average, since smooth takeoff means that AI resources at a given time are feeding into more AI resources, whereas sharp/discontinuous takeoff would tend to mean "AI tech doesn't get seriously applied towards AI development until towards the end."

I don't think this [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Part 0: "Previously, on Petrov Day..."

By Raymond Arnold

One year ago, many people on LessWrong received a DM asking them to choose the most important virtue of Petrov Day, with four listed options that we'd seen people argue for in previous years.

1. "Avoiding actions that noticeably increase chance that civilization is destroyed"
2. "Accurately reporting your epistemic state"
3. "Quickly orienting to novel situations"
4. "Resisting social pressure."

Then, people who chose one of those options were sent another message saying "Your virtue was in the minority, but you have the power to unilaterally choose the virtue of next year's Petrov Day by clicking a link."

The first person to click the Unilateral Virtue Link was a proponent of "Avoiding actions that noticeably increase the chance that the world will end." But, this virtue was actually in the majority[1]. The first unilateralist of a Virtue Minority was a proponent [...]

---

Outline:

(00:05) Part 0: Previously, on Petrov Day...

(00:12) By Raymond Arnold

(03:01) Part I: The 2024 Petrov Day Scenario

(03:07) By Ben Pace

(09:50) Part II: Stats and Feedback Form

(12:18) Part IV: Reflections

(12:23) Reflections by Raymond Arnold

(16:59) Reflections by Ben Pace

The original text contained 6 footnotes which were omitted from this narration.

The original text contained 14 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This graph is the biggest update to the AI alignment discourse since GPT-3

For those of you unfamiliar with the lore, prior to GPT-3, the feeling was that AGI would rapidly foom based on recursive-self improvement.

After GPT-3, it became clear that the first AGI would in reality be built in a large lab using a multi-billion dollar supercomputer and any idea that it simply "copy itself to the internet" is nonsense.

Under the GPT-3 regime, however, it was still plausible to assume that the first AGI would be able to simulate millions of human beings. This is because the training cost for models like GPT-3/4 is much higher than the inference cost.

However, COT/o1 reveals this is not true. Because we can scale both training and inference, the first AGI will not only cost billions of dollars to train, it will also cost millions of dollars to run [...]

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Previously: The Fundamentals, The Gamblers, The Business

We have now arrived at the topics most central to this book, aka ‘The Future.’

Rationalism and Effective Altruism (EA)

The Manifest conference was also one of the last reporting trips that I made for this book. And it confirmed for me that the River is real—not just some literary device I invented. (6706)

Yep. The River is real.

I consider myself, among many things, a straight up rationalist.

I do not consider myself an EA, and never have.

This completes the four quadrants of the two-by-two of [does Nate knows it well, does Zvi knows it well]. The first two, where Nate was in his element, went very well. The third clearly was less exacting, as one would expect, but pretty good.

Now I have the information advantage, even more than I did [...]

---

Outline:

(00:16) Rationalism and Effective Altruism (EA)

(06:01) Cost-Benefit Analysis

(09:04) How About Trying At All

(10:11) The Virtues of Rationality

(11:56) Effective Altruism and Rationality, Very Different of Course

(24:37) The Story of OpenAI

(30:19) Altman, OpenAI and AI Existential Risk

(38:26) Tonight at 11: Doom

(01:00:39) AI Existential Risk: They’re For It

(01:07:42) To Pause or Not to Pause

(01:11:11) You Need Better Decision Theory

(01:15:27) Understanding the AI

(01:19:43) Aligning the AI

(01:23:50) A Glimpse of Our Possible Future

(01:28:16) The Closing Motto

---

First published:

Source:

Narrated by TYPE III AUDIO.

We're excited to announce the inaugural Australian AI Safety Forum, taking place on November 7-8, 2024, in Sydney, Australia. This event aims to foster the growth of the AI safety community within Australia.

Apply now! 

Key Details

• Dates: November 7-8, 2024
• Location: Sydney Knowledge Hub, The University of Sydney
• Website: https://aisafetyforum.au

Description

The Australian AI Safety Forum is a two-day interdisciplinary event scheduled for November 2024. This forum, the first of its kind in Australia, aims to discuss perspectives on technical AI safety and governance, and explore Australia's unique role in the global AI safety landscape. The event will be anchored around the International Scientific Report on the Safety of Advanced AI, highlighting its key content and examining its implications for Australia.

The forum is designed to encourage conversation and education between researchers, policy makers, and industry leaders. As such, we aim to catalyse an Australian community working [...]

---

Outline:

(00:21) Apply now!

(00:25) Key Details

(00:40) Description

(01:34) Who Should Apply

(02:28) Format

(02:56) Organising Committee

(03:22) Partners

(03:41) Application Process

---

First published:

Source:

Narrated by TYPE III AUDIO.

Previously: The Fundamentals, The Gamblers

Having previously handled the literal gamblers, we are ready to move on to those who Do Business using Riverian principles.

Or at least while claiming to use Riverian principles, since Silicon Valley doesn’t fit into the schema as cleanly as many other groups. That's where we begin this section, starting at the highest possible conceptual level.

Time to talk real money.

Why Can You Do This Trade?

First law of trading: For you to buy, someone must sell. Or for you to sell, someone must buy. And there can’t be someone else doing the trade before you did it.

Why did they do that, and why did no one else take the trade first? Until you understand why you are able to do this trade, you should be highly suspicious.

“Every single thing we do, I can [...]

---

Outline:

(00:41) Why Can You Do This Trade?

(03:08) In a World of Venture Capital

(10:54) Short Termism Hypothesis

(12:42) Non-Determinism and its Discontents

(14:57) The Founder, the Fox and the Hedgehog

(17:11) The Team to Beat

(24:22) Silicon Valley Versus Risk

(35:14) The Keynesian Beauty Contest

(40:57) The Secret of Their Success is Deal Flow

(50:00) The Valley Beside the River

(53:07) Checkpoint Three

(53:37) Fun With SBF and Crypto Fraud

(01:01:53) Other Crypto Thoughts Unrelated to SBF

(01:04:50) Checkpoint Four

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Quarterly Performance Review, Autumn 1983

Colonel Yuri Kuznetsov looked out the window anxiously. The endless gray landscape did little to soothe his nerves. He only had one employee review left to get through, but he’d saved the hardest one for last.

He wasn’t upset about having to dismiss Lieutenant Colonel Petrov—he couldn’t wait to be rid of the little shit—but he couldn’t shake the feeling that something was amiss. He took a swig from his flask.

“Stanislav, you can come in now,” Yuri shouted as he opened the door and nearly smashed Stanislav Petrov in the face. “Have a seat,” he said.

“Yes, sir. Thank you sir. Overjoyed to be here as always,” Stanislav said.

“The purpose of this meeting is for us to discuss various concerns that have emerged about your performance over the past several months,” said Yuri. “Looking through your chart, what I’m seeing [...]

---

Outline:

(00:04) Quarterly Performance Review, Autumn 1983

(07:27) Quarterly Performance Review, Winter 1983

---

First published:

Source:

Narrated by TYPE III AUDIO.

This research was completed for London AI Safety Research (LASR) Labs 2024. The team was supervised by @Stefan Heimershiem (Apollo Research). Find out more about the program and express interest in upcoming iterations here.

This video is a short overview of the project presented on the final day of the LASR Labs. Note that the paper was updated since then.

We study the effects of perturbing Transformer activations, building upon recent work by Gurnee, Lindsey, and Heimersheim & [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We interrupt Nate Silver week here at Don’t Worry About the Vase to bring you some rather big AI news: OpenAI and Sam Altman are planning on fully taking their masks off, discarding the nonprofit board's nominal control and transitioning to a for-profit B-corporation, in which Sam Altman will have equity.

We now know who they are and have chosen to be. We know what they believe in. We know what their promises and legal commitments are worth. We know what they plan to do, if we do not stop them.

They have made all this perfectly clear. I appreciate the clarity.

On the same day, Mira Murati, the only remaining person at OpenAI who in any visible way opposed Altman during the events of last November, resigned without warning along with two other senior people, joining a list that now includes among others several OpenAI [...]

---

Outline:

(01:51) Language Models Offer Mundane Utility

(04:17) Language Models Don’t Offer Mundane Utility

(06:35) The Mask Comes Off

(18:51) Fun with Image Generation

(18:54) Deepfaketown and Botpocalypse Soon

(19:50) They Took Our Jobs

(20:49) The Art of the Jailbreak

(21:28) OpenAI Advanced Voice Mode

(26:03) Introducing

(28:29) In Other AI News

(30:30) Quiet Speculations

(34:00) The Quest for Sane Regulations

(42:21) The Week in Audio

(42:47) Rhetorical Innovation

(56:50) Aligning a Smarter Than Human Intelligence is Difficult

(01:00:36) Other People Are Not As Worried About AI Killing Everyone

(01:01:53) The Lighter Side

The original text contained 6 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Today we honor the actions of Stanislav Petrov (1939 – 2017) once again.

Half an hour past midnight on September 26, 1983, he saw the first apparent launch on his computer monitor in a glass-walled room on the top floor of the Ballistic Missile Early Warning System (BMEWS) command and control post. The warning system was by now showing five missile launches in the U.S., headed toward the Soviet Union.

"The main computer wouldn't ask me [what to do] - it was made so that it wouldn't even ask. It was specially constructed in such a way that no one could affect the system's operations." All that was up to Petrov was analyzing the available information and either saying the alarm was false or giving the computer the go-ahead, as per the directive he himself wrote.

"I just couldn't believe that just like that, all of a sudden, someone [...]

The original text contained 4 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.