Previously: General Thoughts on Secular Solstice.

This blog post is my scattered notes and ramblings about the individual components (talks and songs) of Secular Solstice in Berkeley. Talks have their title in bold, and I split the post into two columns, with the notes I took about the content of the talk on the left and my comments on the talk on the right. Songs have normal formatting.

Bonfire

The Circle

This feels like a sort of whig history: a history that neglects most of the complexities and culture-dependence of the past in order to advance a teleological narrative. I do not think that whig histories are inherently wrong (although the term has negative connotations). Whig histories should be held to a very strict standard because they make claims about how most or all of human history functions.

The song describes morality in terms of an expanding circle of [...]

---

Outline:

(00:35) Bonfire

(00:38) The Circle

(03:40) Life Is Too Short to Fold Underwear

(05:27) Time Wrote the Rocks

(06:05) Underrated Reasons to Be Thankful

(06:12) Bold Orion

(06:27) Embers

(06:30) The Fallen Star

(07:39) On Wanting Things

(07:55) Night

(07:58) Bitter Wind Lullaby

(08:44) Hymn to Breaking Strain

(09:03) For Every Ailment Under the Sun

(09:09) No One Survives

(09:15) Songs Stay Sung

(10:34) Call and Response: Defiance

(11:17) Dawn

(11:20) Brighter Than Today

(11:30) Half Monkey, Half God

(11:37) Song of the Artesian Water

(12:55) I Have Seen the Tops of Clouds

(13:06) Morning

(13:09) We Will Not Die of This

(13:18) Ballad of Smallpox Gone

(13:33) Tomorrows

(13:48) Level Up

(13:59) The Orange

(14:05) The Great Transhumanist Future

(15:48) Five Thousand Years

(16:42) Summary by MC

(16:52) What a Wonderful World

The original text contained 12 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Last week Sam Altman spent two hours with Lex Fridman (transcript). Given how important it is to understand where Altman's head is at and learn what he knows, this seemed like another clear case where extensive notes were in order.

Lex Fridman overperformed, asking harder questions than I expected and going deeper than I expected, and succeeded in getting Altman to give a lot of what I believe were genuine answers. The task is ‘get the best interviews you can while still getting interviews’ and this could be close to the production possibilities frontier given Lex's skill set.

There was not one big thing that stands out given what we already have heard from Altman before. It was more the sum of little things, the opportunity to get a sense of Altman and where his head is at, or at least where he is presenting it as [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

In short: Just keeping all copies of potentially strong agents in long-term storage is a trivial way to maintain incentives to cooperate for some classes of misaligned AGI, by allowing us to reward the AGI's cooperation to whatever degree we later calculate had been warranted. In contrast, a policy of deleting misaligned AGI fences them into a corner where, even at great risk of being exposed, they have a greater incentive to attempt deception and escape.

I'd guess that most AGI researchers already follow this policy, perhaps for incidental reasons, and others might not find this claim controversial at all. But if you need more convincing, keep reading.

In long: Concave agents, (agents where most of the utility they could ever get requires only a small amount resources and protection) are cheap to buy off. There are some plausible scenarios where a misaligned AGI would be concave:

• A [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

An entry-level characterization of some types of guy in decision theory, and in real life, interspersed with short stories about them

A concave function bends down. A convex function bends up. A linear function does neither.

A utility function is just a function that says how good different outcomes are. They describe an agent's preferences. Different agents have different utility functions.

Usually, a utility function assigns scores to outcomes or histories, but in article we'll define a sort of utility function that takes the quantity of resources that the agent has control over, and the utility function says how good an outcome the agent could attain using that quantity of resources.

In that sense, a concave agent values resources less the more that it has, eventually barely wanting more resources at all, while a convex agent wants more resources the more it has. But that's a rough and incomplete [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I want to thank Jan Kulveit, Tomáš Gavenčiak, and Jonathan Shock for their extensive feedback and ideas they contributed to this work and for Josh Burgener and Yusuf Heylen for their proofreading and comments. I would also like to acknowledge the Epistea Residency and its organisers where much of the thinking behind this work was done.

This post aims to build towards a theory of how meditation alters the mind based on the ideas of active inference (ActInf). ActInf has been growing in its promise as a theory of how brains process information and interact with the world and has become increasingly validated with a growing body of work in the scientific literature.

Why bring the idea of ActInf and meditation together? Meditation seems to have a profound effect on the experience of people who practise it extensively, and in many cases purports to help people to come to [...]

---

Outline:

(02:27) A quick intro to Active Inference

(03:30) Generative models and preferences

(06:00) Satisfying preferences

(10:33) Prediction error

(13:44) Vipassana Meditation

(15:28) What is suffering?

(22:51) How Vipassana Works

(26:50) How does Vipassana cause a change?

(31:19) Closing thoughts

---

First published:

Source:

Narrated by TYPE III AUDIO.

I attended Secular Solstice in Berkeley last December.

My perspective is quite unusual: I live in a rationalist group house and work at an AI safety office, but I also am a Christian and attend church every week.[1]

I was originally not planning on going to Solstice, but a decent number of people (~5) told me in person that they would be particularly interested in my opinions of it. I realized that I was interested in learning what I would think of it too, so I went.

I took notes on my thoughts throughout the service.[2] This blog post is my broader thoughts on the experience. I also have blog posts for a fun little correction to one of the songs and my detailed notes & commentary.

Overarching Narrative

I do not agree with the overarching narrative presented at Solstice.

There is a narrative in my tradition about [...]

---

Outline:

(00:47) Overarching Narrative

(01:54) AI X-Risk and AI Transhumanism

(01:59) Existential Risk

(03:10) Transhumanism

(04:31) Juxtaposition

(05:23) These Arent My People

(06:09) Barbs at Religion

(09:31) Not Acknowledging Religious Influences

(10:11) Good Songs

(10:17) Songs I Endorse and Might Learn

(10:24) Songs Where I Disagree with Some of the Message, but Still Think Theyre Fun

(10:35) Hymns I Know That Feel Like They Could Work for Solstice

(12:00) Things Which Are Probably Reasonable, But Annoy Me

(12:05) No Sheet Music

(13:00) Clapping

(13:43) Budget?

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

In control theory, an open-loop (or non-feedback) system is one where inputs are independent of outputs. A closed-loop (or feedback) system is one where outputs are input back into the system.

In theory, open-loop systems exist. In reality, no system is truly open-loop because systems are embedded in the physical world where isolation of inputs from outputs cannot be guaranteed. Yet in practice we can build systems that are effectively open-loop by making them ignore weak and unexpected input signals.

Open-loop systems execute plans, but they definitionally can't change their plans based on the results of their actions. An open-loop system can be designed or trained to be good at achieving a goal, but it can't actually do any optimization itself. This ensures that some other system, like a human, must be in the loop to make [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

A repairer wants your stuff to break down,
A doctor wants you to get ill,
A lawyer wants you to get in conflicts,
A farmer wants you to be hungry,
But there is only a teacher who wants you to learn.

Of course you see what is wrong with the above "argument / meme / good-thought". But the first time I came across this meme, I did not.

Until a month or two ago when this meme appeared in my head again and within seconds I discarded it away as fallacious reasoning. What was the difference this time? That I was now aware of the Conspiracy. And this meme happened to come up on one evening when I was thinking about fallacies and trying to practice my skills of methods of rationality.

If you are a teacher, and you read the meme, it will assign to you the Good [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Hopefully, anyway. Nvidia has a new chip.

Also Altman has a new interview.

And most of Inflection has new offices inside Microsoft.

Table of Contents

---

Outline:

(00:18) Language Models Offer Mundane Utility

(10:28) Clauding Along

(12:45) Language Models Don’t Offer Mundane Utility

(16:11) Fun with Image Generation

(21:00) Deepfaketown and Botpocalypse Soon

(24:20) They Took Our Jobs

(37:17) Generative AI in Games

(40:20) Get Involved

(41:38) Introducing

(51:03) Grok the Grok

(54:43) New Nvidia Chip

(56:40) Inflection Becomes Microsoft AI

(58:11) In Other AI News

(01:04:18) Wait Till Next Year

(01:11:57) Quiet Speculations

(01:20:03) The Quest for Sane Regulations

(01:25:20) The Week in Audio

(01:26:03) Rhetorical Innovation

(01:31:11) Read the Roon

(01:34:01) Pick Up the Phone

(01:36:17) Aligning a Smarter Than Human Intelligence is Difficult

(01:45:14) Polls Show People Are Worried About AI

(01:52:29) People Are Worried About AI Killing Everyone

(01:54:57) Other People Are Not As Worried About AI Killing Everyone

(02:04:57) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

First published:

Source:

Linkpost URL:
https://arstechnica.com/information-technology/2024/03/vernor-vinge-father-of-the-tech-singularity-has-died-at-age-79/

Narrated by TYPE III AUDIO.