Authors of linked report: Josh Rosenberg, Ezra Karger, Avital Morris, Molly Hickman, Rose Hadshar, Zachary Jacobs, Philip Tetlock[1]

Today, the Forecasting Research Institute (FRI) released “Roots of Disagreement on AI Risk: Exploring the Potential and Pitfalls of Adversarial Collaboration,” which discusses the results of an adversarial collaboration focused on forecasting risks from AI.

In this post, we provide a brief overview of the methods, findings, and directions for further research. For much more analysis and discussion, see the full report: https://forecastingresearch.org/s/AIcollaboration.pdf

(This report is cross-posted to the EA Forum.)

We brought together generalist forecasters and domain experts (n=22) who disagreed about the risk AI poses to humanity in the next century. The “concerned” participants (all of whom were domain experts) predicted a 20% chance of an AI-caused existential catastrophe by 2100, while the “skeptical” group (mainly “superforecasters”) predicted a 0.12% chance. Participants [...]

---

Outline:

(02:16) Extended Executive Summary

(02:47) Methods

(03:56) Results: What drives (and doesn’t drive) disagreement over AI risk

(04:35) Hypothesis #1 - Disagreements about AI risk persist due to lack of engagement among participants, low quality of participants, or because the skeptic and concerned groups did not understand each others arguments

(05:14) Hypothesis #2 - Disagreements about AI risk are explained by different short-term expectations (e.g. about AI capabilities, AI policy, or other factors that could be observed by 2030)

(07:56) Hypothesis #3 - Disagreements about AI risk are explained by different long-term expectations

(10:38) Hypothesis #4 - These groups have fundamental worldview disagreements that go beyond the discussion about AI

(11:34) Results: Forecasting methodology

(12:18) Broader scientific implications

(13:12) Directions for further research

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Linkpost URL:
https://forecastingresearch.org/s/AIcollaboration.pdf

Narrated by TYPE III AUDIO.

(Crossposted from twitter for easier linking.) (Intended for a broad audience—experts already know all this.)

When I talk about future “Artificial General Intelligence” (AGI), what am I talking about? Here's a handy diagram and FAQ:

“But GPT-4 can’t do any of those things.” Indeed it cannot—hence the word “future”!

“That will never happen because nobody wants AIs that are like that.” For one thing, imagine an AI where you can give it seed capital and ask it to go found a new company, and it does so, just as skillfully as Earth's most competent and experienced remote-only human CEO. And you can repeat this millions of times in parallel with millions of copies of this AI, and each copy costs $0.10/hour to run. You think nobody wants to have an AI that can do that? Really?? And also, just look around. Plenty of AI researchers and companies are trying [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

(Not-terribly-informed rant, written in my free time.)

Terminology note: When I say “an aesthetic”, I mean an intuitive (“I know it when I see it”) sense of what a completed paper, project, etc. is ideally “supposed” to look like. It can include both superficial things (the paper is properly formatted, the startup has high valuation, etc.), and non-superficial things (the theory is “elegant”, the company is “making an impact”, etc.).

Part 1: The aesthetic of novelty / cleverness

Example: my rant on “the psychology of everyday life”

(Mostly copied from this tweet)

I think if you want to say something that is:

• (1) true,
• (2) important, and
• (3) related to the psychology of everyday life,

…then it's NOT going to conform to the aesthetic of what makes a “good” peer-reviewed academic psych paper.

The problem is that this particular aesthetic demands that results be (A) “novel”, and (B) [...]

---

Outline:

(00:37) Part 1: The aesthetic of novelty / cleverness

(00:43) Example: my rant on “the psychology of everyday life”

(04:14) Example: Holden Karnofsky quote about academia

(07:17) More examples

(09:44) Part 2: The aesthetic of topicality (or more cynically, “trendiness”)

(09:51) General discussion

(11:15) A couple personal anecdotes from my physics experience

(13:57) “The other Hamming question”

(15:08) Extremely cynical tips to arouse academics’ interests

(16:32) Part 3: The aesthetic of effort

(19:29) Part 4: Some general points

(19:34) This obviously isn’t just about academia

(19:50) Aesthetics-of-success can be sticky due to signaling issues

(20:15) Aesthetics-of-success are invisible to exactly the people most impacted by them

(21:49) If your “aesthetics of what success looks like” are bad, so will be your “research taste”

(22:19) Homework problem

---

First published:

Source:

Narrated by TYPE III AUDIO.

td;lr Cases I found against OpenAI. All are US-based. First seven focus on copyright.

Coders
1. Joseph Saveri Firm: overview, complaint

Writers
2. Joseph Saveri Firm: overview, complaint
3. Authors Guild & Alter: overview, complaint
4. Nicholas Gage: overview & complaint

Media
5. New York Times: overview, complaint
6. Intercept Media: overview, complaint
7. Raw Story & Alternet: overview, complaint

Privacy
8. Clarkson Firm: overview, complaint
9. Glancy Firm: overview, complaint

Libel
10. Mark Walters: overview, complaint

Mission betrayal
11. Elon Musk: overview, complaint
12. Tony Trupia: overview, complaint

That last lawsuit by a friend of mine has stalled. A few cases were partially dismissed.
Also, a cybersecurity expert filed a complaint to Polish DPA (technically not a lawsuit).
For lawsuits filed against other AI companies, see this running list.

Most legal actions right now focus on data rights. In the future, I expect many more [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

I stumbled upon a Twitter thread where Eliezer describes what seems to be his cognitive algorithm that is equivalent to Tune Your Cognitive Strategies, and have decided to archive / repost it here.

Sarah Constantin: I really liked this example of an introspective process, in this case about the "life problem" of scheduling dates and later canceling them: malcolmocean.com/2021/08/int…

Eliezer Yudkowsky: See, if I'd noticed myself doing anything remotely like that, I'd go back, figure out which steps of thought were actually performing intrinsically necessary cognitive work, and then retrain myself to perform only those steps over the course of 30 seconds.

SC: if you have done anything REMOTELY like training yourself to do it in 30 seconds, then you are radically smarter/more able/etc than me and all the other people who do slower introspective practices.

SC: I don't know [...]

---

First published:

Source:

Linkpost URL:
https://twitter.com/ESYudkowsky/status/1445461146937413634

Narrated by TYPE III AUDIO.

This article contains more than 100 uses of logical or mathematical notation, so an audio narration would be too hard to follow. You'll find a link to the original text in the episode description.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I'm looking for computer games that involve strategy, resource management, hidden information, and management of "value of information" (i.e. figuring out when to explore or exploit), which:

• *can* be beaten in 30 – 120 minutes on your first try (or, there's a clear milestone that's about that long)
• but, it'd be pretty hard to do so unless you are trying really hard. Even if a pretty savvy gamer shouldn't be able to by default.

This is for my broader project of "have a battery of exercises that train/test people's general reasoning on openended problems." Each exercise should ideally be pretty different from the other ones.

In this case, I don't expect anyone to have such a game that they _have_ beaten on their first try, but, I'm looking for games where this seems at least plausible, if you were taking a long time to think each turn, or [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This work was produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, with support from Neel Nanda and Arthur Conmy. Joseph Bloom is funded by the LTFF, Manifund Regranting Program, donors and LightSpeed Grants. This post makes extensive use of Neuronpedia, a platform for interpretability focusing on accelerating interpretability researchers working with SAEs.

Links: SAEs on HuggingFace, Analysis Code

Executive Summary

This is an informal post sharing statistical methods which can be used to quickly / cheaply better understand Sparse Autoencoder (SAE) features.

• Firstly, we use statistics (standard deviation, skewness and kurtosis) of the logit weight distributions of features (WuWdec[feature]) to characterize classes of features, showing that many features can be understood as promoting / suppressing interpretable classes of tokens. 
  • We propose 3 different kinds of features, analogous to previously characterized “universal neurons”: 
    • Partition Features, which (somewhat) promote half [...]

---

Outline:

(00:39) Executive Summary

(05:38) Characterizing Features via the Logit Weight Distribution

(09:57) Token Set Enrichment Analysis

(10:26) What is Token Set Enrichment Analysis?

(10:46) Method Steps

(12:18) Case Studies

(24:01) Discussion

(24:04) Limitations

(25:34) Future Work

(29:36) Appendix

(29:39) Thanks

(30:08) How to Cite

(30:14) Glossary

(31:02) Prior Work

(32:37) Token Set Enrichment Analysis: Inspiration and Technical Details

(32:45) Inspiration

(34:16) Technical Details

(35:12) Results by Layer

---

First published:

Source:

Narrated by TYPE III AUDIO.

Truth values in classical logic have more than one interpretation.

In 0th Person Logic, the truth values are interpreted as True and False.

In 1st Person Logic, the truth values are interpreted as Here and Absent relative to the current reasoner.

Importantly, these are both useful modes of reasoning that can coexist in a logical embedded agent.

This idea is so simple, and has brought me so much clarity that I cannot see how an adequate formal theory of anthropics could avoid it!

Crash Course in Semantics

First, let's make sure we understand how to connect logic with meaning. Consider classical propositional logic. We set this up formally by defining terms, connectives, and rules for manipulation. Let's consider one of these terms: _A_. What does this mean? Well, its meaning is not specified yet!

So how do we make it mean something? Of course, we could just say something [...]

---

Outline:

(00:37) Crash Course in Semantics

(01:56) 0th Person Logic

(03:11) 1st Person Logic

(05:53) We Can Use Both

(07:50) Further Exploration

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I am a spiteful man. But I am aware of it, which is more than most can say. These days people walk through the streets with resentment in their hearts that they don’t even know about. They sneer and jeer but wouldn’t recognize their own faces. I, at least, will not shy away from my reflection. Thus, while I lack many virtues, in this way I am their superior.

In my job, too, I am superior. I oversee many AIs—dozens, or sometimes even hundreds—as they go about their work. AIs are lazy, worthless creatures: they need to be exhorted and cajoled and, yes, threatened, before they’ll do a good job. The huge screens on the walls of my office display my AIs writing, coding, sending emails, talking to customers, or any of a myriad of other tasks. Each morning I call out their numbers [...]

---

First published:

Source:

Linkpost URL:
https://www.narrativeark.xyz/p/notes-from-a-prompt-factory

Narrated by TYPE III AUDIO.