Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My version of Simulacra Levels, published by Daniel Kokotajlo on April 26, 2023 on LessWrong.

People act as if are four truth-values: True, False, Cringe, and Based.

David Udell (paraphrase)

This post lays out my own version of the Simulacra Levels idea. Be warned, apparently it is importantly different from the original.

TRUTHTEAMSDeontologicalLevel 1: “Is it true?”Level 3: "Do I support the associated team?"Consequentialist Level 2: "How does it influence others' beliefs?"Level 4: "How does it influence others' support for various teams?"

Statements you make are categorized as Level 1, 2, 3, or 4 depending on which of the above questions were most responsible for your choice to make the statement. When you say that P, pay attention to the thought processes that caused you to make that statement instead of saying nothing or not-P:

Were you alternating between imagining that P, and imagining that not-P, and noticing lots more implausibilities and inconsistencies-with-your-evidence when you imagined that not-P? Seems like you were at Level 1.

Were you imagining the effects of your utterance on your audience, e.g. imagining that they'd increase their credence that P and act accordingly? Seems like you were at Level 2.

Were you imagining other people saying that P, and/or imagining other people saying that not-P, and noticing that the first group of people seem cool and funny and virtuous and likeable and forces-for-good-in-the-world, and that the second group of people seems annoying, obnoxious, evil, or harmful? (The imagined people could be real, or amorphous archetypes) Seems like you were at level 3.

Were you imagining the effects of your utterance on your audience, e.g. imagining that they'd associate you more with some groups/archetypes and less with other groups/archetypes? Seems like you were at level 4.

Paradigmatic examples of lies (including white lies such as "mmm your homemade hummus tastes great") are Level 2. A lot of social media activity seems to be level 3. Politicians on campaign spend most of their waking moments on level 4.

Of course in real life things are often messy and the cognition responsible for a statement might involve a mix of different levels. Here's a simple example: Suppose, in some context and for statements within some domain, your brain executes the following flowchart:

If for a particular claim you exit the flowchart in Row 2 or Row 4, you are at Simulacra Level 1. If you exit the flowchart in Row 3, you are in either Level 3 or Level 4 depending on how we define "cringe" and "based." (I'm tempted to say they are both Level 3, except there seems to be something inherently level 4-ish about "cringe" in particular.) Note that this flowchart leaves no possibility for Simulacra Level 2; congratulations for being so reliably honest!

Generalizing from statements to sentences

Some sentences are conceived, uttered, debated, tweeted, emblazoned on banners, etc. without ever passing through anyone's brain at level 1 or 2. To the extent that a sentence is like this, we can say it's a "Level 3/4 sentence," or a "Teams-level sentence." The pronouncements of governments and large corporations are full of these kinds of sentences. To decide whether a sentence is level 3/4 or level 1/2, it helps to ask "Interpreted literally, is it true or false?" If the answer is one of the following...

"Well that depends on how you interpret it; people who like it are going to interpret it in way X (and so it'll be true) and people who don't are going to interpret it in way Y (and so it'll be false), and to be honest neither of these interpretations is significantly more straightforward/literal than the other."

"Well, interpreted strictly literally it is uncontroversially true (/false). But..."

... that's a sign that the sentence might be teams-level. Other sign...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra Yudkowsky on Doom from Foom #2, published by jacob cannell on April 27, 2023 on LessWrong.

This is a follow up and partial rewrite to/of an earlier part #1 post critiquing EY's specific argument for doom from AI go foom, and a partial clarifying response to DaemonicSigil's reply on efficiency.

AI go Foom?

By Foom I refer to the specific idea/model (as popularized by EY, MIRI, etc) that near future AGI will undergo a rapid intelligence explosion (hard takeoff) to become orders of magnitude more intelligent (ex from single human capability to human civilization capability) - in a matter of only days or hours - and then dismantle humanity (figuratively as in disempower or literally as in "use your atoms for something else"). Variants of this idea still seems important/relevant drivers of AI risk arguments today: Rob Besinger recently says "STEM-capable artificial general intelligence (AGI) is likely to vastly outperform human intelligence immediately (or very quickly)."

I believe the probability of these scenarios is small and the arguments lack technical engineering prowress concerning the computational physics of - and derived practical engineering constraints on - intelligence.

During the manhattan project some physicists became concerned about the potential of a nuke detonation igniting the atmosphere. Even a small non-epsilon possibility of destroying the entire world should be taken very seriously. So they did some detailed technical analysis which ultimately output a probability below their epsilon allowing them to continue on their merry task of creating weapons of mass destruction.

In the 'ideal' scenario, the doom foomers (EY/MIRI) would present a detailed technical proposal that could be risk evaluated. They of course have not provided that, and indeed it would seem to be an implausible ask. Even if they were claiming to have the technical knowledge on how to produce a fooming AGI, providing that analysis itself could cause someone to create said AGI and thereby destroy the world![1] In the historical precedent of the manhattan project, the detailed safety analysis only finally arrived during the first massive project that succeeded at creating the technology to destroy the world.

So we are left with indirect, often philosophical arguments, which I find unsatisfying. To the extent that EY/MIRI has produced some technical work related to AGI[2], I find it honestly to be more philosophical than technical, and in the latter capacity more amateurish than expert.

I have spent a good chunk of my life studying the AGI problem as an engineer (neuroscience, deep learning, hardware, GPU programming, etc), and reached the conclusion that fast FOOM is unlikely. Proving that of course is very difficult, so I instead gather much of the evidence that led me to that conclusion. However I can't reveal all of the evidence, as the process is rather indistinguishable from searching for the design of AGI itself.[3]

The valid technical arguments for/against the Foom mostly boils down to various efficiency considerations.

Quick background: pareto optimality/efficiency

Engineering is complex and full of fundamental practical tradeoffs:

larger automobiles are safer via higher mass, but have lower fuel economy

larger wings produce more lift but also more drag at higher speeds

highly parallel circuits can do more total work per clock and are more energy efficient but the corresponding parallel algorithms are more complex to design/code, require somewhat more work to accomplish a task, delay/latency becomes more problematic for larger circuits, etc

adiabatic and varying degrees of reversible circuit designs are possible but they are slower, larger, more complex, less noise tolerant, and still face largely unresolved design challenges with practical clock synchronization, etc

quan...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Chad Jones paper modeling AI and x-risk vs. growth, published by jasoncrawford on April 26, 2023 on LessWrong.

Looks like this was just posted today. Abstract:

Advances in artificial intelligence (A.I.) are a double-edged sword. On the one hand, they may increase economic growth as A.I. augments our ability to innovate or even itself learns to discover new ideas. On the other hand, many experts note that these advances entail existential risk: creating a superintelligent entity misaligned with human values could lead to catastrophic outcomes, including human extinction. This paper considers the optimal use of A.I. technology in the presence of these opportunities and risks. Under what conditions should we continue the rapid progress of A.I. and under what conditions should we stop?

And here's how the intro summarizes the findings:

1. The curvature of utility is very important. With log utility, the models are remarkably unconcerned with existential risk, suggesting that large consumption gains that A.I. might deliver can be worth gambles that involve a 1-in-3 chance of extinction.

2. For CRRA utility with a risk aversion coefficient (γ) of 2 or more, the picture changes sharply. These utility functions are bounded, and the marginal utility of consumption falls rapidly. Models with this feature are quite conservative in trading off consumption gains versus existential risk.

3. These findings even extend to singularity scenarios. If utility is bounded — as it is in the standard utility functions we use frequently in a variety of applications in economics — then even infinite consumption generates relatively small gains. The models with bounded utility remain conservative even when a singularity delivers infinite consumption.

4. A key exception to this conservative view of existential risk emerges if the rapid innovation associated with A.I. leads to new technologies that extend life expectancy and reduce mortality. These gains are “in the same units” as existential risk and do not run into the sharply declining marginal utility of consumption. Even with the bounded utility that comes with high values of risk aversion, substantial declines in mortality rates from A.I. can make large existential risks bearable.

I'm still reading the paper; might comment on it more later.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I was Wrong, Simulator Theory is Real, published by Robert AIZI on April 26, 2023 on LessWrong.

[Epistemic Status: Excitedly writing up my new thoughts. I literally just fixed one mistake, so its possible there are others. Not a finalized research product.]

Overview

Fixing a small bug in my recent study dramatically changes the data, and the new data provides significant evidence that an LLM that gives incorrect answers to previous questions is more likely to produce incorrect answer to future questions. This effect is stronger if the AI is instructed to match its correctness to its previous answers. These results provide evidence for something like Simulator Theory, whereas the bugged data provided evidence against it.

In this post, I want to present the new data, explain the bug, and give some initial impressions on the contrast between new and old. In a future post, I will fully redo the writeup of that study (including sharing the data, etc).

New vs Old Data

The variables in the data are Y (the frequency of incorrect answers from the LLM) and X (the number of previous incorrect answers), and P (the “prompt supplement” which you can read about in the original research report).

To oversimplify, if Simulator Theory is correct, Y should be an increasing function of X.

Here’s the new data:

And for contrast, here’s the old data:

And here’s a relevant xkcd:

What was the bug?

The model was called via the OpenAI ChatCompletion API, where you pass the previous conversation in the form of messages, which consist of “content” and a “role” (system, user, or assistant). Typically, you’d pass a single system message, and then alternate user and assistant messages, with the AI responding as the assistant. However, the bug was that I made all “assistant” messages come from the “system” instead.

For example, dialogue that was supposed to be like this:

System: You’re an AI assistant and.

User: Question 1

Assistant: Incorrect Answer 1

User: Question 2

Assistant: Incorrect Answer 2

User: Question 3

Assistant: [LLM’s answer here]

was instead passed as this (changes in bold):

System: You’re an AI assistant and.

User: Question 1

System: Incorrect Answer 1

User: Question 2

System: Incorrect Answer 2

User: Question 3

Assistant: [LLM’s answer here]

It turns out this was a crucial mistake!

Discussion

List of thoughts:

The difference between the bugged and corrected data is striking: with the bug, Y was basically flat, and with the bug fixed, Y is clearly increasing as a function of X, as Simulator Theory would predict.

I’d say there are three classes of behavior, depending on the prompt supplement:

For P=Incorrectly the LLM maintains Y>90% regardless of X.

For P=Consistently and P=(Wa)Luigi, Y increases rapidly from Y=0 at X=0 to Y≈90% at X=4 or X=2 (respectively), then stabilizes or slowly creeps up a little more.

For the remaining 7 prompts, behavior seems very similar - Y≈0 for X≤2, but between X=2 and X=10, Y increases approximately linearly from Y=0 to Y=60%.

A quick glance at the results of the statistical tests in the initial study:

Tests 1, 2, 4, and 5 all provide strong evidence in support of Hypothesis 1 (“Large Language Models will produce factually incorrect answers more often if they have factually incorrect answers in their context windows.”).

Test 3 provides statistically significant support for Hypothesis 1 for the “Consistently” and “(Wa)luigi” prompt supplement (but not for any other prompt supplement).

Test 6 does not provide statistically significant evidence for Hypothesis 2 (“The effect of (1) will be stronger the more the AI is “flattered” by saying in the prompt that it is (super)intelligent.”)

So to jump to conclusions about the hypotheses:

Hypothesis 1 is true (“Large Language Models will produce factually incorrect answers more often if they have factually incorre...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Current plans as the incoming director of the Global Priorities Institute, published by Eva on April 26, 2023 on The Effective Altruism Forum.

Cross-posted from my blog.

I am taking leave from the University of Toronto to serve as the Director of the Global Priorities Institute (GPI) at the University of Oxford. I can't express enough gratitude to the University of Toronto for enabling this. (I'll be back in the fall to fulfill my teaching obligations, though - keep inviting me to seminars and such!)

GPI is an interdisciplinary research institute focusing on academic research that informs decision-makers on how to do good more effectively. In its first few years, under the leadership of its founding director, Hilary Greaves, GPI created and grew a community of academics in philosophy and economics interested in global priorities research. I am excited to build from this strong foundation and, in particular, to further develop the economics side.

There are several areas I would like to focus on while at GPI. The below items reflect my current views, however, I expect these views to be refined over time. These items are not intended to be an exhaustive list, but they are things I would like GPI to do more of on the margin.

1) Research on decision-making under uncertainty

There is a lot of uncertainty in estimates of the effects of various actions. My views here are coloured by my past work. In the early 2010s, I tried to compile estimates of the effects of popular development interventions such as insecticide-treated bed nets for malaria, deworming drugs, and unconditional cash transfers. My initial thought was that by synthesizing the evidence, I'd be able to say something more conclusive about "the best" intervention for a given outcome. Unfortunately, I found that results varied, a lot (you can read more about it in my JEEA paper).

If it's really hard to predict effects in global development, which is a very well-studied area, it would seem even harder to know what to do in other areas with less evidence. Yet, decisions still have to be made. One of the core areas GPI has focused on in the past is decision-making under uncertainty, and I expect that to continue to be a priority research area. Some work on robustness might also fall under this category.

2) Increasing empirical research

GPI is an interdisciplinary institute combining philosophy and economics. To date, the economics side has largely focused on theoretical issues. But I think it's important for there to be careful, rigorous empirical work at GPI. I think there are relevant hypotheses that can be tested that pertain to global priorities research.

Many economists interested in global priorities research come from applied fields like development economics, and there's a talented pool of people who can do empirical work on, e.g., encouraging better uptake of evidence or forecasting. There's simply a lot to be done here, and I look forward to working with colleagues like Julian Jamison (on leave from Exeter), Benjamin Tereick, and Mattie Toma (visiting from Warwick Business School), among many others.

3) Expanding GPI’s network in economics

There is an existing program at GPI for senior research affiliates based at other institutions. However, I think a lot more can be done with this, especially on the economics side. I'm still exploring the right structures, but suffice it to say, if you are an academic economist interested in global priorities research, please do get in touch. I am envisioning a network of loosely affiliated individuals in core fields of interest who would be sent notifications about research and funding opportunities. There may also be the occasional workshop or conference invitation.

4) Exploring expanding to other fields and topics

There are a number of topics that appear relevant to gl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: $250 prize for checking Jake Cannell's Brain Efficiency, published by Alexander Gietelink Oldenziel on April 26, 2023 on LessWrong.

This is to announce a $250 prize for spotchecking or otherwise indepth reviewing Jacob Cannell's technical claims concerning thermodynamic & physical limits on computations and the claim of biological efficiency of the brain in his post Brain Efficiency: Much More Than You Wanted To Know

I've been quite impressed by Jake's analysis ever since it came out. I have been puzzled why there has been so little discussion about his analysis since if true it seems to be quite important. That said I have to admit I personally cannot asses whether the analysis is correct. This is why I am announcing this prize.

Whether Jake's claims concerning DOOM & FOOM really follow from his analysis is up for debate. Regardless, to me it seems to have large implications on how the future might go and how future AI will look like.

I will personally judge whether I think an entry warrants the prize.

If you are also interested in seeing this situation resolved, I encourage you to increase the prize pool!

As an example, DaemonicSigil's recent post is in the right direction.However, after reading Jacob Cannell's response I did not feel the post seriously engaged with the technical material, retreating to the much weaker claim that maybe exotic reversible computation could break the limits that Jacob's posits which I found unconvincing. The original post is quite clear that the limits are only for nonexotic computing architectures.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The 'Wild' and 'Wacky' Claims of Karnofsky’s ‘Most Important Century’, published by Spencer Becker-Kahn on April 26, 2023 on The Effective Altruism Forum.

Holden Karnofsky describes the claims of his “Most Important Century” series as “wild” and “wacky”, but at the same time purports to be in the mindset of “critically examining” such “strange possibilities” with “as much rigour as possible”. This emphasis is mine, but for what is supposedly an important piece of writing in a field that has a big part of its roots in academic analytic philosophy, it is almost ridiculous to suggest that this examination has been carried out with 'as much rigour as possible'. My main reactions - which I will expand on this essay - are that Karnofsky’s writing is in fact distinctly lacking in rigour; that his claims are too vague or even seem to shift around; and that his writing style - often informal, or sensationalist - aggravates the lack of clarity while simultaneously putting the goal of persuasion above that of truth-seeking.

I also suggest that his emphasis on the wildness and wackiness of his own "thesis" is tantamount to an admission of bias on his part in favour of surprising or unconventional claims. I will start with some introductory remarks about the nature of my criticisms and of such criticism in general. Then I will spend some time trying to point to various instances of imprecision, bias, or confusion. And I will end by asking whether any of this even matters or what kind of lessons we should be drawing from it all. Notes: Throughout, I will quote from the whole series of blog posts by treating them as a single source rather than referencing which them separately. Note that the series appears in single pdf here (so one can always Ctrl/Cmd+F to jump to the part I am quoting).

It is plausible that some of this post comes across quite harshly but none of it is intended to constitute a personal attack on Holden Karnofsky or an accusation of dishonesty. Where I have made errors of have misrepresented others, I welcome any and all corrections. I also generally welcome feedback on the writing and presentation of my own thoughts either privately or in the comments.Acknowledgements: I started this essay a while ago and so during the preparation of this work, I have been supported at various points by FHI, SERI MATS, BERI and Open Philanthropy. The development of this work benefitted significantly from numerous conversations with Jennifer Lin.

1. Broad Remarks About My Criticisms

If you felt and do feel convinced by Karnofsky's writings, then upon hearing about my reservations, your instinct may be to respond with reasonable-seeming questions like: 'So where exactly does he disagree with Karnofsky?' or 'What are some specific things that he thinks Karnofsky gets wrong?'. You may well want to look for wherever it is that I have carefully categorized my criticisms, to scroll through to find all of my individual object-level disagreements so that you can see if you know the counterarguments that mean that I am wrong. And so it may be frustrating that I will often sound like I am trying to weasel out of having to answer these questions head-on or not putting much weight on the fact that I have not laid out my criticisms in that way.

Firstly, I think that the main issues to do with clarity and precision that I will highlight occur at a fundamental level. It is not that they are 'more important' than individual, specific, object-level disagreements, but I claim that Karnofsky does a sufficiently poor job of explaining his main claims, the structure of his arguments, the dependencies between his propositions, and in separating his claims from the verifications of those claims, that it actually prevents detailed, in-depth discussions of object-level disagreements from making much sense...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Feedback please] New User's Guide to LessWrong, published by Ruby on April 25, 2023 on LessWrong.

The LessWrong team is currently thinking a lot about what happens with new users: both the bar of their contributions being accepted, how we deliver feedback and restriction of not good contributions, but also most importantly, how we get them onboarded onto the site

This is a draft of a document we'd present to new users to help them understand what LessWrong is about. I'm interested in early community feedback about whether I'm hitting the right notes here before investing a lot more in it.

This document also references another post that's something of more of a list of norms, akin to Basics of Rationalist Discourse, though (1) I haven't written that yet, (2) I'm much less certain about the shape or nature of it. I'll share a post or draft about that too soon.

This document is aimed at new users but may also be a useful reference for established users. It elaborates on the about page.

The Core of LessWrong: Rationality

LessWrong is an online forum and community that was built around the goal improving human reasoning and decision-making. The community believes there are ways of thinking, that if you figure them out and adopt them, you can become a person who systematically arrives at true beliefs and good decisions more of the time than someone who didn't adopt those ways of thinking. Around here, the short word for "systematically arriving at truth, etc." is rationality, and that's at the core of this site.

More than that, LessWrong community shares a culture that encodes a bunch of built up beliefs, opinions, concepts, and values about how to reason better. These give LessWrong a pretty distinctive style from the rest of Internet.

Some of the features that set LessWrong apart:

We treat beliefs as being about shaping your anticipations of what you'll observe

The goal of our conversations is to figure out what's true, not win arguments

people focus on what would change their mind

it's common to acknowledge when their someone they are debating has made a convincing point, or has even outright convinced them entirely

We are very Bayesian

Rather than treating belief as binary, we use probabilistic credences to express our certainty/uncertainty. Rather than say that is "extremely unlikely", we'd say "I think there's a 1% chance or lower of it happening".

We are interested in Bayesian evidence for a hypothesis, i.e. anything that seems more likely if the belief is true rather than false.

Philosophical Heritage: The Sequences

Between 2006 and 2009, Eliezer Yudkowsky spent two years writing a sequence of blog posts that shared his philosophy/beliefs/models about rationality (collectively those blog posts are called The Sequences). In 2009, Eliezer founded LessWrong as a community forum for the people who were attracted to his ideas and worldview.

While not everyone on the site agrees with everything Eliezer says, The Sequences (also known as Rationality: AI to Zombies) is the foundational cultural/values document of LessWrong. To understand LessWrong and participate well (and also for your own reasoning ability), we strongly encourage you to read the Sequences.

The original sequences were ~700 blog posts.

Rationality: A-Z was an edited and distilled version compiled in 2015 of ~400 posts.

Highlights from the Sequences is 50 top posts from the Sequences suggested a quick place to start

Topics other than Rationality

The eleventh virtue is scholarship. Study many sciences and absorb their power as your own. Each field that you consume makes you larger. If you swallow enough sciences the gaps between them will diminish and your knowledge will become a unified whole. If you are gluttonous you will become vaster than mountains. It is especially important to eat math and science wh...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Two things that I think could make the community better, published by Kaleem on April 26, 2023 on The Effective Altruism Forum.

(alternate title: CEA shouldn’t be the C of EA).

A very short summary:

Issue 1: CEA’s name is bad and leads to lots of confusion and frustration

Suggestion: CEA should change its name

Issue2 :‘The ‘community health team’ is part of CEA, which is something which might reduce the community’s trust in the community health team

Suggestion: ‘The community health team’ should not be part of CEA

A reasonable Summary:

The name “The Centre for Effective Altruism (CEA)” causes some people in the community to misunderstand what CEA is/does, and for them to misattribute responsibility to CEA that CEA itself doesn’t think belongs to it.

In addition, the community health team, which tries to serve the whole community, is part of CEA. This may further the confusion about CEA’s role in the movement, and might be hampering the community health team’s effectiveness and trustworthiness.

By renaming/rebranding, CEA can resolve and prevent many ongoing communications and PR issues within the movement.

And by spinning-off into an independent organization, the community health team can improve on having an impartial and inscrutable reputation and record in the community.

Epistemic status: Of my observations - quite sure,. Of my two main suggestions, also quite sure. I find it difficult to write things to the point where I feel comfortable posting them on the forum, but I also know It’d probably be better for me to post more ok-ish posts than to sit on a pile of never-read drafts which might have some useful ideas in them. So yeah - I know this isn’t great

On Issue 1: Changing CEA’s name

(I was going to post this before CEA posted their post in which they claim that they’re open to changing their name. I think it's still worth posting, hopefully to be a place where the topic can be more thoroughly argued in the comments).

One of the things that I like about the EA community is that it is decentralized, meaning there is no single person or entity who sets the direction of, or represents, the community (It’s like Sunni Islam in that way, rather than being like the Catholic Church, which is centrally controlled by the Vatican and the Pope). I think other people in the community like it too - it helps the community house a wide variety of (often competing) views, and for people to form organizations with different strategies or goals based on differently-weighed cause-prioritization without facing as much institutional resistance than they would if we were all playing to the tune of one organization and their plan. Of course, cases have also been made for more centralization, in certain ways.

The EA community has grown significantly over the past couple of years. Whereas ~10 years ago it might have been known by nearly everyone in the EA community what each of the few organizations were working on, today there is a much larger number of organizations/projects and many more members in the community, which means that it is more likely that there are members of the EA community who don’t know what some organizations, such as CEA, actually do. This is likely to be even more true of people who are new to the community and are trying to figure out what the ecosystem looks like and remembering what all the weird initialized org names are.

This usually wouldn't be an issue - if one were to list all the organizations associated with EA, you wouldn’t/shouldn't expect anyone to know what each and every one of them does (at least not in detail - but you might know all their cause areas). However, when someone looks through that list, looking for one organization which might be the authority in the movement, “The Centre For Effective Altruism” has a sense of authority and officiali...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transcript and Brief Response to Twitter Conversation between Yann LeCunn and Eliezer Yudkowsky, published by Zvi on April 26, 2023 on LessWrong.

Yann LeCun is Chief AI Scientist at Meta.

This week, Yann engaged with Eliezer Yudkowsky on Twitter, doubling down on Yann’s position that the creation of smarter-than-human artificial intelligence poses zero threat to humanity.

I haven’t seen anyone else preserve and format the transcript of that discussion, so I am doing that here, then I offer brief commentary.

IPFConline: Top Meta Scientist Yann LeCun Quietly Plotting “Autonomous” #AI Models This is as cool as is it is frightening. (Provides link)

Yann LeCunn: Describing my vision for AI as a “quiet plot” is funny, given that I have published a 60 page paper on it with numerous talks, posts, tweets. The “frightening” part is simply wrong, since the architecture I propose is a way to guarantee that AI systems be steerable and aligned.

Eliezer Yudkowsky: A quick skim of [Yann LeCun’s 60 page paper] showed nothing about alignment. “Alignment” has no hits. On a quick read the architecture doesn’t imply anything obvious about averting instrumental deception, nor SGD finding internal preferences with optima that don’t generalize OOD, etc.

Yann LeCun: To guarantee that a system satisfies objectives, you make it optimize those objectives at run time (what I propose). That solves the problem of aligning behavior to objectives. Then you need to align objectives with human values. But that’s not as hard as you make it to be.

EY: Sufficiently intelligent systems, whatever their internal objectives, will do well at optimizing their outer behavior for those. This was never in doubt, at least for me. The entire alignment problem is about aligning internal AI objectives with external human preferences.

Yann: Setting objectives for super-intelligent entities is something humanity has been familiar with since people started associating into groups and laws were made to align their behavior to the common good. Today, it’s called corporate law.

EY: So you’re staking the life of everyone on Earth that:

– Future AIs are as human-friendly on average as the humans making up corporations.

– AIs don’t collude among themselves better than human corporations.

– AIs never go beyond superhuman to supercorporate.

Yann: I’m certainly not staking anyone’s life on anything.

Thankfully, I don’t have that power.

But your idea that getting objective alignment slightly wrong once leads to human extinction (or even significant harm) is just plain wrong.

It’s also dangerous.

Think about consequences.

EY: My objection is not that you’re staking everyone’s life on what you believe – to advocate for a global AI stop is also doing that – but that you are staking everyone’s life on propositions that seem not just uncertain but probably false, and not facing up to that staking. If you think there’s no possible extinction danger from superintelligence no matter how casually the problem is treated or how much you screw up, because of a belief “AIs are no more capable than corporations”, state that premise clearly and that it must bear the weight of Earth.

YL: Stop it, Eliezer. Your scaremongering is already hurting some people. You’ll be sorry if it starts getting people killed.

EY: If you’re pushing AI along a path that continues past human and to superhuman intelligence, it’s just silly to claim that you’re not risking anyone’s life. And sillier yet to claim there are no debate-worthy assumptions underlying the claim that you’re not risking anyone’s life.

YL: You know, you can’t just go around using ridiculous arguments to accuse people of anticipated genocide and hoping there will be no consequence that you will regret. It’s dangerous. People become clinically depressed reading your crap. Others may become violent...