The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Interview With Cade Metz on His Reporting About Slate Star Codex, published by Zack M Davis on March 26, 2024 on LessWrong.

On 16 March 2024, I sat down to chat with New York Times technology reporter Cade Metz! In part of our conversation, transcribed below, we discussed his February 2021 article "Silicon Valley's Safe Space", covering Scott Alexander's Slate Star Codex blog and the surrounding community.

The transcript has been significantly edited for clarity. (It turns out that real-time conversation transcribed completely verbatim is full of filler words, false starts, crosstalk, "uh huh"s, "yeah"s, pauses while one party picks up their coffee order, &c. that do not seem particularly substantive.)

ZMD: I actually have some questions for you.

CM: Great, let's start with that.

ZMD: They're critical questions, but one of the secret-lore-of-rationality things is that a lot of people think criticism is bad, because if someone criticizes you, it hurts your reputation. But I think criticism is good, because if I write a bad blog post, and someone tells me it was bad, I can learn from that, and do better next time.

So, when we met at the Pause AI protest on February 12th, I mentioned that people in my social circles would say, "Don't talk to journalists." Actually, I want to amend that, because when I later mentioned meeting you, some people were more specific: "No, talking to journalists makes sense; don't talk to Cade Metz specifically, who is unusually hostile and untrustworthy."

CM: What's their rationale?

ZMD: Looking at "Silicon Valley's Safe Space", I don't think it was a good article. Specifically, you wrote,

In one post, [Alexander] aligned himself with Charles Murray, who proposed a link between race and I.Q. in "The Bell Curve." In another, he pointed out that Mr. Murray believes Black people "are genetically less intelligent than white people."

End quote. So, the problem with this is that the specific post in which Alexander aligned himself with Murray was not talking about race. It was specifically talking about whether specific programs to alleviate poverty will actually work or not.

This seems like a pretty sleazy guilt-by-association attempt. I'm wondering - as a writer, are you not familiar with the idea that it's possible to quote a writer about one thing without agreeing with all their other views? Did they not teach that at Duke?

CM: That's definitely true. It's also true that what I wrote was true. There are different ways of interpreting it. You're welcome to interpret it however you want, but those areas are often discussed in the community. And often discussed by him. And that whole story is backed by a whole lot of reporting. It doesn't necessarily make it into the story. And you find this often that within the community, and with him, whether it's in print or not in print, there is this dancing around those areas.

And you can interpret that many ways. You can say, we're just exploring these ideas and we should be able to.

ZMD: And that's actually my position.

CM: That's great. That's a valid position. There are other valid positions where people say, we need to not go so close to that, because it's dangerous and there's a slippery slope. The irony of this whole situation is that some people who feel that I should not have gone there, who think I should not explore the length and breadth of that situation, are the people who think you should always go there.

ZMD: I do see the irony there. That's also why I'm frustrated with the people who are saying, "Don't talk to Cade Metz," because I have faith. I am so serious about the free speech thing that I'm willing to take the risk that if you have an honest conversation with someone, they might quote your words out of context on their blog.

CM: But also, it's worth discussing.

ZMD: It's worth tryin...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Resist the Fading Qualia Argument (Andreas Mogensen), published by Global Priorities Institute on March 26, 2024 on The Effective Altruism Forum.

This paper was published as a GPI working paper in March 2024.

Abstract

The Fading Qualia Argument is perhaps the strongest argument supporting the view that in order for a system to be conscious, it does not need to be made of anything in particular, so long as its internal parts have the right causal relations to each other and to the system's inputs and outputs.

I show how the argument can be resisted given two key assumptions: that consciousness is associated with vagueness at its boundaries and that conscious neural activity has a particular kind of holistic structure.

I take this to show that what is arguably our strongest argument supporting the view that consciousness is substrate independent has important weaknesses, as a result of which we should decrease our confidence that consciousness can be realized in systems whose physical composition is very different from our own.

Introduction

Many believe that in order for a system to be conscious, it does not need to be made of anything in particular, so long as its internal parts have the right causal relations to each other and to the system's inputs and outputs. As a result, many also believe that the right software could in principle allow there to be something it is like to inhabit a digital computer, controlled by an integrated circuit etched in silicon.

A recent expert report concludes that if consciousness requires only the right causal relations among a system's inputs, internal states, and outputs, then "conscious AI systems could realistically be built in the near term." (Butlin et al. 2023: 6) If that were to happen, it could be of enormous moral importance, since digital minds could have superhuman capacities for well-being and ill-being (Shulman and Bostrom 2021).

But is it really plausible that any system with the right functional organization will be conscious - even if it is made of beer-cans and string (Searle 1980) or consists of a large assembly of people with walky-talkies (Block 1978)? My goal in this paper is to raise doubts about what I take to be our strongest argument supporting the view that consciousness is substrate independent in something like this sense.[1] The argument I have in mind is Chalmers' Fading Qualia Argument (Chalmers 1996:

253-263).

I show how it is possible to resist the argument by appeal to two key assumptions: that consciousness is associated with vagueness at its boundaries and that conscious neural activity has a particular kind of holistic structure. Since these assumptions are controversial, I claim only to have exposed important weaknesses in the Fading Qualia Argument.

I'll begin in section 2 by explaining what the Fading Qualia Argument is supposed to show and the broader dialectical context it inhabits. In section 3, I give a detailed presentation of the argument. In section 4, I show how the argument can be answered given the right assumptions about vagueness and the structure of conscious neural activity. At this point, I rely on the assumption that vagueness gives rise to truth-value gaps.

In section 5, I explain how the argument can be answered even if we reject that assumption. In section 6, I say more about the particular assumption about the holistic structure of conscious neural activity needed to resist the Fading Qualia Argument in the way I outline. I take the need to rely on this assumption to be the greatest weakness of the proposed response.

Read the rest of the paper

^

See the third paragraph in section 2 for discussion of two ways in which the conclusion supported by this argument is weaker than some may expect a principle of substrate independence to be.

Thanks for listening. To help us out...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Modern Transformers are AGI, and Human-Level, published by Abram Demski on March 26, 2024 on The AI Alignment Forum.

This is my personal opinion, and in particular, does not represent anything like a MIRI consensus; I've gotten push-back from almost everyone I've spoken with about this, although in most cases I believe I eventually convinced them of the narrow terminological point I'm making.

In the AI x-risk community, I think there is a tendency to ask people to estimate "time to AGI" when what is meant is really something more like "time to doom" (or, better, point-of-no-return). For about a year, I've been answering this question "zero" when asked.

This strikes some people as absurd or at best misleading. I disagree.

The term "Artificial General Intelligence" (AGI) was coined in the early 00s, to contrast with the prevalent paradigm of Narrow AI.

I was getting my undergraduate computer science education in the 00s; I experienced a deeply-held conviction in my professors that the correct response to any talk of "intelligence" was "intelligence for what task?" -- to pursue intelligence in any kind of generality was unscientific, whereas trying to play chess really well or automatically detect cancer in medical scans was OK.

I think this was a reaction to the AI winter of the 1990s. The grand ambitions of the AI field, to create intelligent machines, had been discredited. Automating narrow tasks still seemed promising. "AGI" was a fringe movement.

As such, I do not think it is legitimate for the AI risk community to use the term AGI to mean 'the scary thing' -- the term AGI belongs to the AGI community, who use it specifically to contrast with narrow AI.

Modern Transformers[1] are definitely not narrow AI.

It may have still been plausible in, say, 2019. You might then have argued: "Language models are only language models! They're OK at writing, but you can't use them for anything else." It had been argued for many years that language was an AI complete task; if you can solve natural-language processing (NLP) sufficiently well, you can solve anything. However, in 2019 it might still be possible to dismiss this.

Basically any narrow-AI subfield had people who will argue that that specific subfield is the best route to AGI, or the best benchmark for AGI.

The NLP people turned out to be correct. Modern NLP systems can do most things you would want an AI to do, at some basic level of competence. Critically, if you come up with a new task[2], one which the model has never been trained on, then odds are still good that it will display at least middling competence. What more could you reasonably ask for, to demonstrate 'general intelligence' rather than 'narrow'?

Generative pre-training is AGI technology: it creates a model with mediocre competence at basically everything.

Furthermore, when we measure that competence, it usually falls somewhere within the human range of performance. So, as a result, it seems sensible to call them human-level as well. It seems to me like people who protest this conclusion are engaging in goalpost-moving.

More specifically, it seems to me like complaints that modern AI systems are "dumb as rocks" are comparing AI-generated responses to human experts. A quote from the dumb-as-rocks essay:

GenAI also can't tell you how to make money. One man asked GPT-4 what to do with $100 to maximize his earnings in the shortest time possible. The program had him buy a domain name, build a niche affiliate website, feature some sustainable products, and optimize for social media and search engines. Two months later, our entrepreneur had a moribund website with one comment and no sales. So genAI is bad at business.

That's a bit of a weak-man argument (I specifically searched for "generative ai is dumb as rocks what are we doing"). But it does demonstrate a pat...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Timelines to Transformative AI: an investigation, published by Zershaaneh Qureshi on March 26, 2024 on The Effective Altruism Forum.

This post is part of a series by Convergence Analysis' AI Clarity team.

Justin Bullock and Elliot Mckernon have recently motivated AI Clarity's focus on the notion of transformative AI (TAI). In an earlier post, Corin Katzke introduced a framework for applying scenario planning methods to AI safety, including a discussion of strategic parameters involved in AI existential risk. In this post, I focus on a specific parameter: the timeline to TAI. Subsequent posts will explore 'short' timelines to transformative AI in more detail.

Feedback and discussion are welcome.

Summary

In this post, I gather, compare, and investigate a range of notable recent predictions of the timeline to transformative AI (TAI).

Over the first three sections, I map out a bird's eye view of the current landscape of predictions, highlight common assumptions about scaling which influence many of the surveyed views, then zoom in closer to examine two specific examples of quantitative forecast models for the arrival of TAI (from Ajeya Cotra and Epoch).

Over the final three sections, I find that:

A majority of recent median predictions for the arrival of TAI fall within the next 10-40 years. This is a notable result given the vast possible space of timelines, but rough similarities between forecasts should be treated with some epistemic caution in light of phenomena such as Platt's Law and information cascades.

In the last few years, people generally seem to be updating their beliefs in the direction of shorter timelines to TAI. There are important questions over how the significance of this very recent trend should be interpreted within the wider historical context of AI timeline predictions, which have been quite variable over time and across sources.

Despite difficulties in obtaining a clean overall picture here, each individual example of belief updates still has some evidentiary weight in its own right.

There is also some conceptual support in favour of TAI timelines which fall on the shorter end of the spectrum. This comes partly in the form of the plausible assumption that the scaling hypothesis will continue to hold. However, there are several possible flaws in reasoning which may underlie prevalent beliefs about TAI timelines, and we should therefore take care to avoid being overconfident in our predictions.

Weighing these points up against potential objections, the evidence still appears sufficient to warrant (1) conducting serious further research into short timeline scenarios and (2) affording real importance to these scenarios in our strategic preparation efforts.

Introduction

The timeline for the arrival of advanced AI is a key consideration for AI safety and governance. It is a critical determinant of the threat models we are likely to face, the magnitude of those threats, and the appropriate strategies for mitigating them.

Recent years have seen growing discourse around the question of what AI timelines we should expect and prepare for. At a glance, the dialogue is filled with contention: some anticipate rapid progression towards advanced AI, and therefore advocate for urgent action; others are highly sceptical that we'll see significant progress in our lifetimes; many views fall somewhere in between these poles, with unclear strategic implications.

The dialogue is also evolving, as AI research and development progresses in new and sometimes unexpected ways. Overall, the body of evidence this constitutes is in need of clarification and interpretation.

This article is an effort to navigate the rough terrain of AI timeline predictions. Specifically:

Section I collects and loosely compares a range of notable, recent predictions on AI timelines (taken from su...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Effective Giving Projects That Have (and Haven't) Been Tried Among Christians, published by JDBauman on March 26, 2024 on The Effective Altruism Forum.

TLDR: US Christian giving amounts to hundreds of billions of dollars per year. Much of it is far less effective than it could be. EA for Christians (EACH) is tackling this, but more can be done. Below is a list of projects we have worked on so far related to effective giving. If this excites you, or you'd like to start a new project or incubate a charity improving effectiveness among Christians/Christian orgs, we'd love to partner with or support you.

Context: I recently had a chat with someone at Giving What We Can who thought that people may be less keen to start a new project in the effective giving & Christianity space because they assume Effective Altruism for Christians has already tried it. But there's a lot we haven't tried (or things we have tried that we might not be best at). For more context, EACH is a global community of 500+ Christians in EA. I'm the FT director and I work with numerous excellent and committed PT staff.

Effective-giving related Projects: In no particular order, here's a short list of most of the effective giving projects we've undertaken over the last 3-5 years (while I've worked here). Some of these have cross-over with careers, EA community building, etc.

Projects we're giving proactive attention to (at least 1-2+ staff hours a week) are marked with ()

Projects we're giving even more attention to (2+ staff hours a week) are marked with (+)

1-on-1s with Christians interested in effective giving (we've done 500+ to-date; most of our 1-on-1s at least touch on effective giving) (+)

General EA Christian conferences, retreats, and meetups (+)

A conference organizing Christian impact professionals from large Christian development charities (E.g. Compassion, Hope, etc.) () to discuss EA . We did one in 2023. A video on this here. DM me for a report on how this went. ()

Report about M&E practices at Christian development charities. We have one forthcoming this spring. ()

Published book about effective altruism and surprising ways to have a large impact with one's life. We have one forthcoming in 2025 (+)

A Christian Campaign for (mostly Givewell) effective charities (raised $380,000+) ()

Talks at churches on effectiveness and radical generosity.

Uni internships doing outreach related to radical and effective generosity (We've had 8 interns for this and also a partnership with One-For-The-World)

Articles about EA and Christianity, especially effective Christian charity (We've published dozens of blogs (+)

A podcast heavily featuring Christians who earn-to-give or work at effective charities. We've done one with 10+ episodes (+)

3+ videos with Christian youtubers about effective altruism (especially effective giving)

Social meetups at cities across US coastal cities and London (we've done a couple dozen) (+)

Online discussions on EA and Christian themes (we've done 140, about 30 about effective giving topics with an avg. 10 people at each; youtube videos here) ()

A 5-minute animated video describing effective altruism (and effective giving) from a Christian perspective. See here

Academic workshops on effective giving. We've done some on EA themes, with a few talks on generosity. This year we have one on longtermism. ()

Online talks on effective giving themes. We've done 5-10 ()

M&E advising from Christian EA development professionals to Christian development charities. We're starting a pro bono offering in spring 2024 ()

A report on plausibly highest-impact Christian poverty charities. We have done some related work in this report

An Intro-course to EA/Effective giving for Christians. See our 4-week Intro course ()

Career outreach that promotes effective giving as a primary way to have an impactf...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Should rationalists be spiritual / Spirituality as overcoming delusion, published by Kaj Sotala on March 26, 2024 on LessWrong.

I just started thinking about what I would write to someone who disagreed with me on the claim "Rationalists would be better off if they were more spiritual/religious", and for this I'd need to define what I mean by "spiritual".

Here are some things that I would classify under "spirituality":

Rationalist Solstices (based on what I've read about them, not actually having been in one)

Meditation, especially the kind that shows you new things about the way your mind works

Some forms of therapy, especially ones that help you notice blindspots or significantly reframe your experience or relationship to yourself or the world (e.g. parts work where you first shift to perceiving yourself as being made of parts, and then to seeing those parts with love)

Devoting yourself to the practice of some virtue, especially if it is done from a stance of something like "devotion", "surrender" or "service"

Intentionally practicing ways of seeing that put you in a mindstate of something like awe, sacredness, or loving-kindness; e.g. my take on sacredness

(Something that is explicitly not included: anything that requires you to adopt actual literal false beliefs, though I'm probably somewhat less strict about what counts as a true/false belief than some rationalists are. I don't endorse self-deception but I do endorse poetic, non-literal and mythic ways of looking, e.g. the way that rationalists may mythically personify "Moloch" while still being fully aware of the fact that the personification is not actual literal fact.)

I have the sense that although these may seem like very different things, there is actually a common core to them.

Something like:

Humans seem to be evolved for other- and self-deception in numerous ways, and not just the ways you would normally think of.

For example, there are systematic confusions about the nature of the self and suffering that Buddhism is pointing at, with minds being seemingly hardwired to e.g. resist/avoid unpleasant sensations and experience that as the way to overcome suffering, when that's actually what causes suffering.

Part of the systematic confusion seem to be related to social programming; believing that you are unable to do certain things (e.g. defy your parents/boss) so that you would be unable to do that, and you would fit in better to society.

At the same time, even as some of that delusion is trying to make you fit in better, some of it is also trying to make you act in more antisocial ways. E.g. various hurtful behaviors that arise from the mistaken belief that you need something from the outside world to feel fundamentally okay about yourself and that hurting others is the only way to get that okayness.

For whatever reason, it looks like when these kinds of delusions are removed, people gravitate towards being compassionate, loving, etc.; as if something like universal love (said the cactus person) and compassion was the motivation that remained when everything distorting from it was removed.

There doesn't seem to be any strong a priori reason for why our minds had to evolve this way, even if I do have a very handwavy sketch of why this might have happened; I want to be explicit that this is a very surprising and counterintuitive claim, that I would also have been very skeptical about if I hadn't seen it myself! Still, it seems to me like it would be true for most people in the limit, excluding maybe literal psychopaths whom I don't have a good model of.

All of the practices that I have classified under "spirituality" act to either see the functioning of your mind more clearly and pierce through these kinds of delusions or to put you into mind-states where the influence of such delusions is reduced and you sh...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LessOnline (May 31 - June 2, Berkeley, CA), published by Ben Pace on March 26, 2024 on LessWrong.

A Festival of Writers Who are Wrong on the Internet[1]

LessOnline is a festival celebrating truth-seeking, optimization, and blogging. It's an opportunity to meet people you've only ever known by their LessWrong username or Substack handle.

We're running a rationalist conference!

The ticket cost is $400 minus your LW karma in cents.

Confirmed attendees include Scott Alexander, Eliezer Yudkowsky, Katja Grace, and Alexander Wales.

Less.Online

Go through to Less.Online to learn about who's attending, venue, location, housing, relation to Manifest, and more.

We'll post more updates about this event over the coming weeks as it all comes together.

If LessOnline is an awesome rationalist event,

I desire to believe that LessOnline is an awesome rationalist event;

If LessOnline is not an awesome rationalist event,

I desire to believe that LessOnline is not an awesome rationalist event;

Let me not become attached to beliefs I may not want.

Litany of Rationalist Event Organizing

^

But Striving to be Less So

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Third-party testing as a key ingredient of AI policy, published by Zac Hatfield-Dodds on March 25, 2024 on The AI Alignment Forum.

(nb: this post is written for anyone interested, not specifically aimed at this forum)

We believe that the AI sector needs effective third-party testing for frontier AI systems. Developing a testing regime and associated policy interventions based on the insights of industry, government, and academia is the best way to avoid societal harm - whether deliberate or accidental - from AI systems.

Our deployment of large-scale, generative AI systems like Claude has shown us that work is needed to set up the policy environment to respond to the capabilities of today's most powerful AI models, as well as those likely to be built in the future. In this post, we discuss what third-party testing looks like, why it's needed, and describe some of the research we've done to arrive at this policy position.

We also discuss how ideas around testing relate to other topics on AI policy, such as openly accessible models and issues of regulatory capture.

Policy overview

Today's frontier AI systems demand a third-party oversight and testing regime to validate their safety. In particular, we need this oversight for understanding and analyzing model behavior relating to issues like election integrity, harmful discrimination, and the potential for national security misuse.

We also expect more powerful systems in the future will demand deeper oversight - as discussed in our 'Core views on AI safety' post, we think there's a chance that today's approaches to AI development could yield systems of immense capability, and we expect that increasingly powerful systems will need more expansive testing procedures.

A robust, third-party testing regime seems like a good way to complement sector-specific regulation as well as develop the muscle for policy approaches that are more general as well.

Developing a third-party testing regime for the AI systems of today seems to give us one of the best tools to manage the challenges of AI today, while also providing infrastructure we can use for the systems of the future.

We expect that ultimately some form of third-party testing will be a legal requirement for widely deploying AI models, but designing this regime and figuring out exactly what standards AI systems should be assessed against is something we'll need to iterate on in the coming years - it's not obvious what would be appropriate or effective today, and the way to learn that is to prototype such a regime and generate evidence about it.

An effective third-party testing regime will:

Give people and institutions more trust in AI systems

Be precisely scoped, such that passing its tests is not so great a burden that small companies are disadvantaged by them

Be applied only to a narrow set of the most computationally-intensive, large-scale systems; if implemented correctly, the vast majority of AI systems would not be within the scope of such a testing regime

Provide a means for countries and groups of countries to coordinate with one another via developing shared standards and experimenting with Mutual Recognition agreements

Such a regime will have the following key ingredients [1]:

Effective and broadly-trusted tests for measuring the behavior and potential misuses of a given AI system

Trusted and legitimate third-parties who can administer these tests and audit company testing procedures

Why we need an effective testing regime

This regime is necessary because frontier AI systems - specifically, large-scale generative models that consume substantial computational resources - don't neatly fit into the use-case and sector-specific frameworks of today.

These systems are designed to be 'everything machines' - Gemini, ChatGPT, and Claude can all be adapted to a vast number of do...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How Educational Courses Help Build Fields: Lessons from AI Safety Fundamentals, published by Jamie B on March 25, 2024 on The Effective Altruism Forum.

Cross-posted as there may be others interested in educating others about early-stage research fields on this forum. I am considering taking on additional course design projects over the next few months. Learn more about

hiring me to consult on course design.

Introduction / TL;DR

So many important problems have too few people with the right skills engaging with them.

Sometimes that's because nobody has heard of your problem, and advocacy is necessary to change that. However, once a group of people believe your problem is important, education is the next step in empowering those people with the understanding of prior work in your field, and giving them the knowledge of further opportunities to work on your problem.

Education (to me) is the act of collecting the knowledge that exists about your field into a sensible structure, and transmitting it to others in such a way that they can develop their own understanding.

In this post I describe how the AI Safety Fundamentals course helped to drive the fields of AI alignment and AI governance forward. I'll then draw on some more general lessons for other fields that may benefit from education, which you can skip straight to

here.

I don't expect much of what I say to be a surprise to passionate educators, but when I was starting out with BlueDot Impact I looked around for write-ups on the value of education and found them lacking. This might help others who are starting out with field building and are unsure about putting time into education work.

Case study: The AI Safety Fundamentals Course

Running the AI Safety Fundamentals Course

Before running the AI Safety Fundamentals course, I was running a casual reading group in Cambridge, on the topic of technical AI safety papers.

We had a problem with the reading group: lots of people wanted to join our reading group, would turn up, but would bounce because they didn't know what was going on. The space wasn't for them. Not only that, the experienced members in the group found themselves repeatedly explaining the same initial concepts to newcomers. The space wasn't delivering for experienced people, either.

It was therefore hard to get a community off the ground, as attendance at this reading group was low. Dewi (later, my co-founder) noticed this problem and got to work on a curriculum with Richard Ngo - then a PhD student at Cambridge working on the foundations of the alignment problem. As I recall it, their aim was to make an 'onboarding course for the Cambridge AI safety reading group'. (In the end, the course far outgrew that remit!)

Lesson 1: A space for everyone is a space for no-one.

You should feel okay about being exclusive to specific audiences. Where you can, try to be inclusive by providing other options for audiences you're not focusing on. That could look like:

To signal the event is for beginners, branding your event or discussion as "introductory".

Set clear criteria for entry to help people self-assess, e.g. "assumed knowledge: can code up a neural network using a library like Tensorflow/Pytorch".

There was no great way to learn about alignment, pre-2020

To help expose some of the signs that a field is ready for educational materials to be produced, I'll briefly discuss how the AI alignment educational landscape looked before AISF.

In 2020, the going advice for how to learn about AI Safety for the first time was:

Read everything on the alignment forum. I might not need to spell out the problem with this advice but:

A list of blog posts is unstructured, so it's very hard to build up a picture of what's going on. Everyone was building their own mental framework for alignment from scratch.

It's very hard for amateurs ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On attunement, published by Joe Carlsmith on March 25, 2024 on LessWrong.

(Cross-posted from my website. Podcast version here, or search for "Joe Carlsmith Audio" on your podcast app.

This essay is part of a series that I'm calling "Otherness and control in the age of AGI." I'm hoping that the individual essays can be read fairly well on their own, but see here for brief summaries of the essays that have been released thus far.)

"You, moon, You, Aleksander, fire of cedar logs.

Waters close over us, a name lasts but an instant.

Not important whether the generations hold us in memory.

Great was that chase with the hounds for the unattainable meaning of the world."

~ Czeslaw Milosz, "Winter"

"Poplars (Autumn)," by Claude Monet (image source here)

My last essay examined a philosophical vibe that I (following others) call "green." Green is one of the five colors on the Magic the Gathering Color Wheel, which I've found (despite not playing Magic myself) an interesting way of classifying the sort of the energies that tend to animate people.[1] The colors, and their corresponding shticks-according-to-Joe, are:

White: Morality.

Blue: Knowledge.

Black: Power.

Red: Passion.

Green: ...

I haven't found a single word that I think captures green. Associations include: environmentalism, tradition, spirituality, hippies, stereotypes of Native Americans, Yoda, humility, wholesomeness, health, and yin. My last essay tried to bring the vibe that underlies these associations into clearer view, and to point at some ways that attempts by other colors to reconstruct green can miss parts of it.

In particular, I focused on the way green cares about respect, in a sense that goes beyond "not trampling on the rights/interests of moral patients" (what I called "green-according-to-white"); and on the way green takes joy in (certain kinds of) yin, in a sense that contrasts with merely "accepting things you're too weak to change" (what I called "green-according-to-black").

In this essay, I want to turn to what is perhaps the most common and most compelling-to-me attempt by another color to reconstruct green - namely, "green-according-to-blue." On this story, green is about making sure that you don't act out of inadequate knowledge. Thus, for example: maybe you're upset about wild animal suffering. But green cautions you: if you try to remake that ecosystem to improve the lives of wild animals, you are at serious risk of not knowing-what-you're-doing.

And see, also, the discourse about "Chesterton's fence," which attempts to justify deference towards tradition and the status quo via the sort of knowledge they might embody.

I think humility in the face of the limits of our knowledge is, indeed, a big part of what's going on with green. But I think green cares about having certain kinds of knowledge too. But I think that the type of knowledge green cares about most isn't quite the same as the sort of knowledge most paradigmatically associated with blue. Let me say more about what I mean.

How do you know what matters?

"I went out to see what I could see..."

~ Annie Dillard, "Pilgrim at Tinker Creek"

An 1828 watercolor of Tintern Abbey, by J.M.W. Turner (image source here)

Blue, to me, most directly connotes knowledge in the sense of: science, "rationality," and making accurate predictions about the world. And there is a grand tradition of contrasting this sort of knowledge with various other types that seem less "heady" and "cognitive" - even without a clear sense of what exactly the contrast consists in. People talk, for example, about intuition; about system 1; about knowledge that lives in your gut and your body; about knowing "how" to do things (e.g.

ride a bike); about more paradigmatically social/emotional forms of intelligence, and so on.

And here, of course, the rationalists protest at the idea ...