Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Resources I send to AI researchers about AI safety, published by Vael Gates on June 14, 2022 on LessWrong.

This is my masterlist of resources I send AI researchers who are mildly interested in learning more about AI safety. I pick and choose which resources to send based on the researcher's interests. The resources at the top of the email draft are the ones I usually send, and I add in later sections as seems useful. I'll also sometimes send The Alignment Problem, Human-Compatible, or The Precipice.I've also included a list of resources that I had students read through for the course Stanford first-year course "Preventing Human Extinction", though I'd most recommend sufficiently motivated students read AGISF Technical Agenda. These reading choices are drawn from the various other reading lists; this is not original in any way, just something to draw from if you're trying to send someone some of the more accessible resources.

There's a decent chance that I'll continue updating this post as time goes on, since my current use case is copy-pasting sections of this email to interested parties. Note that "I" and "Vael" are mentioned a few times, so you'll need to edit a bit if you're copy-pasting. Happy to make any edits and take suggestions.

[Crossposted to the EA Forum]

List for AI researchers

Hello X,

Very nice to speak to you! As promised, some resources on AI alignment. I tried to include a bunch of stuff so you could look at whatever you found interesting. Happy to chat more about anything, and thanks again.

Introduction to the ideas

"The case for taking AI seriously as a threat to humanity" by Kelsey Piper (Vox)

The Most Important Century and specifically "Forecasting Transformative AI" by Holden Karnofsky, blog series and podcast. Most recommended for timelines.

A short interview from Prof. Stuart Russell (UC Berkeley) about his book, Human-Compatible (the other main book in the space is The Alignment Problem, by Brian Christian, which is written in a style I particularly enjoyed)

Technical work on AI alignment

Empirical work by DeepMind's Safety team on alignment

Empirical work by Anthropic on alignment

Talk (and transcript) by Paul Christiano describing the AI alignment landscape in 2020

Podcast (and transcript) by Rohin Shah, describing the state of AI value alignment in 2021

Alignment Newsletter and ML Safety Newsletter

Unsolved Problems in ML Safety by Hendrycks et al. (2022)

Alignment Research Center

Interpretability work aimed at alignment: Elhage et al. (2021) and Olah et al. (2020)

AI Safety Resources by Victoria Krakovna (DeepMind) and Technical Alignment Curriculum

Introduction to large-scale risks from humanity, including "existential risks" that could lead to the extinction of humanity

The first third of this book summary (copied below) of the book "The Precipice: Existential Risk and the Future of Humanity" by Toby Ord

Chapter 3 is on natural risks, including risks of asteroid and comet impacts, supervolcanic eruptions, and stellar explosions. Ord argues that we can appeal to the fact that we have already survived for 2,000 centuries as evidence that the total existential risk posed by these threats from nature is relatively low (less than one in 2,000 per century).

Chapter 4 is on anthropogenic risks, including risks from nuclear war, climate change, and environmental damage. Ord estimates these risks as significantly higher, each posing about a one in 1,000 chance of existential catastrophe within the next 100 years. However, the odds are much higher that climate change will result in non-existential catastrophes, which could in turn make us more vulnerable to other existential risks.

Chapter 5 is on future risks, including engineered pandemics and artificial intelligence. Worryingly, Ord puts the risk of engineered pandemics causing an existential ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the New York Effective Altruism Hub: a coworking and event space (+AIS/longtermism office scoping), published by Rockwell Schwartz on June 13, 2022 on The Effective Altruism Forum.

We're excited to announce that EA NYC and The Centre For Effective Altruism (CEA) are opening a new space in Manhattan—with a working name of New York Effective Altruism Hub (NYEAH)—that will serve as a daily coworking space for EA professionals, an event space for after-hours community events, and a meeting space for EA organizations.We are currently in the process of finalizing a lease and anticipate the office opening in the late summer or early fall, depending on renovation needs.SurveyPotentially interested in using the space? Please fill out our latest survey.

This survey is a follow-up to our December questionnaire and will shape the office design, furnishings, and internal layout. Your feedback will directly impact our decisions as we get NYEAH off the ground.

AI Safety/Longtermism OfficeThere is also interest in potentially opening a second and separate office space that would be specifically focused on AI safety and longtermism and you'll find questions related to this in the survey as well.

NY EA Hub Office Manager OpeningWe are seeking an NYEAH Operations Manager who is passionate about helping others make a positive difference by ensuring their working environment is optimized. This role will be a direct employee of CEA and work as a part of the EA NYC team, assisting with EA events. Applications close the night of June 29. You can reach out to EA NYC Director Rocky with any questions and check out this EA Forum post describing what it's like to work as an EA Office Manager!

P.S. EA NYC is currently hiring for several roles and our future employees will have the option of working out of NYEAH!

People have already indicated they want a cooler name, which we are open to. We want the intent of the space to be legible, including for people traveling to NY and trying to plug in quickly and easily.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Steering AI to care for animals, and soon, published by Andrew Critch on June 14, 2022 on The Effective Altruism Forum.

I'm worried that animal welfare advocates might neglect the importance of AI in determining what happens to animals. More specifically, I'm worried that the value

{animal welfare matters}

and the true-according-to-me belief

{AI is going to transform and determine most of what happens on earth}

... don't exist in same person often enough, such that opportunities to steer AI technology toward applications that care for animals could go under-served.

Of course, we could hope that AI alignment solutions would, if effective in protecting human wellbeing, would likely serve animals as well, but I'm not so sure, and I'd like to see more efforts to change the memetic landscape among present-day humans to better recognize the sentience and moral importance of of animal life, especially wild animals that we might not by default think of humanity as "responsible for". The only concrete example I know of is the following, which seems to have had little support from or connection to EA:

/ - a project using ML to translate the language of sperm whale in their natural habitat. As far as I know, they are not fully funded, and could probably use support from EAs, and I think the work they're doing is in-principle feasible from a technical perspective.

Ideally, I'd like to see a lot more support for projects like the above, which increase AI animal welfare bandwidth over the next 3-5 years, before more break-neck progress in AI makes it even harder to influence people and steer where technology and its applications are going.

So! If you care about animals, and are starting to get more interested in importance of AI, please consider joining or supporting or starting projects that steer AI progress toward caring more about animals. I'm sad to say my day job is not addressing this problem nearly as well or as quickly as I'd like (although we will somewhat), so I wanted to issue a bit of a cry for help — or at least, a cry for "someone should do something here".

Whatever you decide, good luck, and thanks for reading!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra EY: Can AGI destroy us without trial & error?, published by Nikita Sokolsky on June 13, 2022 on LessWrong.

NB: I've never published on LW before and apologize if my writing skills are not in line with LW's usual style. This post is an edited copy of the same article in my blog.

EY published an article last week titled “AGI Ruin: A List of Lethalities”, which explains in detail why you can’t train an AGI that won’t try to kill you at the first chance it gets, as well as why this AGI will eventually appear given humanity’s current trajectory in computer science. EY doesn’t explicitly state a timeline over which AGI is supposed to destroy humanity, but it’s implied that this will happen rapidly and humanity won’t have enough time to stop it. EY doesn’t find the question of how exactly AGI will destroy humanity too interesting and explains it as follows:

My lower-bound model of "how a sufficiently powerful intelligence would kill everyone, if it didn't want to not do that" is that it gets access to the Internet, emails some DNA sequences to any of the many many online firms that will take a DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery. (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.)

The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer. Losing a conflict with a high-powered cognitive system looks at least as deadly as "everybody on the face of the Earth suddenly falls over dead within the same second".

Let’s break down EY’s proposed plan for “Skynet” into the requisite engineering steps:

Design a set of proteins that can form the basis of a “nanofactory”

Adapt the protein design to the available protein printers that accept somewhat-anonymous orders over the Internet

Design “diamondoid bacteria” that can kill all of humanity and that can be successfully built by the “nanofactory”. The bacteria must be self replicating and able to extract power from solar energy for self sustainance.

Execute the evil plan by sending out the blueprints to unsuspecting protein printing corporations and rapidly taking over the world afterwards

The plan above looks great for a fiction book and EY is indeed a great fiction writer in addition to his Alignment work, but there’s one unstated assumption: the AGI will not only be able to design everything using whatever human data it has available, but it will also execute the evil plan without needing lots of trial and error like mortal human inventors do. And surprisingly this part of EY’s argument gets little objection. A visual representation of my understanding of EY’s mental model of AGI vs. Progress is as follows:

How fast can humans develop novel technologies?

Humans are the only known AGI that we have available for reference, so we could look at the fastest known examples of novel engineering to see how fast an AGI might develop something spectacular and human-destroying. Patrick Collison of Stripe keeps a helpful page titled “Fast” with notable “examples of people quickly accomplishing ambitious things together”. The engineering entries include:

P-80 Shooting Star, a World War II aircraft designed and built in 143 days by Lockheed Martin.

Spirit of St. Loui...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Is Most Important For Your Productivity?, published by lynettebye on June 13, 2022 on The Effective Altruism Forum.

The Peek behind the Curtain interview series includes interviews with eleven people I thought were particularly successful, relatable, or productive. We cover topics ranging from productivity to career exploration to self-care.

This third post covers “What is most important for your productivity?”

You can find bios of my guests at the end of this post, and view other posts in the series here. This post is cross posted on my blog.

What are the most important things you do for your productivity?

Keep checking in with your high-level goal

I think it really comes back down to writing down and rewriting down and thinking about the high-level goal of the project. Having something that's an easy handle, that's like a sentence or two, and just checking in with myself a lot about where I'm at with respect to the high-level goals. Writing something every month about like, "I did this last month and what was the goal? Have I moved toward the goal or not?"

Ajeya Cotra

Start by breaking the task down

One thing that's really helpful for me is just small-scale planning. When I'm starting a task, write down a list of the things that I would probably want to do. Just having that be clear helps a lot.

Daniel Ziegler

Checklists

I have a checklist and every day when I sign off, I will write every single one of my projects and email them to my boss letting them know status on all the projects. When I was in college, I lived on a checklist. Every day I had the checklist that needed to be completed that day. It makes it so that I don't drop the ball.

Abi Olvera

Decrease anxiety

Decreasing anxiety over things. Because I feel like there's this vicious cycle a lot of people get sucked up in, where if you're really anxious about something, you're focused on the anxiety and assuaging the anxiety rather than on the work that you need to do.

Eva Vivalt

Internalize that procrastinating only makes aversive tasks worse

I think I've gotten better over time at doing what needs to be done that's not easy or particularly enjoyable, and finding out again and again that it's all better if I tackle it head-on rather than trying to put it off for as long as humanly possible.

Similar to being a recovering introvert who always thought going to a party was a terrible thing to do yet always, every time I went, both enjoying it and then on reflection going, "Wow. I definitely make the right call there." Regretting it when I decide not to do so.

I guess over time I've learned that the sense of aversion usually fades after I actually start doing the aversive task. It's not fun, but it's okay. A sense of relief after you've done it and gotten it off your plate. It seems like worth doing rather than maybe like spending a week or two weeks or two months going, "Oh man, I need to do this. It's even worse since I haven't done it for this long." Getting stuck in that sort of cycle.

Greg Lewis

Financial penalties for work within your control

I use Beeminder. I had to put in the initials of the person that I emailed, and it kind of just, "Why don't I aim for five emails a week?" I was trying to hit more than five. It was only counting on what I could control, which was emailing. It doesn't matter if they don't respond, the fact is I emailed. I do it as well to cap my Facebook time. I do it as well to keep my unread inbox below 75.

Abi Olvera

Track if your week actually moved something forward

It's like, "Was there something that I was able to take the time for this week, and that moved forward significantly somehow," which can look like lots of different things. It could be strategic planning for CSET or it could be working on a specific research project and having time to write or other things like tha...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Continuity Assumptions, published by Jan Kulveit on June 13, 2022 on The AI Alignment Forum.

This post will try to explain what I mean by continuity assumptions (and discontinuity assumptions), and why differences in these are upstream of many disagreements about AI safety. None of this is particularly new, but seemed worth reiterating in the form of a short post.

What I mean by continuity

Start with a cliff:This is sometimes called the Heaviside step function. This function is actually discontinuous, and so it can represent categorical change. In contrast, this cliff is actually continuous:

From a distance or in low resolution, it looks like a step function; it gets really steep. Yet it is fundamentally different.

How is this relevant

Over the years, I've come to the view that intuitions about whether we live in a "continuous” or "discontinuous” world are one of the few top principal components underlying all disagreements about AI safety. This includes but goes beyond the classical continuous vs. discrete takeoff debates.A lot of models of what can or can't work in AI alignment depends on intuitions about whether to expect "true discontinuities" or just "steep bits". This holds not just in one, but many relevant variables (e.g. the generality of the AI’s reasoning, the speed or differentiability of its takeoff). The discrete intuition usually leads to sharp categories like:

Before the cliffAfter the cliffNon-general systems. Lack the core of general reasoning, that which allows thought in domains far from training dataGeneral systems. Capabilities generalise farWeak systems - that won't kill you, but also won't help you solve alignmentStrong systems - that would help solve alignment, but unfortunately will kill you by default, if unalignedSystems which may be misaligned, but aren't competently deceptive about it System which is actively modelling you at a level where the deception is beyond your ability to noticeWeak actsPivotal acts..

In Discrete World, empirical trends, alignment techniques, etc usually don't generalise across the categorical boundary. The right is far from the training distribution on the left. Your solutions don't survive the generality cliff, there are no fire alarms - and so on.

Note that while assumptions about continuity in different dimensions are in principle not necessarily related, and you could e.g. assume continuity in takeoff and discontinuity in generality - in practice, they seem strongly correlated.

Deep cruxes

Deep priors over continuity versus discontinuity seem to be a crux which is hard to resolve.

My guess is intuitions about continuity/discreteness are actually quite deep-seated: based more on how people do maths, rather than specific observations about the world. In practice, for most researchers, the "intuition" is something like a deep net trained on a whole lifetime of STEM reasoning - they won't update much on individual datapoints, and if they are smart, they are often able to re-interpret observations to be in line with their continuity priors. (As an example, compare Paul Christiano's post on takeoff speeds from 2018, which is heavily about continuity, to the debate between Paul and Eliezer in late 2021. Despite the participants spending years in discussion, progress on bridging the continuous-discrete gap between them seems very limited.)

How continuity helps

In basically every case, continuity implies the existence of systems "somewhere in between".

Systems which are moderately strong: maybe weakly superhuman in some relevant domain and quite general, but at same time maybe still bad with plans on how to kill everyone.

Moderately general systems: maybe are able of general reasoning, but in a strongly bounded way

Proto-deceptive systems which are bad at deception.

The existence of such systems helps us with two fund...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why and how to start a pilot project in EA, published by Ben Williamson on June 13, 2022 on The Effective Altruism Forum.

In October 2021, I received an initial grant of $9,640 from the EA Infrastructure Fund to launch Effective Self-Help, a project researching more effective wellbeing and productivity advice.

This was, well, a little unexpected. The subsequent 7 months have been an exciting learning curve, if at times also daunting and a little overwhelming. What follows is my attempt to convince you why you should start a pilot project in EA and then offer my advice on how to do this based on my experience launching Effective Self-Help (ESH).

I hope this post can inspire a few people to follow through on ideas they have for projects. Starting any kind of project on your own is scary but incredibly rewarding, and I think is the most impactful thing many students and early-career EAs could be doing.

As it’s based on my experience, it’s probably a little unrepresentative in some aspects. I think the central points are pretty universal though, whether you’re wanting to launch an organisation, start a blog, or set up a new community group.

I plan to publish a follow-up post in the next week or so highlighting the many lessons I’ve learnt while building ESH, attempting to shine a light on the pitfalls I’ve encountered so you can avoid making the same mistakes.

Hopefully, there are a few useful lessons here for people early in their journey as an effective (altruist) entrepreneur. At the very least, writing and posting this is a valuable reflection exercise for me.

Why

A case for impact

I’ll state this briefly as this has been discussed in-depth elsewhere. EA has an increasingly large amount of total funding available relative to the number of cost-effective projects it can spend this money on. High availability of funding isn’t universal but is probably particularly true if you’re passionate about longtermism or community-building, this is probably particularly true.

Starting a pilot project that could scale to absorb some of this funding has huge potential value. Some of the world’s largest companies started in a small niche before progressively expanding. Similarly, EA itself started as a small, niche movement focused on charity evaluation before expanding to what it is today.

While your chances of large-scale success may be low, I think there are few alternative career paths that can produce as large a best-case impact as effective entrepreneurship.

A case for personal value

That all sounds great, but what about if you fail? Isn’t that the most likely outcome?

I would argue that as a student or early professional with any interest in entrepreneurship, starting a pilot project is the most valuable thing that you can do irrespective of the project’s success or failure. In other words, I think the personal value of trying to create, run, and scale your own project is large enough to justify the choice of doing this by itself.

A few reasons to justify this strong claim:

1. Demonstrate soft skills in a particularly robust way

As a project lead, you have to manage your time, build relationships, and stay on track with goals on your own. If you can do these things the hard way, any future employer can be confident that you can do them with the benefits of a manager and organisation structure.

2. Stress-test your approach to work.

Lack an accountability system? Tend to form imprecise goals? Struggle to plan and organise your time?

Running your own project will test your approach to the foundational skills of working effectively and productively. You’ll learn very quickly where you need to improve how you work. Out of this, you’ll build more robust systems that will bring you benefits for years to come.

3. Make many more, and better, contacts

People tend to get excited about g...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why so little AI risk on rationalist-adjacent blogs?, published by Grant Demaree on June 13, 2022 on LessWrong.

I read a lot of rationalist-adjacents. Outside of LessWrong and ACX, I hardly ever see posts on AI risk. Tyler Cowen of Marginal Revolution writes that, "it makes my head hurt" but hasn't engaged with the issue. Even Zvi spends very few posts on AI risk.

This is surprising, and I wonder what to make of it. Why do the folks most exposed to MIRI-style arguments have so little to say about them?

Here's a few possibilities

Some of the writers disagree that AGI is a major near-term threat

It's unusually hard to think and write about AI risk

The best rationalist-adjacent writers don't feel like they have a deep enough understanding to write about AI risk

There's not much demand for these posts, and LessWrong/Alignment Forum/ACX are already filling it. Even a great essay wouldn't be that popular

Folks engaged in AI risk are a challenging audience. Eliezer might get mad at you

When you write about AGI for a mainstream audience, you look weird. I don't think this is as true it used to be, since Ezra Klein did it in the New York Times and Kelsey Piper in Vox

Some of these writers are heavily specialized. The mathematicians want to write about pure math. The pharmacologists want to write about drug development. The historians want to argue that WWII strategic bombing was based on a false theory of popular support for the enemy regime, and present-day sanctions are making the same mistake

Some of the writers are worried that they'll present the arguments badly, inoculating their readers against a better future argument

What they wrote

I'll treat Scott Alexander's blogroll as the canonical list of rationalist-adjacent writers. I've grouped them by their stance on the following statement:

Misaligned AGI is among the most important existential risks to humanity

Explicitly agrees and provides original gears-level analysis (2)

Zvi Mowshowitz

“The default outcome, if we do not work hard and carefully now on AGI safety, is for AGI to wipe out all value in the universe.” (Dec 2017)

Zvi gives a detailed analysis here followed by his own model in response to the 2021 MIRI conversations.

Holden Karnofsky of OpenPhil and GiveWell

In his Most Important Century series (Jul 2021 to present), Holden explains AGI risk to mainstream audiences. Ezra Klein featured Holden’s work in the New York Times.

This series had a high impact on me, because Holden used to have specific and detailed objections to MIRI’s arguments (2012). Ten years later, he’s changed his mind.

Explicitly agrees (4)

Jacob Falkovich of Putanumonit

“Misaligned AI is an existential threat to humanity, and I will match $5,000 of your donations to prevent it.” (Dec 2017)

Jacob doesn’t make the case himself, but he links to external sources.

Kelsey Piper of Vox

“the big overarching challenge is making sure that as our systems get more powerful, we design them so their goals are aligned with those of humans — that is, humanity doesn’t construct scaled-up superintelligent AI that overwhelms human intentions and leads to catastrophe.” (Apr 2021)

Steve Hsu of Information Processing

“I do think that the top 3 risks to humanity, in my view, are AI, bio, and unknown unknowns.” (Apr 2020)

Alexey Guzey of Applied Divinity Studies

“AI risk seems to be about half of all possible existential risk.”

The above quote is from a May 2021 PDF, rather than a direct post. I can’t find a frontpage post that makes the AI risk case directly

Explicitly ambivalent (2)

Tyler Cowen of Marginal Revolution

“As for Rogue AI... For now I will just say that it makes my head hurt. It makes my head hurt because the topic is so complicated. I see nuclear war as the much greater large-scale risk, by far” (Feb 2022).

Julia Galef of Rationally Speaking

Julia int...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Unjournal: Call for participants and research, published by david reinstein on June 13, 2022 on The Effective Altruism Forum.

TLDR

If you are interested in being on the Unjournal's Founding Committee, Advisory Board, or Reviewer Pool please fill out THIS FORM (about 3-5 min).

If you have research (your own or others) you would like us to assess, please fill out THIS FORM (1 per entry), which will also make you eligible for a bounty prize. (This part repeats my earlier call on the forum)

Note: This is a linkpost to my general call in the 'gitbook' here, which mainly appeals to academics. I add a few excerpts and comments below.

Overview and call

I David Reinstein am writing with an open call for committee members, board members, reviewers, and suggestions of relevant work for a new peer-review initiative (not a publication!) called The Unjournal. This was discussed in an earlier EA Forum post. It has taken some shape since then, but the plans are largely the same.

The Unjournal team is building a system for credible, public, journal-independent feedback and evaluation of research. ...

Our initial focus: research relevant to global priorities, particularly in the fields of quantitative social science (including economics) and impact evaluation. [we] support [formats for] replicability and robustness (such as hosted dynamic documents, ... we wish to foster a living and continually-improved approach to research projects...

Our linked project space HERE gives further details about the Unjournal project.

Briefly, the Unjournal’s process (proposed and under-discussion):

Identify or invite contributions of relevant research that is publicly hosted on any open platform...

Pay reviewers to evaluate and give careful feedback on this work. Elicit quantifiable and comparable metrics ...

Publicly post and link all reviews of the work....

The Unjournal project has a small amount of funding and we are in a pilot phase. After our initial meeting, we are now putting out an open call for participants in three different roles:

Founding committee members (help manage and set rules, small honorarium)...

Advisory board members (supporters, no time-commitment)..., and

A pool of Reviewers (who will be paid for their time)...

We are also looking for high-quality, globally-pivotal research projects to evaluate, particularly those embodying open science practices and innovative formats (as noted in previous post, with a bounty).

The roles are explained in more detail in the link, and there is a form to submit your interest.

Roles

Founding committee members ... Minimum of 5-10 hours total over the next 9 months, plus editorial activities ... funding for a $50 per hour honorarium for the first 5-10 hours ... seeking further funding.

... interested in people with a range of experiences and skills, including in global priorities research and related fields, other academic research, policy research and practice, open science and meta-science, and bibliometrics and scholarly publishing....

Advisory board members: more passive advisory board, to offer very light touch guidance. ... no time obligation, but you may occasionally be consulted... looking for active researchers, practitioners, and stakeholders with a strong publication record who are affiliated with an academic, governmental or relevant non-profit institution.

Reviewers: We will reach out to reviewers on a case-by-case basis ... we are also putting together a list of people interested... $250 per review [hoping for funding to increase this]

Feel free to reach out directly

I would 'repaste the whole thing' but I don't want several versions of the same document floating around, in case we want to add clarifications or note changes.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On A List of Lethalities, published by Zvi on June 13, 2022 on LessWrong.

Response to (Eliezer Yudkowsky): A List of Lethalities.

Author’s Note: I do not work in AI Safety, lack technical domain knowledge and in many ways am going to be wrong. I wasn’t going to write this to avoid potentially wasting too much time all around without having enough to offer, and for fear of making stupid errors, but it was clear that many people thought my response would be valuable. I thank those whose anonymous sponsorship of this post both paid for my time and made me update that the post was worth writing. I would be happy for this to happen again in the future.

Eliezer has at long last delivered the definitive list of Eliezer Rants About Why AGI Will Definitely Absolutely For Sure Kill Everyone Unless Something Very Unexpected Happens.

This is excellent. In the past we had to make do with makeshift scattershot collections of rants. Now they are all in one place, with a helpful classification system. Key claims are in bold. We can refer, consider and discuss them.

It would be an even better post if it were more logically organized, with dependencies pointed out and mapped and so on.

One could also propose making it not full of rants, but I don’t think that would be an improvement. The rants are important. The rants contain data. They reveal Eliezer’s cognitive state and his assessment of the state of play. Not ranting would leave important bits out and give a meaningfully misleading impression.

I am reminded of this comment of mine that I dug out of the archives, on another Eliezer post that was both useful and enthused with this kind of attitude:

Most of this applies again. Eliezer says explicitly that the alternative post would have been orders of magnitude harder to write, and that the attitude is important information.

I would expand this. Not only are the attitude and repetition important information in terms of allowing you to understand the algorithm generating the post and create a better Inner Eliezer, but they also are importantly illustrating the cognitive world in which Eliezer is operating.

The fact that this is the post we got, as opposed to a different (in many ways better) post, is a reflection of the fact that our Earth is failing to understand what we are facing. It is failing to look the problem in the eye, let alone make real attempts at solutions.

Eliezer is not merely talking to you, yes you (with notably rare exceptions) when he does this. He is also saying model the world as if it really is forcing him to talk like this.

The only point above that doesn’t seem to apply here is #9.

The core message remains the most important thing. Conveying the core message alone would be a big win. But here it also matters that people grasp as many of the individual points as possible, especially whichever of them happens to be the one bottlenecking their understanding of the scope and difficulty of the problem or allowing them to rationalize.

Thus there needs to be a second version of the document that someone else writes that contains the properly organized details without the ranting, for when that is what is needed.

In terms of timelines, only ‘endgame’ timelines (where endgame means roughly ‘once the first team gets the ability to create an AGI capable of world destruction’) are mentioned in this post, because they are a key part of the difficulty and ‘how long it takes to get there’ mostly isn’t. Talk of when AGI will kill us is distinct from talk of how or why it will, or whether it will be built. That stuff was the subject of that other post, and it doesn’t really matter in this context.

It is central to the doom claim that once one group can build an AGI, other groups also rapidly gain this ability. This forces humanity to solve the problem both on the first try and...