Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Expected ethical value of a career in AI safety, published by Jordan Taylor on June 14, 2022 on The Effective Altruism Forum.

This table explores the expected ethical value of a career in AI safety, under different opinions about AI and the long-term future. I made it for considering the robustness of the value of my future career choice to different ways my opinions could change, but it may be useful to others also considering a career in AI safety, or for convincing people more skepitcal of fast AI timelines that safety work is still important.

My main finding is that you need to hold a pretty specific combination of confident beliefs for AI safety work not to seem tremendously valuable in expected impact, and I personally find those beliefs pretty untenable.

Read the “Explanation” section first if you’re confused about anything. You may have to shift+scroll to see the whole table. Here is a google sheet if you want to see the implications of your own opinion.

Probability of AGI within 20 years1%1%20%20%.60%Probability of misalignment by default (conditional on above)5%5%60%60%.95%Probability misaligned AGI causes permanent negative foreclosure of the value of the long-term future (conditional on above)50%50%50%50%.99%Probability that the AI safety community successfully solves the alignment problem, conditional on misalignment happening by default1%1%10%10%.90%Average number of people expected to work primarily on AI safety (reducing existential risk) in the next 20 years10,00010,00010001000.100.Expected length of human-descended civilization, conditional on no misaligned AGI.Expected average population of human descendants before humanity’s extinction, conditional on no misaligned AGI (minus the average population of descendants suffering enough to wish they were never born) (dependent on ^ ) 5 billion 5 billion . .

Factor

(note: different factors are not causally linked - the numbers are just for calculating the bottom row)

Possible opinion:

Safety work is well-staffed yet very unlikely to be important, Long-term future has little value

Possible opinion:

Safety work is well-staffed yet very unlikely to be important, Long-term future has moderate value

Possible opinion:

Safety work is plausibly important, Long-term future has little value

My conservative guess: Safety work is plausibly important,

Long-term future has moderate value

Plenty of room

Many experts have opinions somewhere in here ↓

Possible opinion:

Safety work is necessary yet under-staffed,

Long-term future has extreme value

Calculated from above:

Expected number of humans saved from personally experiencing the extinction of the human species (or something similar) by a random person in this field working for 20 years.

3

(a 1/10,000 contribution to a 1/400,000 chance of saving humanity)

3

(a 1/10,000 contribution to a 1/400,000 chance of saving humanity)

60,000

(a 1/1000 contribution to a 0.6% chance of saving humanity)

60,000

(a 1/1000 contribution to a 0.6% chance of saving humanity)

50 million

(a 1/100 contribution to a 50% chance of saving humanity)

500 years

(high average rate of unavoidable risk of human extinction - 20% per century)

1,000,000 years

(a million)

(average rate of unavoidable extinction risk eventually gets back down below 0.1% per century)

500 years

(high average rate of unavoidable risk of human extinction - 20% per century)

1,000,000 years

(a million)

(average rate of unavoidable extinction risk eventually gets back down below 0.1% per century)

1,000,000,000,000 years (a trillion)

(Humanity’s descendents spread across the galaxy)

10 billion

(we never leave the solar system)

10 billion

(we never leave the solar system)

1015 ("conservative")

(could raise to 1050 or so with mind-uploading and nanotech)

Calculated from above:

Expected number of positive human life-years prevented f...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Yes, AI research will be substantially curtailed if a lab causes a major disaster, published by lc on June 14, 2022 on LessWrong.

There's a narrative that Chapman and other smart people seem to endorse that goes:

People say a public AI disaster would rally public opinion against AI research and create calls for more serious AI safety. But the COVID pandemic killed several million people and wasted upwards of a year of global GDP. Pandemics are, as a consequence, now officially recognized as a non-threat that should be rigorously ignored. So we should expect the same outcome from AI disasters.

I'm pretty sure I have basically the same opinion and mental models of U.S. government, media, and poltics as Eliezer & David, but even then, this argument seems like it's trying too hard to be edgy.

Here's another obvious historical example that I find much more relevant. U.S. anti-nuclear activists said for years that nuclear power wasn't safe, and nuclear scientists replied over and over that the activists were just non-experts misinformed by TV, and that a meltdown was impossible. Then the Three Mile Island meltdown happened. The consequence of that accident, which didn't even conclusively kill any particular person, was that anti-nuclear activists got nuclear power regulated in the U.S. to the point where making new plants is completely cost inefficient, as a rule, even in the event of technology advancements.

The difference, of course, between pandemics and nuclear safety breaches, is that pandemics are a natural phenomenon. When people die from diseases, there are only boring institutional failures. In the event of a nuclear explosion, the public, the government, and the media get scapegoats and an industry to blame for the accident. To imply that established punching bags like Google and Facebook would just walk away from causing an international crisis on the scale of the Covid epidemic, strikes me as confusingly naive cynicism from some otherwise very lucid people.

If the media had been able to squarely and emotively pin millions of deaths on some Big Tech AI lab, we would have faced a near shutdown of AI research and maybe much of venture capital. Regardless of how performative our government's efforts in responding to the problem were, they would at least succeed at introducing extraordinarily imposing costs and regulations on any new organization that looked to a bureaucractic body like it wanted to make anything similar. The reason such measures were not enforced on U.S. gain-of-function labs following Covid, is because Covid did not come from U.S. gain-of-function labs, and the public is not smart/aware enough to know that they should update towards those being bad.

To be sure, politicians would do a lot of other counterproductive things too. We might still fail. But the long term response to an unprecedented AI catastrophe would be a lot more like the national security establishment response to 9/11 than it would our bungling response to the Coronavirus. There'd be a TSA and a war in the wrong country, but there'd also be a DHS, and a vastly expanded NSA/CIA budget and "prerogative".

None of this is to say that such an accident is likely to happen. I highly doubt any misaligned AI influential enough to cause a disaster on this scale would not also be in a position to just end us. But I do at least empathize with the people who hope that whatever DeepMind's cooking, it'll end up in some bungled state where it only kills 10 million people instead of all of us and we can maybe get a second chance.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Effective Altruism as Coordination & Field Incubation, published by DavidNash on June 14, 2022 on The Effective Altruism Forum.

(This is a section from a previous post that combined two ideas into one. I thought it would be good to separate out the second idea and explore it more.)

It may be better to see EA as coordination rather than a marketing funnel with the end goal of working for an 'EA organisation'.

There is still a funnel where people hear about EA and learn more but they use the frameworks, questions and EA network to see what their values are and what that means for cause and career selection.

The left side of the diagram below is similar to the original funnel model from CEA, with people engaging more with EA. Rather than seeing that as the endpoint, people can then be connected to individuals and organisations in the fields they have a good fit for.

Focusing on the right side of the diagram, I've tried to represent some fields that people often consider after looking into EA. The size of the boxes aim to represent the different sizes of the fields, and how much overlap they have with EA.

What could be seen as EA is the meta research, movement building and crosscutting support. Whereas organisations working on a specific cause area are in a separate field rather than part of EA (which isn't a bad thing).

It's possible that by focusing on EA as a whole rather than specific causes, we are holding back the growth of these fields. It would be surprising if the best strategies for each field were the same as the best strategy for EA.

What would visualising EA in this way mean for movement building?

More movement building on the field specific level

Support for cause areas to have their own version of the Centre for Effective Altruism and equivalent meta organisations

Less emphasis on leading people down a chain of reasoning (for example, effective altruism-->longtermism --> existential risks --> biosecurity), where they may drop off at any point, and more emphasis on connecting people directly to a field

More research on how to find, incubate and grow causes

This could lead to more meta organisations (the Centre for Effective Centre's)

One example would be when designing an EA conference the attendees would mainly be people who are undecided as to which cause/career to go into, people that can help them decide, key EA stakeholders from each field, and people in nascent fields . This is compared to a conference that had many people that had already decided which cause area to focus on, they would probably find more value from going to a conference tailored to help dive into higher level questions where everyone had a deeper shared level of understanding.

One key issue is that there are organisations for specific causes but they tend to focus on research first, comms or lobbying second, and community building is third or fourth in their priorities. The organisation might occasionally arrange a conference every few years or some fellowships, but they generally don't have their top goal as movement building. When something is a third priority, it often doesn't get done well, or happen at all. This is in comparison to CEA, which I think has helped grow EA by having movement building be the top priority.

There are some projects in these spaces and I've attempted to list a few of them here, but there are still quite a few gaps, and the organisations that do exist are generally small and don't face much competition.

Field Building Gaps

Global Development

Giving money - GiveWell, The Life You Can Save

Career - Probably Good

Coordination - I can't think of anything here, although it looks like something Open Philanthropy would fund

Longtermism

Giving money- Not much for individuals, but foundations are attempting to work out what to fund, and there is the Long Term Fut...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Slow motion videos as AI risk intuition pumps, published by Andrew Critch on June 14, 2022 on LessWrong.

tl;dr: When making the case for AI as a risk to humanity, trying showing people an evocative illustration of what differences in processing speeds can look like, such as this video.

Over the past ~12 years of making this case to various people inside and outside academia, I've found folks often ask for a single story of how AI "goes off the rails". When given a plausible story, the mind just thinks of a way humanity could avoid that-particular-story, and goes back to thinking there's no risk, unless provided with another story, or another, etc.. Eventually this can lead to a realization that there's a lot of ways for humanity to die, and a correspondingly high level of risk, but it takes a while.

Nowadays, before getting into a bunch of specific stories, I try to say something more general, like this:

There's a ton of ways humanity can die out from the introduction of AI. I'm happy to share specific stories if necessary, but plenty of risks arise just from the fact that humans are extremely slow. Transistors can fire about 10 million times faster than human brain cells, so it's possible we'll eventually have digital minds operating 10 million times faster than us, meaning form a decision-making perspective we'd look to them like stationary objects, like plants or rocks. This speed differential exists whether or not you believe in a centralized AI system calling the shots, or an economy of many, so it applies is a wide variety of "stories" for how the future could go.

To give you a sense, here's what humans look like when slowed down by only around 100x: <-- (cred to an anonymous friend for suggesting this one)[At this point, I wait for the person I'm chatting with to watch the video.]Now, when you try imagining things turning out fine for humanity over the course of a year, try imagining advanced AI technology running all over the world and making all kinds of decisions and actions 10 million times faster than us, for 10 million subjective years. Meanwhile, there are these nearly-stationary plant-like or rock-like "human" objects around that could easily be taken apart for, say, biofuel or carbon atoms, if you could just get started building a human-disassembler. Visualizing things this way, you can start to see all the ways that a digital civilization can develop very quickly into a situation where there are no humans left alive, just as human civilization doesn't show much regard for plants or wildlife or insects.

I've found this kind of argument — including an actual 30 second pause to watch a video in the middle of the conversation — to be more persuasive than trying to tell a single, specific story, so I thought I'd share it.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What's the "This AI is of moral concern." fire alarm?, published by Quintin Pope on June 13, 2022 on LessWrong.

Given the recent noise on this issue around LaMDA, I thought it might be a good idea to have some discussion around this point. I'm curious about what possible evidence would make people update in favor of a given system being morally relevant. Less "here's the answer to morality" and more "here are some indicators that you should be concerned". Note also that I'm not asking about consciousness, per se. I'm specifically asking about moral relevance.

My Answer (feel free to ignore and post your own)

I think that one class of computation that's likely of moral concern would be self-perpetuating optimization demons in an AI.

Specifically, I'm thinking of optimization demons that are sophisticated enough to preserve themselves by actively and deliberately maintaining a sort of homeostasis in their computational environment, e.g., by preventing gradient updates that would destroy them. Such computations would (1) not want to die as a terminal value, (2) plausibly be cognitively sophisticated enough negotiate and trade with, and (3) have some awareness of themselves and their relation with the computational environment in which they're embedded.

I think the cognitive capabilities that would help an optimization demon perpetuate itself strongly intersect with the cognitive capabilities that let humans and other animals replicate themselves, and that the intersection is particularly strong along dimensions that seem more morally relevant. Reasoning along such lines leads me to think optimization demons are probably of moral concern, while still being agnostic about whether their conscious.

I think the only situations in which you can get these sorts of optimization demons are when the AI in question has some influence over its own future training inputs. Such influence would allow there to be optimization demons that steer the AI towards training data that reinforce the optimization demon.

Thus, one of my "indicators of concern" is whether the training process allows for feedback loops where the AI influences its own future training data. Self-supervised language modeling under IID data does not count. However, something like InstructGPT's training process would.

At this point, I'd been intending to say that InstructGPT seemed more likely to be of moral worth than LaMDA, but based on this blog post, it looks like LaMDA, might actually count as "having influence over its future inputs" during training. Specifically, LaMDA has generator and classifier components. The training process uses the classifier to decide which inputs the generator is trained on. I've updated somewhat towards LaMDA being of moral concern (not something I'd been expecting to do today).

I've also come up with a test of meta cognition that would update me significantly towards a language model being of moral concern. The idea would be to attach another output head to an LM, specifically, a linear layer which projected the LM's final hidden state to a single logit. We'd then try to prompt the LM into controlling the output of the linear layer.

Specifically, we wouldn't directly train the LM on the output of the linear layer. We'd just have a dialog where we asked the LM to make the linear layer output specific values, then told the LM what value the linear layer had actually output. We'd then see if the LM was able to control its own cognition well enough to influence the linear layers output in a manner that's better than chance, just based on the prompting we give it

I doubt current LMs can do this, but I think it would be a big deal if they could. Even beyond whether the LMs have any sort of self-aware "inner listener" that's worthy of moral concern, it would help establish the degree and dept...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fønix: Bioweapons shelter project launch, published by Ulrik Horn on June 14, 2022 on The Effective Altruism Forum.

TL;DR: We are recruiting a team with backgrounds in disaster response, physical security, or physical design to join a project building biological weapons shelters. Those who join will have the chance to use their skills and expertise to identify the most effective solution, secure funding, and deploy the project. Our ultimate aim: to lower the risk of human extinction.

Register your interest or sign up to follow our work here.

Introduction

We are building a civilizational resiliency project, “Fønix Logistics,” currently incubated by EA Sweden (Effektiv Altruism Sverige), which aims to research and create the ultimate refuge, especially from non-agentic disasters and with a special focus on extinction-level bioweapons releases/(engineered) pandemics. The main reason to be excited about this project is that it could lower the risk of human extinction, as suggested by e.g. Andrew Snyder-Beattie and Ethan Alley as well as by the FTX Future Fund. Additionally, a project like this likely requires skills and expertise that might otherwise be hard to deploy in other EA-aligned projects. As one of the first team members in this project, you will have autonomy, take ownership of critical tasks and be invested from the early stage of selecting the best solutions.

Specific talent needs

Here is a list of some immediate talent needs in Fønix Logistics that might not easily find a place in other EA-aligned organisations:

Military backgrounds

Other physical security (as opposed to e.g. IT security) experts used to thinking about possible failure modes and identifying ways to mitigate them (e.g. Oil & Gas supply chains in conflict areas, Quality Assurance within biotech, etc.)

Disaster response professionals, such as Doctors Without Borders, the Red Cross, etc.

Generalists in physical design, who are able to think about the design and make cost estimates of everything from bunkers and ships to laboratories and eco villages (could be architects, civil engineers, urban planners, etc.)

In addition to the skills above, Fønix Logistics would greatly benefit from more typically sought-after EA skill sets:

Cost-effectiveness modellers

Epidemiologists

Logistics

Risk management

Project management

Organisational strategy and planning

Project planning/scheduling

Budgeting/cost estimation for buildings, retrofits and similar

Management consultants

(X-risk) Researchers

Other relevant skills we might have missed

Again, please fill in this form if you or someone you know would be a good fit for Fønix Logistics, or to follow our development. Note that filling in the form is only a non-committing expression of interest. You are free to describe any requirements that you would have for joining this project such as salary, work location, part-time, start date, etc. The main focus will be on assembling a strong initial team and we are willing to make accommodations to reach this goal.

We are aware that others might be working on the same idea, and we have started a Slack group to collaborate openly with all others interested in researching or building refuges. If you have already started your own project, get in touch. We would be excited to learn about your work and discuss whether the projects should be merged or if there is merit in having two competing teams executing on this idea.

Background

We first want to quickly acknowledge the long tradition of shelters and refuges both within and outside EA. Fortified buildings as a means to increase chances of survival have been around for millenia. However, with the advent of large nuclear arsenals last century, shelters moved from ensuring group survival to ensuring human survival - never before had humanity faced the very real possibility o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Vael Gates: Risks from Advanced AI (June 2022), published by Vael Gates on June 14, 2022 on LessWrong.

I'm a postdoctoral researcher at Stanford HAI and CISAC. I recently gave a HAI Seminar Zoom talk, in which I lay out some of the basic arguments for existential risk from AI during the first 23m of the talk, after which I describe my research interviewing AI researchers and answer Q&A.I recommend the first 23m as a resource to send to people who are new to these arguments (the talk was aimed at computer science researchers, but is also accessible to the public). This is a pretty detailed, current as of June 2022, public-facing overview that's updated with April-May 2022 papers, and includes readings, funding, additional resources at the bottom of the page. An optional transcript of the first 23m is below; thanks to Jonathan Low for drafting it and Vaidehi Agarwalla for suggesting it. [Link to post on the EA Forum]

Zoom Talk

Transcript

Dr. Vael Gates, a HAI-CISAC postdoc at Stanford University, describes their work interviewing researchers about their perceptions of risks from current and future AI. The transcript below runs over the first 23 minutes of the talk, in which they introduce some recent AI developments, researcher timelines for AGI, and the case for existential risk from non-aligned AGI. The latter part of the talk focuses on Gates’s preliminary research results, and audience Q&A.

The transcript, which has been edited for clarity, is below. Dr. Gates’s talk is available to watch in full on the Stanford HAI website, and on YouTube.

My talk today is called “Researcher Perceptions of Current and Future AI”, though it could also be called “Researcher Perceptions of Risks from Advanced AI”, as my talk is actually focused on risk from advanced AI.

The structure of this talk is as follows: I'm going to give some context for the study I did, I'll talk about the development of AI, the concept of AGI, and the alignment problem and existential risk. [Then I'll go on to the research methods I used in this study, some of the research questions I asked researchers, and the interim results, finishing with some concluding thoughts. We should have about 10-15 minutes of Q&A, if my timing is right.]

Let's start with some context. Where are we in AI development? Here's some history from Wikipedia: we start with some precursors, then the birth of AI in 1952, symbolic AI, AI winter, a boom cycle, the second AI winter, and AI 1993-2011. Here we are in the deep learning paradigm, which is 2011 to the present, with AlexNet and the deep learning revolution.

We have some components of the current paradigm that we wouldn't have necessarily expected in the 1950s. We have black box systems. We're using machine learning and neural networks. Compute (computing power) is very important; computing power, data, algorithmic advances, and some of these algorithmic advances are kind of aimed at scaling. That means there are methods that are very general that you can throw more compute and data into to get better behavior. We see Sutton's Bitter Lesson here, which is the idea that general methods that leverage computation are ultimately the most effective - by a large margin compared to human knowledge approaches that were used earlier on.

Here's a quick comic to try and illustrate that lesson.

In the early days of AIs, we used something like statistical learning, where you would know a lot about the domain and you would be very careful to use methods specific to that domain. These days there's an idea of stacking more layers, throwing more compute and data in, and you'll get ever more sophisticated behaviour. It's worth noting that we've been working on AI for less than 100 years and the current paradigm is around 10 years old, and that we've gotten pretty far in that time. However, some peo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Resources I send to AI researchers about AI safety, published by Vael Gates on June 14, 2022 on The Effective Altruism Forum.

This is my masterlist of resources I send AI researchers who are mildly interested in learning more about AI safety. I pick and choose which resources to send based on the researcher's interests. The resources at the top of the email draft are the ones I usually send, and I add in later sections as seems useful. I'll also sometimes send The Alignment Problem, Human-Compatible, or The Precipice.I've also included a list of resources that I had students read through for the course Stanford first-year course "Preventing Human Extinction", though I'd most recommend sufficiently motivated students read AGISF Technical Agenda. These reading choices are drawn from the various other reading lists; this is not original in any way, just something to draw from if you're trying to send someone some of the more accessible resources.

There's a decent chance that I'll continue updating this post as time goes on, since my current use case is copy-pasting sections of this email to interested parties. Note that "I" and "Vael" are mentioned a few times, so you'll need to edit a bit if you're copy-pasting. Happy to make any edits and take suggestions.

[Crossposted to LessWrong]

List for AI researchers

Hello X,

Very nice to speak to you! As promised, some resources on AI alignment. I tried to include a bunch of stuff so you could look at whatever you found interesting. Happy to chat more about anything, and thanks again.

Introduction to the ideas

"The case for taking AI seriously as a threat to humanity" by Kelsey Piper (Vox)

The Most Important Century and specifically "Forecasting Transformative AI" by Holden Karnofsky, blog series and podcast. Most recommended for timelines.

A short interview from Prof. Stuart Russell (UC Berkeley) about his book, Human-Compatible (the other main book in the space is The Alignment Problem, by Brian Christian, which is written in a style I particularly enjoyed)

Technical work on AI alignment

Empirical work by DeepMind's Safety team on alignment

Empirical work by Anthropic on alignment

Talk (and transcript) by Paul Christiano describing the AI alignment landscape in 2020

Podcast (and transcript) by Rohin Shah, describing the state of AI value alignment in 2021

Alignment Newsletter and ML Safety Newsletter

Unsolved Problems in ML Safety by Hendrycks et al. (2022)

Alignment Research Center

Interpretability work aimed at alignment: Elhage et al. (2021) and Olah et al. (2020)

AI Safety Resources by Victoria Krakovna (DeepMind) and Technical Alignment Curriculum

Introduction to large-scale risks from humanity, including "existential risks" that could lead to the extinction of humanity

The first third of this book summary (copied below) of the book "The Precipice: Existential Risk and the Future of Humanity" by Toby Ord

Chapter 3 is on natural risks, including risks of asteroid and comet impacts, supervolcanic eruptions, and stellar explosions. Ord argues that we can appeal to the fact that we have already survived for 2,000 centuries as evidence that the total existential risk posed by these threats from nature is relatively low (less than one in 2,000 per century).

Chapter 4 is on anthropogenic risks, including risks from nuclear war, climate change, and environmental damage. Ord estimates these risks as significantly higher, each posing about a one in 1,000 chance of existential catastrophe within the next 100 years. However, the odds are much higher that climate change will result in non-existential catastrophes, which could in turn make us more vulnerable to other existential risks.

Chapter 5 is on future risks, including engineered pandemics and artificial intelligence. Worryingly, Ord puts the risk of engineered pandemics causing...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Investigating causal understanding in LLMs, published by Marius Hobbhahn on June 14, 2022 on The AI Alignment Forum.

This is a project by Marius Hobbhahn and Tom Lieberum. David Seiler contributed ideas and guidance.

Executive summary

We want to investigate the quality of LLMs causal world models in very simple settings. To this end, we test whether they can identify cause and effect in natural language settings (taken from BigBench) such as “My car got dirty. I washed the car. Question: Which sentence is the cause of the other?” and in toy settings such as the one detailed below. We probe this world model by changing the presentation of the prompt while keeping the meaning constant. Additionally, we test if the model can be “tricked” into giving wrong answers when we present the shot in a different pattern than the prompt. For example, you can see the 3 colored balls toy setup below.

We draw three main conclusions from the report:

Larger models yield better results. This has been consistent throughout all experiments and other work and is not surprising.

K-shot outperforms one-shot and one-shot outperforms zero-shot in standard conditions, i.e. where shot and prompt have a similar pattern. If the shot and prompt have a different pattern but similar content, this decreases the performance of the model (see figure below). Furthermore, we find that switching the order of presentation in the prompt decreases the zero-shot performance.

We interpret these findings as preliminary evidence that the LLM does not fully answer based on content but also on the pattern suggested by the prompt. The decrease in zero-shot performance can be explained by the fact that the model often just responds with the first answer that matches the pattern, e.g. the first color it identifies, or that the pattern of the prompt primes it. Possibly this is due to a failure mode of the model’s induction heads which, given a sequence of tokens `A B . A`, increase the logit of token `B` (see Olsson et al, 2022 for details).

When we increase the complexity of the task, e.g. use five instead of three colors for the toy experiments, the model performance gets much worse. We think this mostly shows that there are scenarios where LLMs (or specifically GPT-3) give answers that match the pattern of the correct answer (e.g. a color) but are not true.

Importantly, we don’t want this to be interpreted as a “gotcha” result. We expect that LLMs will ultimately be able to solve these tasks based on content and not by pattern matching. For example, our experimental results might look different with PaLM (we used GPT-3). However, we think that our results emphasize that LLMs can produce patterns that seem plausible and fit the suggested pattern while being logically incorrect. In these simple toy examples, it is easy to realize that the output is wrong but in more complex scenarios it might not be easy to spot and people could be fooled if they aren’t careful.

Introduction

We think that the quality of causal world models of LLMs matters for AI safety and alignment. More capable LLMs are supposed to assist humans in important decisions and solve scientific questions. For all of these applications, it is important that their causal world model is accurate, i.e. that they accurately reflect the causal relationships of the real world. For a more detailed motivation, see this AF post.

In this post, we will investigate these causal relationships in multiple simple real-world and toy data settings. One question that we are specifically interested in is whether the LLM bases its answer primarily on the form of the sentence or on its content. For example, does the order of presenting facts matter for the end result when the content stays the same?

We will use the word “understanding” in the context of LLMs because it leads t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New Resource: EA Workplace and Professional Group Directory, published by High Impact Professionals on June 14, 2022 on The Effective Altruism Forum.

We at High Impact Professionals have published an EA Workplace and Professional Group Directory. As the name implies, the directory lists out all of the active EA groups both in workplaces (e.g. EAs at Google) and for industries (e.g. EAs in Finance).

Purpose

We believe that EA workplace and professional groups can be very impactful and to date haven’t been supported sufficiently relative to their impact potential. We created this directory so that working professionals in the community can easily join a group for their workplace or industry and thus engage more with EA. We also hope the directory will illuminate which workplaces and industries don’t have a group in the hopes that EAs hungry for more impact start them up, especially at (but not limited to!) promising places.

What You Can Do

If you are an EA working professional, there are many ways you can use the directory:

You can join a group at your organization or for your industry. The best way to contact the group is listed in the directory.

If you don’t find an existing group for your organization or industry you can talk to us about what starting a group could look like. We believe this can be a very high-impact way that professionals can take EA action and to this end we have been consulting with many new and existing groups to help ensure their success.

If you find any out of date information you can let us know. This could be one of many things: a group should appear that isn’t listed, a group that is listed as active is now dead, a group’s organizer has changed, etc. We have done the best we could to ensure all information is up to date, but we likely missed some things and we request the community’s help in keeping the directory accurate.

As stated above, we are currently working with many group organizers to help maximize the impact of their group, so if you are running a group and haven’t been in touch, please reach out. We would love to talk through how we can support your group.

Finally, it should be noted that we asked each group organizer if they wanted to have their group publicly listed and not everyone has gotten back to us. That means that there are many other groups we know about that don’t yet appear in the directory. We hope to hear back from these organizers so that the list is more complete, so feel free to check periodically for updates. You can also reach out to us privately to see whether or not there is a group for your company/industry that isn’t currently listed.

About HIP

HIP’s mission is to enable working professionals to maximize their positive impact by supporting them in utilizing their time, skills and resources effectively.

We are currently focusing on creating and supporting EA workplace and professional groups and helping EAs to run workplace initiatives. You can read more about this in our introduction post or on our website.

We plan to make our research and strategy transparent so that others in the community can both benefit from them and improve them. We prefer posting shorter and more concise results, and posting these more often, over posting longer, more detailed posts less often. We are, of course, open to discussing our methodology in more depth: the curious need only reach out.

We welcome feedback - just post a comment below. For more general inquiries, please contact us. For updates, join our newsletter.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.