Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Cause Exploration Prizes] The importance of Intercausal Impacts, published by Sebastian Joy on August 24, 2022 on The Effective Altruism Forum.

This essay was submitted to Open Philanthropy's Cause Exploration Prizes contest.

If you're seeing this in summer 2022, we'll be posting many submissions in a short period. If you want to stop seeing them so often, apply a filter for the appropriate tag!

Defining key cause areas - such as global poverty, animal suffering, and existential risks - is an important element of effective altruism, in part because it has helped effective altruists to identify the most effective organizations and interventions in each area. However, addressing one cause area at a time without considering the ways in which its interventions affect other cause areas - for better or worse - can have a negative impact on the overall effectiveness of EA, a consequence that is often not considered. Invoking an Intercausal Impacts analysis, this article makes a case for establishing food systems transformation/meat reduction as a cause area in its own right.

Effective Altruism’s ‘divide-and-improve’ approach

Effective Altruism aims to improve the world as effectively and efficiently as possible. To achieve this goal, global problems are usually split up into different cause areas. Each cause area is analysed on the criteria triad of importance, tractability, and neglectedness. Such an analysis is what led to causes such as ‘global poverty’ (improving the economic or health situation of the least affluent), ‘animal suffering’ (reducing non-human suffering) and ‘existential risks’ (increasing the chances of survival for future generations) becoming prominent focus areas of EA. And the list of cause areas is growing.

This ‘divide-and-conquer’ or rather ‘divide-and-improve’ approach has proven very useful so far in EA thinking and theorising, and has led to the establishment of several meta-charities for each of the three cause areas highlighted above. The goal of these meta-charities is to research, analyse and promote those organisations and interventions that are best suited to solve the specific problems within their cause area. While the meta-charity ‘GiveWell’ looks into the most effective interventions to help the global poor, ‘Animal Charity Evaluators’ tries to identify the most useful interventions to reduce animal suffering, and the ‘Centre for the Study of Existential Risk’ aims at reducing the risk of humanity becoming extinct.

Here is a simple overview of this process in three steps:

Drawbacks of the divide-and-improve approach

While this approach has many benefits and has helped improve the state of the world tremendously, it comes with crucial and so far often overlooked drawbacks. There are two ways this approach might actually work against EA’s overall goal of creating the most good in the world - both of which involve what I call ‘Intercausal Impacts’, i.e. the overall impact of a given organisation or intervention not only on its respective primary cause area but on all cause areas aggregated.

Overrating: Interventions (and organisations) that are regarded as very efficient in one cause area might have negative spill-over effects and actually do damage in other cause areas. Thus, their overall impact might be significantly less positive (and in some cases even net negative) than is immediately apparent. Ignoring negative spill-over effects on other cause areas implies an overrating of the intervention or organisation in question.

Underrating: Interventions (and organisations) that have a positive impact on various causes but are not amongst the most effective in any particular cause area will receive little or no attention and support although their overall cost-effectiveness across cause areas might be superior. Ignoring positive spill-ov...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Makes A Good Measurement Device?, published by johnswentworth on August 24, 2022 on The AI Alignment Forum.

In some sense, any new measurement device automates a part of research. A thermometer automates the task of sticking one's finger in something to check how hot it is, a scale automates the task of holding something to check how heavy it is, etc. The automated version is not only more convenient, but more precise and reproducible (as is usually the case when automating things). For alignment, one analogue might be interpretability tools which automate the work done when some human looks at a part of a neural net and sees what it's doing.

Let's take that last example and dig into it a bit more: interpretability tools which automate the work done when some human looks at a part of a net and sees what it's doing. We want to leverage the analogy to other measurement tools, like a thermometer or a scale, to better understand automation of interpretability.

Here's one type of proposal I hear a lot: to automate interpretability, have some human researchers look at parts of a net, poke at them, and write up an explanation of how they're interpreting it. Collect data from many such instances, and train a neural net to take net-parts and produce explanations.

We want to leverage the analogy to thermometers or scales, so what would be the analogous strategy for making a thermometer or scale? Well, have a bunch of humans stick their fingers in a bunch of stuff and report how hot the stuff is, then train a neural net to replicate the humans' hotness-reports. Or, have a bunch of humans hold things and report how heavy they are, then train a net to replicate the humans' heaviness-reports.

Hopefully it is obvious that the "train a net to replicate human reports" results would not be nearly as useful, for purposes of scientific progress, as actual thermometers or scales. But what's missing? And how can we carry that insight back to the interpretability problem?

The thermometer has two great powers: a simple legible data type, and reproducilbility. First, simple legible data type: the thermometer's output is a single number (the temperature), and we can compare that number with other thermometer-outputs. That's a kind-of-thing for which we have very precise mathematical understanding: we know exactly what kinds-of-things we can do with these numbers, we have a nice general representation, we're confident that different people mean the same thing by numbers, etc. This is in contrast to natural language, which is typically ambiguous, doesn't necessarily make it obvious what we can do, leads to frequent miscommunication, etc.

Second, reproducibility: if the thermometer says X is hotter than Y, then when I put X and Y in contact, X gets cooler and Y gets hotter (all else equal). I can use the thermometer to rank hotness of a bunch of things, sort them by thermometer reading, and consistently (approximately-deterministically) find that the things on the hotter end feel hotter than the things on the colder end. This is what makes the single-number output (temperature) actually useful: it approximately-deterministically predicts some stuff, across a broad range of contexts, based on just those simple numbers.

Exercise for the reader: walk through the same analogy for a scale.

What would be the analogy of a thermometer for an interpretability tool? Well, something which we can point at part of a net, and get back a simple legible output, which approximately-deterministically predicts some stuff across a broad range of contexts.

When you look at it like that, it's clear that building a measurement tool like a scale or thermometer is itself a task which requires scientific insight. It requires finding some approximately-deterministically-reproducible pattern, which can be predicted by...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Google AI integrates PaLM with robotics: PaLM-SayCan update [Linkpost], published by Evan R. Murphy on August 24, 2022 on The AI Alignment Forum.

Back in April, Google AI announced SayCan, a project which integrated a language model (FLAN) with robotics in order to produce a robot which could follow instructions. For example, cleaning up a mess in a kitchen. (LessWrong post from April which links to a tweet about SayCan.)

This week Google AI has released some new updates, dubbed PaLM-SayCan. This involved upgrading the integrated language model to Google AI's top performing large language model (LLM) of 540-billion parameters, PaLM.

Website for PaLM-SayCan

Blog post announcing PaLM-SayCan

The following updates from this week are quoted from the original SayCan project website (linked in first sentence of this post, bold mine):

[8/16/2022] We integrated SayCan with Pathways Language Model (PaLM), and updated the results. We also added new capabilities including drawer manipulation, chain of thought prompting and multilingual instructions. You can see all the new results in the updated paper.

[8/16/2022] Our updated results show that SayCan combined with the improved language model (PaLM), which we refer to as PaLM-SayCan, improves the robotics performance of the entire system compared to a previous LLM (FLAN). PaLM-SayCan chooses the correct sequence of skills 84% of the time and executes them successfully 74% of the time, reducing errors by a half compared to FLAN. This is particularly exciting because it represents the first time we can see how an improvement in language models translates to a similar improvement in robotics.

[8/16/2022] We open-sourced a version of SayCan on a simulated tabletop environment.

Thanks to Jon Menaster for making me aware of this update from Google AI.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Vingean Agency, published by Abram Demski on August 24, 2022 on The AI Alignment Forum.

I've been involved with several discussions about different notions of agency (and their importance/relationships) lately, especially with the PIBBSS group including myself, Daniel, Josiah, and Ramana; see here.

There's one notion of agency (not necessarily "The" notion of agency, but a coherent and significant notion) which vanishes if you examine it too closely.

Imagine that Alice is "smarter than Bob in every way" -- that is, Bob believes that Alice knows everything Bob knows, and possibly more. Bob doesn't necessarily agree with Alice's goals, but Bob expects Alice to pursue them effectively. In particular, Bob expects Alice's actions to be at least as effective as the best plan Bob can think of.

Because Bob can't predict what Alice will do, the only way Bob can further constrain his expectations is to figure out what's good/bad for Alice's objectives. In some sense this seems like a best-case for Bob modeling Alice as an agent: Bob understands Alice purely by understanding her as a goal-seeking force.

I'll call this Vingean agency, since Vinge talked about the difficulty of predicting agents who are smarter than you. and since this usage is consistent with other uses of the term "Vingean" in relation to decision theory.

However Vingean agency might seem hard to reconcile with other notions of agency. We typically think of "modeling X as an agent" as involving attribution of beliefs to X, not just goals. Agents have probabilities and utilities.

Bob has minimal use for attributing beliefs to Alice, because Bob doesn't think Alice is mistaken about anything -- the best he can do is to use his own beliefs as a proxy, and try to figure out what Alice will do based on that.

When I say Vingean agency "disappears when we look at it too closely", I mean that if Bob becomes smarter than Alice (understands more about the world, or has a greater ability to calculate the consequences of his beliefs), Alice's Vingean agency will vanish.

We can imagine a spectrum. At one extreme is an Alice who knows everything Bob knows and more, like we've been considering so far. At the other extreme is an Alice whose behavior is so simple that Bob can predict it completely. In between these two extremes are Alices who know some things that Bob doesn't know, while also lacking some information which Bob has.

(Arguably, Eliezer's notion of optimization power is one formalization of Vingean agency, while Alex Flint's attraction-basin notion of optimization defines a notion of agency at the opposite extreme of the spectrum, where we know everything about the whole system and can predict its trajectories through time.)

I think this spectrum may be important to keep in mind when modeling different notions of agency. Sometimes we analyze agents from a logically omniscient perspective. In representation theorems (such as Savage or Jeffrey-Bolker, or their lesser sibling, VNM) we tend to take on a perspective where we can predict all the decisions of an agent (including hypothetical decisions which the agent will never face in reality). From this omniscient perspective, we then seek to represent the agent's behavior by ascribing it beliefs and real-valued preferences (ie, probabilities and expected utilities).

However, this omniscient perspective eliminates Vingean agency from the picture. Thus, we might lose contact with one of the important pieces of the "agent" phenomenon, which can only be understood from a more bounded perspective.

On the other hand, if Bob knows Alice wants cheese, then as soon as Alice starts moving in a given direction, Bob might usefully conclude "Alice probably thinks cheese is in that direction". So modeling Alice as having beliefs is certainly not useless for Bob. Still, because Bob ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Beliefs and Disagreements about Automating Alignment Research, published by Ian McKenzie on August 24, 2022 on The AI Alignment Forum.

Epistemic status: Mostly organizing and summarizing the views of others.

Thanks to those whose views I summarized in this post, and to Tamera Lanham, Nicholas Kees Dupuis, Daniel Kokotajlo, Peter Barnett, Eli Lifland, and Logan Smith for reviewing a draft.

Introduction

In my current view of the alignment problem, there are two paths that we could try to take:

Come up with an alignment strategy that allows us to both build aligned AGI and to keep that AGI (or its successors) aligned as they improve towards superintelligence

Come up with an alignment strategy that allows us to build AI systems that are powerful (but not so powerful as to be themselves dangerous) and use that to execute some kind of ‘pivotal act’ that means that misaligned ASI is not built

For the purposes of this post, I am going to assume that we are unable to do (1) – maybe the problem is too difficult, or we don’t have time – and focus on (2).

Within the category of ‘pivotal act’, I see two main types:

Preventative pivotal acts: acts that makes it impossible for anyone to build AGI for a long period of time

Constructive pivotal acts: acts that makes it possible to build aligned ASI

People disagree about whether preventative pivotal acts are possible or even if they were possible, if they’d be a good idea. Again, for the purposes of this post, I am going to assume we can’t or don’t want to execute a preventative pivotal act, and focus on constructive pivotal acts. In particular: can we use AI to automate alignment research safely?

What does ‘automating alignment research’ even mean?

I see three overlapping categories that one could mean when referring to ‘automating alignment research’, ordered in terms of decreasing human involvement:

Level 1: AIs help humans work faster

Examples include brainstorming, intelligent autocomplete, and automated summarization/explanation.

Level 2: AIs produce original contributions

This could be key insights into the nature of intelligence, additional problems that were overlooked, or entire alignment proposals.

Level 3: AIs build aligned successors

Here, we have an aligned AGI that we entrust with building a successor. At this point, the current aligned AGI has to do all the alignment research required to ensure that its successor is aligned.

Mostly I have been thinking about Levels 1 and 2, although some people I spoke to (e.g. Richard Ngo) were more focused on Level 3.

Current state of automating alignment

At the moment, we are firmly at Level 1. Models can produce similar-sounding ideas when prompted with existing ideas and are pretty good at completing code but are not great at summarizing or explaining complex ideas. Tools like Loom and Codex can provide speed-ups but seem unlikely to be decisive.

Whether we get to Level 2 soon or whether Level 2 is already beyond the point where AI systems are dangerous are key questions that researchers disagree on.

Key disagreements

Generative models vs agents

Much of the danger from powerful AI systems comes from them pursuing coherent goals that persist across inputs. If we can build generative models that do not pursue goals in this way, then perhaps these will provide a way to extract intelligent behavior from advanced systems safely.

Timing of emergence of deception vs intelligence

Related to the problem of agents, there is also disagreement about whether we get systems that are intelligent enough to be useful for automating alignment before they are misaligned enough (e.g. deceptive or power-seeking) to be dangerous. My understanding is that Nate and Eliezer are quite confident that the useful intelligence comes only after they are already misaligned, whereas most other people are more unc...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A new view for 80k's job board (plus some features), published by Yonatan Cale on August 24, 2022 on The Effective Altruism Forum.

TL;DR:

The Airtable view can filter by free text (what's your profession?), by date-published (remove very old job ads), and more. We think this alone will make searching for a job much more convenient.

We also built a custom version for software developers that has a few extra optimizations, which you can see here.

Elevator pitch for software developers

If you try filtering the 80k job board for “engineering”, you get 170 jobs, 75% of which are unrelated to software. This version has the other 25%.

How

80k graciously exposed their Airtable data (and Kush, head of the 80k job board, commented about it), so we filtered it by “software” and “developer” (plus a few more tweaks), and left it as an Airtable UI.

We have more plans for this

This is just our version after 1-2 hours of work, we’re trying to launch fast and get feedback before we keep going.

Update:

Added a “Add missing info” button, for things like “the website says the position is already closed”, which others can then filter by.

Added a filter for “this was posted in the last 12 months” by default (but you can remove it).

Do you want to make your own version?

It's really easy: Click “Use this data” on top of 80k's table or our unfiltered version (pulled from 80k).

We are very eager for feedback

Especially from people actually searching for jobs right now. DM, comment, tell us something anonymously, or anything else. We probably won’t build any feature unless someone asks for it.

Pictures

Staying updated

Subscribe to this comment to hear about updates and new features.

Thanks!

This was a slightly edited linkpost from the EA Tech Facebook group.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pressing News Front Page, published by AndreFerretti on August 24, 2022 on The Effective Altruism Forum.

"A newspaper that would cover the most important facts about the last 24 hours would cover the 15,000 child deaths on its cover page every day."—Our World in Data

So here you go:

I hope to inspire a fellow human to write about what matters to humanity, not newspapers' ad departments.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ethan Perez on the Inverse Scaling Prize, Language Feedback and Red Teaming, published by Michaël Trazzi on August 24, 2022 on The AI Alignment Forum.

I talked to Ethan Perez about the Inverse Scaling Prize (deadline August 27!), Training Language Models with Language Feedback and Red-teaming Language Models with Language models.

Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript.

Inverse Scaling Prize

"We want to understand what, in language model pre-training, objectives and data, is causing models to actively learn things that we don’t want them to learn. Some examples might be that large language models are picking up on more biases or stereotypes about different demographic groups. They might be learning to generate more toxic content, more plausible misinformation because that’s the kind of data that’s out there on the internet.

It’s also relevant in the long run because we want to have a very good understanding of how can we find where our training objectives are training models pick up the wrong behavior, because the training objective in combination with the data defines what exactly it is we’re optimizing the models very aggressively to maximize... this is a first step toward that larger goal. Let’s first figure out how can we systematically find where language models are being trained to act in ways that are misaligned with our preferences. And then hopefully, with those insights, we can take them to understand where other learning algorithms are also failing or maybe how we can improve language models with alternative objectives that have less of the limitations that they have now."

Training Language Models With Language Feedback

Why Language Feedback Instead of Comparisons

"The way that RL from human feedback typically works is we just compare two different generations or outputs from a model. And that gives very little information to the model about why, for example, a particular output was better than another. Basically, it's one bit of information or even less than one bit of information to the model that's doing the generation or output about how it should improve. And there are so many ways an output could be wrong or good or bad, and it's hard to do that attribution of which of the hundred words I generated is good or bad... that was kind of the motivation for us to look for other sources of feedback that are much more information-dense, and an easy one for people to give is just writing feedback. We give feedback to each other, verbally or written, e.g. in Google Docs. So this is very natural. It conveys a lot of information. It's not too difficult for us to give."

Measuring the Efficiency of Language Feedback

"This lets you learn from 100 samples of human feedback. So, super data efficient... previous work had gotten something like 64,000 labels that they had to collect and it was a very intensive effort. I think it was a full team at OpenAI working for a year or maybe longer. [...] If we're reducing the amount of samples we need to label by something like 100X, we can apply 100X more thought into effort and time going into evaluating those samples that we're evaluating [..] or maybe we just get 100 times more expertise on that question. Instead of paying crowd workers, we pay a lawyer or a doctor to actually do the evaluation. And that makes it much less likely that we have these failures from RL from human feedback I was describing earlier where the model generates some incorrect medical advice and we don't recognize it's happening."

Red-Teaming Language Models with Language Models

Detecting Power Seeking

"Red teaming is basically finding cases, finding inputs where models fail to produce t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Effective altruism's billionaires aren't taxed enough. But they're trying., published by Luke Freeman on August 24, 2022 on The Effective Altruism Forum.

Dylan Matthews just posted a Vox article "If you’re such an effective altruist, how come you’re so rich?" which addresses critics of effective altruism's billionaires.

My TL;DR

A lot of recent criticism of EA seems to come from the fact that it has a couple of billionaires now as supporters

These billionaires however are some of the biggest donors to US candidates that would increase taxes on them

Open support for raising taxes, e.g. Moskovitz tweeted the other day: "I’m for raising taxes and help elect Dems to do it"

The broader EA community skews heavily left-of-center (typically supportive of higher taxes and social welfare)

Effective altruism was founded explicitly on voluntary redistribution of income from people in high-income countries to low-income countries (e.g. Giving What We Can) and most of the communities founders give a significant portion of their incomes

Given that the billionaires do exist, what else would you rather they spend money on?

That's just my TL;DR – feel free to put in your own summaries, comments and critiques below.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Translating The Precipice into Czech: My experience and recommendations, published by Anna Stadlerova on August 24, 2022 on The Effective Altruism Forum.

[This post is primarily meant for non-English speaking national EA groups who'd like to see translated EA-adjacent books (such as The Precipice) published in their countries. I felt especially motivated to write this after EAGxPrague, during which many people reached out to me and asked me about my experience as a translator of The Precipice into Czech.

Please note that if your group is doing something other than a book translation, such as translating an article or other type of EA-adjacent non-literary text, different considerations might apply. I don't want to intimidate people by how complicated the process of book publishing might seem. In fact, I'd be happy to advise people on their translation projects. I am also drafting an online workshop for EAs who'd like to translate or publish translated content in their countries (see below and please contact me if you're interested).]

Content summary:

In late May 2022, The Precipice was published in the Czech Republic. I am the professional literary translator who translated the book into Czech. In this post, I describe the publishing process and outline other ways the project could have been organized.[1] I end with recommendations for publishing EA-adjacent books, which I think is the most useful part for my target readers.

These considerations and recommendations should be taken as my subjective opinions, albeit based on some experience studying literary translation and working as a literary translator.

In case you want the TLDR, my recommendations for those interested in publishing EA or EA-adjacent books are to:

Find the right publisher

Reach out to the publisher

Create a group of people who will help the translator as they translate

Get on one page with the translator (or find the right translator)

Create a glossary of terms

Proofread, proofread, proofread

Get in touch with other EAs who have done some thinking about translation

The process of publishing The Precipice

[Note about terminology: In this post, I will be talking a lot about "literary translation", "literary translators" etc. Since I've noticed some confusion about this term, I'd like to clarify that "literary translation" means translation of both fiction and non-fiction books. The case of The Precipice, as a non-fiction book, also falls under "literary translation", albeit not under "translation of fiction". The distinction I often make here is between "literary translation" and "translation of other texts", such as articles, documents, leaflets, web content and all things non-literary.]

Description of the process

The Precipice was published in the Czech Republic by Argo Publishers, in a series called Crossover. More information about the series and the publisher is provided in the sections Find the right publisher and Czech literary market.

For any translated book, fiction or nonfiction, in any decent publishing house (at least in the Czech Republic), here's how the process usually is:

1. The translator translates the book and submits it (usually using a word processing software such as Microsoft Word[2]).

2. The language editor reads the translation simultaneously with the original. Not only do they correct errors, they should also actively offer alternative phrasing and comment on the translation.They send it back to the translator. (Non-professionals are sometimes surprised by the quantity of the changes proposed by the language editor, so I thought it might be interesting to mention that even in a very good translation, it's absolutely normal to see ten or more proposed changes per page. I know this from my own experience as a translator and occasional language editor, as well as from many conv...