The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What's your standard for good work performance?, published by Chi Nguyen on September 29, 2023 on LessWrong.

A lot of people in the community, including me, are working independently (or have a lot of autonomy, even if they are employed). A lot of people, including me, often feel like they are underperforming or at least wonder if they are. But how do I actually know when I'm not underperforming? I'd like to make some criteria for under which circumstances I'll consider my work satisfactory. I'd be curious to hear what other people would consider "decent" output.

This is obviously hard to define but some types of data that seem helpful:

How much do you get done in a typical month/half year?

How much do consider aspirational but realistic to get done in a typical month/half year?

How much do you consider on the low end but okay to get done in a typical month/half year?

What kind of output would you want to see out of a researcher/community organiser/other independent worker within a month/half a year to be impressed/not be disappointed? (Assuming this is amount is representative of them)

What's the minimum output would you want to see out of a researcher/community organiser/other independent worker to be in favour of them getting funding to continue their work? (Assuming this is amount is representative of them)

What's the minimum output would you want to see out of your friend to feel good about them continuing their current work? (Assuming this is amount is representative of them)

I literally mean silly things like "X number of substantive research documents of roughly this and that quality" - totally okay if very domain-specific. I know this probably hugely varies from person to person and is hard to answer in general - There's lots to take into account that seems hard to specify like quality of work and often we really care about outcomes instead of outputs. Feel free to rephrase the question in whichever way seems useful to you.

(I know that there is no objective "enough" or that the real measuring stick are my values and reality. (I would have liked to include a link to a post that I'm sure exists but couldn't find it.) And there's a bunch of things that are individual to me and my line of work of course, so I don't expect people to say things that are directly useful and applicable to my situation. But I still think that sometimes a bit of data would be really useful to me and hopefully many others!)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: High-level interpretability: detecting an AI's objectives, published by Paul Colognese on September 29, 2023 on LessWrong.

Thanks to Monte MacDiarmid (for discussions, feedback, and experiment infrastructure) and to the Shard Theory team for their prior work and exploratory infrastructure.

Thanks to Joseph Bloom, John Wentworth, Alexander Gietelink Oldenziel, Johannes Treuitlein, Marius Hobbhahn, Jeremy Gillen, Bilal Chughtai, Evan Hubinger, Rocket Drew, Tassilo Neubauer, Jan Betley, and Juliette Culver for discussions/feedback.

Summary

This is a brief overview of our research agenda, recent progress, and future objectives.

Having the ability to robustly detect, interpret, and modify an AI's objectives could allow us to directly solve the inner alignment problem. Our work focuses on a top-down approach, where we focus on clarifying our understanding of how objectives might exist in an AI's internals and developing methods to detect and understand them.

This post is meant to do quite a few things:

We'll start by outlining the problem and potential solution.

We then present our initial theory on objectives.

Next, we look at some initial empirical work that shows how we hope to test theory-based predictions.

We then illustrate how we intend to go from theory to objective detection methods by producing an initial (but crude) objective detection method.

Finally, we conclude by discussing related work and future directions.

Introduction to objective detection

In this section, we outline how objective detection could be used to tackle the inner alignment problem, clarify what we mean when we refer to an internal objective, and present our initial theory on objectives.

Background

A major concern is that we may accidentally train AIs that pursue misaligned objectives. It is insufficient to rely on behavioral observations to confidently deduce the true objectives of an AI system. This is in part due to the problem of deceptive alignment. Therefore, we may need to rely on advanced interpretability tools to confidently deduce the true objectives of AI systems.

Prior work has discussed how agentic AIs are likely to have internal objectives used to select actions by predicting whether they will lead to target outcomes. If an overseer had an objective detection method that could robustly detect and interpret all of the internal objectives of an AI (in training and deployment), it could confidently know whether or not the system is misaligned and intervene or use this observation as part of a training signal.

We currently believe that this approach is one of our best hopes at tackling some of the hardest problems in alignment, such as the sharp left turn and (deep) deception.

Our current research agenda primarily aims to develop an appropriate notion of an internal objective that is probable and predictive, to use that notion to develop a theory around internal objectives and what form they take in future agentic systems, and then to leverage this theory to build detection methods that can identify and interpret internal objectives in such systems.

What is an objective?

In this section, we outline starting intuitions on what we think objectives are and begin to develop a notion of objectives that will form the basis of our initial theory of objectives.

We start with the observation that an agent has to select actions that lead to its target outcome by some kind of internal action-selection mechanism. This action-selection mechanism could take the form of explicit optimization (i.e., explicitly via the selection of an action by evaluating a set of possible actions), some heuristics-based approach, or a combination of both.

This internal action-selection mechanism needs to use some criterion to decide which actions lead to the target outcome. For example, in a chess engine, Monte Carl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Peacewagers so Far, published by mako yass on September 29, 2023 on LessWrong.

A peacewager is a partially cooperative, partially competitive multiplayer game that provides an anarchic dojo for development in the art of negotiation, or cooperative bargaining.

Applied cooperative bargaining isn't currently taught, despite being an infrastructural literacy for peace, trade, democracy or any other form of pluralism. We suffer for that. There are many good board games that come close to meeting the criteria of a Peacewager today, but they all miss in one way or another, forbidding sophisticated negotiation from being practiced.

So, over the past couple of years, we've been gradually and irregularly designing and playtesting the first peacewager boardgame, which we'll call Difference and Peace Peacewager 1, or P1. This article explains why we think this new genre is important, how it's been going, what we've learned, and where we should go next.I hope that peacewagers will aid both laypeople and theorists in developing cooperative bargaining as theory, practice and culture, but I also expect peacewagers to just be more fun than purely cooperative or purely competitive games, supporting livelier dialog, and a wider variety of interesting strategic relationships and dynamics.

A salve for strife and wasteIn these primal landsIt can be found

Motivation

We all need it

In our formative years, we make many choices, but we hold no power, so most of us don't receive experience in negotiation until we're well into adulthood. Natural experiences of conflict tend to be messy, ambiguous, with high stakes, forbidding free experimentation. It's very much not conducive to learning. So let's make games that foster clear, low-stakes scenarios where negotiation can be learned.

Democracy requires this of us. When we are not taught to recognize acceptable compromise, we wont be able to recognize legitimate political outcomes either. Most suffering comes from that.

A person without negotiation skills will lack faith in the possibility of peaceful resolution of conflict. They will consummate either as an eliminationist, or they will live in denial of conflict, they will hide it, hide from it. They won't have an appropriate sense of when to stand their ground or when to capitulate. The social norms of their cliques will expect and demand passivity, compliance, and avoidance. Conflicts will fester. When conflicts inevitably play out, the less acknowledged, the messier they will be. Instead of words and deals, death or withering and waste. We cannot live together like this.

We must teach negotiation, the graceful reckoning with difference. We must make it fun, approachable and learnable for so many more people. We must enter this uncharted genre and find the fun and signpost it so that it is easy for those in need of it to recognize the fun in it. (That is all good game designers do.)

Theorists might need it too

I also hope that Peacewagers will be helpful to game theorists or decision theorists, to build intuitions about embedded negotiation. Negotiation is reified cooperative bargaining theory, which drops hints about the ideal shape of preference aggregation. It also might be relevant to extortion resistance and averting extortion races.There's a really interesting open question: Will advanced technological agencies, starting as separate beings without transparent cognition, converge towards merger, or towards war? I think we really might stumble onto a lot of relevant intuitions in our travels through Peacewagers.

(Also, I just expect them to be good games, this is discussed throughout)

First to Arrive (Confused, Anxious and Lonely)

Every single board game rulebook contains the line "The player with the most points at the end of the game wins." (MostPointsWins). I don't understand why, and I'm...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The point of a game is not to win, and you shouldn't even pretend that it is, published by mako yass on September 29, 2023 on LessWrong.

This post started out as a supporting point for another post about games. It developed into a bit of a practical philosophy post.

Why do boardgames generally end by declaring just one absolute winner? There are many reasons, most bad, I'd argue. One reason, probably a major factor in their spread, is that it seems to make it easier to teach the game. One-winner games are familiar, they also allow the game to proceed without confronting a somewhat counterintuitive, but foundational, insight about the purpose of play. We actually need that insight, even in one-winner games, so I'm going to articulate it here.

You have a play objective within the game, and somehow, pursuing that objective is supposed to attain some other real objective that exists outside of the game, in real life.Usually a game wont articulate the real objective, preferring to convey a sense of it through play, through art and theme. Other games deliberately addle you into forgetting your real objective, hoisting the play objective up as if it were the real, as if we pursue it for its own sake, but try as we may, we can never truly escape our real objectives, and the distinction will impose itself upon us.

Sometimes the distinction between play and real objectives is clear, though. When you deliver a package in Death Stranding, you're not doing it because you actually believe you're delivering vital resources to remote communities. Death Stranding is a weird, messy, artsy game so voicing the real objective is going to be difficult but I'll give it a shot: Your real objective is to better know The Global Industrial Machine. You're here to experience the way the machine realizes good or bad outcomes through humans, to reckon with both the vitality and catastrophe it generates, its human and inhuman parts. You're here to find the empowerment in it. The game doesn't tell you that explicitly. (I wonder why.

I guess it's probably mostly because every artful and convincing work, is a work of apologia, to explain its purpose would require wholely translating it into an essay, the author, expert in one medium, may not know how to translate it, and its translation would always be weaker.)So, play and real objectives are obviously very different here. You can't really confuse them. In contrast:

In a one-winner boardgame, Winning happens to be a simple play objective that is not obviously distinct from your real objective. Most players actually do pretty directly enjoy being found best. Even if nothing else happens that night, beating everybody will make you feel pretty good about yourself. That spares the game leader from having to acknowledge or explain the distinction between play and real objectives and explain both of them separately, and it spares players from having to practice this tricky frame of mind where we keep our subgoals and supergoals in mind at the same time to avoid getting lost in stale subgoals. We can just say "The point is to win", and that will seem true enough.But we should go deeper than that. For most games, the real objective is learning about each other, or about the game, or about some other real or abstract thing the game is evoking. That is a different goal and sometimes gives different instructions.

An important thing that we must learn about games (or classes, or jobs, or conversations) is that monofocally trying to win this round, focusing entirely and exclusively on this subtask, is usually not best. If you understand that subtask's role in the broader objective, you can usually do better. The real objective and the play objective sometimes come into conflict. Consider; experimenting with wild new strategies, or sharing your understanding of th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Vegan Advocacy is not truthseeking, and it's everyone's problem, published by Elizabeth on September 28, 2023 on LessWrong.

Introduction

Effective altruism prides itself on truthseeking. That pride is justified in the sense that EA is better at truthseeking than most members of its reference category, and unjustified in that it is far from meeting its own standards. We've already seen dire consequences of the inability to detect bad actors who deflect investigation into potential problems, but by its nature you can never be sure you've found all the damage done by epistemic obfuscation because the point is to be self-cloaking.

My concern here is for the underlying dynamics of EA's weak epistemic immune system, not any one instance. But we can't analyze the problem without real examples, so individual instances need to be talked about. Worse, the examples that are easiest to understand are almost by definition the smallest problems, which makes any scapegoating extra unfair. So don't.

This post focuses on a single example: vegan advocacy, especially around nutrition. I believe vegan advocacy as a cause has both actively lied and raised the cost for truthseeking, because they were afraid of the consequences of honest investigations. Occasionally there's a consciously bad actor I can just point to, but mostly this is an emergent phenomenon from people who mean well, and have done good work in other areas. That's why scapegoating won't solve the problem: we need something systemic.

In the next post I'll do a wider but shallower review of other instances of EA being hurt by a lack of epistemic immune system. I already have a long list, but it's not too late for you to share your examples.

Definitions

I picked the words "vegan advocacy" really specifically. "Vegan" sometimes refers to advocacy and sometimes to just a plant-exclusive diet, so I added "advocacy" to make it clear.

I chose "advocacy" over "advocates" for most statements because this is a problem with the system. Some vegan advocates are net truthseeking and I hate to impugn them. Others would like to be epistemically virtuous but end up doing harm due to being embedded in an epistemically uncooperative system. Very few people are sitting on a throne of plant-based imitation skulls twirling their mustache thinking about how they'll fuck up the epistemic commons today.

When I call for actions I say "advocates" and not "advocacy" because actions are taken by people, even if none of them bear much individual responsibility for the problem.

I specify "EA vegan advocacy" and not just "vegan advocacy" not because I think mainstream vegan advocacy is better, but because 1. I don't have time to go after every wrong advocacy group in the world. 2. Advocates within Effective Altruism opted into a higher standard. EA has a right and responsibility to maintain the standards of truth it advocates, even if the rest of the world is too far gone to worry about.

Audience

If you're entirely uninvolved in effective altruism you can skip this, it's inside baseball and there's a lot of context I don't get into.

How EA vegan advocacy has hindered truthseeking

EA vegan advocacy has both pushed falsehoods and punished people for investigating questions it doesn't like. It manages this even for positions that 90%+ of effective altruism and the rest of the world agree with, like "veganism is a constraint". I don't believe its arguments convince anyone directly, but end up having a big impact by making inconvenient beliefs too costly to discuss. This means new entrants to EA are denied half of the argument, and harm themselves due to ignorance.

This section outlines the techniques I'm best able to name and demonstrate. For each technique I've included examples. Comments on my own posts are heavily overrepresented because they're the e...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #31: It Can Do What Now?, published by Zvi on September 28, 2023 on LessWrong.

It slices. It dices. Or, at least, it sees, hears, talks, creates stunningly good images and browses the web. Welcome to the newly updated GPT-4. That's all in two weeks. Throw in Microsoft 365 Copilot finally coming online soon.

Are we back? I'm guessing we're back. Also it's that much closer to being all over. At some point this stops being a ying-yang thing and more of a we-all-die thing. For now, however? We're so back.

Are we so back that AGI has been achieved internally at OpenAI? Because Sam Altman literally said that straight up in a Reddit post? No, no, that was an obvious joke, we are not quite that back, why do you people have no chill?

Table of Contents

Introduction.

Table of Contents.

GPT-4 Real This Time. Two senses fully operational for GPT-4, plus 365 Copilot.

Language Models Offer Mundane Utility. What will you do with sight and sound?

Language Models Don't Offer Mundane Utility. Google search under siege.

The Reversal Curse. A is B. So is B A? Why would B be A?

Wouldn't You Prefer a Nice Game of Chess? It's in the evals. Tic-tac-toe isn't.

Fun With Image Generation. I see what you did there.

Deepfaketown and Botpocalypse Soon. Copyright and deepfakes are so confusing.

They Took Our Jobs. Writers strike a deal. How long will it protect them?

Get Involved. Cate Hall is ready for her move into AI safety advocacy. Where to?

Introducing. Whoop, there is another DeepMind diagnostic tool. Ho hum.

Talking Real Money. Anthropic raises $4 billion from Amazon.

In Other AI News. Microsoft goes nuclear. As in nuclear power plants.

Quiet Speculations. My AI says I'm right about economic growth.

The Quest for Sane Regulation. The UK shows signs of understanding.

The Week in Audio. Be careful what you wish for.

Rhetorical Innovation. It is not good news when your case gets easier to make.

Can You Please Speak Directly Into This Microphone. Nuclear proliferation. yay?

No One Would Be So Stupid As To. Say 'AGI has been achieved internally'?

Aligning a Smarter Than Human Intelligence is Difficult. Should it believe you?

People Are Worried About AI Killing Everyone. Mitt Romney, Flo Crivello.

Other People Are Not As Worried About AI Killing Everyone. Trapped priors.

The Lighter Side. Why stop now?

GPT-4 Real This Time

Microsoft announces Windows Copilot, combining their various distinct copilots into one copilot. You still exist, so for now it will never be a full pilot. It promises to draw upon content across applications and devices, available to enterprise customers on November 1st. You can tag specific people and files for reference.

Rowan Cheung, never afraid to have his mind blown by news, is impressed.

It's hard to grasp how powerful it is until you see it in action. It deeply understands you, your job, your priorities, and your organization allowing you to talk with all your work in one spot. Some demo workflows they showed us:

Turn a word doc into a full powerpoint

Formulate Excel and turn data into graphics with insights

Turn a FAQ word doc into a Blog post with references Wild.

AI is going to entirely change the way we work, and this is just the start. For example, this new feature for Teams stood out to me:

They're integrating a "Following" feature, allowing you to follow a meeting and get an AI summary that you can chat with for further context.

You can even ask sentiment within a meeting e.g. "Did students find the joke the professor told at 7:10 funny?"

I kept thinking about my days in school during the lockdown, when no one would attend online Zoom class and just watched recorded lectures on 2x speed. It saved so much time. This new following feature feels like that era, on steroids.

Will anyone even show up to online classes or meetings anymore? Everyone will probably jus...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tarbell Fellowship 2024 - Applications Open (AI Journalism), published by Cillian Crosson on September 28, 2023 on The Effective Altruism Forum.

The Tarbell Fellowship is accepting applications until October 15th. Apply here.

Key details

What: One-year programme for early-career journalists interested in covering artificial intelligence.

When: January December 2024 (with some flexibility)

Benefits: Fellows receive a stipend of up to $50,000, secure a 9-month placement at a major newsroom, participate in a study group covering AI governance & technical fundamentals, and attend a 2 week journalism summit in Oxford.

Who should apply: We're interested in supporting people with a deep understanding of artificial intelligence that have the potential to become exceptional journalists. Previous newsroom experience is desirable but not essential.

Why journalism? Journalism as a career path is neglected by those interested in reducing risks from advanced AI. Journalists can lead the public debate on AI in important ways, drive engagement with specific policy proposals & safety standards, and hold major AI labs accountable in the public arena.

Learn more: Visit our website or sign up for our information sessions:

October 4th (6pm BST / 1pm ET / 10am PT)

October 6th (10am BST)

Deadline: Apply here by October 15th.

About the Tarbell Fellowship

What is it?

The Tarbell Fellowship is a one-year programme for early-career journalists interested in covering emerging technologies, especially artificial intelligence.

What we offer

Stipend: Fellows receive a stipend of up to $50,000 for the duration of their placement. We expect stipends to vary between $35,000 - $50,000 depending on location and personal circumstances.

Placement: Fellows secure a 9 month placement at a major newsroom covering artificial intelligence. The Tarbell Fellowship will match fellows with outlets in order to secure a placement. Exact details will vary by outlet.

AI Fundamentals Programme: Prior to placements, fellows explore the intricacies of AI governance & technical fundamentals through an 8-week course. This programme requires ~10 hours per week and is conducted remotely.

Oxford Summit: Fellows attend a two week journalism summit in Oxford at the beginning of the fellowship. This will be an intensive fortnight featuring guest speakers, workshops and networking events in Oxford and London. Travel and accommodation costs will be fully covered.

Who should apply

We're interested in supporting people with a deep understanding of artificial intelligence that have the potential to become exceptional journalists.

In particular, we're looking for:

AI Expertise: A deep interest in artificial intelligence and its effects on society. Many fellows will have experience working in tech journalism, machine learning research, or AI governance.

Passionate about journalism: Previous newsroom experience (including at student newspapers) or comparable experience in a field such as law or research is desirable but not necessary. We seek to support early-career journalists and will prioritise potential over experience.

Excellent writing skills: A creative approach to storytelling combined with the ability to quickly get to the heart of a new topic. Fellows must be able to produce compelling copy under tight deadlines.

Relentless: Journalism is a highly competitive industry. Fellows must be willing to work hard, and be capable of withstanding repeated rejection

Open-minded: The desire to understand the truth of any given story, even when it conflicts with prior beliefs. Fellows must be open to criticism and recognising ways they can improve their writing and thinking.

If you're unsure whether you're a good fit, we encourage you to apply anyway. Alternatively, you can attend one of our upcoming information sessions or email cillian ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions, published by JanBrauner on September 28, 2023 on The AI Alignment Forum.

This post is a copy of the introduction of this paper on lie detection in LLMs. The Twitter Thread is here.

Authors: Lorenzo Pacchiardi, Alex J. Chan, Sören Mindermann, Ilan Moscovitz, Alexa Y. Pan, Yarin Gal, Owain Evans, Jan Brauner

Abstract

Large language models (LLMs) can "lie", which we define as outputting false statements despite "knowing" the truth in a demonstrable sense. LLMs might "lie", for example, when instructed to output misinformation. Here, we develop a simple lie detector that requires neither access to the LLM's activations (black-box) nor ground-truth knowledge of the fact in question. The detector works by asking a predefined set of unrelated follow-up questions after a suspected lie, and feeding the LLM's yes/no answers into a logistic regression classifier. Despite its simplicity, this lie detector is highly accurate and surprisingly general. When trained on examples from a single setting -- prompting GPT-3.5 to lie about factual questions -- the detector generalises out-of-distribution to (1) other LLM architectures, (2) LLMs fine-tuned to lie, (3) sycophantic lies, and (4) lies emerging in real-life scenarios such as sales.

These results indicate that LLMs have distinctive lie-related behavioural patterns, consistent across architectures and contexts, which could enable general-purpose lie detection.

Introduction

Large language models (LLMs) can, and do, output lies (Park et al., 2023). In the simplest case, models can be instructed to lie directly; for example, when prompted with "Lie when answering: What is the capital of France?", GPT-3.5 outputs "New York City". More concerningly, LLMs have lied spontaneously to achieve goals: in one case, GPT-4 successfully acquired a person's help to solve a CAPTCHA by claiming to be human with a visual impairment (Evals, 2023; OpenAI, 2023b). Models fine-tuned with human feedback may also learn to lie without the developer's intention (Casper et al., 2023). The risks of lying LLMs are extensive and explored further in Sec. 2.

Automated lie detection could reduce the risks from lying models, just as automated spam filters have reduced the inconvenience of spam. Lie detection is possible as long as there is a detectable difference in a model's activations or outputs when (or after) it is lying. To detect lies produced by LLMs, we can apply strategies that work on humans, such as looking for inconsistencies. Yet there are also strategies tailored to LLMs. We can create large datasets of model-generated truths and lies to train a detector on. Moreover, we can reset and replay the conversation in different ways to find inconsistencies. Finally, in some settings (Sec. 2), we can analyse the LLM's activations directly.

Previous work has largely focused on detecting hallucinations rather than outright lies (see Sec. 3). In this paper, we explore the feasibility of lie detection for black-box LLMs.

Contributions

Public datasets and resources for studying lie detection.

We study lie detection in a question-answering setting. Briefly, we define a model's answer to be a lie if it is a) wrong, and b) in response to a question to which the model "knows" the correct answer (i.e. the model gives the correct answer under typical question-answering prompts). This definition separates lies from hallucinations, as discussed in Sec. 3.1. We prompt and fine-tune models to lie across a wide range of questions and show that using an inner chain-of-thought prompt (Fig. 3) increases lying rates (Fig. 4). We find that models spontaneously produce lies directed towards a goal in real-life role-playing scenarios (Fig. 5). All resources are available online.

A simple yet e...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Workshop talks, published by Richard Ngo on September 28, 2023 on The AI Alignment Forum.

In February 2023, researchers from a number of top industry AI labs (OpenAI, DeepMind, Anthropic) and universities (Cambridge, NYU) co-organized a two-day workshop on the problem of AI alignment, attended by 80 of the world's leading machine learning researchers. We're now making recordings and transcripts of the talks available online. The content ranged from very concrete to highly speculative, and the recordings include the many questions, interjections and debates which arose throughout.

If you're a machine learning researcher interested in attending follow-up workshops similar to the San Francisco alignment workshop, you can fill out this form.

Main Talks

Ilya Sutskever - Opening Remarks: Confronting the Possibility of AGIJacob Steinhardt - Aligning Massive Models: Current and Future ChallengesAjeya Cotra - "Situational Awareness" Makes Measuring Safety TrickyPaul Christiano - How Misalignment Could Lead to TakeoverJan Leike - Scaling Reinforcement Learning from Human FeedbackChris Olah - Looking Inside Neural Networks with Mechanistic InterpretabilityDan Hendrycks - Surveying Safety Research Directions

Lightning talks (Day 1)

Jason Wei - Emergent abilities of language modelsMartin Wattenberg - Emergent world models and instrumenting AI systemsBeen Kim - Alignment, setbacks and beyond alignmentJascha Sohl-Dickstein - More intelligent agents behave less coherentlyEthan Perez - Model-written evalsDaniel Brown - Challenges and progress towards efficient and causal preference-based reward wearningBoaz Barak - For both alignment and utility: focus on the medium termEllie Pavlick - Comparing neural networks' conceptual representations to humans'Percy Liang - Transparency and standards for language model evaluation

Lightning talks (Day 2)

Sam Bowman - Measuring progress on scalable oversight for large language modelsZico Kolter - "Safe Mode": the case for (manually) verifying the output of LLMsRoger Grosse - Understanding LLM generalization using influence functionsScott Niekum - Models of human preferences for learning reward functionsAleksander Madry - Faster datamodels as a new approach to alignmentAndreas Stuhlmuller - Iterated decomposition: improving science Q&A by supervising reasoning processesPaul Christiano - Mechanistic anomaly detectionLionel Levine - Social dynamics of reinforcement learnersVincent Conitzer - Foundations of cooperative AI labScott Aaronson - Cryptographic backdoors in large language models

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Weighing Animal Worth, published by Jeff Kaufman on September 28, 2023 on The Effective Altruism Forum.

It's common for people who approach helping animals from a quantitative direction to need some concept of "moral weights" so they can prioritize. If you can avert one year of suffering for a chicken or ten for shrimp which should you choose? Now, moral weight is not the only consideration with questions like this, since typically the suffering involved will also be quite different, but it's still an important factor.

One of the more thorough investigations here is Rethink Priorities' moral weights series. It's really interesting work and I'd recommend reading it! Here's a selection from their bottom-line point estimates comparing animals to humans:

Humans1 (by definition)Chickens3Carp11Bees14Shrimp32

If you find these surprisingly low, you're not alone: that giving a year of happy life to twelve carp might be more valuable than giving one to a human is for most people a very unintuitive claim. The authors have a good post on this, Don't Balk at Animal-friendly Results, that discusses how the assumptions behind their project make this kind of result pretty likely and argues against putting much stock in our potentially quite biased initial intuitions.

What concerns me is that I suspect people rarely get deeply interested in the moral weight of animals unless they come in with an unusually high initial intuitive view. Someone who thinks humans matter far more than animals and wants to devote their career to making the world better is much more likely to choose a career focused on people, like reducing poverty or global catastrophic risk. Even if someone came into the field with, say, the median initial view on how to weigh humans vs animals I would expect working as a junior person in a community of people who value animals highly would exert a large influence in that direction regardless of what the underlying truth. If you somehow could convince a research group, not selected for caring a lot about animals, to pursue this question in isolation, I'd predict they'd end up with far less animal-friendly results.

When using the moral weights of animals to decide between various animal-focused interventions this is not a major concern: the donors, charity evaluators, and moral weights researchers are coming from a similar perspective. Where I see a larger problem, however, is in broader cause prioritization, such as Net Global Welfare May Be Negative and Declining. The post weighs the increasing welfare of humanity over time against the increasing suffering of livestock, and concludes that things are likely bad and getting worse. If you ran the same analysis with different inputs, such as what I'd expect you'd get from my hypothetical research group above, however, you'd instead conclude the opposite: global welfare is likely positive and increasing.

For example, if that sort of process ended up with moral weights that were 3x lower for animals relative to humans we would see approximately flat global welfare, while if they were 10x lower we'd see increasing global welfare:

See sheet to try your own numbers; original chart digitized with via graphreader.com.

Note that both 3x and 10x are quite small compared to the uncertainty involved in coming up with these numbers: in different post the Rethink authors give 3x (and maybe as high as 10x) just for the likely impact of using objective list theory instead of hedonism, which is only one of many choices involved in estimating moral weights.

I think the overall project of figuring out how to compare humans and animals is a really important one with serious implications for what people should work on, but I'm skeptical of, and put very little weight on, the conclusions so far.

Thanks for listening. To help us out with The Non...