Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anthropic: Core Views on AI Safety: When, Why, What, and How, published by jonmenaster on March 9, 2023 on The Effective Altruism Forum.

We founded Anthropic because we believe the impact of AI might be comparable to that of the industrial and scientific revolutions, but we aren’t confident it will go well. And we also believe this level of impact could start to arrive soon – perhaps in the coming decade.

This view may sound implausible or grandiose, and there are good reasons to be skeptical of it. For one thing, almost everyone who has said “the thing we’re working on might be one of the biggest developments in history” has been wrong, often laughably so. Nevertheless, we believe there is enough evidence to seriously prepare for a world where rapid AI progress leads to transformative AI systems.

At Anthropic our motto has been “show, don’t tell”, and we’ve focused on releasing a steady stream of safety-oriented research that we believe has broad value for the AI community. We’re writing this now because as more people have become aware of AI progress, it feels timely to express our own views on this topic and to explain our strategy and goals. In short, we believe that AI safety research is urgently important and should be supported by a wide range of public and private actors.

So in this post we will summarize why we believe all this: why we anticipate very rapid AI progress and very large impacts from AI, and how that led us to be concerned about AI safety. We’ll then briefly summarize our own approach to AI safety research and some of the reasoning behind it. We hope by writing this we can contribute to broader discussions about AI safety and AI progress.

As a high level summary of the main points in this post:

AI will have a very large impact, possibly in the coming decadeRapid and continuing AI progress is a predictable consequence of the exponential increase in computation used to train AI systems, because research on “scaling laws” demonstrates that more computation leads to general improvements in capabilities. Simple extrapolations suggest AI systems will become far more capable in the next decade, possibly equaling or exceeding human level performance at most intellectual tasks. AI progress might slow or halt, but the evidence suggests it will probably continue.

We do not know how to train systems to robustly behave wellSo far, no one knows how to train very powerful AI systems to be robustly helpful, honest, and harmless. Furthermore, rapid AI progress will be disruptive to society and may trigger competitive races that could lead corporations or nations to deploy untrustworthy AI systems. The results of this could be catastrophic, either because AI systems strategically pursue dangerous goals, or because these systems make more innocent mistakes in high-stakes situations.

We are most optimistic about a multi-faceted, empirically-driven approach to AI safetyWe’re pursuing a variety of research directions with the goal of building reliably safe systems, and are currently most excited about scaling supervision, mechanistic interpretability, process-oriented learning, and understanding and evaluating how AI systems learn and generalize. A key goal of ours is to differentially accelerate this safety work, and to develop a profile of safety research that attempts to cover a wide range of scenarios, from those in which safety challenges turn out to be easy to address to those in which creating safe systems is extremely difficult.

Our Rough View on Rapid AI Progress

The three main ingredients leading to predictable1 improvements in AI performance are training data, computation, and improved algorithms. In the mid-2010s, some of us noticed that larger AI systems were consistently smarter, and so we theorized that the most important ingredient in AI performance m...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why do we assume there is a "real" shoggoth behind the LLM? Why not masks all the way down?, published by Robert AIZI on March 9, 2023 on LessWrong.

In recent discourse, Large Language Models (LLMs) are often depicted as presenting a human face over a vast alien intelligence (the shoggoth), as in this popular image or this Eliezer Yudkowsky tweet:

I think this mental model of an LLM is an improvement over the naive assumption that the AI is the friendly mask. But I worry it's making a second mistake by assuming there is any single coherent entity inside the LLM.

In this regard, we have fallen for a shell game. In the classic shell game, a scammer puts a ball under one of three shells, shuffles them around, and you wager which shell the ball is under. But you always pick the wrong one because you made the fundamental mistake of assuming any shell had the ball - the scammer actually got rid of it with sleight of hand.

In my analogy to LLMs, the shells are the masks the LLM wears (i.e. the simulacra), and the ball is the LLM's "real identity". Do we actually have evidence there is a "real identity" in the LLM, or could it just be a pile of masks? No doubt the LLM could role-play a shoggoth - but why would you assume that's any more real that roleplaying a friendly assistant?

I would propose an alternative model of an LLM: a giant pile of masks. Some masks are good, some are bad, some are easy to reach and some are hard, but none of them are the “true” LLM.

Finally, let me head off one potential counterargument: "LLMs are superhuman in some tasks, so they must have an underlying superintelligence”. Three reasons a pile of masks can be superintelligent:

An individual mask might be superintelligent. E.g. a mask of John von Neumann would be well outside the normal distribution of human capabilities, but still just be a mask.

The AI might use the best mask for each job. If the AI has masks of a great scientist, a great doctor, and a great poet, it could be superhuman on the whole by switching between its modes.

The AI might collaborate with itself, gaining the wisdom of the crowds. Imagine the AI answering a multiple choice question. In the framework of Simulacra Theory as described in the Waluigi post, the LLM is simulating all possible simulacra, and averaging their answers weighted by their likelihood of producing the previous text. For example, if question could have been produced by a scientist, a doctor, or a poet, who would respectively answer (A or B), (A or C), and (A or D), the superposition of these simulacra would answer A. This could produce superior answers than any individual mask.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Windfall Clause for CEO could worsen AI race dynamics, published by Larks on March 9, 2023 on The Effective Altruism Forum.

Summary

This is a response to the Windfall Clause proposal from Cullen O’Keefe et al, which aims to make AI firms promise to donate a large fraction of profits if they become extremely profitable. While I appreciate their valiant attempt to produce a policy recommendation that might help, I am worried about the practical effects.

In this article I argue that the Clause would primarily benefit the management of these firms, resulting in an increased concentration of effective wealth/power relative to a counterfactual where traditional corporate governance was used. This could make AI race dynamics worse and increase existential risk from AI.

What is the Windfall Clause?

The Clause operates by getting firms now to sign up to donate a large fraction of their profits for the benefit of humanity if those profits become very large. The idea is that, right now, profits are not very large, so this appears a ‘cheap’ commitment in the short term. In the future, if the firm becomes very successful, they are required to donate an increasing fraction.

This is an example structure from O’Keefe document:

Many other possible structures exist with similar effects. As an extreme example, you could require all profits above a certain level to be donated.

Typical Corporate Governance

The purpose of a typical corporation is to make profits. Under standard corporate governance, CEOs are given fairly broad latitude to make business decisions. They can determine strategy, decide on new products and pricing, alter their workforces and so on with limited restrictions. If the company fails to make profits, the share price will fall, and it might be subject to a hostile takeover from another firm which thinks it can use the assets more wisely. Additionally, in the meantime the CEO’s compensation is likely to fall due to missed incentive pay.

The board also supplies oversight. They will be consulted with on major decisions, and their consent is required for irreversible ones (e.g. a major acquisition or change of strategy). The auditor will report to them so they can keep apprised of the financial state of the company.

The amount of money the CEO can spend without oversight is quite limited. Most of the firm’s revenue probably goes to expenses; of the profits, the board will exercise oversight into decisions around dividends, buybacks and major acquisitions. The CEO will have more discretion over capital expenditures, but even then the board will have a say on the total size and general strategy, and all capex will be expected to follow the north star of future profitability. A founder-CEO might retain some non-trivial economic interest in the profits (say 10% if it was a small founding team and they grew rapidly with limited need for outside capital), which is truely theirs to spend as they wish; a hired CEO would have much less.

How does the clause change this?

In contrast, the clause appears to give a lot more latitude to management of a successful AGI firm.

Some of the typical constraints remain. The firm must still pay its suppliers, and continue to produce goods and services that others value enough to pay for them more than they cost to produce. Operating decisions will remain judged by profitability, and the board will continue to have oversight over major decisions.

However, a huge amount of profit is effectively transferred from third party investors to the CEO or management team. They go from probably a few percent of the profits to spend as they wish to controlling the distribution of perhaps half. Additionally, some of this is likely tax deductible.

It is true that the CEO couldn’t spend the money on personal yachts and the like. However, extremely rich peopl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anthropic's Core Views on AI Safety, published by Zac Hatfield-Dodds on March 9, 2023 on LessWrong.

We founded Anthropic because we believe the impact of AI might be comparable to that of the industrial and scientific revolutions, but we aren’t confident it will go well. And we also believe this level of impact could start to arrive soon – perhaps in the coming decade.

This view may sound implausible or grandiose, and there are good reasons to be skeptical of it. For one thing, almost everyone who has said “the thing we’re working on might be one of the biggest developments in history” has been wrong, often laughably so. Nevertheless, we believe there is enough evidence to seriously prepare for a world where rapid AI progress leads to transformative AI systems.

At Anthropic our motto has been “show, don’t tell”, and we’ve focused on releasing a steady stream of safety-oriented research that we believe has broad value for the AI community. We’re writing this now because as more people have become aware of AI progress, it feels timely to express our own views on this topic and to explain our strategy and goals. In short, we believe that AI safety research is urgently important and should be supported by a wide range of public and private actors.

So in this post we will summarize why we believe all this: why we anticipate very rapid AI progress and very large impacts from AI, and how that led us to be concerned about AI safety. We’ll then briefly summarize our own approach to AI safety research and some of the reasoning behind it. We hope by writing this we can contribute to broader discussions about AI safety and AI progress.

As a high level summary of the main points in this post:

AI will have a very large impact, possibly in the coming decadeRapid and continuing AI progress is a predictable consequence of the exponential increase in computation used to train AI systems, because research on “scaling laws” demonstrates that more computation leads to general improvements in capabilities. Simple extrapolations suggest AI systems will become far more capable in the next decade, possibly equaling or exceeding human level performance at most intellectual tasks. AI progress might slow or halt, but the evidence suggests it will probably continue.

We do not know how to train systems to robustly behave wellSo far, no one knows how to train very powerful AI systems to be robustly helpful, honest, and harmless. Furthermore, rapid AI progress will be disruptive to society and may trigger competitive races that could lead corporations or nations to deploy untrustworthy AI systems. The results of this could be catastrophic, either because AI systems strategically pursue dangerous goals, or because these systems make more innocent mistakes in high-stakes situations.

We are most optimistic about a multi-faceted, empirically-driven approach to AI safety We’re pursuing a variety of research directions with the goal of building reliably safe systems, and are currently most excited about scaling supervision, mechanistic interpretability, process-oriented learning, and understanding and evaluating how AI systems learn and generalize. A key goal of ours is to differentially accelerate this safety work, and to develop a profile of safety research that attempts to cover a wide range of scenarios, from those in which safety challenges turn out to be easy to address to those in which creating safe systems is extremely difficult.

The full post goes into considerably more detail, and I'm really excited that we're sharing more of our thinking publicly.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper Summary: The Effectiveness of AI Existential Risk Communication to the American and Dutch Public, published by Otto on March 9, 2023 on The Effective Altruism Forum.

This is a summary of the following paper by Alexia Georgiadis (Existential Risk Observatory):

Thanks to @Lara Mani, @Karl von Wendt, and Alexia Georgiadis for their help in reviewing and writing this post. Any views expressed in this post are not necessarily theirs.The rapid development of artificial intelligence (AI) has evoked both positive and negative sentiments due to its immense potential and the inherent risks associated with its evolution. There are growing concerns that if AI surpasses human intelligence and is not aligned with human values, it may pose significant harm and even lead to the end of humanity. However, the general publics' knowledge of these risks is limited. As advocates for minimising existential threats, the Existential Risk Observatory believes it is imperative to educate the public on the potential risks of AI. Our introductory post outlines some of the reasons why we hold this view (this post is also relevant). To increase public awareness of AI's existential risk, effective communication strategies are necessary.

This research aims to assess the effectiveness of communication interventions currently being used to increase awareness about AI existential risk, namely news publications and videos. To this end, we conducted surveys to evaluate the impact of these interventions on raising awareness among participants.

Methodology

This research aims to assess the effectiveness of different media interventions, specifically news articles and videos, in promoting awareness of the potential dangers of AI and its possible impact on human extinction. It analyses the impact of AI existential risk communication strategies on the awareness of the American and Dutch populations, and investigates how social indicators such as age, gender, education level, country of residence, and field of work affect the effectiveness of AI existential risk communication.

The study employs a pre-post design, which involves administering the same intervention and assessment to all participants and measuring their responses at two points in time. The research utilises a survey method for collecting data, which was administered to participants through an online Google Forms application. The survey consists of three sections: pre-test questions, the intervention, and post-test questions.

The effectiveness of AI existential risk communication is measured by comparing the results of quantitative questions from the pre-test and post-test sections, and the answers to the open-ended questions provide further understanding of any changes in the participant's perspective. The research measures the effectiveness of the media interventions by using two main indicators: "Human Extinction Events" and "Human Extinction Percentage."

The "Human Extinction Events" indicator asks participants to rank the events that they believe could cause human extinction in the next century, and the research considers it as effective if participants rank AI higher post intervention or mention it after the treatment when they did not mention it before. If the placement of AI remained the same before and after the treatment, or if participants did not mention AI before or after the treatment, the research considered that there was no effect in raising awareness.

The "Human Extinction Percentage" indicator asks for the participants' opinion on the likelihood, in percentage, of human extinction caused by AI in the next century. If there was an increase in the percentage of likelihood given by participants, this research considered that there was an effect in raising awareness. If there is no change or a decrease in the percentage, this r...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2022 Survey Results, published by Screwtape on March 8, 2023 on LessWrong.

The Data

0. Population

There were 186 respondents over 28 days. The first week of the survey got 172 responses. The second week got 10. The third and fourth weeks got 2 responses each.

Previous surveys have been run over the past decade or so. Their numbers are as follows:

2009: 1662011: 1090 2012: 11952013: 16362014: 1503 2016: 3083 2017: "About 300"2020: 612022: 186

The cheerfully optimistic interpretation of this is something like "cool! I got about as many as Scott did on his first try!" I'll talk about the survey size more in the Conclusion section. For now, on with the numbers.

Previous LessWrong Surveys:No: 131, 72.0%Prefer not to answer: 6, 3.3%Yes: 45, 24.7%

Did you also take the ACX 2022 Survey?No: 93, 50.5%Prefer not to answer: 3, 1.6%Yes: 88, 47.8%

I. Demographics (Not counting sex and gender)

There's a format I'll use in a few places throughout this summation. If you see a sequence of numbers like "30.1 + 8.9 (24, 28, 34) [n=186]" those numbers are "Mean + standard deviation (1st quartile, 2nd quartile, 3rd quartile) [n= number responding]."

Age: 30.1 + 8.9 (24, 28, 34) [n=186]

I broke this out into buckets by decade so I could get a better sense of what was going on.

Under 20: 13, 7.0%20 to 29: 91, 49.2%30 to 39: 57, 30.8%40 to 49: 14, 7.6%50 to 59: 8, 4.3%60 to 69: 2, 1.1%

The 2009 to 2014 surveys consistently found the mean age was 27, and in 2016 the mean age had only gone up to 28. This is an interesting property for a community to have! It's far from unusual: any survey of a college campus will find the mean age stays about the same, and for different reasons so can the mean age of countries. A survey of Britney Spears fans probably finds that the mean age goes up over time, and many churches observe the same pattern. It's one of several high level ways to look at a community: are you replacing yourself about as fast as you lose people?

LessWrong is probably a community people find and age out of. We're a bit older. In 2011, that 20 to 29 bucket was about 60%. Still, teenagers and college students are still finding us.

Race:White (non-Hispanic): 148, 82.7%Asian (East Asian): 6. 3.4%Asian (Indian subcontinent): 6, 3.4%Hispanic: 5, 2.8%Middle Eastern: 3, 1.7%Other: 14, 7.8%

Country:United States: 95, 51.1%United Kingdom: 20, 10.8%Australia: 12, 6.5%Germany: 11, 5.9%New Zealand: 6, 6.6%France: 6, 6.6% Canada: 5, 5.5%Russia, 4, 4.4%Israel: 4, 4.4%Other: 22 , 24%[The Other category is mostly Eastern European countries with one or two respondents. If there were three or less respondents from your country, I changed your country to Other for the public dataset in case being the one person in your country was too identifying.]

II. Sex, Gender, and Relationships

Sex:Female: 21, 11.5%Male: 162, 88.5%

Gender:F (cisgender): 18, 9.8%F (transgender MtF): 6, 3.3%M (cisgender): 152, 83.1%M (transgender FtM): 0, 0%Other: 7, 3.8%

Sexual Orientation:Asexual: 7, 3.9%Bisexual: 24, 13.3%Heterosexual: 145, 80.1%Homosexual: 5, 2.8%Other: 0, 0%

Relationship Style:Prefer monogamous: 106, 58.9%Prefer polyamorous: 29, 16.1%Uncertain/no preference: 44, 24.4%Other: 1, 0.6%

Number of Partners0: 81, 44.51%1: 90, 49.5%2: 9, 4.9%3: 1, 0.5%4: 0, 0%5: 0, 0%Lots and lots: 1, 0.5%Relationship Goals:. . . and currently not looking for more relationship partners: 119, 67.6%. . . and currently looking for more relationship partners: 57, 32.4%[39.0% of people who don’t have a partner aren’t looking for one. ]

To the one person with lots and lots of partners and looking for more, rock on you ambitious fellow.

Relationship Status:Married: 40, 21.9%Relationship: 59, 32.2%Single: 84, 45.9%[Two single people have a non-zero number of partners.]

Living With:Alone: 41, 23.0%With parents and/or guardians: 29, 16.3%With part...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anthropic's Core Views on AI Safety, published by Zac Hatfield-Dodds on March 9, 2023 on The AI Alignment Forum.

We founded Anthropic because we believe the impact of AI might be comparable to that of the industrial and scientific revolutions, but we aren’t confident it will go well. And we also believe this level of impact could start to arrive soon – perhaps in the coming decade.

This view may sound implausible or grandiose, and there are good reasons to be skeptical of it. For one thing, almost everyone who has said “the thing we’re working on might be one of the biggest developments in history” has been wrong, often laughably so. Nevertheless, we believe there is enough evidence to seriously prepare for a world where rapid AI progress leads to transformative AI systems.

At Anthropic our motto has been “show, don’t tell”, and we’ve focused on releasing a steady stream of safety-oriented research that we believe has broad value for the AI community. We’re writing this now because as more people have become aware of AI progress, it feels timely to express our own views on this topic and to explain our strategy and goals. In short, we believe that AI safety research is urgently important and should be supported by a wide range of public and private actors.

So in this post we will summarize why we believe all this: why we anticipate very rapid AI progress and very large impacts from AI, and how that led us to be concerned about AI safety. We’ll then briefly summarize our own approach to AI safety research and some of the reasoning behind it. We hope by writing this we can contribute to broader discussions about AI safety and AI progress.

As a high level summary of the main points in this post:

AI will have a very large impact, possibly in the coming decadeRapid and continuing AI progress is a predictable consequence of the exponential increase in computation used to train AI systems, because research on “scaling laws” demonstrates that more computation leads to general improvements in capabilities. Simple extrapolations suggest AI systems will become far more capable in the next decade, possibly equaling or exceeding human level performance at most intellectual tasks. AI progress might slow or halt, but the evidence suggests it will probably continue.

We do not know how to train systems to robustly behave wellSo far, no one knows how to train very powerful AI systems to be robustly helpful, honest, and harmless. Furthermore, rapid AI progress will be disruptive to society and may trigger competitive races that could lead corporations or nations to deploy untrustworthy AI systems. The results of this could be catastrophic, either because AI systems strategically pursue dangerous goals, or because these systems make more innocent mistakes in high-stakes situations.

We are most optimistic about a multi-faceted, empirically-driven approach to AI safety We’re pursuing a variety of research directions with the goal of building reliably safe systems, and are currently most excited about scaling supervision, mechanistic interpretability, process-oriented learning, and understanding and evaluating how AI systems learn and generalize. A key goal of ours is to differentially accelerate this safety work, and to develop a profile of safety research that attempts to cover a wide range of scenarios, from those in which safety challenges turn out to be easy to address to those in which creating safe systems is extremely difficult.

The full post goes into considerably more detail, and I'm really excited that we're sharing more of our thinking publicly.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Translucent Thoughts Hypotheses and Their Implications, published by Fabien Roger on March 9, 2023 on The AI Alignment Forum.

Epistemic status: Uncertain about the validity of the claims I’m making here, and looking for feedback about the research directions I’m suggesting.

Thanks to Marius Hobbhahn, Johannes Treutlein, Siméon Campos, and Jean-Stanislas Denain for helpful feedback on drafts.

Here is a set of hypotheses:

The first AGIs will have LLMs at their core

Effective plans to defeat humanity can’t be found in a single LLM forward pass

LLMs will solve complex tasks by using English text (self-prompting, scratch pads, combination of expert LLMs, .)

I call these the Translucent Thoughts hypotheses.

I think the Translucent Thoughts hypotheses are likely (around 20% conditioning on AGI before 2030) because:

Text pretraining is more efficient at building algorithms and knowledge required for real-world plan generation and evaluation than alternative methods;

Future models are likely to be like Transformers, which use a limited amount of serial step in a single forward pass, and deception requires many serial steps;

Text pretraining and slight fine-tuning makes model able to use text generation to increase the maximum number of serial steps by a huge factor. Getting this increase through other means is likely to be hard and not competitive.

If these hypotheses are true, it should lead us to prioritize underexplored research directions, such as circumventing steganography or building extremely reliable text-supervision methods. I think those deserve attention, because Translucent Thoughts AIs are not safe by default.

In this post, I argue that we may will in a world where the first AGIs will look like X, and I then describe ways to make the first AGIs safer given X. This is different from most other works in this space, which often directly describe a kind of safe AGI. Despite this, the ideas of this post are close to some other works describing paths to safe AGIs, such as:

Externalized Reasoning Oversight, which describes a class of solutions similar to the one outlined here, but also aims for additional properties which I argue can be replaced with a less stringent hypothesis about AI systems;

Conditioning Predictive Models, which makes assumptions slightly different from the Translucent Thoughts hypotheses, yielding different research directions;

The Open Agency Model and Factored Cognition which describe subsets of AIs with Translucent Thoughts, which might be safe.

The Translucent Thoughts Hypotheses

Here, I sketch a world in which the first AGIs have certain properties. I argue that this world is likely, and thus a subset of all possible futures to care about. But I think it’s not a large part of all possible futures (20% conditioning on AGI before 2030).

The First AGIs Will Have LLMs at Their Core

By “first AGIs” I mean the first systems able to automate all cognitive tasks.

AGI is likely to do reasoning and planning using LLMs. AGI might rely on vision models for some tasks and interactions with the world, and it might use explicit search processes like AlphaGo. But I expect LLMs to do plan generation and evaluation, which are the core of the system (from an Alignment point of view).

Why: Vision systems are bad at coming up with and evaluating deceptive plans. Explicit search processes can’t generate and evaluate plans in the real world. LLMs seem to be able to do both plan generation and evaluation. (Plan generation and evaluation are the core tasks we would like to monitor to make AGIs safe, which is why I focus on those.)

End-to-end neural networks won’t be able to compete with LLMs when it comes to reasoning and planning, or at least, end-to-end networks will use “their LLMs parts” to do their most advanced form of reasoning and planning. This means tha...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Speed running everyone through the bad alignement bingo. $5k bounty for a LW conversational agent, published by ArthurB on March 9, 2023 on LessWrong.

There's a wave of people, of various degrees of knowledge and influence, currently waking up to the ideas of AI existential risk. They seem to be literally going through every box of the bad alignement bingo card takes.

I think there is value in educating those people. I'm aware there's an argument to be made that: education at scale doesn't matter, coordination is too difficult, all that matter is solving alignment and that takes care of the rest.

There's something to that, but I disagree that education at scale doesn't help. It can make progress of frontrunners marginally more safety oriented, it can steer company cultures, it can move the Overton window, change the Zeitgeist, it can buy a bit of time. You likely didn't stumble on these ideas all on your own, so arguing against the value of outreach or education is also arguing against your own ability to do anything.

It's also a matter of ROI, and there are some very low hanging fruit there. The simplest thing would be to write a long FAQ that goes through every common objections. No, people won't read the whole sequences, or Arbital on their own, but they might go through a FAQ.

But we can do better than a FAQ. It's now fairly straightforward, with tools like langchain () to turn a set of documents into a body of knowledge for a conversational agent. This is done by building an index of embedding that a language model can search to bring context to an answer. This doesn't preclude fine tuning, but it makes it unnecessary.

So a straightforward project is to index lesswrong, index arbitral, index the alignment forum, maybe index good alignement papers as well, blog posts, books.

Then hook that up to the ChatGPT API, and prompt it to:

list search queries for relevant material to answer the question

compose an answer that reflects the content and opinion of the data

answer with infinite patience

Some jailbreak prompts may be needed to prevent ChatGPT's conditioning to regurgitate AI risk appeasing propaganda through the API, but there are a bunch of those out there. Or use the API of other models as they become open source or commercially available.

Will this save humanity? No. Will this turn the course of safety research? Also no. Is this using AI to advance alignment? Well, yes, a little bit, don't dismiss very small starts.

Is this worth spending a weekend hacking on this project instead of posting on Twitter? Absolutely.

Will this actually make things worse? No, you're overthinking this.

I'll pay $5k to the best version built by the end of March (if any is built). It's a modest bounty but it's really not all that much work, and it's fun work. And of course if anyone wants to add their own contribution to the bounty please do.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: IRL in General Environments, published by michaelcohen on March 9, 2023 on The AI Alignment Forum.

Here is a proposal for Inverse Reinforcement Learning in General Environments. (2 1/2 pages; very little math).

Copying the introduction here:

The eventual aim of IRL is to understand human goals. However, typical algorithms for IRL assume the environment is finite-state Markov, and it is often left unspecified how raw observational data would be converted into a record of human actions, alongside the space of actions available. For IRL to learn human goals, the AI has to consider general environments, and it has to have a way of identifying human actions. Lest these extensions appear trivial, I consider one of the simplest proposals, and discuss some difficulties that might arise.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.