Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deception?! I ain’t got time for that!, published by Paul Colognese on July 18, 2022 on LessWrong.

Or ... How penalizing computation used during training disfavors deception.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

Thanks to Evan Hubinger, Yonadav Shavit, and Arun Jose for helpful discussions and feedback.

Introduction

We want to use ML to produce “good” programs that perform well on some desired metrics and have some desired properties. Unfortunately, some programs that look good during, and at the end of the training, may not actually be good. This includes deceptive programs that pretend to look good during the training process in order to pursue their own objectives later.

Because deceptive programs have to use computation on tasks like deceptive planning, modifying the ML training process to favor programs that use less computation may help prevent the selection of deceptive programs. A training process that favors such “fast programs” is said to have a speed prior/bias.

The aim of this post is to flesh out an argument for why training processes with speed priors may disfavor deceptive programs, and to consider how speed priors might be implemented in practice.

We begin by looking at how the priors/biases of a training process may be leveraged to help us select good programs rather than programs that just look good. Building on previous work, we then argue that speed priors may disfavor deceptive programs and more general forms of misbehavior. Next, we consider some potential implementations of speed priors before closing with some additional thoughts.

Using priors to help select good programs

An ML training process begins by initializing a space of programs (typically this space corresponds to the parameter space of a neural network) and then searches for a program that looks good to the training process, i.e., that seems to achieve good performance on the relevant metrics and have the desired properties. We can think of “looking good” as a constraint that programs must satisfy in order to be considered for selection.

One problem we face is that programs that look good, may not actually be good. For example, we may want a program that generalizes well to deployment inputs, but if the training process only measures performance on the training inputs, programs that look good to the training process may include programs that generalize well and those that don’t.

The training process may have priors/biases which means that among the programs that look good to the training process, the training process will usually favor selecting certain programs over others.

The presence of priors/biases in the training process can help us find programs that are actually good despite just looking good. To see why this may be, we note that there is evidence that today’s standard ML training processes have a simplicity prior/bias; they favor the simplest functions that fit the data. Simple functions that fit the training data tend to generalize well, partly because they are less likely to overfit. From this example, we see that the presence of a simplicity prior in the training process encourages the process to select programs that not only perform well on the training set (look good) but also generalize well (are actually good).

We note that constructing and analyzing priors that favor multiple desired metrics/properties may be very difficult, especially since some metrics/properties may be anti-correlated. For example, although simple functions tend to generalize well, simple programs may be more likely to be deceptive.

Taking the above discussion into account, the following approach may encourage the selection of good programs:

Spend resources imp...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Survey of the Potential Long-term Impacts of AI, published by Sam Clarke on July 18, 2022 on The Effective Altruism Forum.

Aim: survey the potential long-term impacts of AI, striking a balance between comprehensiveness and parsimony.

Where this fits with similar work: as far as we know, the best existing materials on this topic are this slide deck and sections 2 and 3 of this forum post. In some sense, this paper is aiming to be a more comprehensive version of these pieces. It also includes discussion of the long-term opportunities from AI, as well as the risks.

Audience: people who want an overview of the various ways that AI could have long-term impacts. This was written for an audience that isn't necessarily familiar with longtermism, so some framing points will be obvious to certain readers.

Work done collaboratively with Jess Whittlestone.

Also available here as a PDF.

Summary

Based on surveying literature on the societal impacts of AI, we identify and discuss five areas in which AI could have long-term impacts: in science, cooperation, power, epistemics, and values. Considering both possible benefits and harms, we review the state of existing research in these areas and highlight priority questions for future research.

Some takeaways:

Advanced AI could be very good or very bad for humanity, and it is not yet determined how things will go.

AGI is not necessary for AI to have long-term impacts. Many long-term impacts we consider could happen with "merely" comprehensive AI services, or plausibly also with non-comprehensive AI services (e.g. Sections 3.2 and 5.2).

There are several different pathways through which AI could have long-term impacts, each of which could be sufficient by itself.

These takeaways are not original, but we hope we have added some depth to the arguments and to this community's understanding of the long-term impacts of AI more broadly.

1 Introduction

Artificial intelligence (AI) is already being applied in and impacting many important sectors in society, including healthcare [Jiang et al. 2017], finance [Daníelsson et al. 2021], and law enforcement [Richardson et al. 2019]. Some of these impacts have been positive—such as the ability to predict the risk of breast cancer from mammograms more accurately than human radiologists [McKinney et al. 2020]—whilst others have been extremely harmful—such as the use of facial recognition technology to surveil Uighur and other minority populations in China [Hogarth & Benaich 2019]. As investment into AI research continues, we are likely to see substantial progress in AI capabilities and their potential applications, precipitating even greater societal impacts.

What is unclear is just how large and long-lasting the impacts of AI will be, and whether they will ultimately be positive or negative for humanity. In this paper we are particularly concerned with impacts of AI on the far future:[1] impacts that would be felt not only by our generation or the next, but by many future generations who could come after us. We will refer to such impacts as long-term impacts.

Broadly speaking, we might expect AI to have long-term impacts because of its potential as a general purpose technology: one which will probably see unusually widespread use, tend to spawn complementary innovations, and have a large inherent potential for technical improvement. Historically, general purposes technologies—such as the steam engine and electricity—have tended to precipitate outsized societal impacts [Garfinkel 2022]. In this paper, we consider potential long impacts of AI which could:

Make a global catastrophe more or less likely (i.e. a catastrophe that poses serious damage to human well-being on a global scale, for example by enabling the discovery of a pathogen that kills hundreds of millions of people).[2]

Make premature human e...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Four questions I ask AI safety researchers, published by Akash on July 17, 2022 on The Effective Altruism Forum.

Over the last few months, I’ve been trying to develop a stronger inside view on AI safety research agendas.

As part of this quest, I’ve been having conversations with AI safety researchers. I notice myself often asking the following questions:

What are you working on, and how does it help us get aligned AI?

Imagine you’re talking to a smart high school student. How would you describe the alignment problem? And how would you describe how your work addresses it?

Imagine that you came up with a solution to the specific problem you’re working on. Or even more boldly, imagine your entire program of research succeeds. What happens next? Concretely, how does this help us get aligned AI (and prevent unaligned AI)?

What are the qualities you look for in promising AI safety researchers? (beyond general intelligence)

I find Question #1 useful for starting the conversation and introducing me to the person’s worldview.

I find Question #2 useful for getting clearer (and often more detailed) explanations of the person’s understanding of the alignment problem & how their work fits in. (Note that this is somewhat redundant with question #1, but I find that the questions often yield different answers).

I find Question #3 useful for starting to develop an inside view on the research agenda & the person’s reasoning abilities. (Note that the point is to see how the person reasons about the question. I’m not looking for a “right answer”, but I am looking to see that someone has seriously thought about this and has reasonable takes.)

I find Question #4 useful for building stronger models of AI safety community-building.

I also ask a lot of follow-up questions. But I find that these are the four questions that I ask nearly everyone, and I think they’ve helped me develop a stronger inside view.

If I had to pick one question, I would probably ask #3. I think it’s a rather challenging question, and I’m generally impressed when people have coherent answers.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reducing aquatic noise as a wild animal welfare intervention, published by saulius on July 18, 2022 on The Effective Altruism Forum.

Summary

Aquatic noise comes from ships, seismic surveys to find oil and gas, sonar, and other sources. It causes marine animals stress and masks their communications, among other effects on marine ecosystems.

Aquatic noise can be reduced by lobbying for international agreements to slow down ships, adopting various technologies, and protesting against seismic surveys.

Usually, aquatic noise is considered an environmental or conservationist issue, and these seem to be the dominant concerns of the few organizations working on reducing aquatic noise. In this text, I analyzed whether animal advocates should work on decreasing aquatic noise as it potentially stresses wild fish.

My tentative conclusion is that most likely, ocean noise interventions wouldn’t be cost-effective compared to current farmed animal interventions, although there is some small chance that it’s more cost-effective than them. I attempted to analyzed the cost-effectiveness of a campaign prevent seismic surveys to find oil and gas, but conclusions seem too uncertain to inform decisions.

Pursuing wild animal welfare (WAW) interventions that are also good for the environment and conserving species could help the WAW movement find allies in these important fields. Also, it might be good for the reputation of the WAW movement to have a concrete intervention that they could to point to.

Context and conclusions

I’ve spent some months trying to find a wild animal welfare (WAW) intervention that is:

Tractable (can in principle be funded >$100K/yr starting in 2023 even if we choose not to do so),

Non-controversial (>40% support and <30% oppose in a US poll), and

Directly cost-effective (10%+ as cost-effective in expectation as chicken welfare corporate campaigns).

The intervention that seemed most likely to satisfy these criteria was trying to reduce aquatic noise. I think that most likely, it is much less than 10% as cost-effective as chicken welfare reforms, but there is a small chance that some aquatic noise interventions could be even more cost-effective than corporate chicken welfare campaigns. Also, perhaps the cost-effectiveness bar is too high because the last funded animal welfare intervention by EA donors might not meet it.

Aquatic noise would be my personal choice if someone decided that we should fund some WAW intervention now to grow the movement. But overall, it’s unclear whether the effective animal advocacy movement should spend resources on solving this issue. Experiments investigating how noise impacts species with very many individuals (such as arctic krill and lanternfish) might inform this decision. But based on my conversations with aquatic noise experts, such experiments are likely to cost hundreds of thousands of dollars and might have inconclusive results.

I think that it could be worthwhile for one or two people who care about WAW to try to have an academic career in aquatic noise (if there are no such people already), or to do WAW outreach to academics already working in the area. This could help a lot to identify how promising this cause is from the WAW perspective and to identify the best opportunities to help animals.

What aquatic noise is, and why I thought reducing it could be a promising WAW intervention

The main sources of aquatic noise are:

Ships and boats

Seismic surveys (usually to find oil and gas)

Sonar

Pile driving and other offshore construction

Wind farms

Acoustic deterrent devices used by fish farms

Dynamite fishing

This list is in the order of importance that many articles seem to give to each source (e.g., Duarte et al. (2021), Širović et al. (2021), Hildebrand (2004), Williams et al. (2018)), often implicitly. I haven’t seen any analysi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Probability distributions of Cost-Effectiveness can be misleading, published by Lorenzo on July 18, 2022 on The Effective Altruism Forum.

TL;DR:

The "mean" cost-effectiveness of interventions with uncertain impact can be misleading, sometimes significantly.We want to consider mean(cost)mean(effect), not mean(costeffect)

Epistemic status (how much I'm sure of this): I'm pretty confident about the main claim, but still confused about the details, I end the post with some questions.

Minimal extreme example:

Let's say that you have a magical intervention that has:

33% of saving 1 life

33% of saving 100 lives

33% of saving 199 lives

All for the known cost of $10,000.It would be an amazing intervention! If you run hundreds of similar interventions, you can save lives with cost-effectiveness of $100/life: the expected value is 100 lives saved, and the cost is always $10,000.

But here is what happens if you model it in Guesstimate:

You get $3400 mean cost per life! Underestimating the actual effectiveness by a factor of 34!

This is obvious in hindsight: since Guesstimate shows the "mean" cost-effectiveness

mean(costeffect)=mean(100001,10000100,10000199)≈3400

instead of what we care about, which is:

costeffect=costNeffectN=mean(cost)mean(effect)=10000mean(1,100,199)=100.

Looking at the 5th and 95th percentile helps in many cases, but not in scenarios where there is a very small probability of very high effects and a significant probability of small effects. Minimal Guesstimate example with 4.8% of saving 1000 lives and 95.2% of saving 1 life.

Some practical examples of very small chances of huge value might be deworming or policy interventions. For those, mean(cost/effect) and mean(cost)/mean(effect) might differ by orders of magnitude.

Three recent examples:

(search for "EDIT 22/06/2022:" in this post, it changes a result by an order of magnitude)

/ (in this case the difference is smaller, 54 vs 31, see )

(19 vs 12, see )

If you want to check another Guesstimate model

For most models, you can just manually calculate mean(cost)/mean(effect), since the means are shown in the Guesstimate UI.For more complex cases, if you are comfortable with python, you can port a guesstimate model to numpy using/ and add .mean() liberally (very MVP, let me know if it doesn't work with a model you want to try).

Possible solutions / mitigations:

If you're interested in a single number for some sense of "expected cost-effectiveness", get the expected value and the expected cost and divide those numbers instead of the distributions (if the distributions can be considered independent).

If costs are constant in your model, consider looking at the value per dollar (or per $1,000) instead of dollars per value, so the denominator is constant. The minimal example would become

Other ideas? I'm definitely not an expert in any of this and there's probably a nice mathematical/statistical solution that I can't think of! Please comment if you think of anything!

Some questions I still have:

How can we express the uncertainty around cost/effectiveness if the ratio distribution is hard to reason about and has misleading moments?

How could the UI in guesstimate or some potential alternative indicate to the user when to use mean(f(distribution)) and when to use f(mean(distribution)) for nonlinear functions, to prevent people from making this very common mistake?We might want to use the former for e.g. the value of cash transferseffect=∑klog(cash)→mean(effect)=mean(klog(cash))≠klog(mean(cash))

Really curious to know if anyone has ideas!

Huge thanks to Sam Nolan, Justis Mills, and many others for fleshing out the main idea, editing, and correcting mistakes.

Assuming independence between cost and effect

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Enlightenment Values in a Vulnerable World, published by Maxwell Tabarrok on July 18, 2022 on The Effective Altruism Forum.

Enlightenment Values in a Vulnerable World

Introduction:

The Vulnerable World Hypothesis: If technological development continues then a set of capabilities will at some point be attained that make the devastation of civilization extremely likely, unless civilization sufficiently exits the semianarchic default condition.

The Vulnerable World Hypothesis (VWH) is an influential 2019 paper by philosopher Nick Bostrom. It begins with a metaphor for technological progress: An urn full of balls representing technologies of varying degrees of danger and reward. A white ball is a technology which powerfully increases human welfare, while a black ball is one which “by default destroys the civilization that invents it.” Bostrom stipulates that “the term ‘civilizational devastation’ in the VWH refers to any destructive event that is at least as bad as the death of 15 percent of the world population or a reduction of global GDP by > 50 percent lasting for more than a decade.” Given the dire consequences of such a technology, Bostrom argues for enlarged state capacity, especially in terms of global reach and surveillance, to prevent the devastating technology from being invented.

The VWH is a wet blanket thrown over Enlightenment values; values which are popular with many EAs and among thinkers associated with progress studies such as David Deutsch, Steven Pinker, and Tyler Cowen. These Enlightenment values can be summarized as: political liberty, technological progress, and political liberty ⇒ technological progress. Even if technology has a highly positive expected value on human welfare this can be easily outweighed by a small chance of catastrophic or existential risk. The value of political liberty is often tied to its promotion of technological progress. Large risks from technological progress would therefore confer large risks on political liberty. Bostrom highlights this connection but goes further. Not only is political liberty dangerous because of its facilitation of catastrophic technological risk, but strict political control is good (or at least better than you thought it was before) because it is necessary to prevent these risks.

In response to a black ball technology Bostrom says that “It would be unacceptable if even a single state fails to put in place the machinery necessary for continuous surveillance and control of its citizens.” If Bostrom is right that even a small credence in the VWH requires continuously controlling and surveilling everyone on earth, then Enlightenment values should be rejected in the face of existential risk.

We do not know whether the VWH is true, and it is undecidable via statistical analysis until we draw a black ball or empty the urn. Thus, I consider the implications of the VWH for Enlightenment values both when it is false and when it is true. . If it is false then traditional arguments for Enlightenment values become even stronger. If the VWH is true I find that one can still reasonably believe that unconstrained technological progress and political liberty are important moral goods as both ends and means as long as some properties of the urn are satisfied. Even if these properties are not necessarily satisfied, I show that Bostrom’s proposed solution of empowering a global government likely increases existential risk overall.

Part 1: Outcomes Conditional on VWH Truth Value

VWH Is False

First, we can quickly consider what we should do if we knew that the VWH was false. In this case, the arguments made by progress studies in support of the set of Enlightenment values: political liberty, technological progress and political liberty ⇒ technological progress, are proved stronger. Since we know that there are no bla...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quantilizers and Generative Models, published by Adam Jermyn on July 18, 2022 on The AI Alignment Forum.

Thanks to Evan Hubinger for discussions about quantilizers, and to James Lucassen for discussions about conditioned generative models. Many of these ideas are discussed in Jessica Taylor's Quantilizers: A Safer Alternative to Maximizers for Limited Optimization: this post just expands on a particular thread of ideas in that paper. Throughout I’ll refer to sections of the paper. I have some remaining confusion about the “targeted impact” section, and would appreciate clarifications/corrections!

Abstract

This post explores the relationship between quantilizers and generative models. My main takeaways are:

A natural way to build a quantilizer is by sampling from an appropriately-conditioned generative model.

Unfortunately quantilizing doesn’t seem to confer much advantage over the underlying generative model: to the extent that a quantilizer is more powerful than a generative model, it’s more dangerous, and vice versa.

Quantilizing is pretty computationally expensive relative to the advantage it brings, making it unclear if this is a competitive approach even if it conferred a net safety advantage at fixed power.

Definitions

I’ll follow the setup in “Quantilizers: A Safer Alternative to Maximizers for Limited Optimization” (Section 1, Definition 1) and say that a quantilizer is a model with:

A base distribution γ, which describes a set of outputs we think are “normal” or “unlikely to lead to doom”.

A quantile q∈[0,1].

An ordering over outputs (e.g. a utility or loss function).

A quantilizer returns a random output from the top q fraction of outputs from γ according to the ordering. So for instance a 0-quantilizer is a utility maximizer, a 0.1-quantilizer returns outputs in the best 10%, etc.

Where does the base distribution come from?

One challenge with quantilizers is that they require a base distribution γ that we trust. That is, we want to be able to sample from γ many times and never hit bad outputs. Unfortunately, we don’t know how to explicitly write this distribution down.

A natural alternative to explicit construction is to train a generative model on the distribution of actions a humans takes (Section 3.1). Then for any given task γ can be written as

This seems quite doable. Train GPT-N on text written by humans, then prompt it with the task:

Prompt: You are a good person who wants to accomplish . What follows is your description of how you intend to do this.

The output will probably be text describing how a good person would accomplish that task!

Unfortunately we might get unlucky and sample a person who isn’t so good (but thinks of themselves as good). That still satisfies the prompt. Robustly getting the agent we want seems hard, but might be possible in the limit where we ask for agents that are generally plausible in the training distribution of the generative model.

Power vs Safety

Armed with our base distribution, we can start quantilizing. Naively we can do this by sampling q−1 times from γtask, then returning the top output. This produces a tradeoff between capabilities and safety.

Single Sampler

In the limit q→1 we’re just returning actions a good human would take. That is, our quantilizer just samples once and returns that sample as output, so it’s no more powerful than a random good person.

Such a model is really unlikely to take over the world, but it’s also really unlikely to solve quantum chemistry problems. This is because both of these outcomes unlikely according to the base distribution.

We can improve things a bit by conditioning on being an unusually capable person, e.g.

Prompt: You are a good and extremely capable person who wants to accomplish . What follows is your description of how you intend to do this.

Now we can get top-end human ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Law School: Why and When? – Considerations for Members of the EA Community, published by Locke USA on July 18, 2022 on The Effective Altruism Forum.

In the United States, it’s common for policymakers and other people in positions of social influence to have law degrees. The career capital that law school provides could be valuable for several promising career paths, including paths related to AI governance, biorisk reduction, and animal welfare advocacy.

With those potential benefits in mind, should you (or should others in the EA community) go to law school? This post, which focuses on law schools and career paths in the United States, describes the law school experience and outlines considerations for readers deciding whether obtaining a law degree might help them have the greatest positive impact through their careers.

Companion posts will offer (1) tactical advice for navigating the law school admissions process and paying for a legal education, and (2) overviews and tentative evaluations of several post-law school career options.

If you are interested in getting in touch with members of the EA community who have attended, currently attend, or are considering law school, we encourage you to fill out this short survey. We will use your answers to try to connect you to people with knowledge or networks relevant to your specific interests and career stage.

Summary

Law school is a promising route into high-impact careers in policy and government, especially for people interested in eventually holding senior roles for which formal credentials are highly valued.

Law school is a necessary step to working as a lawyer, which includes some roles that are underexplored but plausibly high-impact, such as becoming a litigator (a lawyer who works on cases that go to court) focused on animal welfare or biosecurity, or designing contracts and governance structures in high-stakes settings.

Some common advice about law school may apply differently to people aiming at policy roles, rather than roles practicing law. For example, earning top grades in law school appears not to be necessary for most policy roles; using marginal hours to pursue relevant experiences and research may be more valuable for people targeting those paths.

Attending law school is not the best path for earning to give; most people considering law school for this reason probably have more promising options that do not require the time and expense of earning a law degree. There is a relatively low ceiling for how much a person can earn in law, where compensation is based on salaries, hourly rates, or contingency fees; roles in business and finance, where some people are compensated in equity, have a much higher ceiling for the highest earners.

Law school is not a good fit for everyone; for example, it is probably not the right choice for people: who don’t have a clear picture of how a law degree would help them have the kind of positive impact they hope to have in the world, who already have a strong personal fit for a high-impact role working on a pressing problem for which there is a short timeline for effective action, who have significant personal commitments that would make spending 50+ hours per week on schoolwork an unusually costly investment, or for whom prolonged periods of stress or frequent discouraging feedback are especially costly.

Readers can test their potential fit for law school by speaking to a current law student or a recent graduate who holds a position they are excited about, attending a law school class, watching a video of a law school lecture, reading a book about the law school experience, or taking a look at a case that they would likely read while in law school.

If you do decide to go to law school, it is valuable to go in with a clear plan about how you will use your legal education ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Effective Altruism as "nish kam karma yoga" [Larry Brilliant], published by annaleptikon on July 17, 2022 on The Effective Altruism Forum.

I recently read Larry Brilliant's memoir about eradicating smallpox. "Sometimes Brilliant" comes with a subtitle that already suggests that his life took an unforeseeable turn from spirituality to a life of pragmatic service: "The Impossible Adventure of a Spiritual Seeker and Visionary Physician Who Helped Conquer the Worst Disease in History". He, among many other things, is an epidemiologist and was part of the successful WHO Smallpox Eradication program. Larry Brilliant is also one of the founders of the (EA recommended) Seva Foundation and an advisor on Pandemic Preparedness.

As expected, I learned a lot about the history of smallpox and the WHO Smallpox Eradication Program. I learned a lot about Dr. Brilliant as a person.

Engaging with the book also reconnected me to my own spiritual longing and the existential questions that brought me to effective altruism myself. Engaging with these foundations again added to my own sense of meaning in the world, so I'm sharing it for others to read, too. I'd love to learn the technical terms of equivalents to this concept from other wisdom traditions. If you have texts that beautifully convey the principles in a story, that'd be wonderful to read.

Here is an excerpt that easily conceptualizes effective altruism as the practice of "nish kam karma yoga":

[Larry Brilliant, initiated by his wife, sought spiritual insight from Maharaji, an Indian guru]

Maharaji rarely said "do this" or "do that" or indicate that one thing was right or another wrong. He taught by parable or by having us focus on a verse from the Gita."See how Krishna tells Arjuna that not even God can take time off and not work," he would say. "God must be in the world. You must be in the world, not hiding away in a cave. Work to help relieve suffering, but don't get a big head." I took this to mean that I wasn't good enough for spiritual development through meditation, or dhyana yoga. I did not understand that Maharaji was teaching me about another form of yoga, karma yoga, working to be one with God through work in the world."Wake up, wake up," he would say in a high-pitched staccato. "Meditation, devotion, and worship are all good. Very, very good. Do these upaya — these methods. But for you, not only meditation or devotion or asanas [postures]. Your yoga is nish kam karma yoga. You will do service, but avoid praise, and give the fruits of your labor to God.

Don't get excited about your role. That is your dharma. Your path is working in the world, not in meditation. You will find your dharma when you get your UN job. Don't get a big head."

[Upon seeing his first smallpox case]

"Bill," I shouted, too loudly in the quiet room, "we'll lose this girl. She probably has pneumonia on top of everything else, and her lesions are getting infected. She needs serious attention or she will die. Who do we call? How do we get her to the hospital?"My agitation was making everyone uncomfortable, especially the family and the Indian doctors."Larry, slow down," Bill said. "We've all been in the place you are in right now. We all wanted to call the ambulance the first time we saw smallpox at this stage of the disease. But there is no ambulance. There is no treatment."Bill motioned me toward the door. "Because there is nothing anyone can do," he said when we got outside, "people all over the world attribute smallpox to a goddess or supernatural force. They have smallpox goddesses in Nigeria and Japan too. The only thing we can do, and the only thing we must do, is prevent another child from getting this disease.

That means we have to find every case, because once there is onecase, there will be a second, a third, and a fourth. We must not let t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Generative Models for Alignment, published by Arun Jose on July 18, 2022 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program. It builds on work done on simulator theory by Kyle McDonell and Laria Reynolds, who came up with the strategy this post aims to analyze; I’m also grateful to them for their comments and their mentorship during the AI Safety Camp, to Johannes Treutlein for his feedback and helpful discussion, and to Paul Colognese for his thoughts.

TL;DR: Generative models don’t operate on preferences in the way agents or optimizers do, instead just modelling a prior over the world that can return worlds satisfying some conditional, weighted by how likely they are given this prior. This implies that these models may be safer to use than more standard optimizers, although we would likely use them in different ways. For example, at the limit of capability we could use it to do alignment research by simulating a world with a superhuman alignment researcher. That isn’t to say that these models are without their own problems however, carrying with them versions of the outer and inner alignment problems that are subtly different from those of optimizers. This post explores how generative models are different, ways in which we might use them for alignment, some of the potential problems that could come up, and initial attempts at solving them.

How do generative models work?

Generative models are trained by self-supervised learning to model a distribution of data. With large language models, the real distribution underlying the textual data comprising its training corpus represents the world as it is now, in theory. As they get stronger, we can assume that they’re getting better at modelling this world prior, as I’ll describe it going forward.

We have the ability to prompt these models, which, at a low-level, means that we can give it the first part of some text that it has to complete. High-level, this means that we can describe some property of the world that may or may not exist, and the model samples from the world prior and uses a weighted distribution of the likely worlds that satisfy that property (for simplicity at the cost of technical accuracy, you can imagine it simulating the most likely world satisfying the property). For example, if you give it a prompt that says an asteroid is about to impact the world and finishes with something like “The following is an excerpt from the last public broadcast before the asteroid struck”, then the model simulates a world where that is true by modelling the way the most likely ways our world would turn out in that conditional.

Powerful generative models would be able to do this with high fidelity, such that their generations would, for example, account for the changes to society that would occur between now and then and structure the broadcast and its timing (how long before the strike does it happen if communications go down?) accordingly.

In other words, generative models have the advantage of being strongly biased toward the prior distribution of the universe. What this means is that there’s a strong bias in its outputs to remain close to the universe described by the training data - this could be viewed as a posterior sampling over all possible universes after updating on the training data, subject to simplicity bias, but in practice it wouldn’t be updating on a prior over all universes but building an incremental understanding of the universe given each datum. In GPT-3’s case, for example, this would be the universe that’s described by the corpus it was trained on.

Therefore, conditionals applied to it sort outcomes based on likelihood, not optimality - aski...