Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Training goals for large language models, published by Johannes Treutlein on July 18, 2022 on The AI Alignment Forum.

This post was written under Evan Hubinger’s mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program. Many of the ideas in this post, including the main idea behind the training goal, are due to Kyle McDonell and Laria Reynolds. In addition, I am grateful for comments and feedback from Arun Jose (who wrote a related post on conditioning generative models for alignment) and Caspar Oesterheld, and for a helpful discussion with James Lucassen.

Introduction

Large language models (LLMs) have recently enjoyed much success, e.g., achieving 50% accuracy on high school math competition questions. These models can solve various tasks using the right prompts or fine-tuning, such as translation, summarization, or question answering. One path to human-level and potentially superhuman AGI might be scaling up LLMs. This raises the question of what an approach to aligned AGI would look like based on these models.

One hypothesis is that, while LLMs are very competent, they are not adequately described as agents. Instead, one might describe them as myopic simulators that model a distribution over text, without understanding their place in the world or their actions' causal impact on it. For this reason, such models might be safer to use than more agentic models that pursue goals in the world.

In this post, I develop a training goal for LLMs, in Evan Hubinger’s terminology. A training goal is a description, as concrete and low-level as possible, of the algorithm that the model should implement, and an explanation of how the model will be used and why it will be aligned for that purpose. I focus on models that generate a distribution over text, conditioned on a prompt, but that are more capable than today’s models. My goal is to provide an overview of potential problems and solutions, most of which have been raised in prior work.

I will not focus on a training rationale for such models, which is the question of how to train a model to implement the described algorithm using machine learning. However, the proposal is only useful if one could actually train such an algorithm, so I will still touch on training. I also won’t discuss the proposal’s competitiveness, and I won’t look into specific prompts one might use. Lastly, note that I am not an expert on LLMs and will mostly analyze things from an abstract and informal perspective.

Using large language models for alignment

To begin, I broadly describe the setup I have in mind. The approach discussed here can be understood as a kind of oracle AI. I focus on modeling text, instead of e.g. answering questions truthfully or predicting the future, because I can more concretely imagine such an AI and its training setup, given that current LLMs are already doing this. Moreover, modeling a distribution over text is better specified, and it is thus less demanding: when answering questions truthfully, answers could be misleading if they are not optimized to be helpful to humans. For instance, an AI optimizing for giving accurate answers may focus on irrelevant details that are easy to predict but not useful to a human. Just modeling text, on the other hand, does not require the AI to learn human intent explicitly. The downside is that it will generally be less useful than an AI trying to be directly helpful to humans.

I am considering a model like GPT-3, which can sample text completions given a prompt. To do so, the model outputs a distribution over tokens, conditional on previous text. I interpret the model as representing a joint distribution over text, which is factorized into conditional distributions over tokens, where the conditioning works in the normal Bayesian way.

The m...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A distillation of Evan Hubinger's training stories (for SERI MATS), published by Daphne W on July 18, 2022 on The AI Alignment Forum.

This post is a distillation of Evan Hubinger's post "how do we become confident in the safety of a machine learning system?", made as part of the summer 2022 SERI MATS program. While I have attempted to understand and extrapolate Evan's opinions, this post has not been vetted. Likewise, I use training stories (and contribution stories) to describe the methodology of proposals for safe advanced AI without the endorsement of those proposals' authors and based on a relatively shallow understanding of those proposals (due to my inexperience and time constraints). The opinions presented in this post are my own unless otherwise noted.

Epistemic status: Exploratory

Some day, all too soon from now, people will deploy the AI that seals humanity's fate. There are many scenarios for how this comes about or what happens afterward. Some have multiple AI negotiating or fighting for dominance, others one. Some think the handoff of power from humans to AI will go slowly, others fast. But whatever the case, there is one question that these people need to get right: "Is the AI we're about to deploy safe?"

For people to get this answer right when it matters, two things need to happen: we need tools to accurately determine whether an advanced AI is safe, and we need an advanced AI those tools approve of that we can deploy in time for it to matter.

Training stories are a tool to evaluate proposals for making this happen. Specifically, they're meant to analyze complete proposals for training prosaic advanced AI: advanced AI which has similar enough architecture to current systems that most research is cross-applicable. The reason to focus on complete proposals is that it gets us to pay attention to what will be important when push comes to shove. From there, we can backpropagate and get an estimate of what research paths are most beneficial. Evan thinks theoretical prosaic alignment is the most beneficial in expectation for him and many others to pursue, though he supports a broad spectrum of research.

Because I think the original post describes the central concept of training stories very well, I have avoided retreading the same ground, and instead focused on expanding its usability. I will start with some more case studies of using training stories to analyze proposals, and then expand on the original post with contribution stories: a method for analyzing research agendas by how they factor into training stories.

Training stories

Case study: Imitative amplification + intermittent oversight

In this case study, Evan's comments in An overview of 11 proposals for building safe advanced AI map well to the training story evaluation, and I don't feel like much was missing. However, the training story itself makes it more clear to me what the training goal might look like, which could help point at potential experiments with toy models that can be done today. This demonstrates how training stories can be used to clarify a path towards working AI.

Training goal specification: The goal is to end up with a model M which accurately answers whether a model similar to M is safe to run, which suggests possible tests to determine if that model is safe to run, and which answers questions about some other topic T. This can take several forms, but let's try to work one out:M can't take all weights of M as input for inspection, so M has to look at sections of a model and share a summary of its conclusions with other M's to form a reliable answer. This could be organized like HCH. Since M is trained to imitate the output of Amp(M), which is prototypically a human consulting M and some selection of transparency tools, these summaries would be human-legible.Mechanically, M woul...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What should you change in response to an "emergency"? And AI risk, published by AnnaSalamon on July 18, 2022 on LessWrong.

Related to: Slack gives you the ability to notice/reflect on subtle things

Epistemic status: A possibly annoying mixture of straightforward reasoning and hard-to-justify personal opinions.

It is often stated (with some justification, IMO) that AI risk is an “emergency.” Various people have explained to me that they put various parts of their normal life’s functioning on hold on account of AI being an “emergency.” In the interest of people doing this sanely and not confusedly, I’d like to take a step back and seek principles around what kinds of changes a person might want to make in an “emergency” of different sorts.

Principle 1: It matters what time-scale the emergency is on

There are plenty of ways we can temporarily increase productivity on some narrow task or other, at the cost of our longer-term resources. For example:

Skipping meals

Skipping sleep

Ceasing to clean the house or to exercise

Accumulating credit card debt

Calling in favors from friends

Skipping leisure time

If I would strongly prefer to address some situation x before time t, I may sometimes want to "borrow from the future" like this. But the time-scales matter. If I’m trying to address x as much as possible in the next five hours, skipping sleep may make sense. If I’m trying to address x as much as possible over the next year, I’ll probably do better to get my usual amount of sleep tonight. Something similar (with different, resource-specific timescales) will hold for other resources.

So, in short time-scale emergencies, it’ll often make sense to suspend a great deal of normal functioning for a short period of time. In longer time-scale emergencies, your life should mostly look closer to normal.

Principle 2: It matters how much we know how to address the emergency

Much of what we do in daily life – especially when we’re feeling “free” and “unstuck,” and as though there is nothing in particular that we “have to” do – has the effect of making us well-resourced and capable in general. For example, by default, a lot of us would spend a lot of time reading interesting books of varied sorts, nerding out about interesting topics, trying our hands at new skills and crafts, etc. Also, we often like making our living spaces nicer (and more functional), forming new friendships, and so on.

If there is a particular thingy that matters hugely, and if you have an accurate model of how exactly to change that thingy, it may make sense to sacrifice some of your general-purpose capacities in trade for increased ability to address that thingy. (E.g., if you know for a fact that you’ll lose your home unless you pay the mortgage, and if keeping your home is important, it may make sense to trade general-purpose capacities for the ability to make mortgage payments by e.g. working over-long hours at a mind-numbing job that leaves you stupid.)

However, if you don’t have an accurate map of how to address a given thingy, then, even if the thingy is very important, and even if its time-scale is short, you’ll probably mostly want to avoid sacrificing general-purpose capacities. (In a low-information context, your general-purpose capacities are perhaps more likely to turn out helpful for your very important thingy than whatever you’d be trading them off to buy.) Thus, in “emergencies” where you do not have an accurate map of how to solve the emergency, your behavior should probably be more like normal than in better-mapped emergencies.

Side-note: “Emergencies” as wake-up calls

A different way an “emergency” can sometimes rearrange priorities is by serving as a “wake-up call” that helps people peel away accidental accumulation of habits, “obligations,” complacency, etc. For example, I’m told that near encoun...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Examples of AI Increasing AI Progress, published by ThomasWoodside on July 17, 2022 on LessWrong.

Recursive self-improvement is already here.

This point is far from original. It’s been described before, for instance here, in Drexler’s Reframing Superintelligence, and (as I was working on this post) in Jack Clark’s newsletter and even by Yann LeCun. But sometimes I still hear people talk about preparing for “when recursive self-improvement kicks in,” implying that it hasn’t already.

The kinds of recursive self-improvement mentioned here aren’t exactly the frequently-envisioned scenario of a single AI system improving itself unencumbered. They instead rely on humans to make them work, and humans are inevitably slow and thus currently inhibit a discontinuous foom scenario.

It may be tempting to dismiss the kind of recursive self-improvement happening today as not real recursive self-improvement. To think about it as some future event that will start to happen that we need to prepare for. Yes, we need to prepare for increasing amounts of it, but it’s not in the future, it’s in the present.

Here are some currently existing examples (years given for the particular example linked):

(2016) Models play against themselves in order to iteratively improve their performance in games, most notably in AlphaGo and its variants.

(2016) Some neural architecture search techniques use one neural network to optimize the architectures of different neural networks.

(2016) AI is being used to optimize data center cooling, helping reduce the cost of further scaling.

(2021) Code generation tools like GitHub Copilot can be helpful to software engineers, including presumably some AI research engineers (anecdotally, I’ve found it helpful when doing engineering). Engineers may thus be faster at designing AI systems, including Copilot-like systems.

(2021) Google uses deep reinforcement learning to optimize their AI accelerators.

(2022) Neural networks, running on NVIDIA GPUs, have been used to design more efficient GPUs which can in turn run more neural networks.

(2022) Neural networks are being used for compiler optimization in the popular LLVM compiler language, which Pytorch’s just-in-time compiler is based on.

Inspired by Victoria Krakovna’s specification gaming spreadsheet, I’ve made a spreadsheet here with these examples. Feel free to submit more here. I think the number of examples will continue to grow, making it useful to keep track of them.

If this feels underwhelming compared with the kinds of recursive self-improvement often written about, you’re right. But consider that the start of an exponential often feels underwhelming. As time goes on, I expect that humans will become less and less involved in the development of AI, with AI automating more and more of the process. This could very well feel sudden, but it won’t be unprecedented: it’s already begun.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How Interpretability can be Impactful, published by Connall Garrod on July 18, 2022 on The AI Alignment Forum.

This post was written as part of the Stanford Existential Risks Initiative ML Alignment Theory Scholars (MATS) program. thanks to Evan Hubinger for insightful discussion.

Introduction

Interpretability tools allow us to understand the internal mechanism and knowledge of a model. This post discusses methods through which interpretability could reduce x-risk from advanced AI. Most of these ideas are from Neel Nanda’s post on the same topic, my contribution comes from elaborating on them further. I have split these into a handful of different categories. These are not disjoint, but the framing helped with the analysis. The last section includes several reasons why interpretability might not be impactful that apply, for the most part, across the board. The conclusion discusses what this analysis implies for how we should approach interpretability research today.

In evaluating a method of impact, I have tried to analyse the following:

Mechanism & Scale: What is the mechanism of impact? how valuable would this be?

External considerations: what things outside of our interpretability tools need to happen in order for this impact to be realised and how likely are they?

Side effects: what possibility is there of a negative impact on x-risk?

Requirements: what do we require of our interpretability tools, as measured by:

Understanding: what properties of our model do we need to be able to understand?

Competitiveness: how cost-effective do our tools need to be?

Performance impact: to what extent can our tools impact the final model performance?

Reliability: how reliable must our tools be?

Typology: what types of interpretability tools can we use?

With regards to understanding, it is useful to split this into two classes: comprehensive tools, which let you evaluate properties of the full model, and partial tools, which let you evaluate properties of only a part of the model. This distinction is the same as worst-case transparency introduced by Evan. Both of these define a spectrum based on what the class of properties that you can evaluate is. I will refer to the size of this class as the ‘strength’ of the tools.

With regards to reliability, this captures whether the tools work across different models, as well as under gaming attempts from deceptive models or overfitting attempts from SGD.

When referring to type, I have in mind the transparency trichotomy. Specifically, we can think of our tools as belonging to one of three types: inspection tools, which can be applied to a model to make it interpretable, training-enforced tools, which can be used in training to encourage the final model to be interpretable, and architecture tools, which are architectures that make the model inherently interpretable.

I hope this post will provide clarity on what kinds of interpretability research we should work on, and allow comparison of interpretability research to other mechanisms for impacting x-risk from AI.

Auditing and oversight

One set of ideas spoken about by Chris Olah is to use interpretability tools to look for unacceptable properties. There are a few ways we can imagine doing this.

Auditing the final model

Suppose we have trained a model on an objective. If we had suitable interpretability tools, we could use them to inspect the model to see if it is aligned. For example, if we had a proxy aligned system, our tools could recognise it is not following the correct objective. We can then try to correct this, or throw the model out and start again. The hope would be that inner alignment failures can be caught in this way, allowing us to never deploy a model that is not inner aligned.

In order for this auditing to be fool proof we need to have comprehensive tools, since any part of...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why you might expect homogeneous take-off: evidence from ML research, published by Andrei Alexandru on July 17, 2022 on The AI Alignment Forum.

This write-up was produced as part of the SERI MATS programme under Evan Hubinger’s mentorship. It is also my first post on LW, so feedback is very welcome!

Introduction

This article aims to draw a connection between recent ML research and the claim that future advanced AI systems may be homogenous. First, I briefly review this article, where the idea of homogenous take-off is introduced. Then, I outline two different arguments why you might update in the direction of homogenous take-off. For each of the arguments I mention key uncertainties that I have about the argument itself, as well as broader open questions.

TL; DR

I present two reasons to believe that as models become larger they also become more homogenous, i.e. they behave more similarly to each other:

Variance between models behaves unimodally in the overparameterised regime: it peaks around the interpolation threshold, then decreases monotonically. Decreased variance means that models make similar predictions across different training runs (captured as variance from initialisation) and different sampling of the training data (variance from sampling);

Neural networks have a strong simplicity bias even before training, which might mean that multiple training runs with different hyperparameters, initialisation schemes etc. result in essentially the same model.

I’ve somewhat updated in the direction of homogenous take-off as a result of these arguments, though I think that there are still ways in which it’s unclear if e.g. decreasing variance with size rules out heterogeneity.

What’s homogeneous take-off?

There are several axes along which different AI takeoff scenarios could differ: speed, continuity, and number of main actors. Homogeneity vs. heterogeneity in AI takeoff scenarios introduces a new way to look at a potential take-off, through the lens of model homogeneity. Homogeneity intuitively refers to how similar models are at any given time given some definition of similarity. We might specifically refer to homogeneity with regards to alignment, which, again intuitively, means “models are more or less aligned to the same degree” (or: “aligned models will not coexist with unaligned models”).

More formally we mean something like models having similar properties, e.g. alignment properties. In my mind, an alignment property might be something like “corrigibility” or “truthfulness”, though it’s unclear to me to what extent two models which are, say, truthful, are also homogenous. I think working toward a clearer, more precise definition of homogeneity is probably useful in determining what actually counts as evidence for homogenous systems being more likely, though I don’t try to do so in this write-up.

The article sets out a list of arguments supporting the idea of homogenous take-off, which I parse as “evidence from the economics of large scale machine learning”. Without going into too much detail – I recommend reading the original article for the full arguments –, these are:

Training a model is more expensive than running it. This is a relatively straightforward claim which extrapolates from the landscape we have today, where some large language models reportedly have had training budgets in the millions of US dollars, with comparatively little cost to run inference/serve the models themselves once trained.

Training models from scratch is not competitive once the first advanced system is released. To me this follows from 1., in the sense that if it is economically useful to deploy more than one model simultaneously, it’s likely that the additional models will be copies of the original (perhaps fine-tuned on different tasks) rather than new models trained from scrat...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When Utilitarianism is Actually ✨Fun✨, published by Hamish Huggard on July 16, 2022 on The Effective Altruism Forum.

Utilitarianism can be a bummer.

The world has so many problems and there's so much you could be doing to help. Donating more. Advocating more. Researching more.

This can make me feel like:

I have to keep giving until there's nothing left for me

Even then, I can never live up to my moral obligations

I am therefore a bad person

But utilitarianism goes both ways. Sometimes the balance is in your favour. Sometimes utilitarianism is ✨Fun✨.

Some examples:

You're hanging out with your utilitarian friends. Alice has a really nice looking muffin. By your best objective estimation, you would enjoy the muffin more. Alice now has to give you the muffin or be sent to 👮 utilitarian jail 👮.

The EAG conference needs volunteers to do a thing. You could do it, but you're already really busy. You happen to know that Bob has a pretty light load and it would be easier for him. Utilitarianism requires you to try and make Bob do it instead of you. "Me? Bossy? Listen, this is out of my hands. Utilitarianism is making me make you do it."

You are depressed. You know that a walk and a swim would cheer you up. But what's even the point? You'd still be a failure and the world would still hate you. Well too bad bucko. Utilitarianism says you have to take care of your mental health. Utilitarianism needs you to be happy and radiating utils, even if you don't see the point right now.

You need help but are afraid to ask for it. Maybe you're down on your luck and need a couch to sleep on. Maybe an important person has some information you need. Maybe you're spiralling and need to unload your troubles. Utilitarianism doesn't care if you're "squeamish about unpayable debts", or "intimidated by status", or "don't want to be a burden". If someone can pay a small price to give you a big benefit, then Utilitarianism requires that you ask for help.

You can increase the total utility by increasing your own utility. In fact there are structural reasons why this might be a high-impact strategy: you have direct access to your own desires and emotional states, and it's easy to run intervention experiments on yourself. So apply the high-powered EA mindset to improving your own life. Think seriously about what you want out of life and how to get it. This can feel self-indulgent, unvirtuous, or even selfish. Doesn't matter. If you can make yourself better off without making anyone else comparably worse off, then utilitarianism says you have to do it. Last night I drank red wine and danced to Taylor Swift in my underwear and it was so much fun. The fact that I had never even tried this before in my 26 years of life makes me think that there’s tons of low-hanging fruit in the Personal Wellbeing Orchard.

For similar reasons, it can also be high-impact to improve the lives of friends and family. Develop a “value theory” for the people around you. What do they most want and care about? You’re allowed to ask them if you’re not sure, but take their reported values with a grain of salt. Look for low-effort, high-utility actions you could take. Be creative. When this works it feels like you’re winning good-will points for free. Example: my dad recently mentioned how much he enjoys being a retired grandparent, and I know he likes having photos of everything, so I took about five minutes over the course of the day to build a Google Photos album of him playing with my nephew, and he loves it. There’s a well of utility in actions which are 1) hard to do by yourself, and 2) not so valuable that you want to ask someone for it and enter an emotionally charged reciprocal transaction.

Compliments! It's nice to get a compliment, and it's fun to give a compliment. Free utility! Spam the compliment button! “Sure, I could ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A summary of every "Highlights from the Sequences" post, published by Akash on July 15, 2022 on The Effective Altruism Forum.

1

I recently finished reading Highlights from the Sequences, 49 essays from The Sequences that were compiled by the LessWrong team.

Since moving to Berkeley several months ago, I’ve heard many people talking about posts from The Sequences. A lot of my friends and colleagues commonly reference biases, have a respect for Bayes Rule, and say things like “absence of evidence is evidence of absence!”

So, I was impressed that the Highlights were not merely a refresher of things I had already absorbed through the social waters. There was plenty of new material, and there were also plenty of moments when a concept became much crisper in my head. It’s one thing to know that dissent is hard, and it’s another thing to internalize that lonely dissent doesn’t feel like going to school dressed in black— it feels like going to school wearing a clown suit.

2

As I read, I wrote a few sentences summarizing each post. I mostly did this to improve my own comprehension/memory.

You should treat the summaries as "here's what Akash took away from this post" as opposed to "here's an actual summary of what Eliezer said."

Note that the summaries are not meant to replace the posts. Read them here.

3

Here are my notes on each post, in order. I also plan to post a reflection on some of my favorite posts.

Thinking Better on Purpose

One difference between humans and mice is that humans are able to think about thinking. Mice brains and human brains both have flaws. But a human brain is a lens that can understand its own flaws. This is powerful and this is extremely rare in the animal kingdom.

Epistemic rationality is about building beliefs that correspond to reality (accuracy). Instrumental rationality is about steering the future toward outcomes we desire (winning). Sometimes, people have debates about whether things are “rational.” These often seem like debates over the definitions of words. We should not debate over whether something is “rational”— we should debate over whether something leads us to more accurate beliefs or leads us to steer the world more successfully. If it helps, every time Eliezer says “rationality”, you should replace it with “foozal” and then strive for what is foozal.

An 8-year-old will fail a calculus test because there are a lot of ways to get the wrong answer and very few ways to get the right answer. When humans pursue goals, there are often many ways of pursuing goals ineffectively and only a few ways of pursuing them effectively. We rarely do seemingly-obvious things, like:

Ask what we are trying to achieve

Track progress

Reflect on which strategies have worked & or haven’t worked for us in the past

Seek out strategies that have worked or haven’t worked for others

Test several different hypotheses for how to achieve goals

Ask for help

Experiment with alternative ways of studying, writing, working, or socializing

Luke Skywalker tried to lift a spaceship once. Then, he gave up. Instantly. He told Yoda it was impossible. And then he walked away. And the audience sat along and nodded, because this is typical for humans. “People wouldn’t try for more than five minutes before giving up if the fate of humanity were at stake.”

If a model can explain any possible outcome, it is not a useful model. Sometimes, we encounter a situation, and we’re like “huh. something doesn’t seem right here”. But then we push this away and explain the situation in terms of our existing models. Instead, we should pay attention to this feeling. We should resist the impulse to go through mental gymnastics to explain a surprising situation with our existing models. Either our model is wrong, or the story isn’t true.

Sometimes, we investigate our beliefs because we are “suppos...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [4] Climatic consequences of regional nuclear conflicts (Robock et al., 2007), published by Will Aldred on July 15, 2022 on The Effective Altruism Forum.

Will's quick summary and hot takes:

I've heard this Robock et al. paper get cited a lot, including twice at the SERI Conference '22. This is unfortunate, because while the paper makes important claims relating to nuclear x-risk, these claims are probably wrong.

To clarify, the paper being cited at all is not unfortunate, and the authors' writing of this paper started an important debate. For this, I commend them. However, I've almost always heard the paper's conclusion cited in isolation, without acknowledgement of the subsequent debate and counterconclusion. More on this below.

Robock et al.'s, paper, published in 2007, concluded that there'd be significant global cooling effects resulting from a regional nuclear war, e.g., between India and Pakistan.

"Significant global cooling effects" as Robock et al. refer to it means nuclear autumn bordering on nuclear winter.

Reisner et al., a separate group from Los Alamos National Lab, in 2018 published a paper titled "Climate Impact of a Regional Nuclear Weapons Exchange: An Improved Assessment Based On Detailed Source Calculations."

Given that the original Robock et al. paper was titled "Climate Impact of a Regional Nuclear Weapons Exchange," this was a spicy move.

The Reisner et al. paper effectively reruns the modelling done by Robock et al., but with some different and arguably more sophisticated starting assumptions.

Reisner et al. conclude that regional nuclear war would result in negligible global cooling.

Robock et al. responded to Reisner et al.'s critique, and Reisner et al. responded to the response.

Both groups make some valid arguments, but overall Reisner et al. are closer to the truth, according to me (or at least, the version of me at the time of writing).

To explain the heart of the disagreement, I'll first lay out my 3-step model of where disagreement occurs when talking about nuclear winter effects:

How much soot gets up into the stratosphere?

Given x quantity of soot in the stratosphere, how much cooling occurs?

Given y amount of cooling, i) how much crop failure? ii) how much human starvation?

It turns out that 2 is pretty uncontroversial, and 1 and 3 are where disagreements lie.

I'll return to 3 in other posts (e.g., I'll discuss Helfand, 2013 and Jagermeyr, 2020).

1 is what the Robock and Reisner groups are disagreeing over.

Reisner et al. write:

While our thorough simulations of the firestorm produce about 3.7 × 109 kg of black carbon, we find that the vast majority of the black carbon never reaches an altitude above weather systems (approximately 12 km). Therefore, our Earth system model simulations conducted with model-informed atmospheric distributions of black carbon produce significantly lower global climatic impacts than assessed in prior studies, as the carbon at lower altitudes is more quickly removed from the atmosphere. In addition, our model ensembles indicate that statistically significant effects on global surface temperatures are limited to the first 5 years and are much smaller in magnitude than those shown in earlier works. None of the simulations produced a nuclear winter effect.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Resolve Cycles, published by CFAR!Duncan on July 16, 2022 on LessWrong.

Epistemic status: Anecdotally strong

This technique was largely developed by Kenzi Amodei in the context of after-workshop followups and pair debugging. It has been refined and iterated, and has proven highly useful to our alumni, but all theorizing is post-hoc and untested, and direct research into (e.g.) an underlying theory of mind has yet to be done.

Consider the following scenarios:

You’ve been assigned a task that feels like it’s going to take about ten or fifteen hours of work, and you’ve been given three weeks to get it done (e.g. a document that needs to be written).

You’re facing a problem that you’ve tried solving off and on again for years, a problem that your friends and family never seem to run into (e.g. a struggle with motivation as you try to learn a new skill).

There’s a thing you need to do, but it seems impossibly huge or vague (e.g. to achieve your goals you’d need to found a company, emigrate to India, or cure a disease), and you don’t know where to begin.

You’re pretty sure you know all the steps between you and your goal, but there are about forty thousand of them (e.g. you’re hoping to run an actual marathon).

You’ve got a to-do list that’s long and growing, and you can only ever manage to get to the ones that are urgent (e.g. getting your car’s registration renewed, two months late).

Problems like the ones above can range from trivial to crucial, from simple to complex, and from one-time bugs to persistent, serious drains on your time, attention, and resources. There are a lot of elements in the mix—motivation, creativity, perseverance, prioritization—and a lot of justifiable reasons for thinking that solutions will be hard to come by.

Sometimes, though—despite every bit of common sense and experience telling us otherwise—those solutions aren’t hard to come by. Or rather, they might be hard, but they’re not elusive or mysterious or complicated.

The resolve cycle technique is one we offer up with a sort of shamefaced shrug, because it doesn’t sound like “real” applied rationality. It doesn’t have the rock-solid research underpinnings of TAPs or inner sim, or a carefully considered model like the ones behind turbocharging and double crux. It sometimes comes across like the worst possible advice—the sort of thing people say when they don’t actually want to help you with your problem:

“Have you tried setting a five-minute timer and just, y’know—solving it?”

But it works. Not always, not perfectly, but shockingly often and surprisingly well. And so we recommend that you suspend your disbelief (it’s justified) and put your objections on hold (we were just as incredulous as you are) and give it an actual, honest shot. In the worst case, if it does you absolutely no good, you’ve only wasted five minutes, and you’ve successfully exercised your Try Things muscles.

Post-hoc and half-baked

We’ll provide more detail in later sections, but the core of the technique— set a timer and solve your problem in five minutes or less—is extremely straightforward. The question is, why does this work? What’s going on?

We don’t have a complete answer yet, but we do have some quasi-models that pseudo- explain parts of what might be happening for some subset of hypothetical people (maybe).

If we look at the line above, it’s clear that, within the context of a given problem or project, we’d like to be operating as close to the upper end as possible. This is assuming that the project is genuinely important, that we aren’t in need of a break or a vacation, that we aren’t neglecting something else, etc.

There are situations which naturally bring out the Actually Try, such as deadline mode or emergencies, but ideally, we’d like to be able to access it at will, rather than by having to trick ourselves int...