Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ukraine Post #2: Options, published by Zvi on March 10, 2022 on LessWrong.

Or: Why the current situation once again proves the need for all of my pre-existing policy preferences.

Writing about the war is continuing to prove difficult, as things move quickly, the epistemic environment continues to be almost maximally hostile and a lot of elements are things where I was not previously up to speed.

I want to build a model of the situation, which leads to the problem of where even to begin, yet the speed premium beckons and although having a newborn is great as you can imagine I’m on less than ideal amounts of sleep and free time.

Thus it seems worthwhile to skip ahead a bit and ask the practical question first. Assume for the sake of discussion that we prefer people doing better and not dying to people doing worse and dying, and agree that we do not want Russia to win the war.

What would then be our options, either individually or collectively? What can we do?

(Where the collective ‘we’ here is some combination of ‘America’, ‘America + EU/NATO’ and ‘everyone everywhere who wants this same general set of outcomes.’)

I indeed strongly prefer such outcomes.

If you don’t prefer such outcomes enough to care, this post is not trying to convince you to change your mind and is probably not for you. That would be a very different post, and I don’t know what the True Objection would be of most people open to being convinced.

If you do prefer such outcomes, there are a lot of different potential approaches.

On a personal level, one could:

Provide military aid (e.g. by giving money directly to Ukraine).

Provide humanitarian aid (e.g. by giving money to those that provide it).

Lobby or advocate for better government policies or corporate actions.

Attempt to build (and ideally share) a better understanding of the situation.

Convince others to also prefer such outcomes, ideally via #4.

I am going to go ahead and advocate here for the policy responses I think are appropriate. The first ones are direct actions, which are pretty obviously:

Provide military assistance to the extent we can without too much escalation.

Humanitarian aid.

Economic sanctions.

The rest fall under the category of ‘things worth doing anyway.’ This stuff is win-win. This means doing things like:

Nuclear power.

Additional other energy production of various kinds.

Reconciling with other oil producers to extent possible.

Carbon tax.

Taking in as many Russians and Ukrainians as possible who want out.

Supporting free flow of information.

Everything else that’s obviously great (e.g. reduced zoning, more public transit, more building and urbanization, reduced occupational licensing, and so on.)

Use better decision theory.

The approach we’ve broadly collectively selected so far seems to be best centrally described as Cancel Russia. This is leading to many useful interventions, but also to actions that are counterproductive, and causing us to miss or not take full advantage of some of the biggest opportunities.

I won’t discuss potential peace terms and what I think would be worth or not worth taking, other than to point out that any offer that fails to leave Ukraine’s armed forces and ability to continue arming intact, or that would replace the government, is an obvious non-starter, but that I think it is wise that Zelensky is willing to consider territorial concessions, and that commitments to not ‘join blocs’ are mostly cheap talk given the history of what has come before, what alternative arrangements could be made and the prospects of joining short term in any case.

Let’s get the obvious first one out of the way first, which is Military Assistance.

Military Assistance

The obvious first thing to do is provide various forms of military assistance. Every little bit helps, both increasing the probability of better direct o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Samotsvety Nuclear Risk Forecasts — March 2022, published by NunoSempere on March 10, 2022 on The Effective Altruism Forum.

Thanks to Misha Yagudin, Eli Lifland, Jonathan Mann, Juan Cambeiro, Gregory Lewis, @belikewater, and Daniel Filan for forecasts. Thanks to Jacob Hilton for writing up an earlier analysis from which we drew heavily. Thanks to Clay Graubard for sanity checking and to Daniel Filan for independent analysis. This document was written in collaboration with Eli and Misha, and we thank those who commented on an earlier version.

Overview

In light of the war in Ukraine and fears of nuclear escalation, we turned to forecasting to assess whether individuals and organizations should leave major cities. We aggregated the forecasts of 8 excellent forecasters for the question What is the risk of death in the next month due to a nuclear explosion in London? Our aggregate answer is 24 micromorts (7 to 61) when excluding the most extreme on either side. A micromort is defined as a 1 in a million chance of death. Chiefly, we have a low baseline risk, and we think that escalation to targeting civilian populations is even more unlikely.

For San Francisco and most other major cities, we would forecast 1.5-2x lower probability (12-16 micromorts). We focused on London as it seems to be at high risk and is a hub for the effective altruism community, one target audience for this forecast.

Given an estimated 50 years of life left, this corresponds to ~10 hours lost, or ~6% of productive time lost per month given a 40 hour work week. The forecaster range without excluding extremes was <1 minute to ~2 days lost. Because of productivity losses, hassle, etc., we are currently not recommending that individuals evacuate major cities.

Methodology

We aggregated the forecasts from eight excellent forecasters between the 6th and the 10th of March. Eli Lifland, Misha Yagudin, Nuño Sempere, Jonathan Mann and Juan Cambeiro are part of Samotsvety, a forecasting group with a good track record — we won CSET-Foretell’s first two seasons, and have great track records on various platforms. The remaining forecasters were Gregory Lewis, @belikewater, and Daniel Filan, who likewise had good track records.

The overall question we focused on was: What is the risk of death in the next month due to a nuclear explosion in London?. We operationalized this as: “If a nuke does not hit London in the next month, this resolves as 0. If a nuke does hit London in the next month, this resolves as the percentage of people in London who died from the nuke, subjectively down-weighted by the percentage of reasonable people that evacuated due to warning signs of escalation.” We roughly borrowed the question operationalization and decomposition from Jacob Hilton.

We broke this question down into:

What is the chance of nuclear warfare between NATO and Russia in the next month?

What is the chance that escalation sees central London hit by a nuclear weapon conditioned on the above question?

What is the chance of not being able to evacuate London beforehand?

What is the chance of dying if a nuclear bomb drops in London?

However, different forecasters preferred different decompositions. In particular, there were some disagreements about the odds of a tactical strike in London given a nuclear exchange in NATO, which led to some forecasters preferring to break down (2.) into multiple steps. Other forecasters also preferred to first consider the odds of direct Russia/NATO confrontation, and then the odds of nuclear warfare given that.

Our aggregate forecast

We use the aggregate with min/max removed as our all-things-considered forecast for now given the extremity of outliers. We aggregated forecasts using the geometric mean of odds.

Note that we are forecasting one month ahead and it’s quite likely that the crisis will ge...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When did EA miss a great opportunity to do good?, published by James Ozden on March 9, 2022 on The Effective Altruism Forum.

Although EA is trying to do the most good possible, it's likely that we've overlooked or missed some of the best opportunities to do so. I'm trying to think of examples of when this has happened, in order to better understand our blindspots as a community. As a fictitious example, if we're continually overlooking media opportunities that then have a sweeping cultural effect on society, this is worth adjusting for to make better decisions in the future.

One more fleshed out but not quite perfect example might EAs not supporting Extinction Rebellion during its incubation phase in 2018. This might not a great example for a couple of reasons:

Whilst I think there's some evidence that Extinction Rebellion has been quite cost-effective in reducing carbon emissions, it's also not conclusive by any means.

There wasn't much EA grant making, if any, on climate related issues in 2018, which might actually be a larger blindspot of its own due to the increased interest in climate risks now.

Extinction Rebellion still ended up happening and being quite successful, so the counterfactual of EA not supporting this was seemingly okay (although we have no idea about how successful it could have been with greater support early on).

Ignoring these points and assuming this example was much stronger, the case would be something like: Extinction Rebellion had significant positive impacts on reducing carbon emissions, potentially outperforming other grantees of EA funding working on climate issues, yet we failed to identify this opportunity a priori. Not only did we fail to identify this, but other charitable organisations did, which indicates they had some information or connection that the EA movement was lacking. So the question becomes, why did we overlook this opportunity, and how do we stop it happening again?

One consideration to keep in mind is that if we can think of great things happened without the EA movement, then maybe it's fine as counterfactually our support wasn't needed for these interventions. So the more interesting question might be, what could have been extremely impactful in doing good, but failed to take off at all, and the EA movement could have changed this?

I've generally been framing this around specific interventions within a cause area that we may have missed e.g. interventions that could have been extremely impactful within AI risk or Animal Welfare we overlooked. However, this could also be true for cause areas, such as EAs updating towards being more concerned about climate change in the past few years, whilst some organisations were somewhat ahead of the curve relative to us (albeit potentially for different reasons to our interests in climate tail risks). This question might be more like: Were some organisations or institutions ahead of the curve in terms of cause prioritisation relative to EA, and why do we think this was?

Plus I worked for them for 2.5 years so there's probably a case of motivated reasoning.

In the specific case of Extinction Rebellion, my somewhat uninformed prior is that as a movement, EA is generally more focused on technocratic theories of change, which focuses on persuading or working with elites, rather than more democratic or people-powered approaches.

However, there's also the case that we could have scaled them up further or initiated them earlier, which is obviously also good.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meadow Theory, published by Duncan Sabien on March 9, 2022 on LessWrong.

Author's note:

The essay below centers mostly on a philosophy of parenting. However, it's not "about" parenting—rather, parenting is a domain in which the concepts I'd like to convey can be easily represented and productively applied. Parenting is one specific instantiation of the broader set of ideas, just as one might use an essay about football or baseball as a vehicle for the concepts of friction or acceleration or collision elasticity.

Author's note II:

Feedback on this draft ran thus:

[this essay seems to be missing something important, namely] stuff that would flip me from the primary activity of "i am trying to download Duncan's meanings into a sandbox in my mind and then make sense of them in the sandbox before i possibly do the further work of rendering them meaningful and/or useful to me if i have the tenacity for that", to the different primary activity of "i am using Duncan's guidance to recognize in my own head and from my own perspective the possibility of relating to the world in this new way, and i am personally engaged in the process of figuring out when and how and why it might matter to me"

I'm not sure how to get readers to do the latter thing; I think it is the correct thing to do and had sort of implicitly assumed that everybody already would. Since I don't know what changes to make to this draft to encourage the latter thing for readers like the one above, I'm just bluntly making the recommendation. Hopefully I will get better at helping in the near future.

I. The meadow

Imagine a wide, level, grassy meadow, stretching to the horizon, empty of all obstacles.

This meadow will be our metaphor for the world/the territory/reality-as-a-whole, and the picture of it will gradually grow more complex as we step through various thought experiments. But for the moment, it's just one big blank plane.

Now imagine a blindfolded child, running through the meadow.

This is safe, because the meadow contains nothing besides soft, springy grass. There are no walls or other obstacles to run into. If they trip or tumble, they'll be fine.

Running will be our metaphor for human activity. It is good to run. More precisely, it's good to be able to run—to have the freedom to move in whatever direction you please, at whatever speed you please, without fear or constraint. The ideal state is one in which the blindfolded child can run as much as they want to, wherever they want to, whenever they feel like.

(The blindfold is our metaphor for the state of human knowledge—our inability to perceive and comprehend the vast majority of what goes on around us, and our uncertainties and confusions about even the tiny slice we do manage to be aware of.)

II. The post

Now imagine a parent, lounging on the grass, idly supervising the child as they run.

The parent is not blindfolded, because the parent has a wealth of experience and knowledge (relative to the child). They can "see" the world more clearly, so in our metaphor they can straightforwardly see.

(Though they are still unable to see what's behind them, or what's very far away, or what they fail to properly attend to, or things that blend in with the grass or the sky.)

In the infinite meadow, the parent's job is extremely easy. The child is safe; they will run and play and eventually tire and come back to the parent for food and rest and conversation.

But imagine for a moment that the meadow is not empty. Imagine instead that it contains one single post, standing upright in the middle of everything.

Running into this post would not be pleasant. If the child does so, they could be seriously hurt, or possibly even killed.

A handful of possible responses to the inclusion of the post:

The parent might simply go sit by the post themselves, such that they could s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Comparison of Donor-Advised Fund Providers, published by MichaelDickens on March 9, 2022 on The Effective Altruism Forum.

Last updated 2022-03-09 to add Charityvest to the list of providers.

A donor-advised fund (DAF) is an investment account that lets you take a tax deduction now and give the money to charity later. When you give money to a DAF, you can deduct that money just as you would deduct a charitable contribution. The DAF invests the money tax-free. At any time, you can direct the DAF to donate some or all of its holdings to the charity of your choice.

You can open a DAF through a donor-advised fund provider. A provider charges an administrative fee to invest your DAF and make donations in accordance with your recommendations.

For donors in the United States, which DAF provider is the best?

The short answer:

All of the big DAF providers offer similar features. For most people, it doesn't really matter which one you choose.

If you already have a DAF, you might as well keep using it.

If you have a brokerage account at Fidelity, Schwab, or Vanguard, then the easiest thing to do is to open a DAF with your brokerage account. That way, you can manage all your investments in one place.

Otherwise, I believe Schwab Charitable is the best DAF provider for most people.

The long answer:

Even if all the major DAF providers are reasonably good, they do have their own strengths and weaknesses. In the rest of this post, let's look at how they compare.

Cross-posted to my website.

My process

I made a list of every United States nationwide DAF provider I could find. I excluded regional DAF providers (example: Silicon Valley Community Foundation), providers that weren't cause-agnostic (example: National Christian Foundation), and providers that don't work with individual donors (example: American Online Giving Foundation). Your local community foundation might offer a better DAF than any of the national providers of my list, but there are too many community foundations for me to look at them all.

I ended up with eight DAF providers (in alphabetical order):

American Endowment Foundation (AEF)

Charityvest (see disclaimer)

Fidelity Charitable

Greater Horizons

National Philanthropic Trust (NPTrust)

Schwab Charitable

T. Rowe Price Charitable

Vanguard Charitable

This list is probably not comprehensive, but it's all the DAF providers I could find that meet my criteria.

I spoke to representatives at these providers to get more information about their offerings. I also spoke to a few financial advisors who manage DAFs at different providers.

I then eliminated four DAF providers:

T. Rowe Price Charitable has the most investment options, but most of the funds have expense ratios of 0.6% or higher. Not worth it unless you really want those extra investment selections.

AEF and NPTrust both look like decent options, but they charge higher administrative fees than Vanguard/Schwab, offer a worse user experience, and don't have any special features to compensate. So we might as well exclude them.

Fidelity is nearly identical to Schwab, but it offers fewer investment options, so let's exclude it.

Four DAF providers remain: Charityvest, Greater Horizons, Schwab, and Vanguard.

Caveat 1: I don't have good firsthand knowledge of any of these eight DAF providers except for Fidelity Charitable, which I used to use, and Greater Horizons, which I currently use. I created accounts at Charityvest, Schwab, and Vanguard to get a sense of how they work, but I haven't tried to do anything fancy like set up an advisor-managed account. I had to make subjective judgments on things like UI, so don't take my claims as definitive.

Caveat 2: This article is not about whether you should open a DAF in the first place. For some people, a foundation would better serve their needs; other people should simply keep their...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concerns with the Wellbeing of Future Generations Bill, published by Larks on March 9, 2022 on The Effective Altruism Forum.

Summary

This is a critique of the current draft of the UK Wellbeing of Future Generations Bill, a recent attempt to influence policy, supported by some EAs. The Bill imposes a range of new duties on public bodies relating to future generations, ‘sustainable development’ and ‘wellbeing’. We support the underlying goals of the authors of the Bill, but believe that if the Bill were passed as currently drafted, it could significantly damage short and long term welfare. Our concerns only relate to this Bill as currently drafted; we believe in general that it is important for EAs to work more on improving policy in both the long and short terms, and John is involved with a new initiative to help with that. Later in this post we set out thoughts on other approaches which could be effective for achieving long term policy goals.

We do not see compelling evidence that simply passing a law that says there is a requirement for long-term thinking, goals or metrics will necessarily lead to better or more long-term thinking in a way that overall benefits future generations. Vague language, subversion of goals, ignoring or simply failing to improve metrics, contrary incentives of policymakers and the law of unintended consequences are all large risks, in light of the history of well-meaning legislative initiatives that have caused damage to long term welfare.

We think there is substantial risk that the Bill could damage the welfare of future generations. Simply passing a law that the government must do ‘good things’ does not in fact ensure that it will do good things. If the goal of the Bill is circularly defined as imposing the duties and other legal measures it proposes, then it will do that. But we have no confidence that, as drafted, it will benefit future generations.

Introduction

The Bill establishes a set of national wellbeing goals, formulated by the Secretary of State with public consultation. It places various duties on public bodies and creates a new requirement of future generations impact assessments. It also requires a futures and forecasting report with a 25 year time horizon. It empowers the head of the National Audit Office to examine whether public bodies have followed their wellbeing duties. It also suggests a new Joint Committee on Future Generations in Parliament to review future legislation, and a new Future Generations Commission. There are numerous other provisions. It is based on the text of, and lessons learned from, the Well-being of Future Generations (Wales) Act 2015.

We enthusiastically endorse the goal of making governments’ actions and perspectives more long term in outlook. Government action is often characterized by a distressingly short term perspective, as shown by the multiple suboptimal responses to the COVID-19 pandemic, and indeed in the multiple failures in pandemic preparation before that.

We also welcome the increasing interest of the EA movement in improving policy and governance in a non-partisan way, as we think these areas are neglected, potentially high impact and more tractable than is generally understood.

However, we are concerned that the proposed Wellbeing of Future Generations Bill may not achieve its stated outcomes. Given the current state of knowledge, we think there is a high risk of unintended and counterproductive consequences from actions based on superficial analysis. We think that the analysis should be taken to a deeper level before action is decided upon.

We know a number of people with experience in government and policy who have expressed the view that perspectives in the EA movement can sometimes – with the very best of intentions – be naïve in terms of the expected practical outcome of partic...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Rephrasing Of and Footnote To An Embedded Agency Proposal, published by Josh Hickman on March 9, 2022 on The AI Alignment Forum.

I wanted to clarify some comments made in this post, which proposed a way to resolve the issues brought up in the Action Counterfactuals subsection of the Embedded Agency paper. I noticed that my ideal edits would end up rewriting the post almost completely (it wasn't particularly clear), and I wanted to more coherently lay out the thinking.

The Problem

The problem is that we'd like agents to consider action counterfactuals, and, perhaps, find a proof that an action leads to the highest reward before taking it. The formulation in the paper leads to some Lobian issues, where agents choose actions by searching over proofs for statements similar to:

[The paper actually has a much cleaner phrasing of the statements it searches for proofs for, to make the proof issue much clearer.]

This is a rather substantial bummer, because something like that would be great to rely on. It'll find the proofs for what you'd like, but it'll also find proofs for nonsense. The Embedded Agency paper is a great resource for this, but you should also work out why this might happen in your head.

The Confusion

One potential intuition for why this problem exists is the following:

The agent who is taking this argmax always chooses the highest reward thing. If I know the rewards for all actions but the last X, well, choosing X means it was the result of the argmax. It's right there in the source code / function definition. So W must have been the biggest! Therefore, X is the result of the argmax being computed, because even if we don't know W, I know that the agent choosing X means W is bigger than all the other W' values. It must be -- the choice was coming from an agent doing an argmax!

This might be a confusing phrasing. The ideal way to explain this might be setting up the 5-10 problem in a theorem prover and just showing people it'll find the bad proof. But I think the narrative version captures something coherent.

The confusion we should have is that this is not how we, humans, engage in action counterfactuals! Not even close. Normally that's not an issue, but let's pay more attention to see if the difference is substantive.

In the original post, I use the example of a professor considering whether to slash the tires on his car. He ponders what it would imply, if he chose to do that. It would imply he must desperately not want the ability to travel, plausibly. It might imply he's driven into a river and needs access to the air within them for safety, like he's seen in movies. He's a sensible person -- him making the choice must mean there's a very good reason he did so! Much better than the standard "I'm just trying to get to work" reasons to not slash the tires.

But the professor, running his weaker, informal decision making process, does not slash his tires. You don't want to know the rewards implied by choosing to take an action. Although the professor has no Lobian issues in his informal decision-making, he views the future differently: he asks what would happen if he was the same type of person who just made the choice, excluding any consideration of why he might be making the decision, and considers what might happen to that imaginary person.

The Intuition

The intuition I was using was: why can't the agents do something similar? Create a new agent that "teleports them forward in time" by just precommiting to making the decision, and seeing what results? This matches our informal understanding of the phrase "action counterfactual" much better, I think, and because we're creating a new agent (one with this one decision pre-set into it), it's possible the self-reference issues go away.

Does it mechanically work to resolve the issue?

It does in the specific exam...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ELK Sub - Note-taking in internal rollouts, published by Hoagy on March 9, 2022 on The AI Alignment Forum.

My ELK submission was labelled under 'Strategy: train a reporter that is useful for another AI'. This is definitely a fair description, though the AI it needs to be useful to is itself - the reporter is essentially internalized. I also agree that the proposed counterexample, hiding information in what seems like human-comprehensible speech, is the biggest flaw.

Nonetheless I think my proposal has enough additional detail and scope for extension that it's worth fleshing out in its own post - so here we are. Some of the responses to counterexamples below also go beyond my original proposal.

For anyone interested, here is my original proposal (Google Doc) which contains the same idea in somewhat less generality.

In this post I'll first flesh out my proposal in slightly more general terms, and then use it to try and continue the example/counter-example dialogue on ELK.

I know there were a number of proposals in this area and I'd be very interested to see how others' could be integrated with my own. In particular I think mine is weak on how to force the actor to use human language accurately. I expect there are lots of ways of leveraging existing data for this purpose but I've only explored this very lightly.

Many thanks to ARC for running the competition and to Misha Wagner for reviewing and discussing the proposal.

Proposal Summary

Creating Human Semantic Space

We should think of the actor as described in ELK as having two kinds of space in which it works:

The space of potential observations of its sensors and actuators, O

The space of internal representations, ie its Bayes Net, I

The machine takes in observations, converts it into its internal representation I, simulates the action in this internal representation, and then cashes this out in its actuators.

What we want to do is force it to convert back and forth between I and a third space, the space of human understandable meaning, which we will call H.

How can we achieve this? My basic idea is that we leverage the world of text and image based models to create a model called a Synonymizer. This device, in its ideal form, would be trained to take in any piece of text, image, video, audio, or combination thereof, and output a piece of media in any of these forms which preserves the semantic content as far as possible while having complete freedom to change the format. (Initial experiments in this vein would probably use GPT-3 primed with 'This document will contain a paragraph, followed by a paragraph with the same meaning but in a different structure..')

The set of equivalency classes according to this Synonymizer (though they would in practice by fuzzy) should then be isomorphic to human semantic space, H.

Learning to use H as a human would

Next, when training, during its internal rollouts, we would periodically force the machine to translate from I into a space which is then passed through the Synonymizer, before being converted back into I, forcing it to learn a mapping between the two:

Of course, at this point is isn't using human semantics, just a very weirdly regularized latent space, but having this should also allow us to bring in a vast amount of possible training data, utilizing labelled image, video, text description of events, etc, which could all be used to train the model, and thereby force it to understand language and video as a human does in order to maximize performance. For example, while training the model to predict future events in a piece of video, text description of future frames can be appended, forcing it to learn to read text into its internal understanding of the situation in order to perform well.

The combination of this training example and the Synonymizer should hopefully go a long...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: It Looks Like You're Trying To Take Over The World, published by gwern on March 9, 2022 on The AI Alignment Forum.

This story was originally posted as a response to this thread.

It might help to imagine a hard takeoff scenario using only known sorts of NN & scaling effects...

In A.D. 20XX. Work was beginning. "How are you gentlemen !!"... (Work. Work never changes; work is always hell.)

Specifically, a MoogleBook researcher has gotten a pull request from Reviewer #2 on his new paper in evolutionary search in auto-ML, for error bars on the auto-ML hyperparameter sensitivity like larger batch sizes, because more can be different and there's high variance in the old runs with a few anomalously high performance values. ("Really? Really? That's what you're worried about?") He can't see why worry, and wonders what sins he committed to deserve this asshole Chinese (given the Engrish) reviewer, as he wearily kicks off yet another HQU experiment...

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ask AI companies about what they are doing for AI safety?, published by michaelchen on March 8, 2022 on The Effective Altruism Forum.

Today, I had the pleasure of attending a talk by Jeff Dean at my university (Georgia Tech). Titled “Five Exciting Trends in Machine Learning,” it was a fascinating, engaging presentation. Midway through the talk, I started to idly wonder, if there ends up being a Q&A after the talk, could I ask a question about AI safety? I started drafting a question on my phone.

After the talk, there was in fact time for questions. The moderator took one question from one person sitting a few rows in front of me, and I wished I had raised my hand earlier and sat closer to the front. Then, the moderator read aloud two questions from people watching the livestream, and they asked their own question. Jeff Dean was still available for questions after the talk, however. I ended up asking him something like the following:

“One major focus of DeepMind is aligning ML models to follow what humans want, rather than narrowly pursuing objectives that are easy to specify. You mentioned the trend of how AI is becoming increasingly capable and general. If this continues, and if we had a highly advanced general AI and want it to cure cancer, one solution to that objective would be to kill everyone, but that would be pretty bad. So it seems important to be able to figure out how to specify the objectives that we actually want, and some exciting approaches to this include reward modeling or iterated amplification. Is this a problem that Google AI is working on or plans to work on, like its sister company DeepMind?”

I don’t think that was the optimal way of asking about AI alignment, but that’s what I ended up asking. (If anyone has suggestions on how to talk about long-term AI risk in a better way, please leave a comment!)

His response was essentially, Google AI is doing some work on AI safety. Google AI focuses more on near-term stuff, while DeepMind starts from the perspective of thinking about super AGI. He’s more optimistic that we'll be able to solve these issues as we go and that we'll have the constraints necessary to prevent an AI from killing everyone, while he does appreciate that some people are approaching this from a more paranoid perspective.

I thanked him for his thoughts. In total, I think Jeff Dean was asked just around ten questions.

I remember reading a quote that went something like, “At every town hall, ask your representatives what they are doing to address the climate crisis.” I don’t know how often Jeff Dean visits universities to give a talk, but if every time, just two or three students from the local EA group asked him a polite, thoughtful question about AI alignment, I think he might pay closer attention to it.

More generally, EAs in a local area may want to have a coordinated effort to ask speakers (carefully considered) questions about topics relevant to EA. Many social movements have gained traction and created change from just a handful of people raising awareness of an issue. Though there are many pitfalls we would want to avoid – appearing uneducated, out-of-touch, aggressive, or polemical, for example – I think we could do more to adopt helpful strategies from traditional social movements.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.