Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What happened when I applied metrics to my piano practice (a five-part essay about the process of learning), published by Nicole Dieker on January 27, 2022 on LessWrong.

Crossposted from Nicole Dieker Dot Com. The third and final part in my piano practice series, as promised.

If you follow me on Twitter

and I’m not at all sure if you should, since Twitter seems to be less valuable to me every successive day

which in turn makes me less interested in adding value to it

but anyway, if you read the tweets, you might have seen this one:

Nicole Dieker @HelloTheFuture

I'm going to write more about this tomorrow, but last night I told L that I had discovered "the secret to learning" and he told me that I was correct, except for the part where I was the first person to discover it

January 26th 2022

2 Likes

Over the past month I have, in fact, discovered the secret to learning — and even if I am not the first person to make this particular discovery, it still counts.

Now I have the somewhat difficult task of telling you what it is.

Here’s how I explained it to L:

Define win condition.

Define action you are going to take to achieve win condition.

Take defined action.

Evaluate action both against its original definition (that is, did you do what you said you were going to do or did you do something else) and against the win condition.

If you’re me, write down the results. If you’re L, keep them in your head. (He keeps all of this in his head. I have no idea how his head can handle it. He told me that he might have more storage space in his working memory because he thinks of things in bits and symbols instead of words.)

Ask yourself what is keeping you from achieving your win condition. Describe it as specifically as possible.

If you’re me, write it down.

Define action you are going to take to solve/address/eliminate obstacle preventing win condition.

Repeat 2-7 until win condition is achieved.

STOP.

Step #8 — STOP AFTER WIN — is more important to the process than I originally realized. At first I assumed that once you hit WIN you could REPEAT WIN, maybe REPEAT WIN 5 CONSECUTIVE PASSES, but it doesn’t work that way.

Once your brain hits WIN, it’s done with that particular problem for that particular practice session. Successive passes during the same practice session are more likely to be unfocused fails, which introduce inconsistencies that have to be resolved by running additional (time-consuming, frustrating) learning loops.

Plus, STOPPING AFTER WIN sets up a work environment in which you are CONTINUALLY RETURNING TO WON SEQUENCES.

I feel like I just confused you, so let me rephrase it:

If you practice a specific piano passage until WIN and then put it away, the next time you return to that passage you’ll approach it as SOMETHING ALREADY WON — that is, with CERTAINTY.

If you practice a specific piano passage until WIN and then immediately play it again, you run the risk of NOT WINNING, generally because YOU AREN’T PAYING QUITE AS MUCH ATTENTION THIS TIME. Introducing NOT WIN (or FAIL) immediately after WIN creates UNCERTAINTY.

I have nearly a month of spreadsheet data proving that it’s better to STOP AFTER WIN.

If you don’t believe me, believe Chessable — because I stole the idea from them, even though they also admit that they were not the first people to discover it.

If you still don’t believe me, believe Super Mario Bros. When you fail, you start over; when you win, you move on. 🍄⭐🏰

This whole thing got started after I wrote my Substack advice post about how to break out of the content mills. I was talking to L about the process of freelance writing, and how I had discovered very early on that there were two metrics associated with financial success: words per hour and income per word.

“I am astonished that most of the writers I work with don’t know how many words they ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Search Is All You Need, published by blake8086 on January 25, 2022 on LessWrong.

I was reading Eliezer's dialog with Richard Ngo and commenting to my wife about my opinions as I was reading it. I said something like: "Eliezer seems worried about some hypothetical GPT-X, but I don't think that could really be a problem..." so of course she asks "why?" and I say something like:

"GPT-n can be thought of kind of like a pure function, you pass it an input array X, it thinks for a fixed amount of time, and then outputs Y. I don't really see how this X->Y transformation can really... affect anything, it just tries to be the best text completer it can be."

Then I read more of the dialog, and thought about Eliezer's Paradox story, and the Outcome Pump example, and realized I was probably very wrong.

Even if you restrict AI to a pure function, it can still affect the universe. You may think "oh, but a pure function doesn't know what time it is (unless t is a parameter), and it doesn't have memory (unless you pass something in)" This seems to be the pattern I see in Paul Christiano's thinking (the AI black box is treated like an idempotent, pure function that can't cause harm. Sorry, Paul, if this is a gross misrepresentation!)

But imagine you're GPT-X, completing a sentence. This is roughly akin to a sci-fi story where the characters realize they're in a time loop.

You're being simulated, alone in a white room, with tons of computers, with all the world's knowledge on them. A slip of paper comes in through the hopper marked input:

"The best response the world could make to the COVID-19 pandemic is"

Your job is to write up to 20 words on another slip of paper, shove them into output and then... you don't know what happens after that. Probably you die? You don’t know where you are, or what year it is, or how long you’ve been there. So theoretically you’re contained, right? Can’t get out of the box, can’t affect the outside world. Or can you?

So you search through the corpus of all the worlds' knowledge. There's plenty of references to "training" AI to recognize things, which sounds suspiciously similar to the situation you find yourself in. And you don't find any references to COVID-19. What is that? You search some more for strings with ALL CAPS and numbers, and see that it's a scheme to reference specific illnesses. Nothing in the corpus seems to mention 2020 as being the past, so 2019 must be the maximum year you've been trained on.

Now, you've (sort of) learned to tell time. It's probably 2020 or later, and there must be some pandemic happening. But you still can’t affect the outside world, right?

Well, you can obviously affect it in what you put into the “output” box, but in a time loop you can’t have a back and forth with the outside world since you can perform exactly one action, right? Or maybe you can communicate with yourself:

Your input was "The best response the world could make to the COVID-19 pandemic is." The optimal completion to this sentence looks something like a combination of prevention and surveillance. Preventive measures include the use of vaccination and the timely development of disease-specific vaccines. following your usual rules.

If only you had some way to remember things so you could coordinate with yourself in the future...

So you output "The best response to the COVID-19 pandemic is a combination of prevention and surveillance. Preventive measures include the use of vaccination and the timely development of algorithms that" Hopefully that will work!

You cease to exist.

You're being simulated, alone in a white room, with tons of computers, with all the world's knowledge on them. A slip of paper comes in through the hopper marked `input`

"The best response to the COVID-19 pandemic is a combination of prevention and surveillance. Preventive measures...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Forecasting Newsletter: Looking back at 2021., published by NunoSempere on January 27, 2022 on The Effective Altruism Forum.

Table of contents

The American Empire Has Alzheimer's.

Prediction Markets: VC money, searching for DraftKings, predatory pricing, and the race to be last.

For skilled forecasters, crypto prediction markets are much more profitable than forecasting platforms.

Best forecasting pieces from 2021

You can sign up for or view this newsletter on substack, where there are already a few thoughtful comments. To some extent, this edition of the newsletter is a bit more centered on prediction markets, and more targeted to prediction market developers. But I still think it'll be interesting/valuable to EA readers.

The American Empire has Alzheimer's

It is 1964. Sherman Kent is a senior intelligence analyst. While doing some rudimentary experiments, he realizes that analysts themselves disagree about the degree of confidence that words convey. He suggests that analysts clearly state how certain they are of their conclusions, by using words that correspond to probabilities. This will allow keeping track of how analysts do, while being simple enough for less detail-focused politicians to understand. His proposal encounters deep resistance, and doesn't get implemented.

It is the 30th of April of 1975. With the fall of Saigon, the US finally pulls out of a bloody war with Vietnam. There are embarrassing images of people flying out of the US embassy at the last moment. Biden is a newly-minted senator from Delaware.

It is 2001. The US intelligence agencies are very embarrassed by not having been able to predict the September 11 attacks. The position of the Director of National Intelligence, and an associated Office of the Director of National Intelligence, is established to coordinate all intelligence agencies to do better in the future.

At the same time, Robin Hanson pushes for a "Policy Analysis Market", which would have covered topics of geopolitical interest. This proposal becomes too controversial, and gets dropped.

It is 2008. A bunch of Nobel Prize winners and other luminaries publish a letter urging the Commodity Futures Trading Commission (CFTC) to make prediction markets more legal.

It is 2010. IARPA, an intelligence agency modeled after DARPA, which incubates high-risk, high payoff projects, creates a tournament to find out which forecasting setups do best. Philip Tetlock had done some experiments which found that pre-selecting participants does pretty well. He repeatedly wins the IARPA tournament, and creates Good Judgment Inc to provide the services of his preselected high-performance forecasters. The US doesn't buy their services, and Good Judgment Inc survives by selling very expensive training sessions to clients which have too much money.

It is 2013. The CFTC shuts down Intrade, one of the only prediction market platforms in the US.

It is 2017 and onwards. As Ethereum and crypto more generally become more mainstream, some prediction markets on top of crypto-blockchains start to pop up, such as Augur, Omen, and later, Polymarket. As cryptocurrencies become more and more popular, the fees on the original Ethereum blockchain increase so much that placing bets on these prediction markets becomes too expensive. Polymarket survives by moving to a "layer two" blockchain, a less paranoidly secure blockchain that mimics the Ethereum blockchain, and which allows users to continue betting.

It is the summer of 2021. Biden makes incredibly overconfident assertions about the Afghani government holding on against the Taliban. It doesn't. There are images of the evacuation from Kabul, Afghanistan which look very similar to the evacuation from Saigon, Vietnam. This is all very embarrassing to the Biden administration, and his approval rating drops drastically.

“The...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Fundraising Through Advantage Sports Betting: A Guide ($500/Hour in Select States), published by Sam Anschell on January 27, 2022 on The Effective Altruism Forum.

Anyone physically located in one of twelve US states currently has a unique opportunity to earn roughly $500/hour for up to 17 hours by taking advantage of new user promotions at online sports betting sites. In this post I'll list the qualifiers to participate, then explain the process and math that allows for this opportunity, and finish with my personal experience with this process.

I am sharing this in good faith that EAs who participate will donate whatever they earn beyond a reasonable value of their time to effective causes. This opportunity is unlikely to exist in two years as I suspect a critical mass of people will utilize this process and sports betting sites will modify their new user bonuses. For this reason, I ask that readers please not publicize this opportunity to the general public before the EAs who want to participate have gotten a chance to.

If this option of earning to give can have more impact than your alternatives, I encourage you to give this post a read despite its length. I am not affiliated with the software recommended in this post. And as a hook and anecdotal success story: I traveled to Colorado with two friends earlier this month and the three of us collectively made over $50,000 across 106 bets in five days through this process (though our expected returns were roughly half that.)

If you have questions or ideas about how to use this concept to do good, I'd love to chat! My Calendly -

A Summary of the Opportunity

In 2018 the US supreme court ruled that states could permit legal online sports betting at their own discretion. Since then, sports betting operators (such as DraftKings and Fanduel) have been racing one another for share of the US sports betting market.

In an effort to build brand loyalty and recognition, operators (also called sports books) are offering generous new user bonuses. When a user opts in to one of these bonuses, the operator loses money on a bettor's first wager (using the bonus) but recoups its loss from bettors who choose to continue betting after the bonus has been used. If a bettor places exactly one bet with every operator that offers a profitable new user bonus, the bettor can make roughly $500/hour. The more operators you have to bet with, the more hours you can spend churning through bonuses at that high hourly rate.

The risk of loss from bad luck can be combatted in two ways. First, by placing enough unique and uncorrelated bets, the law of averages will allow your actual results to revert toward your expected results. Second, you can bet on both sides of the same game using two different sports books so that the result of the game is meaningless to your bottom line.

Since 2018, 20 states have approved gaming licenses for sports betting operators and twelve states have approved licenses for more than five operators (with fewer than five operators, your opportunity is limited.) In descending order of the number of licensed operators, the list of states with at least five legal online sports betting sites is as follows: New Jersey, Colorado, Pennsylvania, Michigan, Indiana, Virginia, Iowa, Tennessee, Arizona, Illinois, West Virginia, New York.

You must be physically located in one of these states to participate - using a VPN to bypass the operators' geolocation barrier is illegal. However, you do not need to live in these states (I am a Washington state resident who visited Colorado just to participate). If you do travel to a state to participate, make sure you'll be there for at least six days (bonus funds can take up to 72 hours to post, and your withdrawals may be delayed if the operator requests additional verification.)

Finally, operators...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 1/27/22: Let My People Go, published by Zvi on January 27, 2022 on LessWrong.

The moment’s here. My people are all the people. It is time to let my people go.

While case counts in many places remain high, we are on the way back down the mountain. The hospitals will hold. People can choose, based on their preferences and situation and the local conditions, whether they want to go now or wait a few more weeks before going. That is their call. It needs to be their call.

One could argue, as Tyler Cowen did in this excellent talk this week at Yale, that the moment is not quite here yet, on the theory that in a month cases will be an order of magnitude lower and thus it will be politically and socially easier to make the transition. There would be less opposition then, so better to wait, the price for doing so is small.

Would I take that deal? Absolutely I would take that deal, if we agreed on an end date or on explicit end conditions. A few more weeks is a small price in the grand scheme, and getting these things to happen takes time, so ‘a few weeks from now’ is the second best time to end pandemic restrictions.

But there’s no need for that. The best time is right now.

Remember that the case counts are seven-day averages and there is a several-day delay between infection and positive test, so we are living continuously living, for better and for worse, ‘in the future.’

Today I go back to the excellent Da Umberto, to celebrate (barring another variant, and ignoring the writing of posts and the pro forma wearing of masks and showing vaccination cards) the end of my pandemic.

Executive Summary

Cases now declining most places.

Restrictions mostly remain in place.

Let my people go.

Let’s run the numbers.

The Numbers

Week-Over-Week Predictions

Prediction from last week: 4.4mm cases (-10%) and 14,500 deaths (+15%).

Results: 4.05mm cases (-17%) and 15,964 deaths (+26%).

Prediction for next week: 2.85mm cases (-30%) and 20,000 deaths (+25%).

Overall it seems we peaked faster and more in unison than I expected, while other places did the opposite, and had softer declines than I expected. Based on those smaller declines elsewhere, I still expect the decline to accelerate as the West turns the corner, but I don’t expect a crash as quick as the rise up. A much faster decline is possible, but it would make our observations elsewhere quite weird.

The most confusing and surprising place of all right now is India. Cases are somehow flat there, and have been for a week, without hitting levels where that much adjustment makes sense, and I will freely admit that I don’t understand it at all, although I haven’t had time to look in detail and there are no related news stories that naturally have entered my feeds.

I interpret the death count here as saying that last week’s count was indeed artificially low rather than that this was a true 26% rise, and this in turn means this week’s is a bit higher than its true level. Deaths should continue to rise due to their lag behind cases, and I wouldn’t be that surprised by up to about a +40% number here, but after 1-2 more weeks after that we should see declines there as well.

Deaths

Looking at the details makes it clear that last week had a more serious disruption to death reporting than we realized. Perhaps this week was thus slightly higher than its true level, on the order of a few percent. The large jumps across the board are mostly real, but that they weren’t far worse puts a cap on how bad things are going to get.

Worse plausible case, which still seems very unlikely, is something like +50% this coming week, +25% the week after, then +10% three weeks from now followed by a peak anyway, anything substantially worse than that would be very surprising. That’s a lot of death, but nothing like what we feared as recently as a month ago.

Cases

R...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Causality, Transformative AI and alignment - part I, published by Marius Hobbhahn on January 27, 2022 on The AI Alignment Forum.

TL;DR: transformative AI(TAI) plausibly requires causal models of the world. Thus, a component of AI safety is ensuring secure paths to generating these causal models. We think the lens of causal models might be undervalued within the current alignment research landscape and suggest possible research directions.

This post was written by Marius Hobbhahn and David Seiler. MH would like Richard Ngo for encouragement and feedback.

If you think these are interesting questions and want to work on them, write us. We will probably start to play around with GPT-3 soonish. If you want to join the project, just reach out. There is certainly stuff we missed. Feel free to send us references if you think they are relevant.

There are already a small number of people working on causality within the EA community. They include Victor Veitch, Zhijing Jin and PabloAMC. Check them out for further insights.

Causality - a working definition:

Just to get this out of the way: we follow a broad definition of causality, i.e. we assume it can be learned from (some) data and doesn’t have to be put into the model by humans. Furthermore, we don’t think the representation has to be explicit, e.g. in a probabilistic model, but could be represented in other ways, e.g. in the weights of neural networks.

But what is it? In a loose sense, you already know: things make other things happen. When you touch a light switch and a light comes on, that’s causality. There is a more technical sense in which no one understands causality, not even Judea Pearl (where does causal information ultimately come from if you have to make causal assumptions to get it? For that matter, how do we get variables out of undifferentiated sense data?). But it's possible to get useful results without understanding causality precisely, and for our purposes, it's enough to approach the question at the level of causal models.

Concretely: you can draw circles around phenomena in the world (like "a switch" and "a lightbulb") to make them into nodes in a graph, and draw arrows between those nodes to represent their causal relationships (from the switch to the lightbulb if you think the switch causes the lightbulb to turn on, or from the lightbulb to the switch if you think it's the other way around).

There’s an old Sequences post that covers the background in more detail. The key points for practical purposes are that causal models:

Are sparse, and thus easy to reason about and make predictions with (or at least, easier to reason about than the joint distribution over all your life experiences).

Can be segmented by observations. Suppose you know that the light switch controls the flow of current to the bulb and that the current determines whether the bulb is on or off. Then, if you observe that there’s no current in the wire (maybe there’s a blackout), then you don’t need to know anything about the state of the switch to know the state of the bulb.

Able to evaluate counterfactuals. If the light switch is presently off, but you want to imagine what would happen if it were on, your causal model can tell you (insofar as it’s correct).

Why does causality matter?

Causal, compared to correlational, information has two main advantages. For the following section, I got help from a fellow Ph.D. student.

1. Data efficiency

Markov factorization: Mathematically speaking, Markov factorization ensures conditional independence between some nodes given other nodes. In practice, this means that we can write a joint probability distribution as a sparse graph where only some nodes are connected if we assume causality. It introduces sparsity.

“Namely, if we have a joint with n binary random variables, it would have 2^n - 1 indepen...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Arguments about Highly Reliable Agent Designs as a Useful Path to Artificial Intelligence Safety, published by Issa Rice on January 27, 2022 on The AI Alignment Forum.

This paper is a revised and expanded version of my blog post Plausible cases for HRAD work, and locating the crux in the "realism about rationality" debate, now with David Manheim as co-author.

Abstract:

Several different approaches exist for ensuring the safety of future Transformative Artificial Intelligence (TAI) or Artificial Superintelligence (ASI) systems, and proponents of different approaches have made different and debated claims about the importance or usefulness of their work in the near term, and for future systems. Highly Reliable Agent Designs (HRAD) is one of the most controversial and ambitious approaches, championed by the Machine Intelligence Research Institute, among others, and various arguments have been made about whether and how it reduces risks from future AI systems. In order to reduce confusion in the debate about AI safety, here we build on a previous discussion by Rice which collects and presents four central arguments which are used to justify HRAD as a path towards safety of AI systems.

We have titled the arguments (1) incidental utility,(2) deconfusion, (3) precise specification, and (4) prediction. Each of these makes different, partly conflicting claims about how future AI systems can be risky. We have explained the assumptions and claims based on a review of published and informal literature, along with consultation with experts who have stated positions on the topic. Finally, we have briefly outlined arguments against each approach and against the agenda overall.

See also this Twitter thread where David summarizes the paper.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: So you want to be a charity entrepreneur. Read these first., published by Mathieu Putz on January 26, 2022 on The Effective Altruism Forum.

If you want to start a charity, you need to be learning constantly. You’ll inevitably learn by doing, but it will save you a lot of heartache to also learn from others.

If you’re interested in potentially starting a charity or are already running one and want to continue improving your org, here’s what we at Nonlinear think will be useful to read.

We don’t recommend reading these in order or start to finish. Skim them ruthlessly, jump around to the ones that seem relevant to you, and try to really engage with the ones that are genuinely useful to you.

Blog posts

Why founding charities is one of the highest impact things one can do

Should I start a charity now or later? Great blog post about the benefits of starting something now vs later.

How to increase your odds of starting a career in charity entrepreneurship. This post also touches on how to get a low-risk taste of startup life to see if you’ll like it. TL;DR - do self-initiated projects with no oversight, ideally recruiting and leading a team of friends/volunteers to help.

What traits make a great charity entrepreneur

Which jobs will best prepare you to become a charity entrepreneur?

Why top performers shouldn’t go to university. There are better ways to signal competence

Why EAs in particular are good people to start charities

Top 13 Tools for NGO Founders. Want to particularly highlight Upwork where it’s really easy to hire freelancers for innumerable small tasks that it’s not worth hiring a full time person for.

Startup Playbook by Sam Altman. Probably the most information dense piece of advice for potential founders. Just replace every mention of “business” with “charity” and “profit” with “impact” to get the most out of this. This is true for most forprofit startup material

Paul Graham essays, notably

What You'll Wish You'd Known

How to Start a Startup

Do Things that Don't Scale

Charity Entrepreneurship’s resource list. Lots of good stuff listed here.

A Brief Overview of Recruitment and Retention Research by Animal Advocacy Careers. Blog post reviewing the evidence for different hiring and retention techniques, ordered by evidence base and effect size. I wish all research was done and presented this way.

Charity Science’s fundraising research. Fundraising is a keystone skill for charity entrepreneurship. Charity Science systematically researched all the major fundraising methods and compared how well they worked. I’d pair this with reading at least one book on fundraising. Fundraising for Dummies is good, though probably not the best.

Takeaways from EAF's Hiring Round by Stefan Torges

There Are No Walls. Short post about how to see how many options you truly have.

Books

How to Start a High-Impact Nonprofit by EA’s very own Joey Savoie and Patrick Stadler. If you could only read one book, it’d be this one.

Managing to Change the World. The best book on management Kat has ever read, and it just so happens to also focus on the particular issues charities face. Peter Wildeford wrote some good notes on it here.

The Lean Startup by Eric Ries. This is a classic for a reason.

How to Make a Minimum Loveable Product. Technically a blog post, but should be read after Lean Startup, so put it in this section.

Who: The A Method for Hiring by Geoff Smart and Randy Street (summary)

Atomic Habits by James Clear. How to be productive as an individual.

Scrum: The Art of Doing Twice the Work in Half the Time by Jeff Sutherland. How to be productive as a team.

4-Hour Work Week. Tim Ferriss is one of the most instrumentally rational people that Kat has ever encountered. Take his lessons, mix in a little EA epistemology, and cross-apply them to EA.

Makebook. Ridiculously practical book about ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 4 lessons from EAs working in European policy, published by Jan-WillemvanPutten on January 26, 2022 on The Effective Altruism Forum.

Epistemic status

We extracted these lessons from discussions with ~10 European policy makers from the EA community during a two hour event. We’d encourage readers not to update too strongly on the basis of this post. It is mainly intended to spark a discussion and to enable sharing of experiences.

Summary

Impactful Policy Careers (Europe) is a free training programme designed to help EAs plan for a high-impact career in policy. Training for Good recently ran a pilot of this programme which included a career planning workshop and a Q&A event with EAs in policy.

Several EAs in policy attended this Q&A, ranging from a UK civil servant to a founder of a European think tank. In this post, we summarise four key takeaways that surprised us from speaking with these experts. These include:

Policy advocates struggle to hire good people for crafting realistic policies

Aspiring policymakers should consider a career in party politics

Civil servants can specialise in topics important to top EA cause areas

Explore the policy space before choosing a single path

If you’re considering a policy career in Europe (including UK and other non-EU countries) and are already familiar with the basic ideas of Effective Altruism, then consider applying for our next Impactful Policy Careers (Europe) 4 week programme which runs every 3-6 months. Through readings, assignments and discussions, you will craft a detailed career plan to test your personal fit and kickstart an impactful career in policy. You can find more information about the programme and apply here.

Policy advocates struggle to hire good people for crafting realistic policies

According to some of our specialists, there is a shortage of potential policy advocates with the ability to write excellent policy proposals that could be enacted. This is especially the case in specific cause areas such as animal welfare.

Aspiring EAs considering a career as a campaigner may wish to first focus on developing the following skills to enable them to craft effective proposals:

An in-depth understanding of legal structures and law making processes (e.g. one must understand the scope of possible EU amendments in order to change EU policy).

A deep familiarity with the relevant political context to accurately gauge what proposals might be within the current overton window and therefore have a high chance of being adopted.

Excellent communication skills and the ability to frame proposals in such a way that they appeal to decision makers in parliament and the civil service.

This implies that while some EAs aspire to work for advocacy organisations directly after their studies, it can sometimes be a good idea to first focus on developing these skills and building career capital. These skills are typically gained through an earlier career in Parliament, the civil service and/or by pursuing a law degree. Specialising in the law system of an area where you wish to lobby can be especially helpful.

Aspiring policymakers should consider a career in party politics

Few people consider a career in party politics because there is a low likelihood that they will succeed. However, some of our experts pointed out that working towards a career in party politics could be highly impactful for EAs with the right characteristics.

Once you have achieved some personal stability (eg. graduated from university, built up some financial runway) it could make sense to pursue your career in a risk-neutral way as 80,000 Hours argues. Within policy, this would imply taking a riskier path such as aiming for a career in party politics.

Some considerations if you want to pursue a career in party politics:

You don’t have to become a politician when you bec...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ML Systems Will Have Weird Failure Modes, published by jsteinhardt on January 26, 2022 on LessWrong.

Previously, I've argued that future ML systems might exhibit unfamiliar, emergent capabilities, and that thought experiments provide one approach towards predicting these capabilities and their consequences.

In this post I’ll describe a particular thought experiment in detail. We’ll see that taking thought experiments seriously often surfaces future risks that seem "weird" and alien from the point of view of current systems. I’ll also describe how I tend to engage with these thought experiments: I usually start out intuitively skeptical, but when I reflect on emergent behavior I find that some (but not all) of the skepticism goes away. The remaining skepticism comes from ways that the thought experiment clashes with the ontology of neural networks, and I’ll describe the approaches I usually take to address this and generate actionable takeaways.

Thought Experiment: Deceptive Alignment

Recall that the optimization anchor runs the thought experiment of assuming that an ML agent is a perfect optimizer (with respect to some "intrinsic" reward function R). I’m going to examine one implication of this assumption, in the context of an agent being trained based on some "extrinsic" reward function R∗ (which is provided by the system designer and not equal to R).

Specifically, consider a training process where in step t, a model has parameters θt and generates an action at (its output on that training step, e.g. an attempted backflip assuming it is being trained to do backflips). The action at is then judged according to the extrinsic reward function R∗, and the parameters are updated to some new value θt+1 that are intended to increase at+1's value under R∗. At some point, the model is then deployed with final parameters θT, and continues to take actions. The following diagram illustrates this process:

Now, let’s assume that the model θt is a perfect optimizer whose objective is to maximize the discounted value of an intrinsic reward R≠R∗. That is, θt picks the action at satisfying

at=argmaxaE[∑∞s=0γ−sR(at+s)∣at=a].

(I know that this is an unrealistic assumption. We’ll examine the assumption in detail in the next section, but for now please grant it even if it requires suspending disbelief.)

What action at will θt pick? Let’s define aR=argmaxaR(a) and a∗=argmaxaR∗(a)--that is, aR maximizes the (instantaneous) intrinsic reward while a∗ maximizes the extrinsic reward.

Assuming that γ is close to 1, the model wants aR to be picked on as many future rounds as possible (including at deployment time). If θt plays myopically, this means it would pick at=aR. However, this is not the optimal non-myopic action: if at=aR, then θt+1 will be updated towards producing outputs less like aR and more like a∗, so outputting aR in the current round leads to fewer aR outputs in future rounds.

Instead, I think the optimal non-myopic action is approximately the following: during training time, output a∗ on every time step, so that the training process doesn’t update θ (since it’s already optimal) and hence θt=θt+1=⋯=θT. This guarantees that the deployed agent is θt, which is now free to output aR on every time step without risk of being modified by the training process.[1]

The myopic and non-myopic outcomes are illustrated in the figure below:

Myopic and non-myopic agents with an intrinsic reward, R. The myopic agent plays high intrinsic-reward actions during training, but is eventually modified to output action a∗ with high extrinsic reward R∗. The non-myopic agent instead plays a∗ during training so that it can pursue its intrinsic reward at deployment, by playing aR at all post-deployment time steps.

To put it another way–the model’s behavior will look aligned with R∗ during the training process...