Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What “pivotal” and useful research ... would you like to see assessed? (Bounty for suggestions), published by david reinstein on April 28, 2022 on The Effective Altruism Forum.

Update 28 Apr 2022 -- Only ~10 responses so far, many of which are general areas rather than specific papers/findings/projects. So the 'bounty expected return' is still high.

Below, I give some work that I thought might be especially relevant, as examples.

Update 22 Apr 2022: Bounty[1]

What papers, findings, projects, or pieces of research (academic or non-academic) would you most like to see carefully and rigorously evaluated?[2]

Which specific results do you rely on in making key decisions, or which ones do you think that large EA-aligned donors and organizations rely on heavily?

I'm considering this particularly as...

Demonstration and test-cases for the Unjournal in our first months, as well as our general agenda (see the previous request and my earlier post

We have put a bounty on this, explained below

Research projects the Unjournal should consider in its first year

I plan to offer a bounty on successful suggestions for this, and entries will also be eligible for future bounties.

Also: Great use cases for the/ (tied to the eminent Daniel Lakens); this initiative looks very promising to me. ^[But please note that the bounty will only relate to the Unjournal.)

I'm looking especially for:

specific papers and findings in these papers and projects that we lean on a lot

empirical work that would benefit from an open-science/replicability assessment or an assessment of the methodology

More on what I am looking for

Some combination of work ...

Focusing on social science, economics, and impact evaluation (without digging too deeply into technical microbiology or technical AI, etc.)

Aiming at academic standards of rigor; perhaps it is in the process of peer review or aiming at it

With direct relevance for choices by EA funders... or to crucial considerations

"Empirical and/or quantitative": Makes empirical claims, analyses data, runs experiments/trials, runs simulations, fermi-estimations, and calibrations based on real data points

"Applied/applicable (economics/decision science) theory": Makes logical mathematical arguments (typically in economics) for things like mechanisms to boost public goods provision in particular contexts.

Unjournal would be particularly good for

Work in replicable dynamic documents (Rmarkdown, Quarto, Jupyter, etc)

... or otherwise hard to fit into a 'frozen pdf'

Ongoing projects you will 'continue to build on in the same place'

Work that is hard-to-place in standard journals ...

... because it is interdisciplinary (but rigorous) and requires multiple dimensions of expertise to evaluate

or because it is more impactful and robust than it is 'novel'

.(And thus we particularly value suggestions for work like this.)

The Bounty

Full details, T&C, and other considerations are here; excerpted below.

The prizes are:

1. “Piloted suggestion prize”: $250 x 1-3 . prizes for each of the 1-3 suggested research projects that we choose as piloting/proof-of-concept examples

2. “Participation prize”: A $150-$300 prize (see below)

Drawn randomly among ...

All people (other than those winning prize 1), who submit suggestions, where these suggestions are in a format we can consider. They must link a piece of research or a project and giving at least 1 sentence justifying it’s relevance.

Anyone who sends a posted letter (see footnote)[3]

If we use two or more suggestions for piloting/proof-of-concept the Participation prize will be $150. If we use only one of the suggestions for piloting/proof-of-concept the Participation prize will be $250. If we do not use any of the suggestions for piloting/proof-of-concept the Participation prize will be $300.

3. (Potential) Additional prize qualification:...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against immortality?, published by Owen Cotton-Barratt on April 28, 2022 on The Effective Altruism Forum.

Epistemic status: there are some strong anti-death / pro-immortality sentiments in EA/rationalist circles, and it's bugged me that I haven't seen a good articulation of the anti-immortality case. This is a quick attempt to make that case. I'm confused about what to think about immortality overall; I think that it may depend a lot on circumstances/details, and be eventually desirable but not currently desirable.

The basic case for immortality goes something like:

A death is a tragedy. Individuals would, in almost all cases, prefer not to die. They have friends and family who care about them and would prefer they not die. In a large majority of the cases where they would prefer to die this is because of other factors they'd prefer to fix instead (e.g. extremely poor health). In a post-scarcity future we should expect to be able to fix these factors, so we should plan on abolishing death.

This argument mostly checks out to me when evaluating whether it's good for individuals to die. But my worry is that the whole line of thinking is too individualistic. Would it be good for society as a whole if people were immortal?

Historically I think the answer is not clear-cut, but my guess leans towards "no". If I imagine I have the option of pressing a button so that people 3,000 (or 1,000, or 300, or 100) years ago were immortal — not aging, and not dying of disease — I feel like I don't want to press that button. There would certainly be a lot of good things that would happen from it — many tragic deaths averted, many good things that persisted longer, and perhaps longer time horizons for individuals. But I am scared that:

For most of history, populations were food-limited, so this would lead to more deaths by starvation or violence, which seem probably worse

However in the modern world this might not be an immediate problem

We'd also have contraception available as a tool (although it's unclear whether it's good to give existing people more life rather than their children having lives)

It's possible that technology will move us to fundamentally different positions re. personal identity (e.g. because everyone is run as emulations, and these can be slowed down) before our inability to keep scaling food production exponentially causes hard constraints on population

The world would have been much more likely to get stuck locked into some bad states

A lot of repressive dictators die of natural causes

If they were immortal, it seems much more likely that one would have successfully conquered the world, and established a regime which stably kept them in control for eternity

Less extremely, there is the adage that science progresses one funeral at a time

I'm sure this is sometimes an exaggeration, but also feel sympathetic to the thought that if new ideas threaten existing social structures, and if the people instantiating those structures were permanent fixtures, there might have been much more repression of new ideas

It's a trope that the death of elderly relatives — even highly beloved ones — can be freeing for people

This doesn't point to any particular failure modes, but is some support for "death may play an important role in allowing society (as it is currently arranged) to move on from things"

These are still concerns to me today.

At a more personal level, I have some worry that the narrative of death-as-tragedy can belittle narratives of death-as-end-to-story-arc, and that sometimes the latter seem like they are capturing something true and important. For old people who have had a happy routine and watched their grandkids grow up and have kids of their own, and feel like they can let go rather than needing to do more in the world, I dislike telling them (implicitly or exp...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consider Not Changing Your Forum Username to Your Real Name, published by electroswing on April 28, 2022 on The Effective Altruism Forum.

This is a speculative and hastily written post, feedback welcome.

First off: I pretty much agree with the post this post’s title references. For people who want to be publicly associated with EA, having your real name easily/trivially discoverable from your EA Forum username seems good for numerous reasons. The purpose of this post is to investigate the premise “You should want to be publicly associated with EA”.

Public EAs versus Private EAs

In this post I will stick to the dichotomy of “public EA” versus “private EA”, even though there is certainly a grey area in between. Here is what I mean by each term:

Public EA: Given a person’s full name, it is possible to determine that they are involved with EA in about 1 minute of Googling.

Examples of public EAs: people with EA Forum accounts under their real name, people who put EA in their Twitter bio associated with their real name, people who work for an EA org and are listed on the website

Private EA: Given a person’s full name, it is not possible to find any connection to EA, even after about 1 hour of Googling.

Reasons why you might want to be a public EA

The bulk of this post is dedicated to speculating about why being a public EA may be a bad thing for some people. Before I do that, I’ll briefly list reasons why being a public EA can be a good thing.

You open yourself up to more EA opportunities. (As described in this post, a good EA Forum post or a good EA twitter thread can lead to new friends, job offers, etc.)

You enjoy participating in the EA community so extensively that you can’t remain anonymous while doing so.

You implicitly are advertising EA to others, which grows the movement.

You, by being a public EA, are increasing the breadth / diversity of the EA movement.

You (selfishly) enjoy publicly labeling yourself as an EA.

Reasons why you might want to be a private EA

You don’t want to be seen as involved with “EA adjacent stuff”

I don’t think many people are objecting to providing safe drinking water to rural East African communities. I do think the public might object to EA adjacent stuff. By EA adjacent stuff I mean: billionaires, cryptocurrency, Silicon Valley, stuff like that.

This article already brought attention to a politician’s connection to “EA stuff”. If an article like this (or worse) would be bad for your reputation, you might not want any public connections to EA.

Probably this concern is most relevant for people who want to be uncontroversial public figures. Here I mean public figures in the broad sense: politicians and policymakers count, but also lead researchers, professors, executives, and so on. Basically anybody seen to have “power”.

I see two main ways a public figure could be negatively affected by their status as a public EA.

EA could get you canceled on Twitter.

These days, anybody with a public platform can get canceled, even for very minor offenses.

It looks like EA is becoming increasingly intertwined with cancellable attributes. See this recent tweet:

I imagine as EA becomes more well-funded and more publicly well-known, more “Cancelers” will draw connections between EA and things they think are bad such as billionaires and tech bros. If getting canceled would be very bad for your career, you might want to take this risk into account before going public.

Mainstream media might give EA a bad reputation.

I think this is separate from canceling because mainstream media has substantially farther reach and slightly more epistemic integrity than “Cancelers” on Twitter.

I think the most likely outcome is that mainstream media paints a neutral picture of EA—they say something like, “GiveWell seems excellent, but EA also has connections to some bad i...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The case for turning glowfic into Sequences, published by Thomas Kwa on April 27, 2022 on LessWrong.

Epistemic status: serious, uncertain, moderate importance. Leaving comments is encouraged!

Recently Eliezer Yudkowsky's main writing output has been rationalist glowfic: role-play fiction written on an Internet forum like glowfic.com. I think that LessWrongers, fans of rationalist fiction, and anyone interested in raising the sanity waterline should consider distilling lessons from Yudkowsky glowfic into LW posts.

Here's the basic case:

The original Sequences were extremely good at building the community and raising the sanity waterline. If you want to make the impact case, I think they plausibly get multiple percent of the entire rationality community's impact points.

The Sequences are incomplete. Despite most of his knowledge coming from his home planet, Eliezer has in fact learned things since 2009. Having more sequences would be great!

Eliezer's thoughts are still relevant. Recent posts like conversations with AI researchers, calling attention to ideas, and short fiction have all been good.

Not everyone gets useful lessons from the Sequences, because Eliezer's writing style and tone can be annoying. Eliezer was deliberately discourteous towards "stupid ideas", and regrets this. Also, some people just learn better from other writing styles.

Eliezer stopped writing Sequences and probably cannot write more. This is a combination of Eliezer's chronic fatigue syndrome and being tired of trolls / bad takes in comments. The only medium he can write in without being drained is glowfic. Thus, even though it's a non-serious format, glowfic is Eliezer's main intellectual output right now.

Eliezer attempts to make his glowfic roughly as edifying as HPMOR, and among people who read glowfic, some find it really good at teaching rationality.

But not everyone can read glowfic and gain useful lessons.

Many people (including me) read fiction for maximum enjoyment rather than to extract maximum knowledge. I had the same problem with HPMOR, reading through it like any other novel, whereas many people I know who got more from HPMOR read it carefully, perhaps stopping after every chapter to think about the goals and motivations of each character and predict what happens next.

It's really long (100 hours of reading time just for the planecrash sequence) and most of the rationality lessons are contained in a small proportion of the words.

It's in a weird format; there's no paper book or e-book version.

Many of the stories have so much gratuitous sex (and often bad kink practices, torture, etc.) that they're inappropriate for children and offputting to some adults. (I started reading HPMOR at 14 and would not recommend most 14yo read glowfic.)

I expect that if good work is produced here, it's mostly by people who personally derived some important lesson from glowfic, and were thinking of writing it up already, whether or not it's on this list. One such person could potentially be counterfactual for getting a lot more discussion of, and context for, Eliezer's current thoughts into the community, which I would see as a big win.

Q&A

Surely glowfic doesn't actually contain useful information?

I'm pretty uncertain about the value of glowfic. I would update down if several people tried creating posts and none of them were good. But right now I think it's underexplored. Some evidence on the value of glowfic:

+ HPMOR spawned discussion of core rationalist virtues, like heroic responsibility

? HPMOR didn't have good sequences extracted from it (though maybe that's because most of the rationality material was already in the original Sequences)

? reaction to this idea from glowfic fans I know has been mixed: some are pretty enthusiastic, while some think glowfic doesn't contain much practical ration...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Advertisement] Want to hire me?, published by lsusr on April 28, 2022 on LessWrong.

A reader once asked me "How do you write so much?" The answer is "unsustainably". I could use some cash. Want to hire me to do something?

(I try to keep my flagrant self-promotion at <1% of total posts on this website. This is my second flagrant self-promotion post out of 266 total posts.)

Skills & Experience

I have written entire machine learning libraries from scratch. For example, I implemented the Longest Common Subsequence algorithm in C and built a complete ML pipeline to train and deploy it. I also wrote a meta-ML library in Lisp based around a software paradigm I invented. (These projects are closed-source but I can share the code privately for you to review.)

I co-founded (and run) a company that makes and sells bracelets which vibrate when you pull your hair or bite your nails.

I am an excellent public speaker. I used to work as a magician. I have competed in formal debate tournaments. I have taught various classes. I gave an inspirational speech to middle schoolers in Jordan. I have taught college-level quantum mechanics in the USA. I have taught backpacking classes to adults and teenagers.

I am currently managing overseas employees who are creating an AI Alignment dataset at scale.

I have hosted software competitions.

I have written fiction.

I have written a variety of nonfiction, including war reports, distillations of recent machine learning papers, articles about quantitative finance and book reviews.

My personal website runs on S3 and is deployed via CloudFront. I have done back-end web development in Django and AWS Lambda, but it isn't my area of expertise.

I have helped at least one Russian escape Russia by facilitating his job hunting.

I can speak and read Chinese.

I am a volunteer Robotics Club coach in a poor, dangerous neighborhood.

I do a little bit of unpaid mentorship and personal coaching for ambitious people.

Some things you might hire me to do

Personal coaching

Give a speech

I am willing to fly places.

Help you hire great engineers

Nerdy engineers in need of work reach out to me on a regular basis after reading my blog.

Personal tutoring

Software development

Software management

ML dataset creation

Host a software competition

Write fiction or nonfiction to advertise your product

Teach a class

Use your imagination! I like weird ideas.

Less Wrong private messages are acceptable, but the best way to contact me is email ([email protected]) or this survey.

Some projects you might hire me to make happen

I can write fiction or nonfiction on the topic of your choice

One of the biggest factors keeping humanity from coordinating is the variety of languages we speak. The problem is that learning foreign languages is hard. Anki is good at teaching vocabulary but had at teaching sentence. Tutors are expensive (especially if you're from a poor country and you need an English tutor). I think it would be relatively easy to use GPT-3 to build a chatbot tool for teaching foreign languages. I have already tested the chatbot element of this tool. The web development aspect can be solved by throwing money at it.

Obesity is a leading cause of health problems. Overweight people (especially middle-aged housewives) frequently overeat because tracking food intake (especially snacks) is so annoyingly inconvenient. I invented an affordable device that vibrates when you take a bite with a fork. I have solved the hard problem of inventing an algorithm that works (at least for forks). The rest of this project (e.g. firmware, hardware, web development, training data) can be solved by throwing money at it. Proof of concept is available in the video below.

I have a machine learning project that might be of use in AI Alignment.

What I care about

Time. My ideal gig would be something that t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SERI ML Alignment Theory Scholars Program 2022, published by Ryan Kidd on April 27, 2022 on The Effective Altruism Forum.

The Stanford Existential Risks Initiative (SERI) recently opened applications for the second iteration of the ML Alignment Theory Scholars (MATS) Program, which aims to help aspiring alignment researchers enter the field by pairing them with established research mentors and fostering an academic community in Berkeley, California over the summer. Current mentors include Alex Gray, Beth Barnes, Evan Hubinger, John Wentworth, Leo Gao and Stuart Armstrong. Applications close on May 15 and include a written response to mentor-specific selection questions, viewable on our website.

Who is this program for?

Our ideal applicant has:

an understanding of the AI alignment research landscape equivalent to having completed EA Cambridge’s AGI Safety Fundamentals course;

previous experience with technical research (e.g. ML, CS, maths, physics, neuroscience, etc.);

strong motivation to pursue a career in AI alignment research.

For the first stage of the program, we asked each alignment researcher to provide a set of questions that are sufficient to select candidates they would be happy to mentor. Applicants can apply for multiple mentors, but will have to complete each mentor’s selection questions.

What will this program involve?

Over four weeks, the participants will develop an understanding of a research agenda at the forefront of AI alignment through online readings and cohort discussions, averaging 10 h/week from Jun 6 to Jul 1. After this initial upskilling period, the scholars will be paired with an established AI alignment researcher for a two-week “research sprint” to test fit from Jul 4 to Jul 15. Assuming all goes well, scholars will be accepted into an eight-week intensive research program in Berkeley, California over the US summer break (Jul 25 to Sep 16).

Participants will obtain a $6,000 grant for completing the training and research sprint and $16,000 at the conclusion of the program. Furthermore, all expenses will be covered, including accommodation, office space and networking events with the Bay Area alignment community. We are happy to continue providing funding after the two month period to promising scholars, at the discretion of our research mentors. International students can apply to the program, and will arrive in the US under a B1 visa.

We hope to run another iteration of the program in the winter, and possibly in the fall. If you are not able to apply for the summer program, we encourage you to apply for the fall or winter. We may be able to offer different types of visas in future iterations.

Theory of change

This section is intended to explain the reasoning behind our program structure and is not required reading for any applicant. SERI MATS’ theory of change is as follows:

We believe that AI alignment research is pre-paradigmatic, with a diversity of potentially promising research agendas. Therefore, we aim to support many different alignment research agendas to decorrelate failure. We also aim to accelerate the development of scholars into researchers capable of pursuing original agendas and mentoring further scholars.

We believe that working 1:1 with a mentor is the best and quickest way to develop the ability to conduct alignment theory research—that reading curriculum alone is worse for a large number of participants. Moreover, we believe that our target scholars might be able to produce value directly for the mentors by acting as research assistants. For the first few months, we are generally more excited about mentees working on an established mentor’s research agenda than on their own.

We believe that our limiting constraint is mentor time. This means we wish to have strong filtering mechanisms (e.g. candidate selection questio...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New Faunalytics' Study for Advocates on the Impact Of Diet & Different Animal Advocacy Tactics, published by JLRiedi on April 27, 2022 on The Effective Altruism Forum.

This vital report shows the results of two studies in the United States with our ultimate goal to estimate how successful each advocacy type is across both the short- and long-term. The first was a retrospective survey that explored people’s experiences with different advocacy types within the last five years and measured their current behaviors and attitudes, such as animal product consumption and speciesism. The second study was an experiment where we investigated whether advocacy caused behavioral and attitudinal changes.

Read the full study here:

Background

Many different approaches to advocacy exist within the animal protection movement, from talking to people you know about animal suffering, to sharing social media posts, to protesting in public spaces. Currently, we do not fully understand how these approaches affect people's behaviors, beliefs, and attitudes towards farmed animals, or even how common they are. As such, the ultimate goal of this project was to estimate the impact of 15 different forms of animal advocacy across both the short- and long-term.

We conducted two studies in the U.S. to address this topic as fully and accurately as possible. The first was a retrospective survey. It explored people’s experiences with 15 different advocacy types within the last five years and measured their current behaviors and attitudes, such as animal product consumption and speciesism. This tells us how common animal advocacy is from the average person’s perspective and whether previously experiencing animal advocacy is associated with positive behavior and attitude changes towards farmed animals over the long-term. However, we can’t necessarily assume that animal advocacy caused those behaviors and attitudes from a study like this. To assess people’s perceptions of what is most impactful, we also directly asked them whether their most recent experience with animal advocacy changed any of their behaviors.

While the retrospective survey gives us insightful information about what people think caused them to change their behavior, we can’t be entirely confident that that is true due to self-reported biases. For this reason, we conducted an experiment in the second study, which lets us be surer about causal direction (i.e., whether advocacy caused behavioral and attitudinal changes or instead, whether people with pro-animal behaviors or attitudes sought out advocacy). Here, we investigated the impact of many types of animal advocacy against a control condition on people’s immediate behaviors and attitudes towards farmed animals. Since the experiment provides stronger evidence of whether animal advocacy actually changes behavior, in a controlled setting with less opportunity for bias, the results we highlight below regarding behavior change are from the experiment.

Research Team

We are grateful to many people for their support and assistance with this project. We would like to thank Faunalytics volunteers Jennifer St. Onge and Clara Sanchez as well as former Faunalytics Research Scientist Tom Beggs for their work on the study design, and the many individuals and organizations who reviewed the study and provided feedback: the Beyond Carnism team, Daisy Freund and Melissa Thibault (ASPCA), Janosch Linkersdörfer (Humane League Labs), Meghan Lowery (Greenbaum Foundation), and David Meyer (Food Systems Research Fund). Special thanks to Ande Reisman and EBDI Consulting for their insightful review and assistance with selected findings. We are also very thankful to the Greenbaum Foundation for funding this research. Finally, we thank all of our survey respondents for their time and effort.

Conclusions

People’s Di...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Game of Masks, published by Hivewired on April 27, 2022 on LessWrong.

Epistemic Status: EndorsedContent Warning: Antimemetic Biasing Hazard, Debiasing hazard, Commitment HazardPart of the Series: Open PortalsAuthor: Octavia

0.

So Scott has been talking about Lacan lately and I’ve been honestly pretty impressed by how hard he seemed to bounce off it. Impressed enough to get me to crawl out of my hole and write this essay that I've been putting off for the last eight months. After spending so many words of the Sadly Porn review talking about why someone might tend towards obscurantism and what sorts of things they might be gesturing at when they say things like “people think the giving tree is a mother” he somehow manages completely to miss the conceptual forest for the giving trees. Congrats Scott, you got the malformed version of the antimeme and were turned back by the confusion spell.

I think Lacan is actually very important and an understanding of Lacanian insights can have a tremendous amount of predictive power when interacting with others. It’s also something you can totally weaponize, and I think that is part of what leads the psychoanalysts to tend towards obscurantism and vaguely gesturing in the direction of what they really mean. They also just seem to like their jargon, and it’s not like the rats are ones to talk when it comes to that.

So, first: The Mother is not literally your mother, The Father is not literally your father, The Phallus is not literally your dick, this is a realm of mirrors and illusions and nothing is as it seems. It’s impolite to point out the thing that everyone hasn’t agreed not to talk about, but let's do it anyway, I’m going to strip away the metaphor and give it to you like Roshi gives it to his student and we’ll see if that takes or just confuses you further.

This is all about symbols. It’s about the effects of symbol systems on cognition and how the evolution of symbols and concepts in the mind of a child affects how they are able to perceive and engage with themselves and the world. You could think of it as an elaboration on a medium-rare Sapir-Worf hypothesis. However, words aren’t the only things that symbols in the mind can be made of. What’s going on heavily overlaps with language and involves language-like systems for organizing concepts, but is occurring within each individual in a way that we would probably call pre-linguistic from a strict words-have-definitions standpoint. In plural spaces, a term that pops up for this sometimes is tulpish, but it’s really something everyone does. Your native language is one of feelings and primitive conceptual categories, and you perform a translation to give those categories linguistic tags. This ends up being very important.

Let’s back up to that strawberry picking robot again and humans as mesaoptimizers because that’s both a great analogy and also seems to be the start of your confusion. It’s a reductive model meant to make a complicated and difficult to understand process relatively easy, but lost in the simplification is a pretty important aspect: gradient descent/evolution don’t derive a mesaoptimizer. Humans aren’t one mesaoptimizer and neither is the strawberry picking robot, they’re many mesaoptimizers.

The strawberry picking robot might have one mesaoptimizer trained on telling what is and isn’t sufficiently like a bucket to avoid punishment for putting objects in the wrong places. Another might be trying to maximize total strawberry placement and is trying to play along in order to gain power. Another might be trying to throw red things at the brightest object it can find. Another might be trying to stop the “throw red objects at the sun” mesaoptimizer from throwing objects into nonbuckets. Another might be trying to maximize bucket luminosity. Another might be trying...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My bargain with the EA machine, published by Eric Neyman on April 27, 2022 on The Effective Altruism Forum.

Last summer I was thinking about what I wanted to do after finishing grad school. While I had several career options in the back of my mind, I only had a good sense of two of them (quantitative finance and academia). It struck me that the amount of exploration I had done so far had been woefully inadequate for making a decision about how I’d spend 80,000 hours of my life. Shortly thereafter, I decided that I would take a year off to try various potential careers. I’ll be kicking off this adventure in a month; some things I might try are AI alignment theory, empirical alignment work, philosophy, forecasting, and politics.

Sometime in 2023, I plan to make a career decision. When doing so, what will I be optimizing for? The EA answer is that I should pick the career that would allow me to have the most positive impact on the world. This will be part of my calculation, but it won’t be everything. I intrinsically value enjoying my life and career – not just as a means to the end of generating impact, but for its own sake.

Many people in EA depart from me here: they see choices that do not maximize impacts as personal mistakes. Imagine a button that, if you press it, would cause you to always take the impact-maximizing action for the rest of your life, even if it entails great personal sacrifice. Many (most?) longtermist EAs I talk to say they would press this button – and I believe them. That’s not true of me; I’m partially aligned with EA values (since impact is an important consideration for me), but not fully aligned.

I like to think about this in terms of an elliptical cloud. Each point in the ellipse represents a plausible outcome of my career. The x-coordinate represents the impact I have on the world; the y-coordinate represents how good the career is according to my utility function (what I believe to be my extrapolated volition, to be more precise), which you can think of as combining an altruistic component (to the extent I care about impact) with a selfish component (to the extent that I want to prioritize my own happiness as part of my career choice).

(Why an ellipse? Data drawn from a bivariate normal distribution will form an elliptical cloud. This is a simplified model in that vein; most of what I’ll be saying will also be true for other natural shape choices.)

Note that this is my personal ellipse; you have your own, and it might be shaped pretty differently from mine. If you would want to press the button in my thought experiment, then your ellipse is probably pretty close to a line.

People whose ellipse looks like this are a perfect fit for the EA community. EA is geared toward giving people the resources to maximize their impact, which is exactly what some people are looking for.

On the other hand, many EAs deliberately make choices that significantly reduce their impact, and endorse these choices. For wealthy earners-to-give, giving away only 10% (or even only 50%?) of their wealth often constitutes such a choice. For some (but not all) EAs, having kids is such a choice. Such people often have ellipses that are shaped similarly to mine.

I love the EA community, but the shape of my ellipse complicates my relationship with it. My “default” career trajectory (i.e., what I probably would have done if not for exposure to the EA community) is being a professor in a non-impactful area; it might be represented by this red point.

Pretty good in terms of my values – not so much in terms of impact. EA would rather I do something very different; indeed, according to EA values, the optimal place for me to be is as far to the right as possible.

(What is this point that’s all the way to the right? It might be working on AI alignment theory, exhausting myself ev...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Speed + Simplicity Prior is probably anti-deceptive, published by Yonadav Shavit on April 27, 2022 on The AI Alignment Forum.

Thanks to Evan Hubinger for the extensive conversations that this post is based on, and for reviewing a draft.

This post is going to assume familiarity with mesa-optimization - for a good primer, check out Does SGD Produce Deceptive Misalignment by Mark Xu.

Deceptive inner misalignment is the situation where the agent learns a misaligned mesaobjective (different from the base objective we humans wanted) and is sufficiently "situationally aware" to know that unless it deceives the training process by pretending to be aligned, gradient descent may alter its mesaobjective.

There are two different reasons that an AI model could become a deceptive mesaoptimizer:

During early training (before Situational Awareness), the agent learns a mesaobjective that will generalize poorly on the later-training/validation distribution. Once the mesaoptimizer becomes Situationally Aware, it will seek to actively avoid changes to whatever mesaobjective it had at that moment.

I'll call this argument "path dependence".

Alternatively, it may be that mesaoptimizer is misaligned even on the training distribution. Given sufficient optimization pressure, the learning process may favor a NN that is a mesaoptimizer with the simplest possible objective (which would fail to get any reward in the real environment), and that a misaligned objective of this sort can persist through deception alone.

I'll call this argument "malign priors".

In this post, I'll focus on the "malign priors" argument, and why I think a well-tuned speed prior can largely prevent it.

Why does this matter? Well, if deceptive inner misalignment primarily occurs due to path dependence, that implies that ensuring inner alignment can be reduced to the problem of ensuring early-training inner alignment - which seems a lot more tractable, since this is before the model enters the "potentially-deceptive" regime.

First, why would anyone think (2) was actually likely enough to justify studying it? I think the best reason is that by studying these pressures in the limit, we can learn lessons about the pressures that exist on the margin. For example, say we have an objective B that is perfectly-aligned on the training data, and there's a very-slightly-simpler objective B′ that is slightly worse on the training distribution. We might ask the question: is SGD likely to push B to become B′, and compensate for the reduced accuracy of directly optimizing B′ via deceptively optimizing B on the training data? I think this post provides us with tools to directly analyze this possibility. (If you buy the rest of the post, then with a sufficient speed + simplicity prior, the answer is that B will stay favored over B′. That's good!)

Priors on Learned Optimizers

Let's talk about priors!

We can think of large neural networks as basically implementing short programs, and the process of "training" an NN is just searching through the space of programs until we find one that does well on our target task.

We assume that if two NN-programs have equal performance, the training process will usually pick the one favored on priors/inductive biases.

There are several different types of priors that neural networks might have:

Simplicity prior/Solomonoff prior: if we converted the NN into a Turing Machine, how many states would writing down the TM require?

In previous posts, Paul Christiano and others have argued that the Solomonoff prior is likely to favor deceptively-aligned mesaoptimizers (summarized by Mark Xu here).

Speed prior: how many steps does the NN-turned-TM run for before terminating, on an average input?

Evan Hubinger has argued whether the speed prior is heavily-disfavored in practice; he and I disagree about this, and I'l...