Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Operations is really demanding, published by jwithing on March 29, 2022 on The Effective Altruism Forum.

Operations is really, really demanding.

I work as a software product manager, but spend two weeks a year as a reservist at a US Navy operations center. There I work 12 hour shifts where I send chat and email messages, make and take phone calls, and otherwise try to be of use. Before I was a full-time software PM, I was a full-time submariner where I project managed our ship's maintenance.

My current workplace is a large e-commerce company with heavy operations footprint. My experiences there confirm what I already knew from the Navy--operations is a really demanding field.

Operations is so, so important! How do we make it less demanding to its practitioners?

What is operations?

I want to play a little bit with the version of operations discussed in the 80,000 Hours career guide. Organizations have people that create things (medicine/widgets/research), people that highlight the value of these goods (salespeople/marketers/outreach), and then people that make sure all of the above goes smoothly--those are the operations folks.

The career guide focuses on operations as the support class of the organization: HR, recruiting, finance, etc. But if GiveDirectly, Against Malaria Foundation, or any org that delivers aid/goods are within the scope of "high-impact organizations," then operations might be the org's core competency, rather than something (relatively) behind-the-scenes. Their operations teams are responsible for delivering the products.

Both forms of operations are hyper challenging and undervalued.

Why is operations especially demanding?

Operations is especially demanding because the scope and impact of its errors. Because operations deals heavily with human systems, the scope of ways things can wrong is nearly unbounded. You can't debug or test human systems. You can provide all the thoughtful written documentation on "How to X" but humans can still find a way to get it wrong. This is unlike code or other forms engineering--those objects do precisely what you tell it to do!

So not only are the failure modes unbounded, the impact of an operations team's error can be quite high. Think about botching payroll, event execution, HR policies, or having an unprepared executive. An organization can't readily undo these things or make them "right." A poor marketer/salesperson/researcher has limited downside by comparison--they just cost you money. Bad ops exacts institutional damage.

OK, but what exactly about operations especially demanding?

Operations staff are keenly aware of the stakes of their work, and tirelessly work to reduce the chance of errors. This is super demanding! Consider the following real-life examples I've witnessed or personally dealt with in my most recent two weeks of operations work. Perhaps these are military heavy, but I'm sure they have analogs elsewhere:

It's 1am on a Thursday night. The flight scheduled for midnight didn't happen, and now your team is running behind schedule. You get a call about this, and need to get some guidance from your boss so now you call them. Both of you will need to be in the office by 8am.

It's 6pm on Sunday night. The scheduler for a Royal Navy admiral calls you to get something on the calendar with your US Navy admiral tomorrow. The earlier in the morning the better--can you accommodate?

Your executive gets a daily report on operations, including on the weekend. The last few times the report hasn't been made to standard so you need to come in Saturday and Sunday mornings to make sure they're done properly.

The above may seem pretty mundane, and it is if you work in ops. But human systems run 24/7. The cost of operations errors is high. Combine those two things and operations staff are never really quite off-the-c...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA should taboo "EA should", published by Lizka on March 29, 2022 on The Effective Altruism Forum.

Tl;dr:

In Forum posts and in conversations, you’ll hear (and find yourself saying) that “EA should” do something. But “EA” is not an agent, and it’s hard for “EA” to do anything. I think we should try to notice these moments and explicitly think through who the subject of that sentence could be.

Disclaimers:

Written quickly. I might have missed important considerations — feel free to point them out, and I may correct the post or add them in!

I am not claiming that all “EA should” posts and statements in the past have been unhelpful. I just think that many of them would be (even) more helpful if they were reworded.

I’m not including specific instances of the issue because I don’t want to single anyone out. My guess is that most of us have said something like this, as I certainly have!

Why “EA should” is an unhelpful phrasing

“EA should” is often vague.

Statements like this can be interpreted in too many ways, which makes them less helpful than they could be.

For instance, “EA should do more to help with COVID” could mean anything from “the EA movement has underestimated the severity of COVID” to “it is your duty as an EA to help with COVID” (and beyond).

Vagueness also makes you much harder to argue with, and I believe that being easy to argue with is a virtue.

Responsibility gets diffused.

Nobody naturally thinks “oh, you’re telling me that I should do this,” and so fewer people take ownership of the issue.

Suppose that you’ve discovered a very significant gap in EA work. If you write a post like “EA should do more to help with COVID,” you might mean, “EA funders should fund vaccine distribution” or maybe “someone with operations skills should improve vaccine distribution systems.” With these rewordings, grantmakers or people with ops skills might sit up and notice that this post or message is for them.

(It’s almost a passive formulation.)

Also:

You are part of EA.

This might not apply in all situations, but— if you think that, say, 80,000 Hours should have a problem profile for a problem that they’ve neglected to list, you might just want to write a version of the problem profile and publish it. Or, if you think someone should start a PA Agency, you could try to directly make that happen.

(Related EA Forum tag.)

Specifying the statement could help you check that it’s true or identify a more specific bottleneck.

Suppose you find yourself saying, “EA should rely more on personal assistants.” Then you think to yourself, “ok, I guess I’m saying EA grantmakers should be willing to fund you for PAs.” So you go check, and see that apparently, grantmakers are funding PAs. So maybe you think to yourself, “But there are still lots of EAs without personal assistants who should probably still have some. Maybe 80K should recommend that people go into that as a career path!” And you check, and see that that, too has happened. I don’t know what the real bottleneck here is, or if it’s even a problem, but it’s possible that at this point you conclude that either there are serious coordination issues (in which case maybe you’ll write a post about how valuable it would be if someone could coordinate a personal assistant service for EAs) or that there’s just not that much demand for one reason or another.

Types of “EA should” moments and how we can turn them into something more useful (examples)

Below are some example “EA should” statements and their translations (rephrasing the statements without using “EA should”), sorted by the types of meanings they would have.

Someone in the EA community should do a specific thing

“EA should have a PA agency!”

Might become “Someone (maybe you!) should start an EA PA agency!”

“EA should fund XYZ”

Might turn into “the Long-Term Future Fund should fund ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Predicting Polygenic Selection for IQ, published by Metaculus on March 28, 2022 on The Effective Altruism Forum.

This is a linkpost for an essay by Ryan Beck appearing in the Metaculus Journal. Readers are encouraged to read the essay in the Journal where they can engage directly with the author and make their own forecasts.

Creating embryos in a petri dish, examining the genes of the resulting embryos, and then implanting the embryo with the highest probability of creating the most intelligent child sounds like something out of science fiction. But it's closer to becoming scientific reality than some might expect. A Metaculus question asks whether genetic engineering techniques will be commercially available by 2050 that can raise IQ by 10 points. Polygenic embryo selection seems to be the most promising technology with a chance of achieving this by far. In the following essay I review the current state of the art of polygenic embryo selection and use this information to provide a forecast for this question.

By 2050, will genetic engineering techniques be available which can raise IQ by 10 points?

Disclaimers and Acknowledgements

This analysis is informed by a fairly superficial literature review. I've spent a decent amount of time trying to understand the information I present below, but I have no background in genetics or biology and my understanding of the following information is still pretty weak, and I've only read some of the papers I've linked, while skimming others. While I tried to cover the general consensus and differing views related to these topics, it's possible the following information is not representative of the current state of the art and consensus of the field. Critiques and corrections are welcome.

This originated as a Metaculus comment which was motivated by the Impactful Forecasting Prize, a really cool initiative to incentivize sharing forecasting rationales on a set of Metaculus questions judged to be impactful.

This Metaculus Journal post is an updated version of the original comment and corrects at least one error that significantly impacts my forecast. See the footnote at the end for a description of the error and see my original comment to see the prior version of this analysis with the error included.¹

I've also updated this analysis to incorporate feedback from @alwaysrinse and @gene_smith. Thanks to both of them for sharing their thoughts. I recommend checking out the original comment thread for their thoughts and perspectives.

My Priors

Before digging into the literature my forecast on the above question was 25% based on a skepticism of high heritability estimates for IQ, a general impression that polygenic scores are still far off from having strong predictive power, and the general view that it was too optimistic to expect things to develop so rapidly. The following analysis may be biased toward these priors.

Background and Primers

This section provides primers that I think are the most helpful for anyone without previous exposure or with limited previous exposure to the topic wanting to understand the background information more thoroughly. This can be skipped for anyone who is already familiar with the subject or is primarily interested in the broader takeaways.

IQ, explained in 9 charts - If you're not sure how important IQ really is or whether it measures something meaningful this is a great article to start with. The takeaway is that higher IQs are linked with many other positive life outcomes, such as increased wealth and reduced mortality.

Polygenic Selection of Embryos - A nice Metaculus Journal post by @galaga which gives a valuable overview of the process of selecting embryos based on polygenic scores and its use for reducing disease risk, as well as potential for increased adoption of this technology in the future.

Twi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ukraine Post #7: Prediction Market Update, published by Zvi on March 28, 2022 on LessWrong.

It has been about two weeks since I last looked at the prediction markets for the invasion of Ukraine. Since then, prices on many markets have shifted quite a lot, so it is once again time to examine what the markets seem to be telling us.

Due to time constraints of various sorts, I have needed to split off both the discussion of how we might improve Metaculus, Polymarket and other prediction markets (other than a few markets I suggest adding) and also discussion of big markets like grain, oil and currencies that can serve as noisy but very useful predictive proxies. I hope to get to all that soon.

Kyiv

I still see some reports that Russia is attempting to ‘encircle’ Kyiv, but I no longer take them seriously. They are setting up defensive positions in various places, and if anything Ukraine is launching counter-offensives. Russia’s own official line is that the attack’s goal was to tie down Ukraine’s forces.

That doesn’t mean they couldn’t possibly ever do it, but it would presumably only happen after a complete Ukrainian collapse elsewhere.

Will Kyiv fall by April 1? (14% → 0%)

No. Can’t happen at this point. This closed so can’t move any lower.

Will Kyiv fall by June 1? (39% → 3%)

Again, no. In theory could happen, but it won’t.

The circles are my predictions. You can see the problem with ‘keeping pace’ with things as something falls over time. If I’d had better tools I’d have been lower along the way.

Will Kyiv fall by 2023? (45% → 9%)

I mean, possibly? But again, not really, no.

If Kyiv falls in 2022, when will it happen?

Notice the drop-off in participation. Here is what the distribution curve looks like (I set the input to flat to avoid distraction, ignore that).

Note that the 6% chance of Kyiv falling after June 1 but during 2022 has remained unchanged. The market already understood that there was little chance of Kyiv falling slowly rather than there being a settlement or things becoming a stalemate. Given that a third of worlds where Kyiv falls earlier have now been ruled out, this could be viewed as being conditionally slightly more likely, which is a mistake, it seems clearly conditionally less likely given Russia’s increasing focus elsewhere and willingness to settle for territorial gains.

It also doesn’t make sense in the same world as a collapse of the chances of it happening in April and May, to then have changes after May to have gone somewhat up, especially given the above graph. So there’s a small contradiction here, likely due to inertia and lack of arbitrage.

The key thing to know is that two weeks ago the market was saying Kyiv was 45% to fall this year, and if Kyiv falls Russia probably ‘wins’ a major victory. Now that is down to 9%.

On a related note: Will Medvedchuk become president of Ukraine before 2024?

There is still a 9% chance Kyiv falls in 2022 and then there’s all of 2023. But he’s also one choice out of many possible choices even if Russia does fully win.

Kharkiv

Will Russia get control of at least three of (Kyiv, Lviv, Odesa, Mariupol, Kherson and Kharkiv) by June 1? 70% → 7%

Will Kharkiv fall by June 1?

Even more than before, the overall market on the six cities is now mostly a market on Kharkiv. Kyiv, Lviv and Odesa are highly unlikely to fall without a general collapse. Kherson and Mariupol will count as having fallen, or if they don’t then Kharkiv was in almost no danger. So these are now the same. It is a good sign that Metaculus users collectively reached the same conclusion.

There are no individual markets on the other four cities. Given what is happening, I do propose that we have a new market on Kherson:

M1. Will Ukraine retake Kherson by June 1?

Further Invasions

Will Belarus invade Ukraine by June 1, 2022? (new → 12%)

If this was go...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Initial research on social movements & protest, published by James Ozden on March 28, 2022 on The Effective Altruism Forum.

Who we are (in a nutshell)

Social Change Lab is a new EA research project that is conducting research into whether protest could be a cost-effective way to achieve positive social change. We’re initially focusing on climate change and animal advocacy, and trying to understand whether protest should play a larger, or smaller, role in accelerating progress in these cause areas. Whilst more speculative, we’re also considering if protest can play a role for other issues, such as pandemic prevention, immigration reform, and existential risks.

We put out some preliminary research in November last year, looking at the role that Extinction Rebellion UK played in making climate change more salient in the UK, and giving an overview of a small amount of the literature on the efficacy of protest. At the moment, we’ve just finished two literature reviews: one on the impact of protests and Social Movement Organisations (SMOs), and one on factors that affect success rates for protests. We are grateful to have received funding from the EA Infrastructure Fund to cover our expenses until July 2022.

Context

Whilst EA has the goal of doing as much good as possible, it’s not clear what the best ways to do this yet are. In a survey of 40 EA community leaders, people were most excited about greater “EA exploration” i.e. greater research into possible causes, interventions and ideas that could allow us to do more good. In particular, surveyed community leaders were interested in bringing in ideas from outside of traditional EA discourse. Similarly, as noted by Charity Entrepreneurship, there is lots of opportunity for organisations that engage in Exploratory Altruism, and we intend to explore the possibility that protest could be more cost-effective than other EA-recommended routes to change.

Generally, as a community, we’ve been pretty research-focused in trying to understand the best ways to improve the world. However, much of this work hasn’t touched on how social movements interact with the world, and how they affect social change. We see this as an opportunity to add value by providing research that addresses some unanswered questions about the role of social movements in improving the world.

Specifically, we think this work can be valuable to two audiences in particular:

Philanthropists seeking to fund the most effective routes to positive change in their respective cause area, where SMOs utilising protest could be a contender. For example, if we discover that animal advocacy protests are more cost-effective in reducing animal suffering relative to existing funded work, it makes sense we also fund these opportunities (all else being equal) if we seek to maximise our impact.

Social movements who utilise protest currently, who can employ and integrate best practices from social science literature to make their campaigns more effective.

Why Protest?

We believe that there is moderate evidence that protest can have an impact on public opinion, voting behaviour, public discourse, legislator behaviour, and corporate behaviour. Our interest in protest began by examining the impact of Extinction Rebellion, whose protest activity coincided with an increase in concern around climate change in the UK - as can be seen in the figure above. The concern about climate change in Europe and the United States has been increasing rapidly, and we believe that it is plausible that this increase has been at least partially due to protest activity by Extinction Rebellion, Fridays for Future, and Insulate Britain. That being said, it is clearly not the case that the increased salience of climate change is sufficient evidence to demonstrate that there is a causal connection between protest...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Primer on God, Liberalism and the End of History, published by Mahdi Complex on March 26, 2022 on LessWrong.

Epistemic status: uncertain, potentially confused.

Medieval Christendom

Medieval Europe’s political life was extremely fractured. The old continent was ruled by a variety of often feuding kings, emperors, lords and princes. Yet, political and cultural life still had a certain level of cohesiveness thanks to unifying force of Christianity and the Catholic Church. Indeed, the Church provided medieval Europe with a common rule of law by setting, for example, rules for marriage and the inheritance of property. This allowed the Church to eventually gain possession of between a quarter and third of all lands in most European countries.

In the late eleventh century, the church also gained the power to appoint bishops, after what would become known as the investiture controversy, a power struggle lasting a few decades between popes and Holy Roman Emperors. Thus, the Church managed to establish itself as an important power broker, commanding loyalties and exercising sovereignty in a manner that transcended borders and traditional allegiances.

The investiture controversy is illustrative of Christianity’s awkward relationship with politics, and the tensions that have existed between secular rulers and religious authorities, ever since Christianity was made the official religion of the Roman Empire in 380 CE. Due to its origins as a minority sect under Roman rule, Christianity offers little guidance in the crafting of a just Christian political order. Unlike in Islam, there is no example for the Christian ruler to follow.

And yet, unexpectedly thrust into the position of dominant religion in Europe, religious and secular rulers had to wrestle with the question of how to apply Christian doctrines to political life. After all, how could a religion purporting to have knowledge of the will of the creator of the Cosmos – the Lord of the Heavens and the Earth – not have anything authoritative to say on the matter of politics?

Thus, Christian political-theology has been characterized by the tension between the “City of Man,” the temporal realm of nature, suffering, sin, and political necessities, and what Saint Augustine called the “City of God,” the eternal realm of salvation, grace and divine justice.

Luther

Europe’s precarious religious unity was shattered with the publication of Martin Luther’s Ninety-five Theses in 1517. He challenged the authority of the pope, and a number of doctrines of an increasingly corrupt Catholic Church, most famously the selling of indulgences.

The printing press allowed Luther’s ideas to spread almost unrestricted across Europe, starting the bloody European wars of religion. The historian and political scientist Mark Lilla describes: “doctrinal differences fuelled political ambitions and vice versa, in a deadly, vicious cycle that lasted a century and a half. Christians addled by apocalyptic dreams hunted and killed Christians with a maniacal fury they had once reserved for Muslims, Jews and heretics. It was madness.” At the height of the conflicts, parts of Germany lost over half of its population.

The Peace of Westphalia of 1648 finally marked the end of the conflicts, and is often credited with being the origin of the international order of independent sovereign states we have today. The conflicts ravaged much of Europe, and yet no agreement on the true religion seemed in sight, and so the signatories of the treaty agreed to the principle of Cuius regio, eius religio, “whose realm, their religion." Each ruler was given authority over religion in their territory, and was not to interfere in the religious affairs of neighbouring countries.

Hobbes

The Peace of Westphalia however did not mean the end of political-theology. Religious minorities w...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Towards a better circuit prior: Improving on ELK state-of-the-art, published by Evan Hubinger on March 29, 2022 on The AI Alignment Forum.

This post is the result of joint work with Kate Woolverton. Thanks to Paul Christiano for useful comments and feedback.

The basic circuit prior setup

We’ll start with the basic setup that we’re trying to improve upon, which is trying to solve ELK via the use of a Boolean circuit size prior. Previously, I summarized Paul, Mark, and Ajeya’s argument for why this might work as follows:

As in the ELK report, there is a plausible argument for why there exists a speed prior that would prefer the direct translator to the human imitator. Naively, the problem with the speed prior here is that the computation required for the human imitator is proportional to the size of the human's Bayes net, whereas the computation required for the direct translator is proportional to the size of the model's Bayes net—and in the superhuman limit we should expect the latter to be substantially larger than the former.

The argument in the ELK report, however, is that while this argument is valid in the limit, there's reason to believe it might be invalid for all the finite cases that we care about. That's because perfect inference in either Bayes net, and thus perfect loss, shouldn't be possible in any finite case. Thus, the performance of the ontology mapping function, and thus its loss, should be proportional to how much computation it puts into its inference task—for which the direct translator has a big advantage, since it gets to reuse the computation performed by the model.

The obvious response here, and the response that is given in the ELK report, is that the above argument is very fragile—it relies on inference in the human's Bayes net being too hard to always get right on the training distribution, which is a strong assumption both about the difficulty of inference and the complexity of the training data.

Furthermore, as the ELK report also notes, it's not enough for the direct translator to just be more efficient than the human imitator: the direct translator has to be a cost-effective improvement (in terms of how much loss/computation it saves per increase in description complexity) compared to all other possible mappings.

Overall, however, despite the issues with this approach, I agree with Paul, Mark, and Ajeya’s conclusion that it is the most promising approach currently on offer:[1]

In order to ensure we learned the direct translator, we would need to change the training strategy to ensure that it contains sufficiently challenging inference problems, and that doing direct translation was a cost-effective way to improve speed (i.e. that there aren’t other changes to the human simulator that would save even more time).

Compared to all our previous counterexamples, this one offers much more hope. We can’t rule out the possibility of a clever dataset where the direct translator has a large enough computational advantage to be preferred, and we leave it as an avenue for further research. In Section: penalizing inconsistencies we discuss an additional ingredient that we think makes computation-complexity-based approaches more plausible.

Furthermore, I agree that, as Paul, Mark, and Ajeya argue, adding consistency might improve this approach as well:

If our training process looks at the relationship between different inputs, then a bad reporter might also need to consider lots of alternative inputs before making its decision. Moreover, if we choose sets of inputs randomly, then a bad reporter might not know which other inputs it needs to think about, and there is an exponentially large space of possible situations it potentially needs to worry about. So we could imagine getting to a situation where “just answer honestly” is the computationally easie...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What I learnt from twenty 1:1s at EAGxOxford, published by Ben Williamson on March 28, 2022 on The Effective Altruism Forum.

This is a quick post I have written up on the train back from the conference so there may be mistakes, poorly-phrased conclusions, etc.

The EAGxOxford conference took place this weekend and was the first EA conference I’ve been to. This is a short post reflecting on my experience at the conference, including how I approached it, how I found it valuable, and things I’d do again/ would do differently next time.

As well as being a useful exercise for me, I hope this can be helpful to other people ahead of attending their first (or next) conference.

In particular, this is a reflection on my experience of focusing on having lots of 1:1 meetings during an EA conference, something that I’ve seen recommended by multiple people in previous posts.

The short answer: I had twenty 1:1s across the two days of the conference and found this incredibly valuable and rewarding, very fun, and surprisingly manageable (if still quite tiring). Echoing previous posts, I definitely recommend scheduling lots of 1:1 meetings with people at an EA conference.

How I approached EAGxOxford

After confirming the logistics for attending the conference (trains, accomodation, etc.), I only started thinking more seriously about EAGx about a week ahead of it starting. Once the Swapcard was up and running, I made an effort to set up a bunch of 1:1s.

Swapcard

To do this, I used the filtered attendees on Swapcard for the areas I work in/ am particularly interested in (entreprepreneurship; mental health & wellbeing). I then went through about 30-50 profiles for maybe a couple of hours, finding a list of about 15 people I was excited to speak to.

From there, I sent each person a message saying that I was excited to connect, how I might be useful to them, and how I hoped they might be useful for me to speak to.

I had expected about half of these people would get back to me looking to meet but instead I received meeting requests from all 15. I think this high success rate was due to a few key things:

I sent these messages maybe 24-48 hours after the Swapcard was set up for the conference, meaning that everyone I messaged likely had lots of times still available to meet

I made it clear in every message why they might find it useful to speak to me, not just why I was excited to speak to them

I run Effective Self-Help (ESH) full-time, a small mental health research project funded by EAIF, which I think made it easier for me to demonstrate why people might want to speak to me (given this project spans both of the filters [entrepreneurship; mental health & wellbeing] that I applied)

While not everyone may be as lucky as me in receiving only positive replies, I hope that my success encourages you to be proactive in reaching out and setting up 1:1s.

Mindset

Beyond this practical work, I spent the few days ahead of EAGxOxford trying to engage with and solidify a mindset of being particularly sociable and proactive. While I tend to be a pretty chatty person, I tend to take a bit of time to warm up and can be hesitant to spark conversations/ introduce myself to new people.

I tried to pre-commit myself to being more proactive than usual in engaging with new people in three ways:

By sharing and explaining this intention with a couple of close friends who I knew would follow up on this with me after the conference, giving me an incentive to actually do what I said I’d do.

By forming some clear steps for how I’d follow through on this plan - e.g. using ‘how has your weekend/ conference been so far?’ as a default icebreaker/ intro for new conversations when stuck

By formulating clear reasons in my head for why I was doing this - e.g. almost every EA I’ve met previously/ spoken to online has been super fr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Vaniver's ELK Submission, published by Matthew "Vaniver" Graves on March 28, 2022 on The AI Alignment Forum.

You can see the actual submission (including a more formalized model) here, and the contest details here. I've reordered things to be more natural as a blog post / explain the rationale / intuition a bit better. This didn't get a prize, tho it may have been because I didn't try to fit the ELK format.

The situation: we have the capacity to train an AI to predict physical reality quite well. We'd like to train an AI (the "SmartVault") that uses that understanding of reality to protect a valuable diamond inside the vault. Ultimately, we'd like the system to protect the diamond and not deceive the owner of the diamond; proximally, we'd like to make a Reporter that can figure out, from the latent knowledge of the SmartVault, whether it's protecting the diamond or deceiving the owner.

I roughly think the proximal goal is doomed, and that the hope needs to be prospective instead of retrospective. [That is, we need to have a system that we can see ahead of time will try to protect the diamond instead of us checking after the fact for deception, because that can just move the meta-level on which deception occurs.] So I'm going to try to propose a solution that de-emphasizes the Reporter, and see how far I get.

Setup

We can roughly model SmartVault as being composed of some subsystems: the Predictor (which, given a trajectory of actions, estimates their likely result), a Reporter (which, given access to the predictor, answers specific questions about the result), and an Optimizer (which, given a situation, tries to decide which trajectory of actions to take). Obviously, the Predictor and the Optimizer are going to be intimately connected to each other, and the ontology of the Predictor is going to be jointly optimized by reality and the training feedback (i.e. the structure of the model's knowledge will reflect both the underlying reality and the salience landscape of the humans labelling examples as 'good' and 'bad'). I'm going to assume that 'the human value function' is part of the 'latent knowledge relevant to prediction (of training reward)', even if it's implicit instead of explicit.

While we’d like to train our machine on (reward, state of reality) pairs, unfortunately we only have access to (reward, human observation) pairs for training; that is, we only know how to communicate the lower-level salience landscape. This means a core challenge is ontological identification, as each ‘human observation’ state could mean many possible ‘machine observation’ states. Furthermore, let’s assume the machine ontology is more detailed and relevant (as it’s the one driving actions, including actions that might corrupt sensors).

There are two main classes of reporters: the 'human simulator' (which answers the question: "what would the human think if they saw their observations?") and the 'direct translator' (which answers the question: "what would a hypothetical correct human think if they saw my observations?").

Similarly, I think there are two main classes of inferred human value functions, which seem pretty analogous: the 'finger' (which answers the question: "did the human observe things that they approved of?"), and the 'moon' (which answers the question: "did the thing the human approved of happen?", or "would a hypothetical correct human approve of the thing that I infer occurred?"). I am tempted to just call them the 'human simulator' and the 'direct translator', but it's probably better to reserve those terms for reporters and introduce new terms for talking about the inferred value functions, even tho I think they use basically the same mechanisms. [One reporter you could imagine, for example, doesn't answer arbitrary questions but just implements the human value func...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some observations about deceptive mesaoptimization, published by leogao on March 28, 2022 on The AI Alignment Forum.

Editor’s note: I’m experimenting with having a lower quality threshold for just posting things even while I’m still confused and unconfident about my conclusions, but with this disclaimer at the top. Thanks to AI_WAIFU for discussions.

I think of mesaoptimization as being split into different types: aligned (which is fine), misaligned (which is bad), and deceptively aligned (which is really bad - I draw the line for deceptively aligned when it actually tries to hide from us instead of us just not noticing it by accident). I mostly think about deceptive mesaoptimization, and I think of regular misaligned optimization as basically an easier case because at last it’s not trying to hide from us (what it means for it to actually be "trying to hide" is kinda fraught anyways so this sidesteps that). In particular, I think of mesaoptimizers as being dangerous because they fail in a malign way (I don't think mesaoptimization itself is necessarily bad, and preventing it doesn't seem obviously good).

OOD-focused framework of deception

One framework for thinking about deceptive mesaoptimizers that I used to use a lot was that they would be good on the training distribution, but then do bad things out of distribution. In this framework, what we would do is then to try to somehow tell during training whether a model is going to do bad things out of distribution, and only keep the ones that don’t, or something along those lines. The intuition is that the mesaoptimizer cares about something in the real world, but it knows that if we catch it then it will not be able to achieve its goals in the real world, so it behaves normally on the training distribution when we’re looking at it but then in deployment it goes and does evil things to satisfy its mesaobjective. Importantly, pursuing the mesaobjective hurts the base objective (since we’re only focusing on inner alignment here we can assume that the base objective is accurate), so deployment is the only time where it can do evil things without getting itself removed.

The main reason here that deception happens is that your base objective isn’t being integrated over the set of all possible states. In theory, if we could enumerate the entire (huge) input space, we could eliminate deceptive behavior entirely. We'd just simply test every single input and see how whether there are any regions where the model does a treacherous turn and consequently tanks the base objective (for now I'm trying to isolate inner alignment so I'm assuming we've solved outer alignment and have a value function that we can just query). If there is nowhere in the entire input source where the model does a treacherous turn, then congratulations, your model is, for all intents and purposes, indistinguishable from being not deceptively aligned. (Non-deceptive inner misalignment is very similar except the mesaoptimizer won’t be trying to hide from you.)

In practice, the input space is absolutely enormous, and it's completely and utterly infeasible to enumerate it. Still, we might think to approximate that "perfect" red teaming by doing a really good approximation with loads of input instances sampled from a broad distribution, possibly with the help of other NNs. Unfortunately, I think this approach is fundamentally doomed, no matter how good the red ranking distribution is made. For this to work, the entire distribution of possible inputs in the real world would have to be covered by the red teaming distribution, and any small imperfections are potential points of leverage for the mesaoptimizer.

Aside from some contrived existence proofs like the RSA2048 example (which are admittedly very useful for thinking about these things so I use them a lot desp...