Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: High-level hopes for AI alignment, published by HoldenKarnofsky on December 15, 2022 on LessWrong.

In previous pieces, I argued that there's a real and large risk of AI systems' aiming to defeat all of humanity combined - and succeeding.

I first argued that this sort of catastrophe would be likely without specific countermeasures to prevent it. I then argued that countermeasures could be challenging, due to some key difficulties of AI safety research.

But while I think misalignment risk is serious and presents major challenges, I don’t agree with sentiments1 along the lines of “We haven’t figured out how to align an AI, so if transformative AI comes soon, we’re doomed.” Here I’m going to talk about some of my high-level hopes for how we might end up avoiding this risk.

I’ll first recap the challenge, using Ajeya Cotra’s young businessperson analogy to give a sense of some of the core difficulties. In a nutshell, once AI systems get capable enough, it could be hard to test whether they’re safe, because they might be able to deceive and manipulate us into getting the wrong read. Thus, trying to determine whether they’re safe might be something like “being an eight-year-old trying to decide between adult job candidates (some of whom are manipulative).”

I’ll then go through what I see as three key possibilities for navigating this situation:

Digital neuroscience: perhaps we’ll be able to read (and/or even rewrite) the “digital brains” of AI systems, so that we can know (and change) what they’re “aiming” to do directly - rather than having to infer it from their behavior. (Perhaps the eight-year-old is a mind-reader, or even a young Professor X.)

Limited AI: perhaps we can make AI systems safe by making them limited in various ways - e.g., by leaving certain kinds of information out of their training, designing them to be “myopic” (focused on short-run as opposed to long-run goals), or something along those lines. Maybe we can make “limited AI” that is nonetheless able to carry out particular helpful tasks - such as doing lots more research on how to achieve safety without the limitations. (Perhaps the eight-year-old can limit the authority or knowledge of their hire, and still get the company run successfully.)

AI checks and balances: perhaps we’ll be able to employ some AI systems to critique, supervise, and even rewrite others. Even if no single AI system would be safe on its own, the right “checks and balances” setup could ensure that human interests win out. (Perhaps the eight-year-old is able to get the job candidates to evaluate and critique each other, such that all the eight-year-old needs to do is verify basic factual claims to know who the best candidate is.)

These are some of the main categories of hopes that are pretty easy to picture today. Further work on AI safety research might result in further ideas (and the above are not exhaustive - see my more detailed piece, posted to the Alignment Forum rather than Cold Takes, for more).

I’ll talk about both challenges and reasons for hope here. I think that for the most part, these hopes look much better if AI projects are moving cautiously rather than racing furiously.

I don’t think we’re at the point of having much sense of how the hopes and challenges net out; the best I can do at this point is to say: “I don’t currently have much sympathy for someone who’s highly confident that AI takeover would or would not happen (that is, for anyone who thinks the odds of AI takeover . are under 10% or over 90%).”

The challenge

This is all recapping previous pieces. If you remember them super well, skip to the next section.

In previous pieces, I argued that:

The coming decades could see the development of AI systems that could automate - and dramatically speed up - scientific and technological advancement, getting us more q...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How "Discovering Latent Knowledge in Language Models Without Supervision" Fits Into a Broader Alignment Scheme, published by Collin on December 15, 2022 on LessWrong.

Introduction

A few collaborators and I recently released a new paper: Discovering Latent Knowledge in Language Models Without Supervision. For a quick summary of our paper, you can check out this Twitter thread.

In this post I will describe how I think the results and methods in our paper fit into a broader scalable alignment agenda. Unlike the paper, this post is explicitly aimed at an alignment audience and is mainly conceptual rather than empirical.

Tl;dr: unsupervised methods are more scalable than supervised methods, deep learning has special structure that we can exploit for alignment, and we may be able to recover superhuman beliefs from deep learning representations in a totally unsupervised way.

Disclaimers: I have tried to make this post concise, at the cost of not making the full arguments for many of my claims; you should treat this as more of a rough sketch of my views rather than anything comprehensive. I also frequently change my mind – I’m usually more consistently excited about some of the broad intuitions but much less wedded to the details – and this of course just represents my current thinking on the topic.

Problem

I would feel pretty optimistic about alignment if – loosely speaking – we can get models to be robustly “honest” in a way that scales even to superhuman systems. Moreover, I think a natural sub-problem that captures much or most of the difficulty here is: how can we make a language model like GPT-n “truthful” or “honest” in a way that is scalable? (For my purposes here I am also happy to make the assumption that GPT-n is not actively deceptive, in the sense that it does not actively try to obscure its representations.)

For example, imagine we train GPT-n to predict news articles conditioned on their dates of publication, and suppose the model ended up being able to predict future news articles very well. Or suppose we train GPT-n to predict the outcomes of particular actions in particular situations, all described (imperfectly by humans) in text. Then I would expect GPT-n would eventually (for large enough n) have a superhuman world model in an important sense. However, we don’t currently know how to recover the “beliefs” or “knowledge” of such a model even in principle.

A naive baseline for trying to make GPT-n truthful is to train it using human feedback to output text that human evaluators believe to be true. The basic issue with this is that human evaluators can’t assess complicated claims that a superhuman system might make. This could lead to either competitiveness problems (if GPT-n only outputs claims that humans can assess) or misalignment issues (if GPT-n outputs false claims because human evaluators can’t assess them correctly).

In many ways this problem is similar to Eliciting Latent Knowledge (ELK), but unlike ELK I am happy to take a “non-worst-case” empirical perspective in studying this problem. In particular, I suspect it will be very helpful – and possibly necessary – to use incidental empirical properties of deep learning systems, which often have a surprising amount of useful emergent structure (as I will discuss more under “Intuitions”).

On the other hand, if we want to study scalable alignment empirically, I think it’s very important for us to also have good reason to believe that our experiments will say something meaningful about future models – and it’s not immediately clear how to do that.

This raises the question: how do we even approach doing research on this sort of problem, methodologically?

Methodology

I worry that a lot of theoretical alignment work is either ungrounded or intractable, and I worry that a lot of empirical alignment work does...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The next decades might be wild, published by mariushobbhahn on December 15, 2022 on The Effective Altruism Forum.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The winners of the Change Our Mind Contest—and some reflections, published by GiveWell on December 15, 2022 on The Effective Altruism Forum.

Author: Isabel Arjmand, GiveWell Special Projects Officer

In September, we announced the Change Our Mind Contest for critiques of our cost-effectiveness analyses. Today, we're excited to announce the winners!

We're very grateful that so many people engaged deeply with our work. This contest was GiveWell's most successful effort so far to solicit external criticism from the public, and it wouldn't have been possible without the participation of people who share our goal of allocating funding to cost-effective programs.

Overall, we received 49 entries engaging with our prompts. We were very happy with the quality of entries we received—their authors brought a great deal of thought and expertise to engaging with our cost-effectiveness analyses.

Because we were impressed by the quality of entries, we've decided to award two first-place prizes and eight honorable mentions. (We stated in September that we would give a minimum of one first-place, one runner-up, and one honorable mention prize.) We also awarded $20,000 to the piece of criticism that inspired this contest.

Winners are listed below, followed by our reflections on this contest and responses to the winners.

The prize-winners

Given the overall quality of the entries we received, selecting a set of winners required a lot of deliberation.

We're still in the process of determining which critiques to incorporate into our cost-effectiveness analyses and to what extent they'll change the bottom line; we don't agree with all the critiques in the first-place and honorable mention entries, but each prize-winner raised issues that we believe were worth considering. In several cases, we plan to further investigate the questions raised by these entries.

Within categories, the winners are listed alphabetically by the last name of the author who submitted the entry.

First-place prizes – $20,000 each[1]

Noah Haber for "GiveWell's Uncertainty Problem." The author argues that without properly accounting for uncertainty, GiveWell is likely to allocate its portfolio of funding suboptimally, and proposes methods for addressing uncertainty.

Matthew Romer and Paul Romer Present for "An Examination of GiveWell’s Water Quality Intervention Cost-Effectiveness Analysis." The authors suggest several changes to GiveWell's analysis of water chlorination programs, which overall make Dispensers for Safe Water's program appear less cost-effective.

To give a general sense of the magnitude of the changes we currently anticipate, our best guess is that Matthew Romer and Paul Romer Present's entry will change our estimate of the cost-effectiveness of Dispensers for Safe Water by very roughly 5 to 10% and that Noah Haber's entry may lead to an overall shift in how we account for uncertainty (but it's too early to say how it would impact any given intervention). Overall, we currently expect that entries to the contest may shift the allocation of resources between programs but are unlikely to lead to us adding or removing any programs from our list of recommended charities.

Honorable mentions – $5,000 each

Alex Bates for a critical review of GiveWell's 2022 cost-effectiveness model

Dr. Samantha Field and Dr. Yannish Naik for "A critique of GiveWell’s CEA model for Conditional Cash Transfers for vaccination in Nigeria (New Incentives)"

Akash Kulgod for "Cost-effectiveness of iron fortification in India is lower than GiveWell's estimates"

Sam Nolan, Hannah Rokebrand, and Tanae Rao for "Quantifying Uncertainty in GiveWell Cost-Effectiveness Analyses"

Isobel Phillips for "Improving GiveWell's modelling of insecticide resistance may change their cost per life saved for AMF by up to 20%"

Tanae Rao and Ricky Huang for "...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Key Mostly Outward-Facing Facts From the Story of VaccinateCA, published by Zvi on December 14, 2022 on LessWrong.

I had the privilege of getting an advance draft of Patrick McKenzie’s very long story founding and running VaccinateCA, an organization dedicated to providing Americans information on where and how to get vaccinated against Covid-19.

It is an amazing document. Despite its length, I am going to flat out say to Read the Whole Thing if you have the capability of doing that. It is full of things worth knowing and understanding. I benefited from reading it twice, once as a draft and again as a finished document.

Since the VaccinateCA post remains very long and difficult to navigate or draw reference from, this post is an attempt to pull out the most long-term-important passages and facts and give an outline of the events and insights that reflect the world outside VaccinateCA, to keep this bounded. I have a short bullet point section on the inside lessons as well, but to keep this bounded am focusing elsewhere.

A central theme is the author, Patrick McKenzie, being surprised and confused by how dysfunctional were government operations, and how deprioritized was the cause of having less citizens get sick and die. As you read, notice these repeated surprises and confuses, and notice that the correct model would not be so reliably surprised in the same directions.

Similarly, he notes that he believes some of his statements present the situation as so dysfunctional that they sound crazy. The statements do not sound crazy to me at all. They sound accurate, and like the type of thing one would expect. It seems important, if we want things to improve, to realize that such statements are accurate, normal and expected.

Again, seriously, read the original if you can, at least the parts about how government systems (largely didn’t) track the vaccine and ensure it was distributed to citizens.

The Logistical Problems With Distribution Were Unnecessary

For some reason we decided that our standardized cold chain, sufficient to keep the vaccines good for ten weeks, was not good enough, despite that obviously being good enough. So for no real physical reason, we didn’t use our existing infrastructure.

The Pfizer vaccine keeps for ten weeks when transported or stored unopened between two degrees Celsius and eight degrees Celsius (36°F and 46°F), which is the exact same range as the flu vaccine and overlaps that of milk.

Transportation was easily achievable via cold-chain logistics, abundantly commercially available in the US and already used pervasively in the pharmaceutical industry.

The product rollout for the vaccine should have been approximately as logistically straightforward as any product rollout in the United States. It was not, principally because it did not use the US’s formidable advantages in coordinating product rollouts.

The Hyperlocal Information Problem

The core problem that VaccinateCA was created to solve was that our vaccine distribution system:

Took some vaccine doses.

Distributed them to various places.

Did not track how much was distributed to which places when.

Did not track how much vaccine was available at which places.

In practice, let each place decide what subset of the people legally eligible was eligible to get their particular vaccine shots. Some restricted by geography, some were more strict about age or condition, others less so.

Individuals were left to call various places to see who had vaccine shots.

Good news is calling pharmacists and asking ‘do you have the vaccine?’ works.

In California this system managed, in the face of overwhelming demand, to administer only 27% of delivered vaccine shots into arms by mid-January.

The remaining 73% were sitting there. In a freezer. Waiting.

Had no plan to remedy the lack of information. Did not much ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The case for transparent spending, published by Jeroen W on December 15, 2022 on The Effective Altruism Forum.

Context

Recently, the purchase of Wytham Abbey came in the spotlight: The purchase by Effective Ventures of a (15 million pound?) manor house. On Twitter and on the forum, people have held extensive discussions on whether the expense was justified. I personally was very surprised, which is why I made a forum post requesting an explanation. (Currently, I’m still slightly sceptical but I’m happy I now understand the reasoning behind it a lot better.)

Because of the purchase, the idea that Effective Ventures/CEA should be more transparent about large expenses has been brought up more often. I agree with that idea, and try to defend it here.

I added footnotes/links when I have a source for the (counter) argument. They may have been made regarding a different formulation of my proposal or a slightly different proposal, but are nonetheless worth mentioning. I sometimes rewrote arguments and may have misunderstood their original meaning, so don’t interpret my rewritings as the original authors’ intentions. I just want to credit where I got the ideas from. I have no experience at all with funding projects, so there might be practical things I’ve overlooked.

My proposal

All spending above a certain threshold by EA organisations should be publicly explained

The threshold should be high enough so it doesn’t take away too much time from grantmakers/employees (ex. $500k, 1 million pounds)

The more concerned people might be about the expense, or the more influential the expense is, the more time and effort should be put into the explanation

The public explanation should be some kind of cost-benefit analysis and clearly state the reasons for the purchase including positives, potential negatives and counterfactuals in layman's terms. Precise numbers are admirable but not always necessary. “Worst case” and “best case” scenarios with some kind of probability distribution might be helpful.

The explanations should be published within a reasonable timeframe (ex. within a month after the purchase/grant is made, every quarter,...). This timeframe should be made clear so that people can expect when to get an explanation of something. The sooner after the purchase, the better.

The EA organisations could be: Effective Ventures and the organisations that fall under it (CEA, 80,000 Hours, Forethought Foundation, EA Funds, Giving What We Can, The Centre for the Governance of AI, Longview Philanthropy, asterisk, non-trivial, BlueDot Impact), Open Philanthropy, GiveWell, Animal Charity Evaluators,... I think the case is the strongest for Effective Ventures and CEA since they represent the EA movement, so they should be held to the highest standards.

I haven’t reviewed every organisation. Some might already do a great job. For example, I get the impression that GiveWell and Open Philanthropy do a better job at explaining grants than EA Funds does (except for EAIF they often only use one sentence, even with million dollar grants).

I’m highly confident some variation of my proposal should be done if the grant is made using individual/small donations, I think it’s reasonable to claim you owe explanations to your donors. The case is less strong when the donor of the grant is just one or two people (like with Wytham Abbey/Open Philanthropy), but I’m still quite confident it’s important in those cases too. You may not owe an explanation to individual/small donors, but rather to the EA community as a whole.

Should this be some kind of official rule every EA organisation must follow? No, but I’d be happy to see some organisations (especially Effective Ventures/CEA) try it out. Different organisations can try different thresholds and use different rules for their public explanations.

Similar propos...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The next decades might be wild, published by Marius Hobbhahn on December 15, 2022 on LessWrong.

I’d like to thank Simon Grimm and Tamay Besiroglu for feedback and discussions.

This post is inspired by What 2026 looks like and an AI vignette workshop guided by Tamay Besiroglu. I think of this post as “what would I expect the world to look like if these timelines (median compute for transformative AI ~2036) were true” or “what short-to-medium timelines feel like” since I find it hard to translate a statement like “median TAI year is 20XX” into a coherent imaginable world.

I expect some readers to think that the post sounds wild and crazy but that doesn’t mean its content couldn’t be true. If you had told someone in 1990 or 2000 that there would be more smartphones and computers than humans in 2020, that probably would have sounded wild to them. The same could be true for AIs, i.e. that in 2050 there are more human-level AIs than humans. The fact that this sounds as ridiculous as ubiquitous smartphones sounded to the 1990/2000 person, might just mean that we are bad at predicting exponential growth and disruptive technology.

Update: titotal points out in the comments that the correct timeframe for computers is probably 1980 to 2020. So the correct time span is probably 40 years instead of 30. For mobile phones, it's probably 1993 to 2020 if you can trust this statistic.

I’m obviously not confident (see confidence and takeaways section) in this particular prediction but many of the things I describe seem like relatively direct consequences of more and more powerful and ubiquitous AI mixed with basic social dynamics and incentives.

Taking stock of the past

Some claims about the past

Tech can be very disruptive: The steam engine, phones, computers, the internet, smartphones, etc; all of these have changed the world in a relatively quick fashion (e.g. often just a few decades between introduction and widespread use). Furthermore, newer technologies disrupt faster (e.g. smartphones were adopted faster than cars) due to faster supply-chain integration, faster R&D, larger investments and much more. People who were born 10 years before me grew up without widespread smartphones and the internet, I grew up with the benefits of both and people who are 10 years younger than me play high-resolution video games on their smartphones. Due to technology, it is the norm, not the exception, that people who are born 10 years apart can have very different childhoods.

AI is getting useful: There are more and more tasks in which AI is highly useful in the real world. By now certain narrow tasks can be automated to a large extent, e.g., protein folding prediction, traffic management, image recognition, sales predictions and much more. Some of these, e.g. sales predictions sometimes rely on technology developed before 2000, but many of the more complex and less structured tasks such as traffic predictions are built on top of Deep Learning architectures only developed in the 2010s. Thus we are either one or multiple decades deep in the AI disruption period. Based on this, I consider another AI winter unlikely. You can make money with AI right now and the ML systems can automate a lot of tasks today even if we didn’t find any additional breakthroughs.

Transformers work astonishingly well: Originally transformers were used mostly on language modeling tasks but by now they have been successfully applied to many computer vision tasks, reinforcement learning, robotics, time series and much more. The basic recipe of “transformer+data+compute+a bit of engineering” seems to work astonishingly well. This was not true for previous architectures such as CNNs or LSTMs.

People use AI to build AI: The first large language models (LLMs) that assist human coders seem to genuinely improve their performance, ther...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA career guide for people from LMICs, published by Surbhi B on December 15, 2022 on The Effective Altruism Forum.

Executive Summary

Individuals from Low and Middle Income Countries (LMICs) engaging with EA often find that existing EA career advice does not address various frequently arising questions and challenges. This post attempts to address that gap by sharing the tentative outcomes of discussions between the authors (who are all from LMICs) on the pros and cons for various career paths.

We hope that this guide will serve as a tool for individuals in LMICs to prioritize career paths. This post may be particularly useful as material for localized introductory or in depth fellowships.

The advice for paths to impact depends on the individual, but broadly speaking, we recommend EAs, and especially those from LMICs, to:

Build career capital early on

Work on top global issues instead of local ones, unless there are clear impact-related reasons to opt for the latter

Some Individuals -we further discuss who and when- may have impact by doing:

local community building

local priorities research and charity-related activities

local career advising

There are many other paths the authors did not explore in-depth but consider promising, such as LMIC-relevant content creation to research on regional comparative advantages in addressing global priorities

For each path, the authors discuss pros and cons and suggest actionable next steps (lots of them!)

Some general advice around how to think about paths to impact:

Assess and prioritize paths by the problem’s scale / neglectedness / tractability, the incremental value you can contribute, and your personal fit.

Consider global priorities first. Look for local comparative advantage.

New EAs with a bias towards acting now risk subpar outcomes by hastily jumping into direct work, so instead first upskill and engage deeply with (and critique!) EA ideas.

Note that this post is not meant to be the final word on how EAs from LMICs should think about career paths, but rather a conversation-starter and call-to-action to improve LMIC-specific diversity in EA. It builds upon excellent prior work by several LMIC groups and community builders, which the authors have compiled in this resource bank.

Acknowledgements

Thanks to Alejandro Acelas, Agustin Covarrubias, Claudette Salinas, Siobhan McDonough, Vaishnav Sunil, and Yi-Yang Chua for all of your feedback on previous drafts.

Introduction

Engaging with EA when you come from a Low or Middle Income Country (LMIC) raises some challenges that might not be obvious to the current EA majority, composed of people from higher income countries, often in the Western world. EA principles framed from a high-income audience can fall flat in contexts where individuals lack the ability to donate, don’t have access to large EA networks for continued engagement, and where EA principles may be at odds with typical cultural norms and histories.

If you come from an LMIC, you have probably wondered how your strategy and prioritization fits in with the EA movement at-large. Some common questions/concerns we hear from LMICs audiences (that you might have at the moment) are:

Why should I care about suffering in other countries if I am surrounded by suffering, poverty, and disease in my own city or country?

What are the most effective causes and charities in my country? Does global EA or GiveWell have adequate information to assess local interventions?

Is it worthwhile to focus on identifying effective charities in my own country? Will it increase overall impact or divert funding from higher impact interventions?

What other high leverage interventions can increase impact? How should we think about localized career advising, research, community building, founding new charities?

What are the comparative advantages of my...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Small EA Groups and Invisible Impact, published by brook on December 14, 2022 on The Effective Altruism Forum.

Epistemic Status: personal observation (n=1)

tl;dr: EA groups not in central hubs (Loxbridge, Berkeley) generate a lot of value they don't see when members move to central hubs to work. Intuitively, this may feel like losing compared to members who stay near the peripheral group.

If you're running an EA group anywhere outside of a large hub (e.g. Oxford, London, Berkeley), what does the best case look like for an individual who interacts with your group?

Obviously there are several answers to this group, but I'll focus on two:

Somebody who stays in the city and continues to help the group as an advisor or senior member, who also works at an EA org or earns to give.

Somebody who leaves the city for an EA hub and has little/no contact with the group, but works at an EA org.

I won't debate which of these is more valuable at length, because I don't think it's that important to my central point. It seems plausible that the additional movement building of (1) could be comparable to the productivity gains of (2), and what's more impactful for one person might not be for another.

My central point is this question: which do you feel good about? At EA Edinburgh, there's something of an undercurrent that Oxford is 'taking' our best members, or an implicit assumption that having people work in Edinburgh and set up EA organisations in Edinburgh is better than them setting up elsewhere. I bought into both of these assumptions for a long time! But I think they are, fundamentally, tribal.

Another model that isn't just a black box labelled 'tribalism' is that the impact produced by people who leave Edinburgh for greener pastures is completely invisible to you, while the people who hang around turn up to meetings every month telling you about all the exciting, impressive things they are doing. It seems pretty easy to not update sufficiently for this selection bias.

Some suggested actions for your group:

Notice if your discussions/thoughts/plans run this way

Question the assumption that things happening nearer to you is more impactful than them happening elsewhere

Stay in contact with members who move away! If you're going to be tribal, you might as well get them to give a talk now and again about what they're doing, or give advice about how to run your group

I'd be interested to hear any other suggestions, and if anybody else has noticed this effect (maybe it's just Edinburgh for some reason?).

Special thanks to Q for giving a talk about egregores which triggered this thought initially.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: «Boundaries», Part 3b: Alignment problems in terms of boundaries, published by Andrew Critch on December 14, 2022 on LessWrong.

AI alignment is a notoriously murky problem area, which I think can be elucidated by rethinking its foundations in terms of boundaries between systems, including soft boundaries and directional boundaries. As anticipated previously in this «Boundaries» sequence, I'm doing that now, for the following problem areas:

Preference plasticity & corrigibility

Mesa-optimizers

AI boxing / containment

(Unscoped) consequentialism

Mild optimization & impact regularization

Counterfactuals in decision theory

Each of these problem areas probably warrants a separate post, from the perspective of making research progress within already-existing framings on the problem of aligning AI technology with humanity. However, with this post, my goal is mainly just to continue conveying the «boundaries» concept, by illustrating how to start using it in a bunch of different problem areas at once. An interest score for each problem area was determined by voting on an earlier LessWrong comment, here, and the voting snapshot is here.

You many notice that throughout this post that I've avoided saying things like "the humans prefer that {some boundary} be respected". That's because my goal is to treat boundaries as more fundamental than preferences, rather than as merely a feature of them. In other words, I think boundaries are probably better able to carve reality at the joints than either preferences or utility functions, for the purpose of creating a good working relationship between humanity and AI technology.

Alright, let's begin by analyzing:

Preference Plasticity & Corrigibility

Preference plasticity is the possibility of changes to the preferences of human preferences over time, and the challenge of defining alignment in light of time-varying preferences (Russell, 2019, p.263).

Interest score: 12/12

Corrigibility is the problem of constructing a mind that will cooperate with what its creators regard as a corrective intervention (Soares et al, 2015).

Interest score: 3/12

I think these two problems are best discussed together, because they are somewhat dual to each other: corrigibility has to do with the desirability of humans making changes to AI systems, and preference plasticity has to do with the undesirabiltiy — or least confusingness — of AI make changes to humans, or sometimes humans making changes to each other or themselves.

Preference plasticity

When is it good to change your preferences based on experience? When is it bad? Do these questions even make sense? And what about changing the preferences of others?

Allowing someone or something else to change your preferences is a profound kind of vulnerability, and in particular is a kind of boundary opening. How should it work, normatively speaking?

Contrast preferences with beliefs. Beliefs have nice rules for when they should update, e.g., Bayes' theorem and logical induction. If we had similar principles for how preferences should update, we could ask AI to respect those principles, or at to least help us uphold them, in the process of affecting our preferences. But from where could we derive rules for "ideal preference updating", other than just asking our preferences what rules we prefer?

Well, for one thing, preferences updates are a mechanism for cooperation: if two agents share the same preferences over the state of the world it's easier for them to get along. Does this happen in humans? I think so. Does it happen optimally? Probably not.

Consider the adage: "You are the average of the N people closest to you" (or similar). I don't know of any satisfying comprehensive study of this kind of claim, and it's probably even wrong in important ways (e.g., by neglecting dynamics around people who are low on the ...