Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Landscape in the UK, published by DavidNash on December 13, 2022 on The Effective Altruism Forum.

Summary

In this post I’m attempting to give an overview of what EA looks like in the UK, including communities and organisations with a range of affiliation with EA.

EA Hubs in the UK

Most of the people working at EA related organisations are in London, Oxford or Cambridge (sometimes known as Loxbridge), with some remote workers around the UK. It’s quite easy to get from London to Oxford or Cambridge, being about an hour by train, and people regularly travel between them.

London

London probably has the largest number of people interested in EA. There are roughly 150-250 people working at EA related organisations although the majority of people interested in EA are working in the private sector, government, academia or (non EA) nonprofits. There are also four EA university groups in London at Imperial College London, King's College London, University College London and London School of Economics.

There isn't one main EA community in London, there are quite a few subgroups based around different causes, workplaces and interests. There are also people who have an interest but attend an event or get involved with a relevant opportunity once every few years.

Subcommunities

Workplace/Cause Communities

Charity Entrepreneurship are near Queens Park and with each new cohort there are more organisations set up that work from the CE office, mainly on global health & development, health security or animal welfare

The Centre on Long-Term Risk are near Primrose Hill and there is a community there for people working on suffering risks

Conjecture are close to London Bridge and support the SERI-MATS fellowship nearby, both focusing on AI Safety

Profession communities

The civil service has people interested in EA working there and they have regular meetups

EA Finance has quite a few members in London and they have meetups every few months

EA Tech have meetups every month or two

There is also a group for people who are interested in politics and EA

There are usually 1 or 2 meetups a year for entrepreneurs and consultants

University groups

There are 4 quite active groups, roughly 20 volunteer organisers as well as the London EA Hub (LEAH) supporting students interested in EA

ICL

KCL

UCL

LSE

Interest communities

EA for Christians have had meetups in London every few months

There are also meetups for people to play football, dance, go to gigs, stitch and climb

There are now quite a few EA related organisations in London, in 2015 there was just Founders Pledge and some Givewell recommended charities.

EA Meta

Charity Entrepreneurship

80,000 Hours

Founders Pledge

Impactful Government Careers

Let’s Fund

SoGive

Non-trivial

EA UK

LEAH - supporting EA groups in London. They are also running a co-working space for students and professionals working on EA projects

Effective Giving

Rethink Priorities ~18 people based in UK

Better Matters

Social Change Lab

Global Development

Against Malaria Foundation

Schistosomiasis Control Initiative

Malaria Consortium

GiveDirectly - London Office

Suvita

Canopie

Lead Exposure Elimination Project

Animal Welfare - Most of these are the UK based teams for international orgs

The Humane League

Animal Equality

Open Cages

The Good Food Institute Europe

Animal Advocacy Careers

Animal Ask

Fish Welfare Initiative

Cellular Agriculture UK

AI Alignment

Conjecture

The Cooperative AI Foundation

Deepmind Safety team

Safe AI London

Long Term Future/Existential Risks

Longview Philanthropy

London Existential Risk Initiative (aimed at students)

All-Party Parliamentary Group For Future Generations

The Centre for Long-Term Resilience

1Day Sooner

Center on Long-Term Risk

There are some remote staff in London for various EA related organisations, such as Animal Charity Evaluators, Centre ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GiveWell’s Moral Weights Underweight the Value of Transfers to the Poor, published by Trevor Woolley on December 13, 2022 on The Effective Altruism Forum.

Ethan Ligon and I submitted this to GiveWell for their "Change Our Minds" contest this year (2022). They will be announcing winners later this week (Dec 15, I think). But before they do, we wanted to share our submission here in case anyone is interested! Ethan has done some amazing work measuring real-world marginal utility by estimating demand systems using consumer purchases data. Seeing this prior work of his and thinking it may be particularly relevant to GiveWell's evaluations (and EA in general), I approached him and we wrote a short paper which we ended up submitting for the challenge.

Rather than post the entire paper here and risk losing info in the reformatting process, I ask that you see the Google Doc of it here. While it does get a bit technical, I've tried to include non-technical summaries for most of the major sections. So in case you feel like some of the details are getting intense, I encourage you to at least read through the intro paragraphs and conclusions as you skim over it. If you get anything out of it, let it be that while total utility levels (over consumption) are unidentifiable, marginal utilities (over total consumption) are not--they can be measured with data!

In case you aren't convinced yet to read the full article, below is a quick excerpt from the Summary:

GiveWell bases much of their cost-effectiveness analysis on the value of doubling consumption. Since increasing consumption expenditures is the primary effect of the GiveDirectly cash transfers program, GiveWell uses the effectiveness (value generated per dollar) of cash transfers as a metric for evaluating the effectiveness of all other programs. However, by valuing “doubling consumption”, GiveWell has assumed the functional form of utility over “real consumption” x to be log(x) and the functional form of marginal utility over consumption to be 1/x (since this is the derivative of log(x)). This is a valid utility function in the sense that it is one of many functions that satisfies the conditions of rationality, but there is strong evidence that it is not a good representation of the preferences of the Kenyan beneficiaries of the GiveDirectly experiment.

The purpose of this article is to explain why GiveWell should reconsider using “doubling consumption” as the basis for assessing the value of consumption (or income) changes and instead value “halving marginal utility of expenditure”—what we think GiveWell actually intends to value. Using data from GiveDirectly’s cash transfers program in Kenya (Haushofer and Shapiro 2016), we provide empirical evidence that rejects the use of any function that implies homothetic preferences (including marginal utility of 1/x). We then empirically estimate the true marginal utility over consumption (λ) as revealed by Kenyan beneficiaries of GiveDirectly’s cash transfers program and show how the value per dollar of cash transfers is actually 2.6 times GiveWell’s current number (from 0.0034 to 0.009). This is because 1/x is quickly dwarfed by revealed marginal utility, , at low levels of consumption.

Therefore, valuing “doubling consumption” underweights the value of cash transfers to the very poor if we let them “speak” for themselves.

Our "headline figure":

FIGURE 5: Utilitarian ROI. The curves labeled cfe use the estimated marginal utilities of expenditure λ(x, p), with p either the values at baseline, or endline. The curve labeled log uses the marginal utility of expenditures implied by a logarithmic utility function used (implicitly) by GiveWell.

4.5. Utilitarian Return on Investment. For every dollar given to a particular household, there’s some increase in utility, which we can think of a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Forecasting Research Institute (we’re hiring), published by Tegan on December 13, 2022 on The Effective Altruism Forum.

The Forecasting Research Institute (FRI) is a new organization focused on advancing the science of forecasting for the public good.

All decision-making implicitly relies on prediction, so improving prediction accuracy should lead to better decisions. And forecasting has shown early promise in the first-generation research conducted by FRI Chief Scientist Phil Tetlock and coauthors. But despite burgeoning popular interest in the practice of forecasting (especially among EAs), it has yet to realize its potential as a tool to inform decision-making.

Early forecasting work focused on establishing a rigorous standard for accuracy, in experimental conditions chosen to provide the cleanest, most precise evidence possible about forecasting itself—a proof of concept, rather than a roadmap for using forecasting in real-world conditions. A great deal of work, both foundational and translational, is still needed to shape forecasting into a tool with practical value.

That’s why our team is pursuing a two-pronged strategy. One is foundational, aimed at filling in the gaps in the science of forecasting that represent critical barriers to some of the most important uses of forecasting—like how to handle low probability events, long-run and unobservable outcomes, or complex topics that cannot be captured in a single forecast. The other prong is translational, focused on adapting forecasting methods to practical purposes: increasing the decision-relevance of questions, using forecasting to map important disagreements, and identifying the contexts in which forecasting will be most useful.

Over the next two years we plan to launch multiple research projects aimed at the key outstanding questions for forecasting. We will also analyze and report on our group’s recently completed project, the Existential Risk Persuasion Tournament (XPT). This tournament brought together over 200 domain experts and highly skilled forecasters to explore, debate, and forecast potential threats to humanity in the next century, creating a wealth of rich data that our team is mining for forecasting and policy insights.

In our upcoming projects, we’ll be conducting large, high-powered studies on a new research platform, customized for the demands of forecasting research. We’ll also work closely with selected organizations and policymakers to create forecasting tools informed by practical use-cases. Our planned projects include:

Developing a forecasting proficiency test for quickly and cheaply identifying accurate forecasters

Identifying leading indicators of increased risk to humanity from AI by building “AI-risk conditional trees” with the help of domain experts (overview of conditional trees here, pg. 13)

Exploring ways of judging (and incentivizing) answers to unresolvable and far-future questions, such as reciprocal scoring

Conducting “Epistemic Audits” to help organizations reduce uncertainty, identify action-relevant disagreement, and guide their decision processes.

(For more on our research priorities, see here and here.)

We’re excited to begin FRI’s work at such an auspicious time for the field of forecasting, with the many great projects, people and ideas that currently inhabit it—spanning the gamut from heavyweight organizations like Metaculus and GJI, to the numerous innovative projects run by small teams and individuals. This environment presents a wealth of opportunities for collaboration and cooperation, and we’re looking forward to being a part of such a dynamic community.

Our core team consists of Phil Tetlock, Michael Page, Josh Rosenberg, Ezra Karger, Tegan McCaslin, and Zachary Jacobs. We also work with various contractors and external collaborators in the forec...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CEEALAR: 2022 Update, published by CEEALAR on December 13, 2022 on The Effective Altruism Forum.

Tldr: we are still going, currently have lots of space, and have potential for further growth. Please apply if you have EA-related learning or research you want to do that requires support.

Update

It’s been a while since our last update, but, suffice to say, we are still here! During 2021 we gradually increased our numbers again, from a low of 4 grantees mid-pandemic, to 15 by the end of the year (our full capacity, with no room sharing and 2 staff living on site). We lifted all Covid restrictions in March 2022, and things started to again feel like they were pre-pandemic. However, our building and its contents are old, and in mid-May this year we closed to new grantees for building repairs and maintenance. We reopened bookings again at the end of July, by which time we had once again got down to very low numbers - we are now up and running again and starting to fill up with new grantees, but we still have plenty of spare capacity. Please apply if you are interested in doing EA work at the hotel. We are offering (up to full) subsidies on accommodation and board for those wishing to learn or work on research or charitable projects (in fitting with our charitable objects).

See our Grant Making Policy for more details.

Along with the ups and downs in numbers, we’ve had ups and downs in other ways. We were delighted to receive our largest grant to date, from the FTX Future Fund, in May ($125k or ~ a year of runway), but this is now bittersweet given recent events. We condemn the actions of SBF and the FTX/Alameda inner circle, and are ashamed of the association. It’s possible the grant will be subject to clawbacks as a result of the commenced bankruptcy proceedings. As with many FTX grantees in the EA Community, we are following and discussing the situation as it unfolds. We intend to follow the consensus that emerges around any voluntary returning of unspent funds.

Despite the significant funding, with the ongoing energy crisis, inflation in general, and increased spending on building maintenance and salaries, our costs have risen rapidly, and we were recently down to ~4 months of runway again. Enter the Survival & Flourishing Fund (SFF). We are extremely grateful to have been awarded a grant of $224,000(!) by Jaan Tallinn as per their most recent announcement.

In order to attract and retain talent, with the last grant we upped our management salaries to ~the UK median salary (£31,286), plus accommodation and food (worth about £6k).

It’s now 4.5 years since we first opened. Since then we have supported ~100 EAs aspiring to do direct work with their career development, and hosted another ~200 visitors from the EA community participating in events, networking and community building. We’ve established an EA community hub in a relatively low cost location. We believe there is plenty of potential demand for it to scale, but we still need to get the word out (which we are doing in part with this blog post).

Our impact

Grantee work

There are two main aspects to our potential impact: the direct work and career building of our grantees, and the community building and networking we facilitate.

We are open to people working on all cause areas of EA, with the caveat that the work we facilitate is desk-based and mostly remote. In practice, this has meant that longtermist topics, especially x-risks, and in particular AI Alignment, have been foremost amongst the work of the grantees we have hosted. But we have also had grantees interested in animal welfare, global health, wellbeing, development and progress, and meta topics related to EA community building.

Since our last update, we have had a number of grantees go on to internships, contracts and jobs at the likes of SERI, CHAI, Alvea, Re...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI alignment is distinct from its near-term applications, published by paulfchristiano on December 13, 2022 on LessWrong.

I work on AI alignment, by which I mean the technical problem of building AI systems that are trying to do what their designer wants them to do.

There are many different reasons that someone could care about this technical problem.

To me the single most important reason is that without AI alignment, AI systems are reasonably likely to cause an irreversible catastrophe like human extinction. I think most people can agree that this would be bad, though there’s a lot of reasonable debate about whether it’s likely. I believe the total risk is around 10–20%, which is high enough to obsess over.

Existing AI systems aren’t yet able to take over the world, but they are misaligned in the sense that they will often do things their designers didn’t want. For example:

The recently released ChatGPT often makes up facts, and if challenged on a made-up claim it will often double down and justify itself rather than admitting error or uncertainty (e.g. see here, here).

AI systems will often say offensive things or help users break the law when the company that designed them would prefer otherwise.

We can develop and apply alignment techniques to these existing systems. This can help motivate and ground empirical research on alignment, which may end up helping avoid higher-stakes failures like an AI takeover. I am particularly interested in training AI systems to be honest, which is likely to become more difficult and important as AI systems become smart enough that we can’t verify their claims about the world.

While it’s nice to have empirical testbeds for alignment research, I worry that companies using alignment to help train extremely conservative and inoffensive systems could lead to backlash against the idea of AI alignment itself. If such systems are held up as key successes of alignment, then people who are frustrated with them may end up associating the whole problem of alignment with “making AI systems inoffensive.”

If we succeed at the technical problem of AI alignment, AI developers would have the ability to decide whether their systems generate sexual content or opine on current political events, and different developers can make different choices. Customers would be free to use whatever AI they want, and regulators and legislators would make decisions about how to restrict AI. In my personal capacity, I have views on what uses of AI are more or less beneficial and what regulations make more or less sense, but in my capacity as an alignment researcher I don’t consider myself to be in the business of pushing for or against any of those decisions.

There is one decision I do strongly want to push for: AI developers should not develop and deploy systems with a significant risk of killing everyone. I will advocate for them not to do that, and I will try to help build public consensus that they shouldn’t do that, and ultimately I will try to help states intervene responsibly to reduce that risk if necessary. It could be very bad if efforts to prevent AI from killing everyone were undermined by a vague public conflation between AI alignment and corporate policies.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 10: Fine-tuning with RLHF is aesthetically unsatisfying., published by Charlie Steiner on December 13, 2022 on The AI Alignment Forum.

As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes. This take owes a lot to the Simulators discussion group.

Fine-tuning a large sequence model with RLHF creates an agent that tries to steer the sequence in rewarding directions. Simultaneously, it breaks some nice properties that the fine-tuned model used to have. You should have a gut feeling that we can do better.

When you start with a fresh sequence model, it's not acting like an agent, instead it's just trying to mimic the training distribution. It may contain agents, but at every step it's just going to output a probability distribution that's been optimized to be well-calibrated. This is a really handy property - well-calibrated conditional inference is about as good as being able to see the future, both for prediction and for generation.

The design philosophy behind RLHF is to train an agent that operates in a world where we want to steer towards good trajectories. In this framing, there's good text and bad text, and we want the fine-tuned AI to always output good text rather than bad text. This isn't necessarily a bad goal - sometimes you do want an agent that will just give you the good text. The issue is, you're sacrificing the ability to do accurate conditional inference about the training distribution. When you do RLHF fine-tuning, you're taking a world model and then, in-place, trying to cannibalize its parts to make an optimizer.

This might sound like hyperbole if you remember RL with KL penalties is Bayesian inference. And okay; RLHF weights each datapoint much more than the Bayesian inference step does, but there's probably some perspective in which you can see the fine-tuned model as just having weird over-updated beliefs about how the world is. But just like perceptual control theory says, there's no bright line between prediction and action. Ultimately it's about what perspective is more useful, and to me it's much more useful to think of RLHF on a language model as producing an agent that acts in the world of text, trying to steer the text onto its favored trajectories.

As an agent, it has some alignment problems, even if it lives totally in the world of text and doesn't get information leakage from the real world. It's trying to get to better trajectories by any means necessary, even if it means suddenly delivering an invitation to a wedding party. The real-world solution to this problem seems to have been a combination of early stopping and ad-hoc patches, neither of which inspire massive confidence. The wedding party attractor isn't an existential threat, but it's a bad sign for attempts to align more high-stakes AI, and it's an indicator that we're probably failing at the "Do What I Mean" instruction in other more subtle ways as well.

More seems possible. More capabilities, more interpretability, and more progress on alignment. We start with a perfectly good sequence model, it seems like we should be able to leverage it as a model, rather than as fodder for a model-free process. Although to any readers who feel similarly optimistic, I would like to remind you that the "more capabilities" part is no joke, and it's very easy for it to memetically out-compete the "more alignment" part.

RLHF is still built out of useful parts - modeling the human and then doing what they want is core to lots of alignment schemes. But ultimately I want us to build something more self-reflective, and that may favor a more model-based approach both because it exposes more interpretable structure (both to human designers and to a self-reflective AI), and because it preserves...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Revisiting algorithmic progress, published by Tamay on December 13, 2022 on LessWrong.

How much progress in ML depends on algorithmic progress, scaling compute, or scaling relevant datasets is relatively poorly understood. In our paper, we make progress on this question by investigating algorithmic progress in image classification on ImageNet, perhaps the most well-known test bed for computer vision.

Using a dataset of a hundred computer vision models, we estimate a model—informed by neural scaling laws—that enables us to analyse the rate and nature of algorithmic advances. We use Shapley values to produce decompositions of the various drivers of progress computer vision and estimate the relative importance of algorithms, compute, and data.

Our main results include:

Algorithmic progress doubles effective compute budgets every ~9 months (95% CI: 4 months to 25 months)

Roughly, progress in image classification has been ~40% due to the scaling of compute, ~40% due to better algorithms, ~10% due to scaling data

The majority (>75%) of algorithmic progress is compute-augmenting (not data-augmenting)

In our work, we revisit a question previously investigated by Hernandez and Brown (2020), which had been discussed on LessWrong by Gwern, and Rohin Shah. Hernandez and Brown (2020) re-implement 15 open-source popular models and find a 44-fold reduction in the compute required to reach the same level of performance as AlexNet, indicating that algorithmic progress outpaces the original Moore’s law rate of improvement in hardware efficiency, doubling effective compute every 16 months.

A problem with their approach is that it is sensitive to the exact benchmark and threshold pair that we choose. Choosing easier-to-achieve thresholds makes algorithmic improvements look less significant, as the scaling of compute easily brings early models within reach of such a threshold. By contrast, selecting harder-to-achieve thresholds makes it so that algorithmic improvements explain almost all of the performance gain. This is because early models might need arbitrary amounts of compute to achieve the performance of today's state-of-the-art models. We show that the estimates of the pace of algorithmic progress with this approach might vary by around a factor of ten, depending on whether an easy or difficult threshold is chosen.

Our work sheds new light on how algorithmic efficiency occurs, namely that it primarily operates through relaxing compute-bottlenecks rather than through relaxing data-bottlenecks. It further offers insight on how to use observational (rather than experimental) data to advance our understanding of algorithmic progress in ML.

That said, our estimates is consistent with Hernandez and Brown (2020)'s estimate that algorithmic progress doubles the amount of effective compute every 16 months, as our 95% confidence interval ranges from 4 to 25 months.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Psychological Disorders and Problems, published by adamShimi on December 12, 2022 on LessWrong.

There are two types of psychological issues: disorders and problems.

Disorders are so massive, so life-altering that we can’t miss them: schizophrenia, crippling social anxiety, destructive addiction. They scream “danger” in neon red blinking letters. We know and fear them because they destroy lives. People with such disorders are often unable to function in society, and see themselves ostracized, shunned, and institutionalized.

On the other hand, psychological problems appear quite benign and harmless: fear of rejection, not liking social gatherings, stress eating/smoking. They’re not good, but they’re also not that big of a deal. Everybody has them, and most people function just fine despite a bunch of psychological problems. They don’t forbid a normal life.

Yet they subtly restrict you.

Take low social anxiety. Maybe you don’t really like meeting new people. Maybe you don’t like group settings. Or you’re uncomfortable talking with others, because you might say something stupid. Whatever your reason, this anxiety makes you:

Avoid parties if you can

Stay less time if you have to go

Be less enjoyable in conversations when there

None of these consequences will ruin your life. Even when put all together, psychological problems still don’t amount to disorders. But as soon as you start to optimize, they get in your way. You start bending the best action that you can find in order to deal with your problems, instead of going for it directly. If networking is the best use of your time but you have social anxiety, you’ll find alternatives that don’t reap all the benefits, often without even realizing it. You’ll just flinch away.

Psychological disorders are obvious liabilities. But if you neglect psychological problems, they will invisibly block your path again and again. Avoiding this doesn’t look like waiting until all your psychological problems are solved, but instead to acknowledge them and endeavour to fix them when they get in the way.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Altruism and Development - It's complicated..., published by DavidNash on December 12, 2022 on The Effective Altruism Forum.

An interesting post from Shruti Rajagopalan looking at the trade off between legibility and complexity in evaluating philanthropic efforts.

As December rolls in, my inbox is filled with requests for donations, often from organizations I have given to in the past. This holiday season is also bittersweet because I cannot visit Delhi, where I was born and my parents still live, because of the air pollution and smog during the winter months. In Delhi, I find it hard to breathe, and usually lose my voice because of inflammation caused by particulate matter pollution. This year, I am under doctor’s orders to avoid travelling to Delhi in the winter; I’ve been struggling with respiratory problems from long Covid.

With air pollution dominating my thoughts and nudges for charitable giving in my inbox, my first instinct is to give to causes that help mitigate pollution in Delhi. But I am also aware of the literature on emotional giving or ineffective altruism. In their 2021 paper, Caviola, Schubert and Greene explain why both effective and ineffective causes may attract dollars. People often give emotionally to a cause that has personally impacted them in some way.

This paper resonated with me because I am exactly the sort of irrational dog lover likely to support the best training programs for guide dogs. These super dogs have my lifelong admiration. My Labrador retrievers can barely fetch a ball.

We all know air pollution is bad. But how bad? And compared to what?

As an alternative, I looked up the top charities recommended by GiveWell—the top two work on reducing Malaria deaths. Malaria kills between 600,000 and 700,000 each year. And GiveWell is considered one of the most credible evaluators in the philanthropic space. Should I be thinking less about air pollution in Delhi and more about malaria in Africa?

So, I thought it best to evaluate 1) my priors on air pollution, 2) whether air pollution mitigation in Delhi merits my dollars/rupees. And if Delhi air pollution merits intervention, then it would be good to 3) identify the reasons air pollution became such a big problem in Delhi (you would be surprised), which would lead to uncovering 4) how to mitigate the problem of air pollution so I can decide where to send my dollars. And since I got to #4, to understand 5) why people think that giving to malaria charities is “higher impact” than solving air pollution.

6. Back to Malaria—Is It Really That Simple?

While writing this post, I also thought more about malaria and whether malaria prevention is more complex than impact evaluations lead us to believe. If legibility is the consequence of a narrowing of vision to make a complex problem tractable, then are these malaria mitigation interventions too simple?

95% of all malaria deaths are in Africa, and that malaria disproportionately kills children.

This is probably why the effective altruism community, which believes in helping those far removed from one’s situation, measures by lives saved per dollar when thinking about long-term and high-impact efforts, and rates malaria prevention charities so highly. In his latest column, Ezra Klein defends the basic principles of effective altruism and separates it from the SBF-FTX mess.

It is impossible not to feel for the children in Africa dying from malaria. But suggesting that distributing nets and antimalarial medication is the best way to save lives and prevent illness compared to anything else we know is narrow. Regions outside Africa only account for 4% of malaria deaths. But I don’t see high use of mosquito nets and antimalarial medication in Europe and the U.S. Outside of camping equipment stores, I don’t think I have seen any mosquito nets bought or sold...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 9: No, RLHF/IDA/debate doesn't solve outer alignment., published by Charlie Steiner on December 12, 2022 on LessWrong.

As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day (ish) for 25 days. Or until I run out of hot takes. And now, time for the week of RLHF takes.

I see people say one of these surprisingly often.

Sometimes, it's because the speaker is fresh and full of optimism. They've recently learned that there's this "outer alignment" thing where humans are supposed to communicate what they want to an AI, and oh look, here are some methods that researchers use to communicate what they want to an AI. The speaker doesn't see any major obstacles, and they don't have a presumption that there are a bunch of obstacles they don't see.

Other times, they're fresh and full of optimism in a slightly more sophisticated way. They've thought about the problem a bit, and it seems like human values can't be that hard to pick out. Our uncertainty about human values is pretty much like our uncertainty about any other part of the world - so their thinking goes - and humans are fairly competent at figuring things out about the world, especially if we just have to check the work of AI tools. They don't see any major obstacles, and look, I'm not allowed to just keep saying that in an ominous tone of voice as if it's a knockdown argument, maybe there aren't any obstacles, right?

Here's an obstacle: RLHF/IDA/debate all incentivize promoting claims based on what the human finds most convincing and palatable, rather than on what's true. RLHF does whatever it learned makes you hit the "approve" button, even if that means deceiving you. Information-transfer in the depths of IDA is shaped by what humans will pass on, potentially amplified by what patterns are learned in training. And debate is just trying to hack the humans right from the start.

Optimizing for human approval wouldn't be a big deal if humans didn't make systematic mistakes, and weren't prone to finding certain lies more compelling than the truth. But we do, and we are, so that's a problem. Exhibit A, the last 5 years of politics - and no, the correct lesson to draw from politics is not "those other people make systematic mistakes and get suckered by palatable lies, but I'd never be like that." We can all be like that, which is why it's not safe to build a smart AI that has an incentive to do politics to you.

Generalized moral of the story: If something is an alignment solution except that it requires humans to converge to rational behavior, it's not an alignment solution.

Let's go back to the perspective of someone who thinks that RLHF/whatever solves outer alignment. I think that even once you notice a problem like "it's rewarded for deceiving me," there's a temptation to not change your mind, and this can lead people to add epicycles to other parts of their picture of alignment. (Or if I'm being nicer, disposes them to see the alignment problem in terms of "really solving" inner alignment.)

For example, in order to save an outer objective that encourages deception, it's tempting to say that non-deception is actually a separate problem, and we should study preventing deception as a topic in its own right, independent of objective. And you know what, this is actually a pretty reasonable thing to study. But that doesn't mean you should actually hang onto the original objective. Even when you make stone soup, you don't eat the stone.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.