Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rhetorical Abusability is a Poor Counterargument, published by Cullen OKeefe on January 8, 2022 on The Effective Altruism Forum.

This post is a bit less polished than ideal, but I wanted to get it out of my drafts

Introduction

One sometimes sees the following type of objection to consequentialism or utilitarianism:

P1. Consequentialist reasoning would justify X.

P2. X is not justifiable.

C. Therefore, consequentialism is wrong.

Similar argument structures can also be applied to any number of ideologies/philosophies:

P1. Libertarianism would justify X.

P2. X is not justifiable.

C. Therefore, Libertarianism is wrong.

Call this general argument format the Justifiability Argument. Some recent criticisms of longtermism arguably follow this structure. We can also call the ideology in P1 the "target ideology" for generic purposes.

However, note the potential ambiguity of "justify" as used in the Justifiability Argument. Depending on the sense in which it is used, P1 could mean either of the following:

A. Consequentialism, properly applied, implies X is justifiable.

B. A speaker can use consequentialist reasoning (e.g., consequentialist argumentation, consequentialist phraseology) to make a rhetorically convincing case for X.

A. is a normal premise in a modus tollens. A lot has been said on when and whether moral arguments of this form are compelling, especially where P2 is supported by ethical intuitions alone. I will not add to that discussion here.

B., however, is much more interesting. For the Justifiability Argument above to work under interpretation B., one needs an additional premises or argumentation. This post explores the Justifiability Argument under interpretation B, which I will call the Rhetorical Abusability Argument.

To focus solely on that question, for the sake of argument we can assume that P2. is true, though in reality we may also disagree with it.

Supplying the Missing Premises

Rhetorical Abusability

To make the Rhetorical Abusability Argument work, one could try to insert a premise like:

P3. It should not be possible for a speaker to make a rhetorically convincing case fora false conclusion using the reasoning of the true moral theory.

Call this the "Rhetorical Abusability Premise." I think most people who make the Justifiability Argument implicitly assume something like the Rhetorical Abusability Premise.

Unfortunately, the Rhetorical Abusability Premise seems incredibly weak to me. For one thing, what is rhetorically convincing to people is a product of moral psychology, not of moral truth. It would be very odd if first-order moral truth depended on the contingencies of how convincing different rhetoric is to different audiences. Different people will find different things rhetorically convincing, and different speakers are differently skilled at rhetoric. It should not be surprising if there are many false (by supposition) moral conclusions that a skilled rhetorician could make some number of people believe. Laypeople's moral beliefs are not especially coherent, and for most of history (and possibly still now) people have believed morally unjustifiable things. So, the mere fact that someone can rhetorically abuse some theory to convince some listener of a false moral conclusion cannot be a good reason to reject that theory.

There are other reasons to reject the Rhetorical Abusability Argument as formulated here. One advantage to the person using the Rhetorical Abusability Argument is that the Rhetorical Abusability Argument imposes no obligation to present the criticized moral theory correctly. What's important to the Rhetorical Abusability Argument is that the cited arguments or phraseology merely purport to apply the target ideology. This is beneficial to the person making the Justifiability Argument for a number of reasons. One is that it simply ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rational Breaks: a better way to work, published by bfinn on January 7, 2022 on LessWrong.

HOW can you use your work time better? Many people waste their days only half-working, and not accomplishing much. It's more productive to alternate between bouts of highly focused work, and breaks to recover.

Various time management techniques propose working like this for fixed time periods, e.g. 25 minutes, followed by a set break. But I think they're mistaken. Fixed time periods are unnatural and mechanical; thoughtful, creative work doesn’t watch clocks. And you can’t work with other people this way either, as their meetings, calls and interruptions won’t fit in with your timeslots.

Here's a different approach: let’s ignore how long you work, and consider breaks instead. Your day consists of both; they’re two sides of the same coin. So instead of working in fixed periods, why not just limit your break time? This guarantees how long you’ll work for – the rest of the day – while leaving you complete freedom in how to divide it up.

This is the idea behind my new technique, provisionally called Rational Breaks. You work for as long or as short as you like, until you want a break. Then break for up to one-third of the time you've just worked.

So after 15 minutes’ work, you can stop for up to 5 minutes. If you work for an hour, you've earned a good 20-minute break. And if a task bores you after 3 minutes, you can even take a break then – but only for 1 minute! Breaks reward you for working, but long breaks have to be earned.

This kind of pattern is natural; research confirms that people tend to take longer breaks after working for longer. Don't take five, take one-third!

Rational Breaks have many benefits (which I'll go into), but the key one is flexibility. They adapt to your attention span, energy, and schedule, as well as to other people. You can use ratios other than one-third, if you like. (Ratio, Rational, see?) And Rational Breaks aren’t just for your day-job – they suit anything that needs focus or effort, such as studying, practicing an instrument, personal admin, writing, or fitness training.

But before going into depth, let’s look at other time management systems. (If you don’t use these, you can skip to the following section, Rational Breaks.)

Clock-work

The best-known system is Pomodoro, named after a tomato-shaped timer. It advocates working in 25-minute stints, ending with an alarm and a 5-minute break, or occasionally 20–30 minutes. (Which is also how long it takes to read Pomodoro’s trademark guidelines.)

In fact, there’s nothing new about working for fixed periods until an alarm goes off. For centuries, schools have taught lessons this way, ending with a bell.[1] At work, fixed periods simplify scheduling; create short-term deadlines to focus your mind; and provide regular breaks, to maintain your energy, attention, decision-making, performance and well-being.

But working against the clock like this – let’s call it ‘clock-work’ – has problems.

How long should each stint last? There’s no consensus between experts, research and systems at all:

Productivity author Mark Forster recommends starting with 5-minute bursts, and progressively extending them to 40 minutes.

Medical professor Dr James Levine suggests 15 minutes’ work at a time.

Pomodoro, as mentioned, advocates 25 minutes.

30- and 60-minute stints worked equally well for computer operators in one study.

School classes typically last 30-90 minutes, and it’s unclear what length is best.

Time-tracking software DeskTime found their most productive users average 52 minutes’ work, plus a 17-minute break.

The best music students practised for about 80 minutes at a time in a well-known study.

The Ultradian system advocates working for 90 minutes, and it’s based on biological cycles lasting up to two hours.

This hug...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Omicron Post #13: Outlook, published by Zvi on January 7, 2022 on LessWrong.

Omicron is now the dominant strain, so much of what would otherwise have gone into this Omicron post ended up in the latest weekly post instead, and I’m considering the right way to merge the two streams together.

In addition to some amount of the usual business, the focus of this post will be an explicit attempt to look at what’s likely to happen from here, and give my baseline scenario.

I recently had reason to chart out what I expected that baseline scenario to be around the world, so it makes sense to share it here.

The world can be divided into five (highly unequally sized) parts, with increasing levels of uncertainty, some places are unclear on whether they’re in groups two, three or four:

South Africa and other already-peaked areas.

Uncontrolled spread: Europe, North America, India, Israel, Australia and Argentina.

Without an Omicron wave, or without good data: Third world, Russia.

Previous containment: South Korea, Japan, Taiwan and New Zealand.

Special case of previous containment: China.

South Africa and Other Already-Peaked Areas

This is the easy story to tell. Predictions about the past are often still hard, but they’re a lot easier.

A few African countries have already finished their Omicron waves, and got through mostly unscathed. Since they’re in the southern hemisphere, one can worry that there’s a wave coming in their winter a few months from now, but I would expect any such wave to be mild even if relatively little is done to prepare, given how things went this time, and that those infections are now baked in.

Uncontrolled spread: Europe, North America, India, Israel, Australia and Argentina.

This is where most of my readers live, and where I live, so it’s been my main focus. There is now a very clear baseline scenario.

Infection rates increase exponentially until they hit a peak. In major cities, this plateau will usually be quite high, with more than 1% of people infected each day, and likely 10%+ of the population testing positive at any given time. In rural areas, the plateau is likely going to be somewhat lower, but not dramatically lower.

Once you hit that peak, there will be a short period (a week or less probably in most cases) before the decline starts in earnest, then picks up speed. Once it’s gone down to something that will usually be between 25% and 50% of its maximum over the course of a few weeks, the decline starts to slow in percentage terms as behaviors adjust back somewhat. Then over time, there is a slow decline, which mostly continues until such time as other dynamics (like weather, or another variant) change the game once more, or you stabilize at low endemic levels.

There are substantial disruptions during the peak period, lasting roughly a month, but nothing as scary or severe as early 2020, and no breakdowns of civil order. Schools end up closed for a few weeks to a month in many places, others have to quickly close for a few days due to staffing issues, but some manage to stay open and it’s fine, they don’t meaningfully make anything worse.

As long as the hospitals manage to continue to provide care, there’s a ton of cases, but not that many deaths. The hospitals come under stress about six days after the relevant cases, through a mix of a lack of staffing and a flood of new patients. The question is what the true capacity is, and what happens when you exceed it, and what our ability is to do triage. Even when we ‘get through’ those weeks, life in the hospitals is still terrible during this period.

Once hospital admissions peak and start turning downward again, and it’s clear this isn’t simply hitting capacity limits and turning people away, we can know that we’re going to make it through in a given area.

We do know that London and New York in part...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Please complete a survey to influence EU animal protection policies, published by PeterSlattery on January 7, 2022 on The Effective Altruism Forum.

Sharing this here quickly, as I have been made aware of it, and it seems very important to publicise. Please complete/share it where relevant.

The EU is currently in the process of revising its animal protection policies and using a survey to collect input between 15 October 2021 and 21 January 2022.

I have been told that it is one of the most significant opportunities for animal welfare progress on a legislative level ever seen in the EU.

Anyone can participate, including research institutions.

In many countries, there have been relatively few responses. Only 21 people in Ireland have responded to the survey, for instance. People from these countries can potentially have a more outsized response on their representatives or local policy. [Not sure how it works, but I imagine EU representatives will be particularly sensitive to responses from their citizens.]To participate, you need to sign up for an account by email. It took about five minutes from asking to create an account, to being sent the sign-up email and arriving at the survey page.

How should I fill out the survey?

Please see this guide from Anima International, or contact animal advocacy organisations in the Open Wing Alliance (OWA) and Eurogroup for Animals (EfA) networks (the links show who the member organisations in your country are). [Thanks to Neil Dullaghan for making me aware of the organisations and initatives].

Contents of the survey

The survey probably take 5-15 minutes to complete. The most important questions appear to be near the bottom

Revision of EU animal welfare legislation (EN)

Fields marked with are mandatory.

Introduction

Why are we consulting?

This initiative will explore several options for addressing the shortcomings identified in the recent evaluation of the EU legislation on animal welfare. The aim is to revise this legislation.

We would like to hear your views and experience on the fitness of the current rules and on how they could be improved.

Target audience

any group directly affected by the legislation, such as farmers and other food business operators

members of the public

(possibly) animal welfare NGO’s and consumer organisations.

About you

[removed]

Fitness check and revision of current legislation:

In 2020, the European Commission adopted its Farm to Fork Strategy (F2F), to promote a shift towards a sustainable food system.

Because animal welfare is a cornerstone of sustainable food production, Farm to Fork committed the Commission to revising the current EU animal welfare legislation by 2023, and to consider options for animal welfare labelling.

The purpose is to improve animal welfare and broaden the scope of the legislation, by aligning it with the latest scientific evidence, current political priorities and public expectations – all while making the legislation easier to enforce.The EU legislation under review is:

a Directive concerning the protection of animals kept for farming purposes

4 Directives laying down minimum standards for protecting:

laying hens

broilers

pigs

calves

a Regulation on animal transport

a Regulation on the protection of animals at the time of killing.

These EU acts regulate animal welfare at farm level, during transport and at slaughter, and cover animals – including fish – that are bred and kept for farming purposes, as well as cats and dogs that are transported for commercial purposes.

The acts do not cover wild animals (except when they are killed for the purpose of depopulation), experimental or laboratory animals (except when they are being transported) or any invertebrate animals.

The goal is to improve the welfare of animals while ensuring sustainable production and fair competition for EU bu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AMA: I was unschooled for most of my childhood, then voluntarily chose to leave to go to a large public high school., published by iceplant on January 7, 2022 on LessWrong.

I've seen a lot of discussion in the LW community about alternatives to traditional school for kids. Given how few kids actually get to experience the alternatives (<4% of kids in the US are homeschooled, <10% of those are unschooled), I think the success and failure modes are less well understood than for traditional schools. I want to offer myself as a datapoint to anyone interested in my subjective experience. Eventually, I want this to be a standalone blog post, but my thoughts still feel pretty disparate, so I'm hoping this will help me find a more cohesive narrative.

About me: I'm now in my mid 20s, went to a "good college" and now have a "good job" in tech (despite a brief gap derping around as an artist...what can I say I was unschooled). I'm not a teacher. I don't have kids. I do feel that unschooling had a big impact on my life, and I'm hoping this will help me understand it better, and how it can empirically affect others.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Bioethicists are (Mostly) Alright, published by Devin Kalish on January 7, 2022 on The Effective Altruism Forum.

Author’s Note: this post is based in part on Discord and Slack rants, as well as various conversations I have had recently, but is largely original

Special thanks to Applied Divinity Studies, for conversation on this matter, as well as commentary on earlier drafts. Some of the wording and sources in the final piece are based on their input

This is my second post on this forum. My co-blogger Nicholas Kross posted my first one here from his account, but it was a little weird to just have him be a proxy for me. I explained my reasons a bit in this comment, but even so, I decided to just get an account of my own this time. Sorry that this is another criticism of Effective Altruists, I swear I don’t hate you guys or anything, I’m an Effective Altruist myself, it’s just a coincidence that the first two pieces I was ready to write and post here are both critical.

It is a recent truism of the circles I run in that you should blog about your career. It is also a recent truism of the circles I run in that bioethicists are crazy and dumb. Bryan Caplan has compared them to astrologers. Robert Wiblin implied it is typical for them to think licking ice-cream in public is wrong. Byrne Hobart made a widely retweeted joke that bioethicists wouldn’t be approved by bioethicists if they were a treatment. Alec Stapp flatly asserted that bioethicists are grifters who stopped us from having challenge trials early in COVID. Scott Alexander even called bioethicists the rationalist “outgroup” along with evil robots back in 2016.

Well I have great news guys, I have infiltrated the enemy. Through clever subterfuge and sheer nerve, I have embedded myself inside a real life bioethics graduate program! And.look, I can’t keep this up. I am in bioethics because it is related to my academic interests. I have always found the reaction to the field prominent in EA/rat/adjacent spaces [1] pretty weird.

I have had informal conversations about this disparity in impressions several times recently. Twice in the last several months when mentioning my degree to others from my ingroup, once at an ACX meetup and once in correspondence with Applied Divinity Studies, the reaction was relief that someone from their background might be able to add some sense to the field on the margin. I am unusual in my bioethics program, but I am unusual just about everywhere except, to an extent, Effective Altruist spaces. I don’t feel unusually unusual in the bioethics program. I am not sure how to prove that bioethics just isn’t that bad, I am not even certain that it isn’t, but I want to at least register some evidence.

A key issue that I see people from my ingroup berating bioethicists about is challenge trials. Human challenge trials for COVID vaccines are popular with the public, so why do bioethicists oppose them?

This question came up semi-recently in the Rationally Speaking interview with Matthew Yglesias,

Matt: “Like, what is the field of bioethics? I don’t understand how that’s a purported domain of expertise. Because I’ve clashed with bioethics Twitter, on both this vaccine distribution thing and on human challenge trials for vaccines. . And, I don’t know, the ethics experts just disagree about the big picture, obvious controversies. The trolley problem, et cetera. I was blown away, on the human challenge trials, that Christine Korsgaard, my former professor – I think the leading Kantian deontological thinker – she had her name on the 1Day Sooner challenge trials thing. And then there’s these, I don’t know who, being like, ‘Well that’s not good ethics.’ And I’m like, ‘Well, according to whom?’ Right? Obviously in consequentialist terms, it’s good ethics. I happened to know the top expert in Kantian eth...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing Effective Self-Help, published by Ben Williamson on January 6, 2022 on The Effective Altruism Forum.

Effective Self-Help (ESH) is a pilot initiative researching the most effective ways people can improve their wellbeing and productivity.

This post explains what Effective Self-Help is, what I hope the project can achieve, and why I believe this work may be highly impactful. As this is a long post, I suggest reading the Summary section below and then skipping to whatever sections seem of greatest interest.

Based on rough Fermi calculations made using Guesstimate, I estimate a mean impact per Effective Self-Help article of 180 hours of high-impact work and 4.2 QALYs. However, these numbers are very uncertain and best taken as an indication that ESH could be highly impactful and cost-effective, rather than a signal that it will be.

Thank you to the many people who have discussed ideas, critiqued drafts, and provided invaluable insights in recent weeks. This project has received initial funding from the EA Infrastructure Fund.

Summary

Effective Self-Help is currently an ongoing series of articles on relevant topics posted to the EA Forum. I intend to develop the project into a home for the best guidance on how individuals can improve their wellbeing and productivity. The project is currently solely the work of me, Ben Williamson, with feedback, comments, and ideas from lots of other lovely people involved in EA.

In the What is Effective Self-Help? section, I discuss how I think ESH can be unique as a self-help resource by combining several key components:

Prioritisation: Highlighting the most effective interventions and how effective they are, rather than just listing interventions that may be effective.

Research Quality: Providing more rigorous and well-rounded advice that combines the findings of scientific literature with more Bayesian forms of evidence.

Presentation: Maximising engagement through curated recommendations and the use of digital media (video, audio, and interactive programmes).

Breadth: Covering topics across the full breadth of wellbeing and productivity, rather than just specific domains of expertise.

Practicality: A focus on enabling behaviour change by highlighting practical actions and tailoring content to maximise the success of habit adoption.

Positive mental health: Looking at how to improve the wellbeing of people who are not struggling with mental health issues as well as those who are.

Free Access: Avoiding any influence of paywalls, advertising, or affiliate marketing on the quality of advice provided and full access to advice.

Under Why Effective Self-Help could be highly impactful, I envisage three main routes to impact which are discussed in greater depth in the ‘Theory of Change’ sections. These are:

Increasing the productivity of people doing high-impact work

Helping a large audience make small improvements to their wellbeing

Attracting new people to the EA community through an interest in self-improvement.

I created Guesstimate models for the first two of these routes to impact, producing best-guess estimates that ESH could add 180 hours of productive work and 4.2 QALYs per article. However, these results are highly uncertain and best interpreted as an indication that ESH has the potential to be highly impactful if things go well, rather than a strong endorsement that it will be.

Downside risks and arguments for ESH being ineffective discusses a range of ways in which Effective Self-Help could be of limited value or of active harm. The main downside risks discussed are that ESH could:

Give advice that is of net harm by badly accounting for risks and side-effects

Serve as a weak introduction to EA

Make the EA community more insular.

The main ways listed for how that Effective Self-Help could be of limited effectiveness are:

S...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] Eric Schwitzgebel: Against Longtermism, published by ag4000 on January 6, 2022 on The Effective Altruism Forum.

This is a linkpost for

Eric Schwitzgebel, a philosophy professor at UC Riverside, just posted a criticism of longtermism on his blog. In short, his arguments are:

We live in a dangerous time in history, but there's no reason to think that the future won't be at least as dangerous. Thus, we'll likely go extinct sooner rather than later, so the expected value of the future is not nearly as great as many longtermists make it out to be.

It's incredibly hard to see how to improve the longterm. For example, should we almost destroy ourselves (e.g., begin a cataclysmic yet survivable nuclear war) to avoid the risks from even more dangerous anthropogenic threats?

Apart from temporal discounting, there are reasonable ethical positions from which one might still have greater reason to help those temporally closer than farther. For example, confucianism says we should focus more on those "closer" to us in the moral circle (friends, family, etc.) than those "farther" (including, presumably, future people).

There's a risk that longtermism could make people ignore the plight of those currently suffering. (Although, Schwitzgebel acknowledges, prominent longtermists like Ord also work in more neartermist areas.)

Overall, the critiques don't seem to be original. The third argument seems to me to be a reminder that it is important to examine the case for longtermism from other ethical perspectives.

If you enjoyed reading Schwitzgebel's post, he has another EA-related post about AI alignment (as well as many posts on consciousness, e.g., in AI).

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 1/6/22: The Blip, published by Zvi on January 6, 2022 on LessWrong.

Cases are hitting record highs around the world. That’s going to continue for another few weeks. While it does, and for a bit after it stops, there are disruptions all around as too many people are sick at one time. Thus, the CDC guideline adjustments, the hospitals under pressure, many schools forced to not meet in person (and then deciding to torture kids with ‘remote learning’ rather than giving them a vacation.)

No question it’s annoying, and for those in health care it’s overwhelming and terrible. I don’t want to minimize it. And if you care about avoiding this it’s going to be a lonely few months.

But compared to March 2020, this is nothing. We got this.

There are dissenting voices, but mostly the consensus seems to be growing that Omicron is milder, the center will hold, and our lives beckon.

Executive Summary

Record high case numbers likely to continue for a few weeks.

Hospitals under pressure but holding, deaths strangely low.

If you get sick you should still do your best to get a negative test before ending your isolation no matter what the CDC says.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 3.5mm cases (+94%) and 10,800 deaths (+25%).

Results: 3.57mm cases (+96%) and 8,814 deaths (+2%).

Prediction for next week: 6mm cases (+71%) and 9,700 deaths (+10%).

On cases that’s another random bullseye. On deaths it’s a big miss, and given how little of the past cases were Omicron I don’t understand how the number came in this low, still below the level 2-3 weeks ago despite much higher case counts.

This is the dog that did not bark. The number damn well should have gone up a bunch, and it didn’t, and it makes me think that Omicron took over faster than I previously thought. I can’t think of another way to explain it, and the fact that I’m using that to explain it seems like very good news going forward.

I do expect deaths to start climbing, how could they not, but we keep not seeing big jumps there, so I’m going to downshift my expectations there even as lagged cases start rising faster.

Deaths

Report that deaths among young people are up 40% from pre-pandemic levels, in the context of life insurance. This sounds terrible and is terrible, but in terms of how one should live one’s life young people are still very unlikely to die and shouldn’t take minimizing death risk as a major life task except when considering doing actively risky things like skydiving, or putting oneself at risk of violence.

Cases

It looks like Manhattan has indeed peaked, and maybe Brooklyn, and maybe NYC overall. If that’s right, then things will have held together well.

Boston on the other hand looks like it’s still headed upwards, currently at more than five times the previous peak.

Washington D.C. schools did systematic tests, found 5.8% of the tests came back positive. This was a lot of kids, so probably a lower rate than the city overall.

A week ago fully one third of NBA refs were in Covid protocols.

Vaccinations

CDC shortens length of time to wait for a Pfizer booster to five months. Six months was, of course, yet another arbitrary round number. And now we have this handy easy, not confusing chart, note that in some places boosters are mandatory soon after they are permitted.

The FDA decision to do thid bypassed VRBPAC. They didn’t consult their advisory committee. This new thing where when we don’t need their input we don’t ask for it? Excellent.

ACIP, alas, could not as easily be avoided, here is a recap of that.

Which includes this unrelated gem:

So, anyway, yeah, now you’re worried about boosters limiting vaccine supplies for other countries? What changed, exactly?

Also, it’s one thing that we haven’t approved Novavax, we have plenty of supply ourselves so we don’t need it. It’s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Importance of foresight evaluations within ELK, published by Jonathan Uesato on January 6, 2022 on The AI Alignment Forum.

This post makes a few basic observations regarding the importance of foresight-based evaluations within ELK [Christiano et al 2021]:

Without foresight-based evaluations, (narrow) ELK is insufficient to avoid subtle manipulation (we’d require a more ambitious version)

Hindsight creep can occur even with seemingly foresight-based evaluations. One way this could happen is if human evaluators primarily defer to evaluations made by future humans in the predicted outcomes.

Because of both points, even if we are counting on a technique like narrow ELK, improving the quality of foresight-based evaluations is very valuable.

The points here are largely covered in ELK appendices on narrow elicitation, defining a utility function, and subtle manipulation. The aim here is to call out some of these points explicitly, and facilitate further focused discussion.

Subtle manipulation

The ELK report focuses on what it calls narrow ELK, which does not aim to address subtle manipulation via ELK:

[In narrow ELK,] We won’t ask our AI to tell us anything at all about subtle manipulation..A mundane [example of subtle manipulation] is that they may have talked with someone who cleverly manipulates them into a position that they wouldn’t really endorse, or emotionally manipulated them in a way that will change their future conclusions.

Subtle manipulation is clearly problematic. Within the overall RL training approach of the ELK report, foresight evaluations are necessary to avoid subtle manipulation. That is, even if we solve narrow ELK, we still need foresight-based evaluations to avoid subtle manipulation.

With hindsight-based policy evaluations, narrow ELK is insufficient

The ELK report provides a simple policy training algorithm based on approval-in-foresight combined with hindsight-based outcomes predictors:

Still, even though humans can’t directly follow along with the actions, they can evaluate the predicted consequence of an action sequence:[image]We can then train a model to predict these human evaluations, and search for actions that lead to predicted futures that look good.

(See also: Appendix: Policy training pseudocode.)

This uses a hybrid of foresight and hindsight feedback. The outcomes predictor is trained in hindsight: trying actions, observing what outcomes result, and training a model to predict those outcomes. Conditioned on these predictions, the policy feedback on the action uses foresight — we don’t wait until the outcomes have happened to decide which action is better (we use the predicted outcomes instead).

The use of approval-in-foresight here is crucial for avoiding subtle manipulation. Consider, what happens if we make the policy feedback also hindsight-based? That is, we have the human choose which outcomes are best after the decision is made. Rather than optimizing for “actions that lead to predicted futures that look good [in foresight]” (emphasis mine), we would optimize for “actions that lead to [realized] futures that look good [in hindsight]”.

In this case, the futures which look best in hindsight are going to be ones where the future humans have been manipulated into liking and/or saying good things about the affairs that arise. The basic point is that the foresight-based policy feedback seems to be necessary with narrow ELK; for a purely hindsight-based scheme we should expect subtle manipulation.

This is the same point the ELK report makes with the paperclip example (bolding mine):

“For example, suppose I watch a 10 second ad that is carefully chosen by a brilliant paperclip-maximizing-marketer. Five years after watching this ad, I decide that paperclips are great so I dedicate my time to making lots of them, and if you evaluate outcom...