Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gradient descent doesn't select for inner search, published by Ivan Vendrov on August 13, 2022 on LessWrong.

TL;DR: Gradient descent won't select for inner search processes because they're not compute & memory efficient.

Slightly longer TL;DR: A key argument for mesa-optimization is that as we search over programs, we will select for "search processes with simple objectives", because they are simpler or more compact than alternative less dangerous programs. This argument is much weaker when your program search is restricted to programs that use a fixed amount of compute, and you're not optimizing strongly for low description length - e.g. gradient descent in modern deep learning systems. We don't really know what shape of programs gradient descent selects for in realistic environments, but they are much less likely to involve search than commonly believed.

Note on terminology (added in response to comments): By "search" I mean here a process that evaluates a number of candidates before returning the best one; what Abram Demski calls "selection" in Selection vs Control . The more candidates considered, the more "search-like" a process is - with gradient descent and A being central examples, and a thermostat being a central counter-example.

Recap: compression argument for inner optimizers

Here's the argument from Risks From Learned Optimization: [emphasis mine]

In some tasks, good performance requires a very complex policy. At the same time, base optimizers are generally biased in favor of selecting learned algorithms with lower complexity. Thus, all else being equal, the base optimizer will generally be incentivized to look for a highly compressed policy.

One way to find a compressed policy is to search for one that is able to use general features of the task structure to produce good behavior, rather than simply memorizing the correct output for each input. A mesa-optimizer is an example of such a policy. From the perspective of the base optimizer, a mesa-optimizer is a highly-compressed version of whatever policy it ends up implementing: instead of explicitly encoding the details of that policy in the learned algorithm, the base optimizer simply needs to encode how to search for such a policy. Furthermore, if a mesa-optimizer can determine the important features of its environment at runtime, it does not need to be given as much prior information as to what those important features are, and can thus be much simpler.

and even more forceful phrasing from John Wentworth:

We don't know that the AI will necessarily end up optimizing reward-button-pushes or smiles; there may be other similarly-compact proxies which correlate near-perfectly with reward in the training process. We can probably rule out "a spread of situationally-activated computations which steer its actions towards historical reward-correlates", insofar as that spread is a much less compact policy-encoding than an explicit search process + simple objective(s).

Compactness, Complexity, and Compute

At face value, it does seem like we're selecting programs for simplicity. The Deep Double Descent paper showed us that gradient descent training in the overparametrized regime (i.e. the regime of all modern deep models) favors simpler models. But is this notion of simplicity the same as "compactness" or "complexity"? Evan seems to think so, I'm less sure. Let's dive into the different notions of complexity here.

The most commonly used notion of program complexity is Kolmogorov complexity (or description length), basically just "length of the program in some reference programming language". This definition seems natural... but, critically, it assumes away all computational constraints. K-complexity doesn't care if your program completes in a millisecond or runs until the heat death of the universe. This makes it a ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fix Prison Telecom, published by Benj Azose on August 13, 2022 on The Effective Altruism Forum.

Summary

People in prisons and jails pay extremely high rates to stay in contact with their family in many parts of the country. 32 states had an average cost for a 15 minute phone call from a county jail of more than $5, and only 2 states had a cost less than $2.

Families spend $1.4 billion a year on calls.

Around 10 million people in the US are affected by poor prison and jail communication systems: ~2 million incarcerated, ~6 million spouses and parents, ~3 million children.

The effect on families is large. One in three families with an incarcerated parent go into debt to pay for prison communication.

Solutions are surprisingly reasonably priced, when areas have uptake. Ameelio (a non-profit working in this area) estimates deployment costs at ~$500k per state. Considering a current estimate of the prison telecom industry making $1.4b a year in profit, a comparatively very small amount of money here could have huge effects on millions of Americans.

Importance

People in prisons and jails pay extremely high rates to stay in contact with their family in many parts of the country. 32 states had an average cost for a 15 minute phone call from a county jail of more than $5, and only 2 states had a cost less than $2, according to the latest data from Prison Policy Initiative.

In addition, federal prisoners are paid sub-minimum-wage rates: $0.12 - $0.40 for jobs paid by the prison system, $0.23 - $1.15 for jobs for companies outside of the prison system. This can mean that people in prison pay their entire day’s wages for a 15 minute phone call!

In aggregate, families pay $1.4 billion a year, according to Business Insider.

There are three main constituencies that are affected by the high cost of prison communications: incarcerated people, their adult support network, and their children.

Note that I will not attempt DALY estimates here, because it’s not clear of a principled way to estimate them. There are good estimates on the population sizes (which I have included) and I also speak to some of the effects on those populations.

Incarcerated People

1.9 million people are currently in prison or jail in the US in 2022, according to the Prison Policy Initiative.

Numerous studies suggest that closer contact with family and outside support networks reduces rates of recidivism or violating parole. Communication with family members also makes the experience more tolerable.

Parents and Spouses

6.5 million adults have an immediate family member currently in jail or prison, according to the Equal Justice Initiative.

They are also the ones who most directly bear the high cost of communication. “The high cost of maintaining contact with incarcerated family members led more than one in three families (34%) into debt to pay for phone calls and visits alone,” according to the Ella Baker Center.

Children

2.7 million children currently have an incarcerated parent, according to the Urban Institute.

Phone calls, and other virtual communication methods, are essential for keeping in touch with an absent parent. “Over half of parents in state prisons (59%) and just under half (45%) of parents in federal prisons did not have any personal visits with their minor children while in prison in 2004,” reported The Sentencing Project based on data from the US Bureau of Justice Statistics.

Neglectedness

(Note: some material in this section is from the policy paper “Improving Outcomes for Incarcerated People by Reducing Unjust Communication Costs” that I previously published with the Day One Project.)

This is a particularly interesting angle for this problem. Prison telecommunication is publicly bid on, and the cost of equivalent services outside of correctional facilities is free or nearly free. Why is this mar...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Credence Calibration Icebreaker Game, published by Ruby on August 14, 2022 on LessWrong.

The Aussie mega-meetup took place this past weekend. For it, a new kind of icebreaker was needed: one which is was not merely fun and sociable, but also instilled with the Way. Thus was the Credence Calibration Icebreaker forged.

A marriage of the credence game and the classic icebreaker, ‘Say three things about yourself, one of them a lie’, the game allows players to learn about each other, test their ability to deceive and detect deception, and discover just how calibrated they are.

How to play

Playing instructions here: docx pdf. Scoring spreadsheet.

Each turn a player makes three statements about themselves. One and only one the statement must be intentionally untrue. All others players assign probabilities of being false to each statement. These probability sum to 1: P(A’) + P(B’) + P(C’ ) = 1. The game is scored in the same manner as the credence game, but with reference to 33% rather than 50%.

The way we played it, a player would reveal which was the lie immediately after everyone else had assigned probabilities. The immediate feedback is more fun and allows players to recalibrate as they learn about their performance. Revealing which statements were lies at the end would require reminding everyone what the other statements were.

Many meetup groups have played the Aumann agree game where groups collectively assign credences to a collection of statements, however that game requires a collection of statements to be collected in advance. Once played, new statements must be collected for a new game. The credence calibration icebreaker has the advantage that players generate the statements allowing for easy replay.

Improvements

Restrictions should be placed on the nature of the lies in order to control which skills are tested. We played without restrictions and most players generated a lie by altering a minor detail of a true statement which didn’t affect its plausibility, e.g. ‘My father’s brain is frozen’1 vs. ‘My uncle’s brain is frozen’. This resulted in the game being less about appraising the plausibility of statements and more about detecting deception by tells and other clues.

Following the original icebreaker game, three statements were used. Reducing the number of statements to two would have the following benefits:

The game is currently data entry intensive, requiring two numbers per question per player to be entered. Two statements would halve this number.

Assigning probabilities of falsehood is counter-intuitive to many, using two statements would allow for the typical direct assignment of truth.

People find generating three statements difficult, two statements would reduce the effort.

Statistics

Various statistics are computed in the scoring spreadsheet. Results from our game showed a high correlation between number correct and score, 0.72, and that players improved over the course of the game thanks to diminishing overconfidence.

1. True statement. As was 'I have three kidneys'.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Appendix: Building a Bugs List prompts, published by CFAR!Duncan on August 13, 2022 on LessWrong.

Prompt 0:

Think about the way computer programmers talk about “bugs” in the program, or “feature requests” that would make a given app or game much better. Bugs are things-that-are-bad: frustrations, irritations, frictions, problems. Feature requests are things-that-could-be-great: opportunities, possibilities, new systems or abilities.

Write down as many “bugs” and “feature requests” as you can, for your own life.

Prompt 1:

A genie has offered to fix every bug you’ve written down, and to give you every feature you’ve requested, but then it will freeze your personality—you won’t be able to grow or add or improve anything else.

Hearing that, are there other things you’d like to write down, before the genie takes your list and works its magic?

Prompt 2:

Imagine someone you know well, like your father or your best friend or a longtime boss or colleague or mentor. Choose someone specific, and really imagine them sitting right there beside you, looking over your shoulder, and reading your list. You say to them, “Look! That’s everything!” and they’re skeptical. What things do they think you’ve forgotten, and should add?

Prompt 3:

Do a mental walkthrough of your day and/or week, starting from waking up on Monday and going step by step through all of your routines and all of the places you go, things you do, and people you interact with. Look for snags or hiccups, as well as for exciting opportunities. Be as thorough as you can—often, actually taking the day or week step by step will cause you to remember things you hadn’t thought of.

Prompt 4:

Take a nice, long, slow thinking pause for each of the following broad domains, one at a time (at least ten seconds and maybe as many as thirty):

Work/career

Education/learning/curiosity

Family

Money/finances

Exercise

Food/diet

Sleep

Scheduling and time management

Hobbies

Friends

Romance

Communication

Social interaction

Emotions

Boundaries

Prompt 5:

Now read this sentence, nice and slow, letting it percolate (maybe read it two or three times, or pause in the middle if it sparks thoughts, and then come back and read the rest).

For this last prompt, think of ways you want the world itself to be different, opportunities you haven’t seized (and what’s kept you from seizing them), problems you haven’t solved (and what makes them sticky), things you knew would go wrong (and then you didn’t do anything about it), times you’ve lost connections or dropped out of things (and then were sad about it), things you wish you’d said but didn’t, things you did say but regretted, places where you’ve never ever felt satisfied or okay, and anything you’d be embarrassed about if your heroes and idols and role models knew.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All the posts I will never write, published by Self-Embedded Agent on August 14, 2022 on LessWrong.

This post has been written for the first Refine blog post day, at the end of the week of readings, dicussions, and exercises about epistomology for doing good conceptual research.

(/with courtesy to Adam Shimi who suggested the title and idea. )

Rationality, Probability, Uncertainty, Reasoning

Failures of The Aumann Agreement Theorem

The famous Aumann Agreement Theorem states that rational reasoners can never agree-to-disagree. In day-to-day life we clearly have many situations where rational reasoners cannot agree-to-disagree. Are people just bad rationalist or are there more fundamental reasons that the Aumann Agreement Theorem can fail.

I review all the ways in which the Aumann Agreement Theorem can fail that I know of - including failures based on indexical information, computational-complexity obstacles, divergent-interpretations-of-evidence, Hansonian non-truth-seeking and more.

Warren Buffet: The Alpha of Wall Street

If we observe a trader that consistenly beat the market that should be evidence against the Efficient Market Hypothesis.

A trader could also just have been lucky. How much should we update against the EMH and how much should we expect the trader to beat the market in the future?

Can we quantify how much information the market absorbed? This is very reminiscent of Wows Bayesian surprise in Bayesian statistics.

The Bid-Ask Spread and Epistemic Uncertainty/ Prediction Markets as Epistemic Fog of War

If you know A will resolve you should buy shares on A, if you know not A will happen you should buy shares on not A. If you think A will not resolve you should sell shares on A. The Bid-ask Spread measures bet resolution uncertainty

Suppose an adversary has an interest in showing you A and t. There is a -> selective reporting

When an earnings call come in.. bid-ask spread increases.

Where Forecasting goes Wrong...

Forecasting is now a big deal in the rationalist community. I argue that a slavish adherence to Bayesian Orthodoxy leads to missing most of the value of prediction markets.

What do we mean when we talk about Probability?

Possibility theory is prior to Probability Theory. Probability theory is possibility theory + Cournot's principle.

Cournot's principle that an epsilon/zero probability possible event will never happen is the fundamental principle of probability theory.

cf Shafer on history of Cournot's principle.

But what happens when we observe a epsilon/zero probability event? We obtain a contradiction requiring belief revision.

Wow!1! I made a productive mistake

An exposition of 'Bayesian Surprise attract Human Attention', focused on the notion of 'Bayesian surprisal' measured in wows. There has been a ton of interest in Predictive Processing and Friston's Free Energy Principle. The discussion is often hampered by equivocation between different quantitites that are the log of something. This post will try to clearly disambiguate between these notions and give both mathematical and intuitive explanations.

I argue that the notion of a 'productive mistake' can be formalized as the choice to engage with a high wow high entropy source.

Compare risk-seeking behaviour in MaxCausalEnt and Schmidthuber's Artificial Curiosity?

Foundations of Reasoning

Atocha Aliseda's Axioms of Abduction

A book review of Aliseda's underappreciated 'Abductive Reasoning'.

From SEP: In the philosophical literature, the term “abduction” is used in two related but different senses. In both senses, the term refers to some form of explanatory reasoning. However, in the historically first sense, it refers to the place of explanatory reasoning in generating hypotheses, while in the sense in which it is used most frequently in the modern literature it refers to the place of explan...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Will MacAskill: The Beginning of History, published by Zach Stein-Perlman on August 13, 2022 on The Effective Altruism Forum.

Will has published some other pieces in advance of What We Owe the Future, but The Beginning of History: Surviving the Era of Catastrophic Risk is the deepest.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open Phil is hiring for a program officer in global health R&D, published by JacobTref on August 14, 2022 on The Effective Altruism Forum.

Open Philanthropy is looking for Program Officers to lead our work on global health research and development (R&D).

Historically, health technologies like vaccines and drugs have saved millions of lives and improved people’s health around the world. However, diseases primarily affecting the world’s poorest people receive much less R&D spending relative to health burden than diseases affecting wealthier people. This discrepancy has left many interesting and challenging technical problems unsolved, which have enormous stakes.

We believe that preventable deaths and ill health caused by neglected diseases will persist for decades without further investment, and that funding such global health R&D is competitive with the best opportunities available to Open Philanthropy.

Open Philanthropy’s scientific research grantmaking began in 2016; in recent years, roughly half of that program’s budget ($25M/year of $50M/year) has been spent in areas that we would classify as global health R&D. Based on our early evaluation of that portfolio’s performance to date, and the remaining opportunities for impact in global health R&D, we have decided to increase our spending in the area to at least $100M/year.

We plan to build a global health R&D team over time and eventually hire more grantmaking staff. Right now, we are hoping to hire an excellent person or several excellent people to lay the foundation for that team.

How to apply

You can find the application link here, alongside more details on the position — including the skills and experience we're looking for, salary, and benefits. The job can be done remotely.

Please apply by 11:59 PM PST on Saturday, September 10th, 2022 to be considered. We will evaluate all applicants as a cohort after that date.

Any questions?

If you have questions about this role, or our thoughts on global health R&D, ask them here or contact me at [email protected]. I'm traveling right now, so may be sporadic in answering, but I'm excited about this role and hope you'll consider applying if it sounds fun to you!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Historical EA funding data, published by TylerMaule on August 14, 2022 on The Effective Altruism Forum.

Summary

I have consolidated publicly available grants data from EA organizations into a spreadsheet, which I intend to update periodically. Totals pictured below.

Observations

$2.6Bn in grants on record since 2012, about 63% of which went to Global Health.

With the addition of FTX and impressive fundraising by GiveWell, Animal Welfare looks even more neglected in relative terms—effective animal charities will likely receive something like 5% of EA funding in 2022, the smallest figure since 2015 by a wide margin.

Notes on the data

NB: This is just one observer's tally of public data. Sources are cited in the spreadsheet; I am happy to correct any errors as they are pointed out.

GiveWell:

GiveWell uses a 'metrics year' starting 1 Feb (all other sources were tabulated by calendar year).

GiveWell started breaking out 'funds directed' vs 'funds raised' for metrics year 2021. Previous years refer to 'money moved', which is close but not exactly the same.

I have excluded funds directed through GiveWell by Open Phil and EA Funds, as those are already included in this data set.

Open Phil

Open Phil labels their grants using 25 'focus areas'. My subjective mapping to broader cause area is laid out in the spreadsheet.

Note that about 20% of funds granted by Open Phil have gone to 'other' areas such as Criminal Justice Reform; these are omitted from the summary figures but still tabulated elsewhere in the spreadsheet.

General

2022 estimates are a bit speculative, but a reasonable guess as to how funding will look with the addition of the Future Fund.

The total Global Health figure for 2021 (~$400M) looks surprisingly low considering e.g. that GiveWell just reported over $500M funds directed for 2021 (including Open Phil and EA Funds). I think that this is accounted for by (a) GiveWell's metrics year extending through Jan '22, (Open Phil reported $26M of Global Health grants that month), and (b) the possibility that some of this was 'directed' i.e. 'firm commitment of $X to org Y' by Open Phil in 2021, but paid out or recorded to the grants database months later; still seeking explicit confirmation here.

Future work

If there is any presently available data that seems worth adding, let me know and I may consider it.

I may be interested in a more comprehensive analysis on this topic, e.g. using the full budget of every GiveWell-recommended charity. I'd be interested to hear if anyone has access to this type of data, or if this type of project seems particularly valuable.

Thanks to Niel Bowerman for helpful comments

Currently the bottleneck to synchronizing data is the GiveWell annual metrics report, which is typically published in the second half of the following year. I may update more often if that is useful.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prioritisation should consider potential for ongoing evaluation alongside expected value and evidence quality, published by freedomandutility on August 13, 2022 on The Effective Altruism Forum.

One argument in favour of funding deworming, despite uncertain effects, is that

the expected value is high because the estimated effect size is slightly positive and the direct costs of deworming are low

the quality of evidence is high (RCTs) (compared to many other initiatives which get funded by EA)

I think this excludes a third key consideration in prioritising causes, which is the potential for ongoing evaluation, both in terms of quality of evaluation and timescales for evaluation.

Imagine that we have 2 interventions, intervention A and intervention B.

Both have the same estimated effect size where the central estimate is positive, but there's a chance the effect is negative: +0.05 (95% Confidence Interval from -0.25 to +0.35)

Both have the same quality of evidence: Effect size estimated from a meta-analysis of 5 RCTs

But Intervention A has marginally greater expected value because it's cheaper to implement

and

Intervention B would be much cheaper to evaluate through ongoing observational studies

In this case, I think we should fund Intervention B because of the value of being able to course-correct and update our estimated expected value based on more cheaply accessible new evidence.

If EA continues to fund deworming in the long-term, but further testing of effects via RCTs or monitoring of effects via observational studies doesn't occur, there's a risk that millions could be spent sub-optimally.

I think EA should either:

also fund further evaluation of deworming, or

prioritise interventions with better expected value or better quality of evidence or potential for cheaper, higher quality or faster further evaluation, over deworming

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prioritizing x-risks may require caring about future people, published by elifland on August 14, 2022 on The Effective Altruism Forum.

Introduction

Several recent popular posts (here, here, and here) have made the case that existential risks (x-risks) should be introduced without appealing to longtermism or the idea that future people have moral value. They tend to argue or imply that x-risks would still be justified as a priority without caring about future people. I felt intuitively skeptical of this claim and decided to stress-test it.

In this post, I:

Argue that prioritizing x-risks over near-term interventions and global catastrophic risks may require caring about future people. More

Disambiguate connotations of “longtermism”, and suggest a strategy for introducing the priority of existential risks. More

Review and respond to previous articles which mostly argued that longtermism wasn’t necessary for prioritizing existential risks. More

Prioritizing x-risks may require caring about future people

I’ll do some rough analyses on the value of x-risk interventions vs. (a) near-term interventions, such as global health and animal welfare and (b) global catastrophic risk (GCR) interventions, such as reducing risk of nuclear war. I assume a lack of caring about future people to test whether it’s necessary for prioritizing x-risk above alternatives. My goal is to do a quick first pass, which I’d love for others to build on / challenge / improve!

I find that without taking into account future people, x-risk interventions are approximately as cost-effective as near-term and GCR interventions. Therefore, strongly prioritizing x-risks may require caring about future people; otherwise, it depends on non-obvious claims about the tractability of x-risk reduction and the moral weights of animals.

InterventionRough estimated cost-effectiveness, current lives only ($/human-life-equivalent-saved)General x-risk prevention (funding bar)$125 to $1,250AI x-risk prevention$375Animal welfare$450Bio x-risk prevention$1,000Nuclear war prevention$1,250GiveWell-style global health (e.g. bednet distribution)$4,500

Estimating the value of x-risk interventions

This paper estimates that $250B would reduce biorisk by 1%. Taking Ord’s estimate of 3% biorisk this century and a population of ~8 billion, we get: $250B / (8B .01 .03) = $104,167/life saved via biorisk interventions. The paper calls this a conservative estimate, so a more optimistic one might be 1-2 more OOMs as effective at ~$10,000 to ~$1,000 / life saved; let’s take the optimistic end of $1,000 / life saved as a rough best guess, since work on bio x-risk likely also reduces the likelihood of deaths from below-existential pandemics and these seem substantially more likely than the most severe ones.

For AI risk, 80,000 Hours estimated several years ago that another $100M/yr (for how long? let’s say 30 years) can reduce AI risk by 1%; unclear if this percentage is absolute or relative, relative seems more reasonable to me. Let’s again defer to Ord and assume 10% total AI risk. This gives: ($100M 30) / (8B .01 .1) = $375 / life saved.

On the funding side, Linch has ideated a .01% Fund which would aim to reduce x-risks by .01% for $100M-$1B. This implies a cost-effectiveness of ($100M to $1B) / (8B .0001) = $125 to 1,250 / life saved.

Comparing to near-term interventions

GiveWell estimates it costs $4,500 to save a life through global health interventions.

This post estimates that animal welfare interventions may be ~10x more effective, implying $450 / human life-equivalent, though this is an especially rough number.

Comparing to GCR intervention

Less obviously than near-term interventions, a potential issue with not caring about future people is over-prioritizing global catastrophic risks (that might kill a substantial percentage o...