Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Immanuel Kant and the Decision Theory App Store, published by Daniel Kokotajlo on July 10, 2022 on LessWrong.

[Epistemic status: About as silly as it sounds.]

Prepare to be astounded by this rationalist reconstruction of Kant, drawn out of an unbelievably tiny parcel of Kant literature!

Kant argues that all rational agents will:

“Act only according to that maxim whereby you can at the same time will that it should become a universal law.” (421)

“Act in such a way that you treat humanity, whether in your own person or in the person of another, always at the same time as an end and never simply as a means.” (429)

Kant clarifies that treating someone as an end means striving to further their ends, i.e. goals/values. (430)

Kant clarifies that strictly speaking it’s not just humans that should be treated this way, but all rational beings. He specifically says that this does not extend to non-rational beings. (428)

“Act in accordance with the maxims of a member legislating universal laws for a merely possible kingdom of ends.” (439)

Not only are all of these claims allegedly derivable from the concept of instrumental rationality, they are supposedly equivalent!

Bold claims, lol. What is he smoking?

Well, listen up.

Taboo “morality.” We are interested in functions that map [epistemic state, preferences, set of available actions] to [action].

Suppose there is an "optimal" function. Call this "instrumental rationality," a.k.a. “Systematized Winning.”

Kant asks: Obviously what the optimal function tells you to do depends heavily on your goals and credences; the best way to systematically win depends on what the victory conditions are. Is there anything interesting we can say about what the optimal function recommends that isn’t like this? Any non-trivial things that it tells everyone to do regardless of what their goals are?

Kant answers: Yes! Consider the twin Prisoner's Dilemma--a version of the PD in which it is common knowledge that both players implement the same algorithm and thus will make the same choice. Suppose (for contradiction) that the optimal function defects. We can now construct a new function, Optimal+, that seems superior to the optimal function:

IF in twin PD against someone who you know runs Optimal+: Cooperate

ELSE: Do whatever the optimal function will do.

Optimal+ is superior to the optimal function because it is exactly the same except that it gets better results in the twin PD (because the opponent will cooperate too, because they are running the same algorithm as you).

Contradiction! Looks like our "optimal function" wasn't optimal after all. Therefore the real optimal function must cooperate in the twin PD.

Generalizing this reasoning, Kant says, the optimal function will choose as if it is choosing for all instances of the optimal function in similar situations. Thus we can conclude the following interesting fact: Regardless of what your goals are, the optimal function will tell you to avoid doing things that you wouldn’t want other rational agents in similar situations to do. (rational agents := agents obeying the optimal function.)

To understand this, and see how it generalizes still further, I hereby introduce the following analogy:

The Decision Theory App Store

Imagine an ideal competitive market for advice-giving AI assistants. Tech companies code them up and then you download them for free from the app store. There is AlphaBot, MetaBot, OpenBot, DeepBot.

When installed, the apps give advice. Specifically they scan your brain to extract your credences and values/utility function, and then they tell you what to do. You can follow the advice or not.

Sometimes users end up in Twin Prisoner’s Dilemmas. That is, situations where they are in some sort of prisoner’s dilemma with someone else where there is common knowledge that they both are likely t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Taste & Shaping, published by CFAR!Duncan on July 10, 2022 on LessWrong.

Author's note: while it does not belong in the handbook proper, Duncan Sabien's essay Goodhart's Imperius is very close to (and partially derived from) the content in this chapter, and is a good extension or follow-up.

Epistemic status: Mixed

Many of the concepts presented in Taste & Shaping (such as operant conditioning, hyperbolic discounting, and theories of intrinsic vs. extrinsic motivation) are all well-known and well-researched. Similarly, the problem the unit seeks to address (of experiencing an emotional disconnect between actions and goals) is widely known and discussed. There is some academic support for the efficacy of priming and narrative reframings, and strong anecdotal support (but no formal research) for the overall conceptual framework.

English uses the word “want” to mean both the declarative, persistent desire to achieve a long-term goal, and also the immediate, visceral desire to satisfy an in-the-moment urge. That’s how we end up saying sentences like “I want to exercise, but I don’t want to exercise...know what I mean?”

Part of the reason that we don’t want to exercise, in the moment (or that we don’t want to work on page 37 of our dissertation, or that we don’t want to call up our parents even though it’s been a while, or that we don’t want to look at our work email backlog, or any number of other things) has to do with emotional valence.

Objects, actions, and experiences often carry with them a sort of aggregated positive-negative or approach-avoid rating, which we at CFAR casually refer to as a “yuck” or a “yum.” Consider, for instance:

The sound that a laptop makes when it falls on a hard, concrete floor

The feelings evoked by the “pinwheel of death” on an Apple computer

Your emotional reaction to the sound you typically use as your alarm

The sight of flashing lights in your rear view mirror

For many people, the sound of a falling laptop is actually painful in some hard-to-define way—it’s as though the sound is somehow intrinsically bad, just as the pinwheel of death is fundamentally frustrating. On an empirical, logical level, it’s clear that these associations were picked up after the fact— many people grow to hate their alarm clock sound over time; few people choose an unpleasant sound to begin with.

But on an emotional level, the yucks and yums can feel inherent and essential. We want to exercise, because exercise (and health and attractiveness and capability) are all very yummy—they’re all attractive qualities that create desire and motivation.

Exhaustion, though? And sweat? And aching joints and burning muscles and gasping for breath and feeling smelly and sticky and uncomfortable? For many people, the component pieces of this very yummy concept are icky, ugh-y, and more or less repulsive. Yuck.

This is problematic, because motivation often speaks the more immediate and emotional language of System 1. When we want to do something in the eat-a-bunch-of-cake sense, we don’t usually feel like we’re spending willpower or using up motivation or “making” ourselves follow through. Following our immediate sense of “yum” is like being in freefall—the default state is forward movement, and it takes some sort of active effort to put on the brakes.

In contrast, many of our more ambitious goals (like earning a postgraduate degree) require a large number of more-or-less difficult, more-or-less arbitrary, and more-or-less thankless steps, few of which are intrinsically desirable on their own. We want the end goal, but we have to consciously marshal our resources in order to take steps toward it, fighting against “yuck” factors much of the way.

People often power through these yuck factors through methods like watching inspiring videos, soliciting support from peers, and setting...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Let's stop saying 'funding overhang', published by Benjamin Todd on July 10, 2022 on The Effective Altruism Forum.

Last year people were saying there’s a funding overhang – an excess of funding relative to opportunities – and now it seems to have gone away. What’s going on?

It’s natural to talk in terms of funding overhangs, but I don’t think it’s the best way to think about the situation. I used to use the term myself, but have now stopped.

Funding ‘overhang’ makes it sound like there’s a ‘correct’ amount of funding, and we can have too little or too much. There are similar issues with terms like ‘funding constrained’ or ‘funding gap’. It’s very natural to think in terms of absolutes, but in reality, everything is on a continuum.

More funding is always more useful – it’s just that its value diminishes as you have more and more. There’s no single optimal amount.

I think the better concept is the bar – the level of cost-effectiveness at which it’s worth funding or working on an opportunity.

Effective altruism will hopefully exist for many decades, and so the challenge we face is figuring out how many resources to give each year in order to maximise our impact over those decades.

To do that, ideally we could put all the opportunities we’ll face over those decades in order of cost-effectiveness, fund or work on the most effective ones first, and then keep going till the entire pot of resources is allocated.

The bar is then given by the cost-effectiveness of the final grant (the ‘last dollar’) or final job. Any grant that’s more effective than the bar should be supported, and all others shouldn’t.

Entrepreneurs should aim to create organisations that are more cost-effective than the current bar – the higher they are above the bar, the easier they will find it to fundraise

If the bar for funding goes down, but the bar for direct work stays the same, then earning to give becomes relatively less attractive. Where the bar ends up depends on (i) how many resources we have (ii) how many opportunities we discover.

This means the bar will change from year-to-year depending on changes in expectations about (i) how many new donors entering the community (ii) stockmarket returns (iii) the discovery of new donation opportunities (iv) the world changing to create or remove opportunities.

All this is hard to estimate, but we have to do our best.

For instance, what’s happened recently is that lots more resources entered the community (as tech stocks and crypto went up), so the bar was dropped, and then the market crash decreased the amount of resources available, so the bar was raised a bit. Overall I think the bar is still lower than it was before 2019, but it’s not as low as it seemed 6 months ago.

This is easiest to quantify with GiveWell recommended charities. Around 2015, the cost-effectiveness of the marginal charity was roughly 10x cash transfers. Last year it dropped to 7x, and this year it increased to 8x – the latter was partly due to the market decline and party due to GiveWell discovering more highly effective opportunities than expected.

I think a similar thing has happened within longtermist issues (as opposed to global health), but my sense is the ups and downs are somewhat greater (e.g. maybe it was 10 units in 2015, then dropped to 2 units, and is now back up to 3).

Either way, these ups and downs are part of the normal course of events.

At all points, additional funding has been useful, it’s just that its value has changed somewhat.

And it has never been ‘easy’ to get funding – rather, how difficult it is goes up and down over time.

Most of the time, these differences are not super decision-relevant. The uncertainties involved in effective altruism are so huge, that a change in the bar by a factor of 2 isn’t going to make the difference in almost any decisions – and s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: User research as a barometer of software design, published by adamzerner on July 8, 2022 on LessWrong.

(Cross-posted on my personal blog.)

In UX design, they do something called user research.

Consider an example. You have two designs: A and B. You think A is better but John thinks B is better. Who's right?

Well, you can debate a bunch of stuff. Design A breaks this guideline, Design B does breaks that guideline. Or... you can actually sit down next to users and watch them use each design. See how much difficulty they have and stuff. If users are all confused by Design A and content with Design B, then Design B is the winner, despite whatever the theoretical arguments might say.

I think the same thing is true in software design. For example, consider two designs: C and D. Design C has a lot of duplicate code, whereas Design D is, well, DRY.

Design D is better, right? Isn't DRYing up duplicate code is a good thing?

Maybe. Maybe not. They might have taught you about DRY code in school, but that doesn't matter. The barometer here is user research, where the users are the programmers. Imagine that you had some programmers try to build the app using "wet" code (Design C). And then imagine that you had other programmers try to build the app using DRY code (Design D). Who would have an easier time?

Why am I making this point? Isn't it obvious? Well, I think that it can be easy to get lost in the weeds a bit with software design and lose sight of the bigger picture. The bigger picture is that you want code that is designed in such a way that it is easy to work with. Imagine having to implement a new feature and being able to easily read through the existing code, understand it, and figure out where you need to make the change for your feature. That is what we are aiming for.

What do I mean when I say I think people get lost in the weeds? Here's what I mean. I propose that they fall victim to Lost Purposes.

Imagine a high school student who starts off trying to get good grades in order to get into a good college, in order to eventually get into a career that will make them happy, but who finds themselves stressed about getting a B+ in a class during the last semester of their senior year when they have already been accepted into the college they are going to attend. This student now cares about the grade in and of itself and has lost sight of the fact that it originally was just a means to the end of leading them to a happy and fulfilling life.

Similarly, the purpose of different design patterns like DRY is to make the codebase easy to work with. But, like the high school student, we lose sight of this. We start seeing them as ends in and of themselves, and we judge code according to how DRY, SOLID and ACIDic it is. Unfortunately the acronyms are just a proxy for what we are truly interested in. They're not the real thing.

That doesn't mean the acronyms aren't useful though. I think that they are useful. I just think that they need to be combined with... something else. It is easy to lose sight of the forest for the trees when you think about the acronyms. User research, at least as a thought experiment, helps you keep sight of that forest, I propose.

Consider the DRY vs "wet" example. At first it might seem obvious that you'd want to dry up that duplicate code. But what happens when you do a little thought experiment. Imagine, in your minds eye, yourself working with the DRY code. And now, again, in your minds eye, imagine yourself working with that "wet" code.

What does it feel like? Maybe that "wet" code isn't actually that bad? Maybe it's kinda nice. Maybe you find that, despite what the professors said, the duplication isn't actually causing you difficulty. Maybe you enjoy the fact that you get to deal with things at a more object/ground level rather than at the more abs...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wheeling and dealing: An internal bargaining approach to moral uncertainty, published by MichaelPlant on July 10, 2022 on The Effective Altruism Forum.

In this post, I explore and evaluate an internal bargaining (IB) approach to moral uncertainty. On this account, the appropriate decision under moral uncertainty is the one that would be reached as the result of negotiations between agents representing the interests of each moral theory, who are awarded your resources in proportion to your credence in that theory. This has only been discussed so far by Greaves and Cotton-Barratt (2019), who give a technical account of the approach and tentatively conclude that the view is inferior to the leading alternative approach to moral uncertainty, maximise expected choiceworthiness (MEC). I provide a more intuitive sketch of how internal bargaining works, and do so in a wide range of cases. On the basis of the cases, as well as considering some challenges for the view and its theoretical features, I tentatively conclude it is superior to MEC.

I close by noting one implication relevant to effective altruists: while MEC seems to push us towards a (fanatical) adherence to longtermism, internal bargaining would provide a justification for something like worldview diversification.

Notes to reader: (1) I’m deliberately writing this in a fairly rough-and-ready way rather than as a piece of polished philosophy. If I had to write it as the latter, I don’t think it would get written for perhaps another year or two. I’ll shortly begin working on this topic with Harry Lloyd, an HLI Summer Research Fellow, and I wanted to organise and share my thoughts before doing that. In the spirit of Blaise Pascal, I should say that if I had had more time, I would have written something shorter. (2) This can be considered a ‘red-team’ of current EA thinking.

1. Introduction

When philosophers introduce the idea of moral uncertainty - uncertainty about what we ought, morally, to do - they often quickly point out that we are used to making decisions in the face of empirical uncertainty all the time.

Here’s the standard case of empirical uncertainty: it might rain tomorrow, but it might not. Should you pack an umbrella? The standard response to this is to apply expected utility theory: you need to think about the chance of it raining, the cost of carrying an umbrella, and the cost of getting wet if you don’t carry an umbrella. Or, more formally, you need to assign credences (strengths of belief) and utilities (numbers representing value) to the various outcomes.

Hence, when it’s pointed out that we’re also often uncertain about what we ought to do - should we, for example, be consequentialists or deontologists? - the standard thought is that our account of moral uncertainty should probably work much like our account of empirical uncertainty. The analogous account for moral uncertainty is called maximise expected choiceworthiness (MEC) (MacAskill, Ord, Bkyvist, 2020). The basic idea is that we need to assign credences to the various theories as well as a numerical value on how choiceworthy the relevant options are on those theories. The standard case to illuminate this is:

Meat or Salad: You are choosing whether to eat meat from a factory farm or have a salad instead. You have a 40% credence in View A, a deontological theory on which eating meat is seriously morally wrong and a 60% credence in View B, a consequentialist theory on which both choices are permissible. You’d prefer to eat meat.

Intuitively, you ought to eat the salad. Why? Even though you have less credence in A than B, when we consider the relative stakes for each view, we notice that View A cares much more about avoiding the meat. Hence, go for the salad as that maximises choiceworthiness.

MEC is subject to various objections (see MacAskill, O...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Center for Space Governance, published by Space Governance on July 10, 2022 on The Effective Altruism Forum.

We are excited to be launching the Center for Space Governance, a non-profit research organization dedicated to exploring current, emerging, and future issues in space policy. We aim to define and map the field of space governance, advancing effective solutions to address the greatest challenges and opportunities for humanity’s future in space.

We are grateful to have received support from the FTX Future Fund’s regranting program, under the space governance area of interest. In the coming months, we will be developing our research agenda with a focus on the long-term implications of space governance, building on the recently published 80,000 Hours problem profile. We would also like to thank Fin Moorhouse for his initiative in bringing together the first EA space community this year, from which several members have already joined our team.

We are looking to bring on board a number of Research Fellows and Research Affiliates to contribute to our activities this summer, from July to September. We welcome applications from all backgrounds and levels of experience. Apply to join here.

If you are interested in supporting our mission or advising us, please get in touch at [email protected]. Thank you, Bartu Kaleagasi and Gustavs Zilgalvis.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ars D&D.sci: Mysteries of Mana, published by aphyer on July 9, 2022 on LessWrong.

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

STORY (skippable)

For thousands of years the Order of Athena has preserved magical knowledge, passing it down from masters to apprentices, in a lineage of knowledge unmarred by countless centuries.

Those hidebound old fools! Your thesis, 'Calculating Levels of Ambient Mana,' could have revolutionized all magic! You found a way to determine how powerful various types of mana used to power spells were at various times, via bold modern Data Science techniques these antediluvian wizards would never have dreamed of.

Sadly, these senile old mages failed to appreciate your contribution, and have assigned you to a standard apprenticeship under a training mage, intending to have you spend the next fifteen years studying primeval books under his tutelage before they let you call yourself a mage. Ridiculous! Modern techniques are far more applicable, and only by looking forward can we hope to become more powerful than the past! These mages are doomed to only ever pass on the knowledge of the past, but you can learn more and grow stronger than they could ever be!

Of course, first you'll need to convince your new 'master' to let you pass early. He has at least agreed to give you a chance to demonstrate the use of your new techniques. Among these mages' traditions is to hold frequent non-lethal duels between one another, and he has one such duel coming up. He's offered to allow you to use your new techniques to choose which spells he should prepare for that duel.

With your new knowledge of mana levels, and an extensive history of duels, you expect you can choose excellent spells and maximize his odds of winning. Then, perhaps, you'll be able to spend the next fifteen years pursuing your own research instead of learning Latin so that you can read cobwebbed old books!

DATA & OBJECTIVES

The dataset contains a list of past duels between mages. Each row contains:

The mana levels at the time of the duel.

The spells each mage brought to that duel.

Which mage won that duel.

Using that dataset, you need to select which spells your master should prepare for this imminent duel to maximize his odds of winning. He can prepare three of the following spells:

Alatar's Abyssal Armor

Balefire Blast of Bonisagus

Crystalline Citadel of Criamon

Flambeau's Flying Fireball

Hylenion's Hammer of the Heavens

Logain's Lava Levee

Merinita's Mud Missiles

Olorin's Obsidian Onslaught

Rainbow Rays of Rahl

Solomon's Solar Shield

Vaporous Vambrace of Verditius

Wicked Wiskeria's Wall of Wailing Winds

Current mana levels are:

Fire 24

Water 39

Earth 26

Air 26

Light 18

His opponent will be preparing the following spells:

Flambeau's Flying Fireball

Merinita's Mud Missiles

Solomon's Solar Shield

Based on how stuck-in-the-past all these wizards are, and how stable the Laws of Magic seem to be, you don't expect that wizard strategies/spell selection/how a duel works/what each spell does should change over time - the only differences you expect between duels are what spells each wizard brought and what the mana levels were.

I'll aim to post the ruleset and results on July 18th (giving one week and both weekends for players). If you find yourself wanting extra time, comment below and I can push this deadline back.

As usual, working together is allowed, but for the sake of anyone who wants to work alone, please spoiler parts of your answers (type a '>' followed by a '!' at the start of a line to open a spoiler block) that contain information or questions about the dataset.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please vis...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is Operations Management?, published by Joseph Lemien on July 10, 2022 on The Effective Altruism Forum.

Epistemic Status: While I feel fairly confident that the core viewpoints expressed in this post are correct, I haven’t shared drafts of this post with other people focused on operations, and so it’s possible I’ve missed important context from other perspectives. There is lots of simplification here; many details and complexities are brushed over to make this more accessible. A good deal of this post is more opinion rather than clear cut fact, so view this more like an op-ed.

Feel free to skip all the examples if you like. They are here for context, but you can understand the core ideas decently even if you skip over the examples.

Caveats, limitations, etc.

I don’t view myself as an expert on the field of operations. Thus, you should view this post as coming from a fairly well-read individual with a decent amount of practice, not as a highly authoritative account from a person with world-class training.

Overview

I'm going to provide a brief overview of what operations is and what operations management is, mention some gentle critiques of how operations is viewed in EA, give several examples of operations management in different contexts:

a factory with materials coming in and products going out, which is a very classic situation for operations management.

a sandwich shop, which serves to show how operations management can be applied to non-industrial situations.

a “digital factory” of video production, which demonstrates both the extent to which the details of the situation matter as well as how easily systems can get complicated.

a health clinic, showing a very realistic application of operations management in which the resources flowing through the system are people.

the Berlin airlift, a historical example in which operations management saved lives.

I’ve purposely included many examples in order to illustrate the wide variety of what operations and operations management looks like in different contexts.

Why I wrote this

I wrote this for a few different reasons:

I have seen a few pieces of writing within EA that conflate operations work and personal assistant work, and I want to prevent that from happening. I’ve also seen multiple job postings titled as operations managers that list personal assistant-type tasks. My stance is this: in certain scenarios you could make a reasonable argument that a personal assistant falls within the operations team/department, but in general we should not be conflating PA work with operations work. An operations manager should be creating/designing systems and processes, optimizing existing processes and establishing new processes, helping business flow smoothly. An operations manager job generally shouldn’t include updating social media pages, handling someone else's emails, or managing their travel schedule.

My rough impression is that people in the EA community (as well as people in general) seem to have a hard time describing what operations management is. The 80,000 Hours article doesn’t mention resource flows, or bottlenecks, or other standard concepts in operations management. I'm hoping that by providing several examples of operations management in distinct contexts I can help to solidify the concept a bit, specifically the manufacturing/production aspect of it.

I very much enjoyed eirine’s two posts on So you want to do operations, and I wish that there were several more posts in that series giving more in-depth descriptions. I was disappointed to learn that the series ended after only two posts.

If people in EA want to describe “everything that supports the core work and allows other people to focus on the core work” then I’d recommend using a term other than operations, because operations already has a meaning in the...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Report from a civilizational observer on Earth, published by owencb on July 9, 2022 on LessWrong.

[translated into English by the author]

I have arrived in [2022]. There is one species, humans, that is building civilization. (A few other species exhibit potential but have not crossed the threshold of runaway economic development or cultural accumulation; it is now unlikely that they ever will unless a deliberate choice is made to allow them to do so.)

There are around 129.2 humans. They are organized into loose and overlapping political structures, including firms, [countries with systems of law one cannot opt out of], and [religions and hobby groups]. There is a degree of [law, treaties, and custom] constraining these groups, but there is no strong political unification, and sometimes there is significant [violence causing destruction of persons or property].

Most of the physical activity of civilization is done by machines operated by humans, although a significant minority is still done directly by humans, and a significant-and-increasing minority is done by machines operated by [rule-governed thinking-machines].

In some domains a large majority of the intellectual labour of civilization is done by thinking-machines, but these tend to be [rule-governed thinking-machines] following precise instructions laid down by humans. Most key intellectual work is still done by humans (although a lot of the communication of this is mediated by [rule-governed thinking-machines]).

In the last econoyear, civilization has built nontrivial [patterning thinking-machines]. And in the last economonth, these have been applied to human language with significant success. [Patterning thinking-machines] are now capable of doing sophisticated tasks that humans might previously have been paid to do. And since the language they learned encoded a lot of civilizational knowledge, these [patterning-thinking-machines] now implicitly contain a large fraction of civilizational knowledge (at least at a [shallow/indexing] level).

Intellectual automation

The next several economonths will likely represent a period of acceleration for civilization. It is unclear whether they will contain an emergent [post-civilization], though it is most likely that that will occur within the next 24 econoyears, and maybe much sooner. It is currently unclear what the nature of any emergent [post-civilization] will be.

The reason for acceleration is:

They can now build [patterning thinking-machines] which are strong at general pattern recognition and generation

Most of the intellectual labour that humans do is just these two operations, specialized to particular domains

(Indeed most human cognition is just these two operations, applied appropriately, e.g. to pattern recognition of valid reasoning steps)

It is therefore mostly the lack of major efforts, rather than technological inability, which prevents a majority of human intellectual work being automated.

(Some human intellectual work is automation-proof for now for economic reasons. Earth’s best [patterning-thinking-machines] are still significantly slower to learn and to generalize than humans are (in terms of the number of examples they need to see), so for specialized intellectual labour the training involved would currently be prohibitive.)

Over the next few economonths we can expect a few domains to start having large swathes of low-level work automated by [patterning thinking-machines]. This is most likely in areas that give human supervisors time to intervene in cases of errors (bemusingly, the humans have fixated on having thinking-machines take over the controls of dangerously-fast mobile machines, which have very little margin for error).

Humans working in these domains will increasingly devote time to training and overseeing these systems rather than doing ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comment on "Propositions Concerning Digital Minds and Society", published by Zack M Davis on July 10, 2022 on LessWrong.

I will do my best to teach them About life and what it's worth I just hope that I can keep them From destroying the Earth

Jonathan Coulton, "The Future Soon"

In a recent paper, Nick Bostrom and Carl Shulman present "Propositions Concerning Digital Minds and Society", a tentative bullet-list outline of claims about how advanced AI could be integrated into Society.

I want to like this list. I like the kind of thing this list is trying to do. But something about some of the points just feels—off. Too conservative, too anthropomorphic—like the list is trying to adapt the spirit of the Universal Declaration of Human Rights to changed circumstances, without noticing that the whole ontology that the Declaration is written in isn't going to survive the intelligence explosion—and probably never really worked as a description of our own world, either.

This feels like a weird criticism to make of Nick Bostrom and Carl Shulman, who probably already know any particular fact or observation I might include in my commentary. (Bostrom literally wrote the book on superintelligence.) "Too anthropomorphic", I claim? The list explicitly names many ways in which AI minds could differ from our own—in overall intelligence, specific capabilities, motivations, substrate, quality and quantity (!) of consciousness, subjective speed—and goes into some detail about how this could change the game theory of Society. What more can I expect of our authors?

It just doesn't seem like the implications of the differences have fully propagated into some of the recommendations?—as if an attempt to write in a way that's comprehensible to Shock Level 2 tech executives and policymakers has failed to elicit all of the latent knowledge that Bostrom and Shulman actually possess. It's understandable that our reasoning about the future often ends up relying on analogies to phenomena we already understand, but ultimately, making sense of a radically different future is going to require new concepts that won't permit reasoning by analogy.

After an introductory sub-list of claims about consciousness and the philosophy of mind (just the basics: physicalism; reductionism on personal identity; some non-human animals are probably conscious and AIs could be, too), we get a sub-list about respecting AI interests. This is an important topic: if most our civilization's thinking is soon to be done inside of machines, the moral status of that cognition is really important: you wouldn't want the future to be powered by the analogue of a factory farm. (And if it turned out that economically and socially-significant AIs aren't conscious and don't have moral status, that would be important to know, too.)

Our authors point out the novel aspects of the situation: that what's good for an AI can be very different from what's good for a human, that designing AIs to have specific motivations is not generally wrong, and that it's possible for AIs to have greater moral patienthood than humans (like the utility monster of philosophical lore). Despite this, some of the points in this section seem to mostly be thinking of AIs as being like humans, but "bigger" or "smaller"

Rights such as freedom of reproduction, freedom of speech, and freedom of thought require adaptation to the special circumstances of AIs with superhuman capabilities in those areas (analogously, e.g., to how campaign finance laws may restrict the freedom of speech of billionaires and corporations). [...]

If an AI is capable of informed consent, then it should not be used to perform work without its informed consent.

Informed consent is not reliably sufficient to safeguard the interests of AIs, even those as smart and capable as a human adult, particularl...