The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - When Did EA Start? by Jeff Kaufman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When Did EA Start?, published by Jeff Kaufman on January 25, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - Sapir-Whorf for Rationalists by Duncan Sabien
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sapir-Whorf for Rationalists, published by Duncan Sabien on January 25, 2023 on LessWrong.
    Casus Belli: As I was scanning over my (rather long) list of essays-to-write, I realized that roughly a fifth of them were of the form "here's a useful standalone concept I'd like to reify," à la cup-stacking skills, fabricated options, split and commit, setting the zero point, and sazen. Some notable entries on that list (which I name here mostly in the hope of someday coming back and turning them into links) include: red vs. white, walking with three, seeding vs. weeding, hidden hinges, reality distortion fields, and something-about-layers-though-that-one-obviously-needs-a-better-word.
    While it's still worthwhile to motivate/justify each individual new conceptual handle (and the planned essays will do so), I found myself imagining a general objection of the form "this is just making up terms for things," or perhaps "this is too many new terms, for too many new things." I realized that there was a chunk of argument, repeated across all of the planned essays, that I could factor out, and that (to the best of my knowledge) there was no single essay aimed directly at the question "why new words at all?"
    So ... voilà.
    Core claims/tl;dr
    New conceptual distinctions naturally beget new terminology.Generally speaking, as soon as humans identify a new Thing, or realize that what they previously thought was a single Thing is actually two Things, they attempt to cache/codify this knowledge in language. Subclaim: this is a good thing; humanity is not, in fact, near the practical limits of its ability to incorporate and effectively wield new conceptual handles.
    New terminology naturally begets new conceptual distinctions.Alexis makes a new distinction, and stores it in language; Blake, via encountering Alexis's language, often becomes capable of making the same distinction, as a result. In particular, this process is often not instantaneous—it's not (always) as simple as just listening to a definition. Actual practice, often fumbling and stilted at first, leads to increased ability-to-perceive-and-distinguish; the verbal categories lay the groundwork for the perceptual/conceptual ones.
    These two dynamics can productively combine within a culture.Cameron, Dallas, and Elliot each go their separate ways and discover new conceptual distinctions not typical of their shared culture. Cameron, Dallas, and Elliot each return, and each teach the other two (a process generally much quicker and easier than the original discovery). Now Cameron, Dallas, and Elliot are each "three concepts ahead" in the game of seeing reality ever more finely and clearly, at a cost of something like only one-point-five concept-discovery's worth of work.(This is not a metaphor; this is in fact straightforwardly what has happened with the collection of lessons learned from famine, disaster, war, politics, and science, which have been turned into words and phrases and aphorisms that can be successfully communicated to a single human over the course of mere decades.)
    That which is not tracked in language will be lost.This is Orwell's thesis—that in order to preserve one's ability to make distinctions, one needs conceptual tools capable of capturing the difference between (e.g.) whispers, murmurs, mumbles, and mutters. Without such tools, it becomes more difficult for an individual, and much more difficult for a culture or subculture, to continue to attend to, care about, and take into account the distinction in question.
    The reification of new distinctions is one of the most productive frontiers of human rationality.It is not the only frontier, by a long shot. But both [the literal development of new terminology to distinguish things which were previously thought to be the same thing, or which were previously invisible] and ...
    32 min
  • LW - Sapir-Whorf for Rationalists by Duncan Sabien
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sapir-Whorf for Rationalists, published by Duncan Sabien on January 25, 2023 on LessWrong.
    Casus Belli: As I was scanning over my (rather long) list of essays-to-write, I realized that roughly a fifth of them were of the form "here's a useful standalone concept I'd like to reify," à la cup-stacking skills, fabricated options, split and commit, setting the zero point, and sazen. Some notable entries on that list (which I name here mostly in the hope of someday coming back and turning them into links) include: red vs. white, walking with three, seeding vs. weeding, hidden hinges, reality distortion fields, and something-about-layers-though-that-one-obviously-needs-a-better-word.
    While it's still worthwhile to motivate/justify each individual new conceptual handle (and the planned essays will do so), I found myself imagining a general objection of the form "this is just making up terms for things," or perhaps "this is too many new terms, for too many new things." I realized that there was a chunk of argument, repeated across all of the planned essays, that I could factor out, and that (to the best of my knowledge) there was no single essay aimed directly at the question "why new words/phrases/conceptual handles at all?"
    So ... voilà.
    (Note that there is some excellent pushback + clarification + expansion to be found in the comments.)
    Core claims/tl;dr
    New conceptual distinctions naturally beget new terminology.Generally speaking, as soon as humans identify a new Thing, or realize that what they previously thought was a single Thing is actually two Things, they attempt to cache/codify this knowledge in language. Subclaim: this is a good thing; humanity is not, in fact, near the practical limits of its ability to incorporate and effectively wield new conceptual handles.
    New terminology naturally begets new conceptual distinctions.Alexis makes a new distinction, and stores it in language; Blake, via encountering Alexis's language, often becomes capable of making the same distinction, as a result. In particular, this process is often not instantaneous—it's not (always) as simple as just listening to a definition. Actual practice, often fumbling and stilted at first, leads to increased ability-to-perceive-and-distinguish; the verbal categories lay the groundwork for the perceptual/conceptual ones.
    These two dynamics can productively combine within a culture.Cameron, Dallas, and Elliot each go their separate ways and discover new conceptual distinctions not typical of their shared culture. Cameron, Dallas, and Elliot each return, and each teach the other two (a process generally much quicker and easier than the original discovery). Now Cameron, Dallas, and Elliot are each "three concepts ahead" in the game of seeing reality ever more finely and clearly, at a cost of something like only one-point-five concept-discovery's worth of work.(This is not a metaphor; this is in fact straightforwardly what has happened with the collection of lessons learned from famine, disaster, war, politics, and science, which have been turned into words and phrases and aphorisms that can be successfully communicated to a single human over the course of mere decades.)
    That which is not tracked in language will be lost.This is Orwell's thesis—that in order to preserve one's ability to make distinctions, one needs conceptual tools capable of capturing the difference between (e.g.) whispers, murmurs, mumbles, and mutters. Without such tools, it becomes more difficult for an individual, and much more difficult for a culture or subculture, to continue to attend to, care about, and take into account the distinction in question.
    The reification of new distinctions is one of the most productive frontiers of human rationality.It is not the only frontier, by a long shot. But both [the literal development of n...
    32 min
  • AF - Quick thoughts on "scalable oversight" / "super-human feedback" research by David Scott Krueger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quick thoughts on "scalable oversight" / "super-human feedback" research, published by David Scott Krueger on January 25, 2023 on The AI Alignment Forum.
    The current default view seems to roughly be:
    Inner alignment is more important than outer alignment (or, alternatively, this distinction is bad/sub-optimal, but basically it's all about generalizing correctly)
    Scalable oversight is the only useful form of outer alignment research remaining.
    We don't need to worry about sample efficiency in RLHP -- in the limit we just pay everyone to provide feedback, and in practice even a few thousand samples (or a "constition") seems ~good enough.
    But maybe it's not good? Because it's more like capabilities research?
    A common example used for motivating scalable oversight is the "AI CEO".
    My views are:
    We should not be aiming to build AI CEOs
    We should be aiming to robustly align AIs to perform "simpler" behaviors that unaided humans (or humans aided with more conventional tools, not, e.g. AI systems trained with RL to do highly interpretive work) feel they can competently judge.
    We should aim for a situation where there is broad agreement against building AIs with more ambitious alignment targets (e.g. AI CEOs).
    From this PoV, scalable oversight does in fact look mostly like capabilities research.
    However, scalable oversight research can still be justified because "If we don't, someone else will". But this type of replaceability argument should always be treated with extreme caution. The reality is more complex: 1) there will be tipping points where it suddenly ceases to apply, and your individual actions actually have a large impact on norms. 2) The details matter, and the tipping points are in different places for different types of research/applications, etc.
    It may also make sense to work on scalable oversight in order to increase robustness of AI performance on tasks humans feel they can competently judge ("robustness amplification"). For instance, we could use unaided human judgments and AI-assisted human judgments as safety filters, and not deploy a system unless both processes conclude it is safe.
    Getting AI systems to safely perform simpler behaviors safely remains an important research topic, and will likely require improving sample efficiency; the sum total of available human labor will be insufficient for robust alignment, and we probably need to use different architectures / hybrid systems of some form as well.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - Quick thoughts on "scalable oversight" / "super-human feedback" research by David Scott Krueger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quick thoughts on "scalable oversight" / "super-human feedback" research, published by David Scott Krueger on January 25, 2023 on The AI Alignment Forum.
    The current default view seems to roughly be:
    Inner alignment is more important than outer alignment (or, alternatively, this distinction is bad/sub-optimal, but basically it's all about generalizing correctly)
    Scalable oversight is the only useful form of outer alignment research remaining.
    We don't need to worry about sample efficiency in RLHP -- in the limit we just pay everyone to provide feedback, and in practice even a few thousand samples (or a "constition") seems ~good enough.
    But maybe it's not good? Because it's more like capabilities research?
    A common example used for motivating scalable oversight is the "AI CEO".
    My views are:
    We should not be aiming to build AI CEOs
    We should be aiming to robustly align AIs to perform "simpler" behaviors that unaided humans (or humans aided with more conventional tools, not, e.g. AI systems trained with RL to do highly interpretive work) feel they can competently judge.
    We should aim for a situation where there is broad agreement against building AIs with more ambitious alignment targets (e.g. AI CEOs).
    From this PoV, scalable oversight does in fact look mostly like capabilities research.
    However, scalable oversight research can still be justified because "If we don't, someone else will". But this type of replaceability argument should always be treated with extreme caution. The reality is more complex: 1) there will be tipping points where it suddenly ceases to apply, and your individual actions actually have a large impact on norms. 2) The details matter, and the tipping points are in different places for different types of research/applications, etc.
    It may also make sense to work on scalable oversight in order to increase robustness of AI performance on tasks humans feel they can competently judge ("robustness amplification"). For instance, we could use unaided human judgments and AI-assisted human judgments as safety filters, and not deploy a system unless both processes conclude it is safe.
    Getting AI systems to safely perform simpler behaviors safely remains an important research topic, and will likely require improving sample efficiency; the sum total of available human labor will be insufficient for robust alignment, and we probably need to use different architectures / hybrid systems of some form as well.
    EtA: the main issue I have with scalable oversight is less that it is advancing capabilities, per se, and more that it seems to raise a "chicken-and-egg" problem, i.e. the arguments for safety/alignment end up being somewhat circular: "this system is safe because the system we used as an assistant was safe" (but I don't think we've solved the "build a safe assistant" part yet, i.e. we don't have the base case for the induction).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Doing EA Better: grant-makers should consider grant app peer review along the public-sector model by ben.smith
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Doing EA Better: grant-makers should consider grant app peer review along the public-sector model, published by ben.smith on January 24, 2023 on The Effective Altruism Forum.
    Epistemic status: speculative
    This is a response to recent posts including Doing EA Better and The EA community does not own its donors' money. In order to make better funding decisions, some EAs have called for democratizing EA's funding systems. This can be problematic, because others have raised questions such as (paraphrasing heavily) such as: "how do we decide who gets a vote?" and "would funders still give if they were forced to follow community preferences"? The same EAs have argued that EA decision-making is “highly centralised, opaque, and unaccountable”, and said that to improve our impact on the world, the effective altruism movement should be more decentralized and there should be greater transparency amongst EA institutions.
    To meet both of the families of concerns expressed over the last week, I propose a grant-assessment system that improves transparency, decentralizes decision-making, and could better inform grant allocation by drawing information from a wider section of the community whilst maintaining funders' prerogatives to select the areas they wish to donate to. The proposal is to adopt a peer-review process used by the grant-making system run by public bodies in the United States, such as the National Institutes of Health and the National Science Foundation.
    In this model, the funder’s program manager makes decisions about grant awards based on reviews and numerical scores allocated by peer reviewers coordinating in expert panels to evaluate grant applications. This would be a positive-sum change that benefits both funders and the community: the community has more input into the grant-making process, and funders benefit from expertise in the community to better achieve their objectives.
    In the rest of this post, I will describe the National Institutes of Health grant evaluation process, describe why I think now is the right time for the effective altruism movement to consider peer review as part of a more mature grant evaluation process, give some notes on implementation in EA specifically, and describe how this approach can both maintain funders’ prerogative to spend their own money as they wish, while giving the community a greater level of decision-making.
    The grant peer review process at the NIH and NSF
    National Institutes of Health
    The National Institutes of Health (NIH) uses a peer review process to evaluate grant applications. This process involves the formation of ‘study sections’, which are groups of experts in the relevant field who review and evaluate grant applications.
    When an application is received, it is assigned to a study section based on its scientific area of focus. Each study section is composed of scientists, physicians, and other experts who have experience in the field related to the research proposed in the application. These would be drawn from the scientific community at large. Study section members are typically compensated for participation, but participation isn’t a full time job–it’s generally a small additional duty researchers can choose to take on, as part of their broader set of research activities.
    The study section members more-or-less independently review the applications and provide written critiques that are used to evaluate the strengths and weaknesses of each application. The study section then meets to discuss the applications, and each member provides a priority score and written summary of the application. These scores and summaries are used to determine which applications will be funded.
    In summary, the NIH uses study sections composed of experts in the relevant field to review and evaluate grant applications through a peer rev...
    15 min
  • LW - Pessimistic Shard Theory by Garrett Baker
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pessimistic Shard Theory, published by Garrett Baker on January 25, 2023 on LessWrong.
    It is often said that a partial alignment solution will bring about an S-risk, as your agent cares enough about humans to keep them around, but not enough about them to allow them to flourish. This is usually not worried about because the thought is that we are currently very far away from even a partial solution, and the insight we gain by developing a partial solution will be easily applied to take us into full solution territory.
    The short & simple reasoning for this is that conditional on humans being around, most things your agent could do to your humans are bad.
    Shard theory makes a weak claim of alignment by default, and the strong claim that shard theoretic agents will be easily partially aligned to their human overseers (human-like values will get a seat at the shard negotiating table). In this regime, we get partial alignment for free, and must work for the full alignment. Per the above paragraph, this is a worrying world to be in!
    In humans, partial altruism leading to devastating consequences is a regular event. Children refuse to kill their dying parents and instead send them to various torturous hospice centers, or else send them to slightly better (but still bad) nursing homes. They do this out of a sense of care to their parents, but not enough of a sense of care to prioritize those parents above other goals they have, such as work or relaxation.
    Out of a sense of fairness, justice, and empathy, many very smart people often advocate for net-harmful policies such as the establishment of Communism on one extreme, and rent ceilings on the other.
    Out of a sense of community, many otherwise caring & helpful people end up complicit in genocides & the establishment of boarders between countries, and the enforcement & continuation of harmful social norms.
    In humans we see many instances of such mis-placed caring, and assuming shard theory is essentially correct except for the part where it says human values will be transferred exactly into our agent, I am skeptical that our shard theoretic agent will correct these flaws. In each instance of the flaw, there is the choice for it to either say 'no, this is a misapplication of the principle I'm trying to follow here', and change the action, or 'actually, I prefer worlds with rent-ceilings, and so I guess I don't care so much about net-harm in these circumstances afterall', and change the principle.
    In practice, humans contend with these decisions all the time, so I don't think its just a matter of always sticking with the principle.
    The hard part lies in the establishment & verification of a reflection mechanism which would resolve these tradeoffs in ways I'd like, and is itself safe against deals among other shards and itself made to short-circuit it so it is not subject to lost-purposes cases like the above. The CEV problem (pdf warning).
    An argument I anticipate: I want to make a corrigibility-bot, it doesn't need to understand all of human values, it just needs to listen to me when I tell it to go do something in the world. The things I tell it to do are going to be piecemeal and small, so there is a fast feedback loop between its actions and me telling it to do stuff.
    My response: I agree this seems like a good thing to aim for. I anticipate someone saying small and slow actions do not make a pivotal act, and another saying this does not ultimately defend you against the agent catastrophically misunderstanding your feedback, or incorporating it in ways you did not intend. I myself think both are good criticisms if you succeed, but also that I do not see a way of making such an agent using current techniques.
    Give me a concrete training story for this, and perhaps we can have further discussion. Extending arguments in Diamo...
    5 min
  • EA - Open Philanthropy Shallow Investigation: Tobacco Control by Open Philanthropy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open Philanthropy Shallow Investigation: Tobacco Control, published by Open Philanthropy on January 25, 2023 on The Effective Altruism Forum.
    1. Preamble
    This document is a shallow investigation, as described here. As we noted in the civil conflict investigation and telecommunications in LMICs investigation we shared earlier this year, we have not shared many shallow investigations in the last few years but are moving towards sharing more of our early stage work.
    This investigation was written by Helen Kissel, a PhD candidate in economics at Stanford who worked at Open Philanthropy for 10 weeks in summer 2022 as one of five interns on the Global Health and Wellbeing Cause Prioritization team. We’ve also included the peer foreword, written by Strategy Fellow Chris Smith. The peer foreword, which is a standard part of our research process, is an initial response to a piece of research work, written by a team member who is not the primary author or their manager.
    A slightly earlier draft of this work has been read and discussed by the cause prioritization team. At this point, we plan to learn more about this topic by engaging with philanthropists who are already working on tobacco, extending the depth of this research (particularly on e-cigarettes), and digging deeper into countries which have seen big declines in their smoking burden (e.g. Brazil).
    2. Peer foreword
    Written by Chris Smith
    It was in 1964 that the US Surgeon General published a report which linked smoking cigarettes with lung cancer, building on research going back more than a decade. The report told readers that smokers had a 9-10x relative risk of developing lung cancer; that smoking was the primary cause of chronic bronchitis; that pregnant people who smoked were more likely to have underweight newborns, and that smoking was also linked to emphysema and heart disease.
    In this shallow, Helen reports that nearly sixty years later, there are ~1.3B tobacco users, and that smoking combustible tobacco remains an extraordinary contributor to the global burden of disease, responsible for some 8 million deaths (including secondhand smoke) and ~230M normative disability-adjusted life years (DALYs) (~173M OP descriptive DALYs) making it a bigger contributor to health damages in our terms than HIV/AIDS plus tuberculosis plus malaria. Moreover, the forward-looking projections are for only modest declines in the total burden as population increases offset a decline in smoking rates. As our framework puts it, we’ve got an important problem.
    Helen walks through the conventional orthodoxy on tobacco control at a population level (higher taxes, marketing restrictions, warning labels) and on smoking cessation support (nicotine replacement therapy, pharmaceutical support). She estimates that a campaign for a cigarette tax which increased the retail price of cigarettes in Indonesia by 10% (a large country with a high attributable disease burden) would reduce tobacco consumption (and attributable DALYs) by 5%, having an expected social return on investment (SROI) of ~3,300x, assuming a 3-year speedup, 10% success rate, and $3M campaign cost. Taxes are considered the single most effective policy measure, but going down the ladder to a moderate advertising ban, the subsequent expected 1% reduction in tobacco consumption and associated DALYs would have an SROI of ~500x. As with any of our shallow back-of-the-envelope-calculations (BOTECs), there is room to debate both the structure and the parameter choices.
    But I think that this is — when combined with the other material — a strong indicator that there could be relatively mainstream tobacco advocacy work which is above our bar in expectation. This suggests a somewhat tractable problem.
    Ok, but isn’t this addressed? Don’t people already know that cigarettes are bad for you? We...
    1 hr 10 min
  • AF - Alexander and Yudkowsky on AGI goals by Scott Alexander
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alexander and Yudkowsky on AGI goals, published by Scott Alexander on January 24, 2023 on The AI Alignment Forum.
    This is a lightly edited transcript of a chatroom conversation between Scott Alexander and Eliezer Yudkowsky last year, following up on the Late 2021 MIRI Conversations. Questions discussed include "How hard is it to get the right goals into AGI systems?" and "In what contexts do AI systems exhibit 'consequentialism'?".
    1. Analogies to human moral development
    [Yudkowsky][13:29]
    @ScottAlexander ready when you are
    [Alexander][13:31]
    Okay, how do you want to do this?
    [Yudkowsky][13:32]
    If you have an agenda of Things To Ask, you can follow it; otherwise I can start by posing a probing question or you can?
    We've been very much winging it on these and that has worked... as well as you have seen it working!
    [Alexander][13:34]
    Okay. I'll post from my agenda. I'm assuming we both have the right to edit logs before releasing them? I have one question where I ask about a specific party where your real answer might offend some people it's bad to offend - if that happens, maybe we just have that discussion and then decide if we want to include it later?
    [Yudkowsky][13:34]
    Yup, both parties have rights to edit before releasing.
    [Alexander][13:34]
    Okay.
    One story that psychologists tell goes something like this: a child does something socially proscribed (eg steal). Their parents punish them. They learn some combination of "don't steal" and "don't get caught stealing". A few people (eg sociopaths) learn only "don't get caught stealing", but most of the rest of us get at least some genuine aversion to stealing that eventually generalizes into a real sense of ethics. If a sociopath got absolute power, they would probably steal all the time. But there are at least a few people whose ethics would successfully restrain them.
    I interpret a major strain in your thought as being that we're going to train fledgling AIs to do things like not steal, and they're going to learn not to get caught stealing by anyone who can punish them. Then, once they're superintelligent and have absolute power, they'll reveal that it was all a lie, and steal whenever they want. Is this worry at the level of "we can't be sure they won't do this"? Or do you think it's overwhelmingly likely? If the latter, what makes you think AIs won't internalize ethical prohibitions, even though most children do? Is it that evolution has given us priors to interpret reward/punishment in a moralistic and internalized way, and entities without those priors will naturally interpret them in a superficial way? Do we understand what those priors "look like"? Is finding out what features of mind design and training data cause internalization vs. superficial compliance a potential avenue for AI alignment?
    [Yudkowsky][13:36]
    Several layers here! The basic gloss on this is "Yes, everything that you've named goes wrong simultaneously plus several other things. If I'm wrong and one or even three of those things go exactly like they do in neurotypical human children instead, this will not be enough to save us."
    If AI is built on anything like the present paradigm, or on future paradigms either really, you can't map that onto the complicated particular mechanisms that get invoked by raising a human child, and expect the same result.
    [Alexander][13:37]
    (give me some sign when you're done answering)
    [Yudkowsky][13:37]
    (it may be a while but you should probably also just interrupt)
    especially if I say something that already sounds wrong
    [Alexander: ]
    the old analogy I gave was that some organisms will develop thicker fur coats if you expose them to cold weather. this doesn't mean the organism is simple and the complicated information about fur coats was mostly in the environment, and that you could expose an organism from a differe...
    41 min
  • AF - Alexander and Yudkowsky on AGI goals by Scott Alexander
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alexander and Yudkowsky on AGI goals, published by Scott Alexander on January 24, 2023 on The AI Alignment Forum.
    This is a lightly edited transcript of a chatroom conversation between Scott Alexander and Eliezer Yudkowsky last year, following up on the Late 2021 MIRI Conversations. Questions discussed include "How hard is it to get the right goals into AGI systems?" and "In what contexts do AI systems exhibit 'consequentialism'?".
    1. Analogies to human moral development
    [Yudkowsky][13:29]
    @ScottAlexander ready when you are
    [Alexander][13:31]
    Okay, how do you want to do this?
    [Yudkowsky][13:32]
    If you have an agenda of Things To Ask, you can follow it; otherwise I can start by posing a probing question or you can?
    We've been very much winging it on these and that has worked... as well as you have seen it working!
    [Alexander][13:34]
    Okay. I'll post from my agenda. I'm assuming we both have the right to edit logs before releasing them? I have one question where I ask about a specific party where your real answer might offend some people it's bad to offend - if that happens, maybe we just have that discussion and then decide if we want to include it later?
    [Yudkowsky][13:34]
    Yup, both parties have rights to edit before releasing.
    [Alexander][13:34]
    Okay.
    One story that psychologists tell goes something like this: a child does something socially proscribed (eg steal). Their parents punish them. They learn some combination of "don't steal" and "don't get caught stealing". A few people (eg sociopaths) learn only "don't get caught stealing", but most of the rest of us get at least some genuine aversion to stealing that eventually generalizes into a real sense of ethics. If a sociopath got absolute power, they would probably steal all the time. But there are at least a few people whose ethics would successfully restrain them.
    I interpret a major strain in your thought as being that we're going to train fledgling AIs to do things like not steal, and they're going to learn not to get caught stealing by anyone who can punish them. Then, once they're superintelligent and have absolute power, they'll reveal that it was all a lie, and steal whenever they want. Is this worry at the level of "we can't be sure they won't do this"? Or do you think it's overwhelmingly likely? If the latter, what makes you think AIs won't internalize ethical prohibitions, even though most children do? Is it that evolution has given us priors to interpret reward/punishment in a moralistic and internalized way, and entities without those priors will naturally interpret them in a superficial way? Do we understand what those priors "look like"? Is finding out what features of mind design and training data cause internalization vs. superficial compliance a potential avenue for AI alignment?
    [Yudkowsky][13:36]
    Several layers here! The basic gloss on this is "Yes, everything that you've named goes wrong simultaneously plus several other things. If I'm wrong and one or even three of those things go exactly like they do in neurotypical human children instead, this will not be enough to save us."
    If AI is built on anything like the present paradigm, or on future paradigms either really, you can't map that onto the complicated particular mechanisms that get invoked by raising a human child, and expect the same result.
    [Alexander][13:37]
    (give me some sign when you're done answering)
    [Yudkowsky][13:37]
    (it may be a while but you should probably also just interrupt)
    especially if I say something that already sounds wrong
    [Alexander: ]
    the old analogy I gave was that some organisms will develop thicker fur coats if you expose them to cold weather. this doesn't mean the organism is simple and the complicated information about fur coats was mostly in the environment, and that you could expose an organism from a differe...
    41 min