The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - A Case for Voluntary Abortion Reduction by Ariel Simnegar
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Case for Voluntary Abortion Reduction, published by Ariel Simnegar on December 20, 2022 on The Effective Altruism Forum.
    Trigger warning: Abortion is a delicate topic, especially for those of us who've had abortions or otherwise feel strongly on this issue. I've tried to make the following case with care and sensitivity, and if it makes anyone feel uncomfortable, I wholeheartedly apologize.
    Disclaimer: This essay specifically concerns voluntary abortion reduction. Any discussion of involuntary intervention is outside of this post's scope.
    Thanks to Ives Parrhesia, Marcus Abramovitch, Ruth Grace Wong, and several anonymous helpers. Their help does not constitute an endorsement of this essay's conclusions.
    Summary
    Many EA principles point us towards supporting voluntary abortion reduction:
    Moral circle expansion.
    We're receptive to arguments that we should expand our moral circle to include animals and future people.
    We should be open to the possibility that fetuses—the future people closest to us—could be included in our moral circle too.
    Our concern for neglected and disenfranchised moral groups.
    If fetuses are moral patients, then they are relatively neglected and disenfranchised, with more abortions occuring each year than deaths by all causes combined.
    The metric of (adjusted) life years.
    We commonly use (adjusted) life years as a measure of the disvalue of problems and the value of interventions.
    This metric arguably doesn't distinguish between fetal deaths and infant deaths.
    Singerian duties to give to help those in need.
    We're typically sympathetic to arguments that we should proactively help those in need, even if it reduces our personal autonomy.
    We should consider whether we should help our children the same way.
    Longtermist philosophical views.
    Longtermists are typically receptive to total / low critical level views, non-person-affecting views, and pro-natalism.
    Just as these views seem to imply that we should care for people in the far future, they also seem to imply that we should care for fetuses, the future people closest to us.
    Moral uncertainty's implications for a potential problem of massive scale.
    Given abortion's massive scale, even a small chance that fetuses are moral patients could imply that we should do something about it.
    In that regard, we should carry out the following interventions:
    Shift our family-focused interventions to spotlight mothers' physical and mental health, and support adoption as an option.
    Suspend our support for charities which reduce the amount of near-term future people until we can systematically review the effect of the above moral considerations on the morality of the charities' interventions.
    In our personal lives, we should:
    Understand the situations of people we know who are considering abortion and do whatever we can to support them in having their babies the way they would like.
    Help each other to be loving parents and raise thriving children, whether or not some of us have abortions or choose to not have children.
    Introduction: Moral Circle Expansion
    Future people count, but we rarely count on them. They cannot vote or lobby or run for public office, so politicians have scant incentive to think about them. They can't bargain or trade with us, so they have little representation in the market. And they can't make their views heard directly: they can't tweet, or write articles in newspapers, or march in the streets. They are utterly disenfranchised.
    The idea that future people count is common sense. Future people, after all, are people. They will exist. They will have hopes and joys and pains and regrets, just like the rest of us. They just don't exist yet.
    Will MacAskill, What We Owe The Future (2022), pp. 9-10.
    As EAs, we're no strangers to expanding our moral circle. We’re rooted in the idea that distance shou...
    27 min
  • AF - An Open Agency Architecture for Safe Transformative AI by davidad (David A. Dalrymple)
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Open Agency Architecture for Safe Transformative AI, published by davidad (David A. Dalrymple) on December 20, 2022 on The AI Alignment Forum.
    Note: my quality bar here is "this is probably worth the reader's time" rather than "this is as clear, compelling, and comprehensive as I can make it."
    In the Neorealist Success Model, I asked:
    What would be the best strategy for building an AI system that helps us ethically end the acute risk period without creating its own catastrophic risks that would be worse than the status quo?
    This post is a first pass at communicating my current answer.
    Bird's-eye view
    At the top level, it centres on a separation between
    learning a world-model from data and eliciting desirabilities within that ontology
    planning against a world-model and associated desirabilities
    acting in real-time
    We see such a separation in, for example, MuZero, which can probably still beat GPT-4 at Go—the most effective capabilities do not always emerge from a fully black-box, end-to-end, generic pre-trained policy.
    Hypotheses
    Scientific Sufficiency Hypothesis: It's feasible to train a purely descriptive/predictive infra-Bayesian world-model that specifies enough critical dynamics accurately enough to end the acute risk period, such that this world-model is also fully understood by a collection of humans (in the sense of "understood" that existing human science is).
    MuZero does not train its world-model for any form of interpretability, so this hypothesis is more speculative.
    However, I find Scientific Sufficiency much more plausible than the tractability of eliciting latent knowledge from an end-to-end policy.
    It's worth noting there is quite a bit of overlap in relevant research directions, e.g.
    pinpointing gaps between the current human-intelligible ontology and the machine-learned ontology, and
    investigating natural abstractions theoretically and empirically.
    Deontic Sufficiency Hypothesis: There exists a human-understandable set of features of finite trajectories in such a world-model, taking values in (−∞,0], such that we can be reasonably confident that all these features being near 0 implies high probability of existential safety, and such that saturating them at 0 is feasible with high probability, using scientifically-accessible technologies.
    I am optimistic about this largely because of recent progress toward formalizing a natural abstraction of boundaries by Critch and Garrabrant. I find it quite plausible that there is some natural abstraction property Q of world-model trajectories that lies somewhere strictly within the vast moral gulf of
    Model-Checking Feasibility Hypothesis: It could become feasible to train RL policies such that a formally verified, infra-Bayesian, symbolic model-checking algorithm can establish high-confidence bounds on its performance relative to the world-model and safety desiderata, by using highly capable AI heuristics that can only affect the checker's computational cost and not its correctness—soon enough that switching to this strategy would be a strong Pareto improvement for an implementation-adequate coalition.
    Time-Bounded Optimization Thesis: RL settings can be time-bounded such that high-performing agents avoid lock-in.
    I'm pretty confident of this.
    The founding coalition might set the time bound for the top-level policy to some number of decades, balancing the potential harms of certain kinds of lock-in for that period against the timelines for solving a more ambitious form of AI alignment.
    If those hypotheses are true, I think this is a plan that would work. I also think they are all quite plausible (especially relative to the assumptions that underly other long-term AI safety hopes)—and that if any one of them fails, they would fail in a way that is detectable before deployment, making an attempt to execute the p...
    10 min
  • EA - SoGive's 2023 plans + funding request by Sanjay
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SoGive's 2023 plans + funding request, published by Sanjay on December 20, 2022 on The Effective Altruism Forum.
    We at SoGive are excited about what we plan to achieve in 2023. We still have material funding gaps, so if readers are keen for these things to happen, then please get in touch by emailing [email protected].
    Our work will involve the following:
    ActivityRough indication of how much effort goes on each40%-45%40%-45%10%-20%
    EA-aligned research
    Supporting the EA talent pipeline
    Supporting major donors, and running SoGive grants
    1. EA-aligned research: We plan to do work akin to the research of organisations like Rethink Priorities. There are enough thorny questions about how to do good / give effectively that an enormous amount more research is needed, and the existing ecosystem would benefit from growing in both capacity and the number of voices, hence we are keen for SoGive to contribute.
    Work which we plan to complete and publish in 2023:
    Strong Minds:
    SoGive’s founder identified Strong Minds as a possible high impact charity in 2015 when thinking about his own giving, and had an initial call with Sean Mayberry of Strong Minds then. We have been engaging with Strong Minds over some years and we plan to finalise a report early in 2023.
    No Means No:
    An investigation of No Means No Worldwide was suggested to us by a member of the EA London community, who was excited to have an EA-aligned recommendation for a charity which prevents sexual violence. We have mostly completed a review of this charity, and were asked not to publish it yet because it used a study which is not yet in the public domain.
    Pratham
    We have been working on a review of Pratham, which works on education in the developing world.
    Note that our model involves having multiple pieces of analysis ongoing, and these may be progressed by someone working for 1 day per week.
    Work which we plan to initiate in 2023:
    Redteaming of John Halstead’s report on climate change
    Counterfactuals vs credit-sharing/Shapley values
    Note: this is not a comprehensive list, just a few examples. For a fuller list, see this public copy of our research agenda.
    Past track record: People within the SoGive community have successfully written a number of valuable pieces of research.
    Cool Earth: In order to support SoGive’s work helping major donors, SoGive produced a review of Cool Earth which argued that the EA community had overvalued Cool Earth. It received an EA Forum prize, was nominated for a Review of Decade prize, and has been cited 14 times on the EA Forum and an unknown number of times elsewhere.
    Giving Green: A SoGive volunteer, with support from SoGive’s founder, wrote a piece critiquing the analytical approach of Giving Green. Prior to this write-up, there had been little critical analysis of the quality of Giving Green’s work.
    Malaria nets: We wrote a piece on GiveWell’s modelling of insecticide resistance, which argued that there were issues with the way that synergists were modelled (synergists are things which help nets overcome insecticide resistance). We also argued that GiveWell were missing an important source of data. GiveWell awarded this analysis an honourable mention prize.
    If you would like to know more about SoGive’s research work, feel free to read our strategy document: section 1 covers our research work, section 1.3 explains how we set our research agenda, section 1.5 provides more examples of our track record of research, section 1.7 includes people from other EA orgs who are willing to serve as references.
    2. Supporting the EA talent pipeline: Our volunteer programme was originally set up to help us analyse charities, however we are now leveraging our experience to serve orgs in the EA community who are looking to hire. In 2023, we aim to recruit c50-100 new volunteers, and of those probably a m...
    7 min
  • EA - High-level hopes for AI alignment by Holden Karnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: High-level hopes for AI alignment, published by Holden Karnofsky on December 20, 2022 on The Effective Altruism Forum.
    See here for an audio version.
    In previous pieces, I argued that there's a real and large risk of AI systems' aiming to defeat all of humanity combined - and succeeding.
    I first argued that this sort of catastrophe would be likely without specific countermeasures to prevent it. I then argued that countermeasures could be challenging, due to some key difficulties of AI safety research.
    But while I think misalignment risk is serious and presents major challenges, I don’t agree with sentiments1 along the lines of “We haven’t figured out how to align an AI, so if transformative AI comes soon, we’re doomed.” Here I’m going to talk about some of my high-level hopes for how we might end up avoiding this risk.
    I’ll first recap the challenge, using Ajeya Cotra’s young businessperson analogy to give a sense of some of the core difficulties. In a nutshell, once AI systems get capable enough, it could be hard to test whether they’re safe, because they might be able to deceive and manipulate us into getting the wrong read. Thus, trying to determine whether they’re safe might be something like “being an eight-year-old trying to decide between adult job candidates (some of whom are manipulative).”
    I’ll then go through what I see as three key possibilities for navigating this situation:
    Digital neuroscience: perhaps we’ll be able to read (and/or even rewrite) the “digital brains” of AI systems, so that we can know (and change) what they’re “aiming” to do directly - rather than having to infer it from their behavior. (Perhaps the eight-year-old is a mind-reader, or even a young Professor X.)
    Limited AI: perhaps we can make AI systems safe by making them limited in various ways - e.g., by leaving certain kinds of information out of their training, designing them to be “myopic” (focused on short-run as opposed to long-run goals), or something along those lines. Maybe we can make “limited AI” that is nonetheless able to carry out particular helpful tasks - such as doing lots more research on how to achieve safety without the limitations. (Perhaps the eight-year-old can limit the authority or knowledge of their hire, and still get the company run successfully.)
    AI checks and balances: perhaps we’ll be able to employ some AI systems to critique, supervise, and even rewrite others. Even if no single AI system would be safe on its own, the right “checks and balances” setup could ensure that human interests win out. (Perhaps the eight-year-old is able to get the job candidates to evaluate and critique each other, such that all the eight-year-old needs to do is verify basic factual claims to know who the best candidate is.)
    These are some of the main categories of hopes that are pretty easy to picture today. Further work on AI safety research might result in further ideas (and the above are not exhaustive - see my more detailed piece, posted to the Alignment Forum rather than Cold Takes, for more).
    I’ll talk about both challenges and reasons for hope here. I think that for the most part, these hopes look much better if AI projects are moving cautiously rather than racing furiously.
    I don’t think we’re at the point of having much sense of how the hopes and challenges net out; the best I can do at this point is to say: “I don’t currently have much sympathy for someone who’s highly confident that AI takeover would or would not happen (that is, for anyone who thinks the odds of AI takeover . are under 10% or over 90%).”
    The challenge
    This is all recapping previous pieces. If you remember them super well, skip to the next section.
    In previous pieces, I argued that:
    The coming decades could see the development of AI systems that could automate - and dramatically speed up - scientif...
    30 min
  • AF - Take 12: RLHF's use is evidence that orgs will jam RL at real-world problems. by Charlie Steiner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 12: RLHF's use is evidence that orgs will jam RL at real-world problems., published by Charlie Steiner on December 20, 2022 on The AI Alignment Forum.
    As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day some days for 25 days. I have now procrastinated enough that I probably have enough hot takes.
    I felt like writing this take a little more basic, so that it doesn't sound totally insane if read by an average ML researcher.
    Use of RLHF by OpenAI is a good sign in that it shows how alignment research can get adopted by developers of cutting-edge AI. I think it's even a good sign overall, probably. But still, use of RLHF by OpenAI is a bad sign in that it shows that jamming RL at real-world problems is endorsed as a way to make impressive products.
    If you wandered in off the street, you might be confused why I'm saying RL is bad. Isn't it a really useful learning method? Hasn't it led to lots of cool stuff?
    But if you haven't wandered in off the street, you know I'm talking about alignment problems - loosely, we want powerful AIs to do good things and not bad things, even when tackling the whole problem of navigating the real world. And RL has an unacceptable chance of getting you AIs that want to do bad things.
    There's an obvious problem with RL for navigating the real world, and a more speculative generalization of that problem.
    The obvious problem is wireheading. If you're a clever AI learning from reward signals in the real world, you might start to notice that actions that affect a particular computer in the world have their reward computed differently than actions that affect rocks or trees. And it turns out that by making a certain number on this computer big, you can get really high reward! At this point the AI starts searching for ways to stop you from interrupting its "heroin fix," and we've officially hecked up and made something that's adversarial to us.
    Now, maybe you can do RL without this happening. Maybe if you do model-based reasoning, and become self-reflective at the right time to lock in early values, you'll perceive actions that manipulate this special computer to be cheating according to your model, and avoid them. I'll explain in a later take some extra work I think this requires, but for the moment it's more important to note that a lot of RL tricks are actually working directly against this kind of reasoning. When a hard environment has a lot of dead ends, and sparse gradients (e.g. Montezuma's Revenge, or the real world), you want to do things like generate intrinsic motivations to aid exploration, or use tree search over a model of the world, which will help the AI break out of local traps and find solutions that are globally better according to the reward function.
    Maxima of the reward function have nonlocal echoes, like mountains have slopes and foothills. These echoes are the whole reason that looking at the local gradient is informative about which direction is better long-term, and why building a world model can help you predict never-before-seen rewarding states. Deep models and fancy optimizers are useful precisely because their sensitivity to those echoes helps them find good solutions to problems, and there's no difference in kind between the echoes of the solutions we want our AI to find, and the echoes of the solutions we didn't intend.
    The speculative generalization of the problem is that there's a real risk of an AI sensing these echoes even if it's not explicitly intended to act in the real world, so long as its actions are affecting its reward-evaluation process, and it benefits from building a model of the real world. Suppose you have a language model that you're trying to train with RL, and your reward signal is the rating of a human who happens to be really eas...
    6 min
  • LW - AGI Timelines in Governance: Different Strategies for Different Timeframes by simeon c
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI Timelines in Governance: Different Strategies for Different Timeframes, published by simeon c on December 19, 2022 on LessWrong.
    Summarization Table
    TimelinesPre-2030Post-2030ExpectationsAGI will be built by an organization that’s already trying to build it (85%) Compute will still be centralized at the time AGI is developed (60%)More companies will be in the race (90%)National government policy won’t have strong effects (70%) The best strategies will have more variance (75%)There will be more compute suppliers (90%)Comparatively More Promising Strategies (under timelines X)Aim to promote a security mindset in the companies currently developing AI (85%)Focus on general community building (90%)Focus on corporate governance (75%)Build the AI safety community in China (80%)Target outreach to highly motivated young people and senior researchers (80%)Avoid publicizing AGI risk (60%)Coordinate with national governments (65%)Beware of large-scale coordination efforts (80%)
    Some governments will be in the race (80%)
    China is more likely to lead than pre-2030 (85%)
    Introduction
    Miles Brundage recently argued that AGI timeline discourse might be overrated. He makes a lot of good points, but I disagree with one thing. Miles says: “I think the correct actions are mostly insensitive to timeline variations.”
    Unlike Miles, I think that if the timeline differences are greater than a couple of years, the choice of actions does depend on timeline differences. In particular, our approach to governance should be very different depending on whether we think that AGI will be developed in ~5-10 years or after that. In this post, I list some of the likely differences between a world in which AGI is developed before, and after ~2030 and discuss how those differences should affect how we approach AGI governance. I discuss most of the strategies and considerations in relative terms, i.e. arguing why they’re likely to be significantly more crucial under certain timelines than others. I am discussing these specific strategies and considerations because I believe they are important for AI governance, or at least likely to be effective strategies under one of the two timelines I am considering.
    I chose 2030 as a cut-off point because it is easy to remember and it seems to make sense to differentiate between the actions that should be prioritized in the time leading up to 2030 (~5-10 years timelines) and those that should be prioritized after 2030 (15-20 years and beyond timelines). But perhaps a better way to read this post is ‘the sooner you think AGI will be developed, the more likely my points about the pre-2030 AGI world are to be true’, and vice versa.
    Epistemic status and reasoning behind publishing this:
    It seems to me that not many people have given detailed thought to this issue, which is why I wrote this post. To me, this is a major consideration. I included probability estimates to help you assess my level of certainty for each claim, even though not all claims may be easy to verify.
    If there were no trade-offs to consider, we could implement almost all interventions at the same time. However, in practice, talent is limited and I expect that there will be trade-offs in terms of how resources are allocated among different timelines. I expect to update a significant portion of the claims I have made based on feedback and new information that becomes available in the coming years. This is because there may be considerations that I have missed and AI governance is complex and uncertain. Therefore, my probability estimates have low resilience.
    Thanks to Nicole Nohemi, Felicity Reddel, Andrea Miotti, Fabien Roger and Gerard van Smeden for the feedback on this post.
    If AGI is developed before 2030, the following is more likely to be true:
    AGI will be built by an organization that’s already...
    19 min
  • EA - Process for Returning FTX Funds Announced by Molly
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Process for Returning FTX Funds Announced, published by Molly on December 20, 2022 on The Effective Altruism Forum.
    FTX has put out a press release announcing a “process for voluntary return of avoidable payments.” This may be a useful option for grantees looking to urgently return any FTX-associated funding rather than wait for the bankruptcy process to play out. But anyone interested in returning money should keep in mind that in order to avoid being subject to redundant clawbacks or other legal claims later, it’s crucial to receive proper release-of-claims paperwork in exchange for returning funding. I strongly recommend you consult with a bankruptcy lawyer before starting this process. The Open Phil legal team is putting together a list of legal service providers for grantees who want to explore this option; we’ll follow up after the holidays with more information.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - The "Minimal Latents" Approach to Natural Abstractions by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The "Minimal Latents" Approach to Natural Abstractions, published by johnswentworth on December 20, 2022 on The AI Alignment Forum.
    Background: The Language-Learning Argument
    How many functions are there from a 1 megabyte image to a yes/no answer to the question “does this image contain an apple?”.
    Well, there are 8M bits in the image, so 28000000 possible images. The function can assign “yes” or “no” independently to each of those 28000000 images, so 228000000 possible functions. Specifying one such function by brute force (i.e. not leveraging any strong prior information) would therefore require 28000000 bits of information - one bit specifying the output on each of the 28000000 possible images. Even if we allow for some wiggle room in ambiguous images, those ambiguous images will still be a very tiny proportion of image-space, so the number of bits required would still be exponentially huge.
    Empirically, human toddlers are able to recognize apples by sight after seeing maybe one to three examples. (Source: people with kids.)
    Point is: nearly-all the informational work done in a toddler’s mind of figuring out which pattern is referred to by the word “apple” must be performed by priors and general observations of the world, not by examples of apples specifically. Before the toddler ever hears the word, they must already have some internal data structure which represents the-things-we-call-apples as a natural category (though possibly implicitly rather than explicitly), just waiting to have a word attached to it. In ML terms, nearly-all the informational work of learning what “apple” means must be performed by unsupervised learning, not supervised learning. Otherwise the number of examples required would be far too large to match toddlers’ actual performance.
    Furthermore, humans are empirically able to communicate; one human can say words, the other can guess what those words refer to, and then the two humans will both agree on what physical thing was referred to by the words. Admittedly this outcome is less robust than we might like (i.e. miscommunication still happens a lot), but it happens ridiculously more often than chance, to the point where exceptions are more notable than non-exceptions.
    So, not only must unsupervised learning perform nearly-all the informational work of learning what words mean, but that unsupervised learning method must also somehow coordinate on (at least roughly) the same potential-word-meanings across different humans.
    We can even go a step further: other kinds of minds, like dogs or neural nets, are also able to coordinate on (at least roughly) the same potential-word-meanings as humans. For now they usually require a lot more examples than human children, but nowhere near the exponentially huge number required to learn meanings by full brute force. Most of the informational work must still be done by some unsupervised process.
    What We’ll Do In This Post
    The language-learning argument adds up to a weak version of the Natural Abstraction Hypothesis: a wide variety of minds converge to using the same “natural abstractions”, i.e. the potential-word-meanings. Those abstractions are “natural” in the sense that they’re a fairly general property of minds (i.e. priors) plus the environment; a wide variety of minds are able to learn them in an unsupervised way just from the environment plus whatever priors are common to all these different kinds of minds.
    My other writing on natural abstraction usually starts from thinking of natural abstractions as summaries of information relevant at a distance, or of information which is highly redundant - both properties which are mostly informational features of the environment, and mostly avoid explicitly talking about minds. They’re “bottom-up” notions of natural abstraction: they start from a low-level e...
    21 min
  • LW - Shard Theory in Nine Theses: a Distillation and Critical Appraisal by LawrenceC
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shard Theory in Nine Theses: a Distillation and Critical Appraisal, published by LawrenceC on December 19, 2022 on LessWrong.
    TL;DR: Shard theory is a new research program started by Quintin Pope and Alex Turner. Existing introductions tend to be relatively long winded and aimed at an introductory audience. Here, I outline what I think are the nine main theses of shard theory, so as to give a more concrete introduction and critique.
    Acknowledgements: Thanks to Thomas Kwa for the conversation that inspired this post. Thanks also to Alex Turner for several conversations about shard theory and Charles Foster and Teun Van Der Weij for substantial feedback on this writeup.
    Epistemic status: As I’m not a member of Team Shard, I’m probably misrepresenting Team Shard’s beliefs in a few places here.
    Introduction
    Shard theory is a research program that aims to build a mechanistic model between training signals and learned values in agents. Drawing large amounts of inspiration from particular hypotheses about the human reward learning system, shard theory posits that the values of agents are best understood as sets of contextually activated heuristics shaped by the reward function.
    In this post, I’ll attempt to outline nine of the main theses of shard theory, as of late 2022. I’ll explain the novelty of each thesis, how it constrains expectations, and then give my opinions and suggest some few experiments that could be done to test the theses.
    Existing explainers of shard theory tend to be aimed at a relatively introductory audience and thus are relatively verbose. By default, I’ll be addressing this post to someone with a decent amount of AI/ML research background, so I’ll often explain things with reference to AI/ML terminology or with examples from deep learning. This post is aimed primarily at explaining what I see as the core claims, as opposed to justifying them; any missing justifications should not be attributed to failures of Team Shard. It’s also worth noting that shard theory is an ongoing research program and not a battle-tested scientific theory, so many of these claims are likely to be revised or clarified over time.
    Related work
    Alex Turner’s Reward is not the optimization target is probably the first real “shard theory” post. The post argues that, by default, the learned behavior of an RL agent is not well understood as maximizing reward. Instead, the post argues that we should try to study how reward signals lead to value formation in more detail. He uses a similar argument to argue against the traditional inner and outer alignment split in Inner and outer alignment decompose one hard problem into two extremely hard problems.
    David Udell’s Shard Theory: An Overview first introduces the shard theory research program, as well as the terminology and core claims of shard theory. Turner and Pope’s The shard theory of human values applies the shard theory to human value formation. It outlines three assumptions that the shard theory of human values makes regarding humans (the cortex is randomly initialized, the brain does self-supervised learning, and the brain does reinforcement learning). Other posts flesh out parts of the shard theory of human values (e.g. “Human value and biases are inaccessible to the genome”) and justify the use of human values as a case study for alignment. (“Humans provide an untapped wealth of evidence about alignment”, “Evolution is a bad analogy for AGI”.) Geoffrey Miller’s The heritability of human values: A behavior genetic critique of Shard Theory argues that the high heritability of many kinds of human values contradicts the core claims of shard theory.
    Thomas Kwa’s Failure modes in a shard theory alignment plan gives definitions for many of the key terms of shard theory and outlines a possible shard theory alignment plan, before raisin...
    26 min
  • EA - AGI Timelines in Governance: Different Strategies for Different Timeframes by simeon c
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI Timelines in Governance: Different Strategies for Different Timeframes, published by simeon c on December 19, 2022 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min