The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Announcing aisafety.training by JJ Hepburn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing aisafety.training, published by JJ Hepburn on January 21, 2023 on LessWrong.
    Crossposted to EA ForumTo help people find what to apply to, aisafety.training acts as a well-maintained living document of AI safety programs, conferences, and other events. This will smooth the experience of people working on and joining AI safety and reduce the burden on word-of-mouth transmission of available programs. This can also be helpful for field builders planning events to see when other things are happening to plan around. We at AI Safety Support have been internally maintaining this document for some time and using it in our free career coaching calls. We now have a public-facing version, a form to add anything we’ve missed, and an email to alert us to corrections.
    For example, below, you will find static / will-soon-be-outdated images without clicking on the website link.
    Application Deadlines
    Program Timeline
    Program table
    If you’re interested in helping to maintain this and can consistently dedicate a few hours per week to reading places where things might have been announced, accepting additions from the form and corrections by email, or if you have feedback or feature requests, leave us a comment or drop by the Alignment Ecosystem Development Discord, where this is worked on along with related projects to improve the information ecosystem such as aisafety.community and the upcoming aisafety.world and ea.domains.
    Additionally, if you wish to have a monthly reminder of upcoming events sent directly to your inbox, subscribe to AI Safety Support’s newsletter here.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Animal Advocacy Digest #4: Dec 24th- Jan 20th by James Ozden
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Animal Advocacy Digest #4: Dec 24th- Jan 20th, published by James Ozden on January 20, 2023 on The Effective Altruism Forum.
    Note: This digest will be once per month now, as EA Forum animal content isn’t as frequent as we expected!
    Welcome to this edition of the Animal Advocacy Digest, where we aim to help people in the animal welfare community quickly keep up with the most important information in the space. As a reminder, you can get this digest as an email by signing up here. =
    I’m summarising the following posts:
    Video: A Few Exciting Giving Opportunities for Animal Welfare in Asia - Jack Stennett, Jah Ying Chung (Good Growth)
    Why Anima International suspended the campaign to end live fish sales in Poland - Jakub Stencel, Weronika Zurek (Anima International)
    Don’t Balk at Animal-friendly Results - Bob Fischer (Rethink Priorities)
    Longtermism and Animal Farming Trajectories - Michael Dello (Sentience Institute)
    Abolitionist in the Streets, Pragmatist in the Sheets: New Ideas for Effective Animal Advocacy - Dhruv Makwana
    Quick links:
    Linkpost: Big Wins for Farm Animals This Decade (Lewis Bollard)
    Video: A Few Exciting Giving Opportunities for Animal Welfare in Asia - Jack Stennett, Jah Ying Chung (Good Growth)
    Good Growth is an Effective Altruist-aligned organisation focusing on animal welfare and food systems in Asia. They focus on growing the animal advocacy ecosystem in Asia, by conducting research prioritisation on key topics within Asian animal advocacy, developing accessible databases on the alternative protein market in Asian countries, as well as building the research capacity of Asian animal advocacy organisations.
    In this video, Good Growth highlights five organisations (including themselves) who might be exciting donation opportunities for those interested in supporting Asian animal advocacy. They are:
    Green Camel Bell, who are working to end the farming of wild animals, such as Giant Salamanders, in China
    Sinergia Animals, who are currently working on cage-free eggs advocacy, institutional outreach for plant-based meals, and investigations in Thailand
    Animal Alliance Asia, who are working on a regranting programme for international funders to reach local advocates in Asia
    Animal Empathy Philippines who are working on capacity-building initiatives to grow the farm animal welfare movement in the Philippines.
    Why might you donate to Asian animal advocacy opportunities? Good Growth has made this handy graphic below which might provide some insight!
    Why Anima International suspended the campaign to end live fish sales in Poland - Jakub Stencel, Weronika Zurek (Anima International)
    Anima International recently took the (brave!) decision recently to suspend their campaign which focuses on ending the live sale of carp in Poland. This campaign was specifically against the purchasing of live carp around Christmas time, a tradition in some post-communist countries in East Europe. Animal advocacy groups had achieved significant wins with this campaign, leading to major retailers withdrawing from the sale of live fish.
    However, Anima International became worried about the consumer trend to replace carp with other fish, such as salmon. As salmon is a carnivorous fish (and carp mostly isn’t), Anima International was increasingly concerned this campaign could actually be leading to more animal deaths, as salmon farming often requires fish feed. Anima calculates that buying one Atlantic Salmon requires around 11 times more fish deaths than buying the same weight in common carp, due to considerations around weight, mortality rates and number of feed fish required to bring these animals to sale weight. As a result, Anima International took the difficult decision to suspend this long standing campaign, in light of potential negative impacts on fish.
    Longtermism and Animal ...
    9 min
  • LW - What’s going on with ‘crunch time’? by rosehadshar
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What’s going on with ‘crunch time’?, published by rosehadshar on January 20, 2023 on LessWrong.
    Also on EAForum here.
    Imagine the set of decisions which impact TAI outcomes.
    Some of the decisions are more important than others: they have a larger impact on TAI outcomes. We care more about a decision the more important it is.
    Some of the decisions are also easier to influence than others, and we care more about a decision the easier it is to influence.
    Decisions in the past are un-influenceable
    Probably some decisions in the future are too, but it may be hard to identify which ones
    How easy it is to influence a decision depends on who you are
    People have access to different sets of people and resources
    Influence can be more or less in/direct (e.g. how many steps removed are you from the US President)
    The influenceable decisions are distributed over time, but we don’t know the distribution:
    Decisions might happen earlier or later (timing/timelines)
    There might be more or fewer decisions in the set
    Decisions might be more or less concentrated over time (urgency/duration)
    There might be discontinuities (only one decision matters, at the right of the distribution suddenly there are no more decisions because takeover has happened.)
    It’s possible that in the future there might be a particularly concentrated period of important decisions, for example like this:
    People refer to this concentrated period of important decisions as ‘crunch time’. The distribution might or might not end up actually containing a concentrated period of important decisions - or in other words, crunch time may or may not happen.
    Where the distribution does contain a concentrated period of important decisions (or at least is sufficiently likely to in expectation), crunch time might be a useful concept to have:
    It’s a flag to switch modes (e.g. aiming for direct effects rather than getting ourselves into a better position in future)
    It’s a way of compressing lots of information about the world (e.g. these 15 things have happened with various levels of confidence, which makes us think that these 50 things will happen soon with various probabilities)
    It’s a concept which can help to coordinate lots of people, which in some crunch time scenarios could be very important
    But we don’t know what the underlying distribution of decisions is. Here are just some of the ways the distribution might look:
    The fact that the underlying distribution is uncertain means that there are several ways in which crunch time might be an unhelpful concept:
    Different sorts of concentrated periods of important decisions might differ so much that it’s better to think about them separately.
    A crunch time lasting a day seems pretty different to one lasting 5 years
    A crunch time where one actor makes one decision seems pretty different to one where many thousands of actors make many thousands of decisions
    The eventual distribution may not contain a single concentrated period of important influenceable decisions.
    Maybe the most important decisions are in the past
    Maybe there will be a gradual decline in the importance of influenceable decisions as humanity becomes increasingly disempowered
    Maybe the distribution is lumpy, and there will be many discrete concentrated periods of important decisions
    Distributions in different domains, or even for different individuals, might diverge sufficiently that there’s no meaningful single period of important decisions
    It seems unlikely that distributions diverge wildly at the individual level, but I think it’s possible that there might be reasonably different distributions for people who work in government versus in AI labs, for example.
    It might still be useful to have crunch time as a concept for a particularly concentrated period of important decisions, but only apply it in specific domains: crun...
    7 min
  • LW - Transcript of Sam Altman's interview touching on AI safety by Andy McKenzie
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transcript of Sam Altman's interview touching on AI safety, published by Andy McKenzie on January 20, 2023 on LessWrong.
    Sam Altman, CEO of OpenAI, was interviewed by Connie Loizos last week and the video was posted two days ago. Here are some AI safety-relevant parts of the discussion, with light editing by me for clarity, based on this automated transcript:
    [starting in part two of the interview, which is where the discussion about AI safety is]
    Connie: So moving on to AI which is where you've obviously spent the bulk of your time since I saw you when we sat here three years ago. You were telling us what was coming and we all thought you were being sort of hyperbolic and you were dead serious. Why do you think that ChatGPT and DALL-E so surprised people?
    Sam: I genuinely don't know. I've reflected on it a lot. We had the model for ChatGPT in the API for I don't know 10 months or something before we made ChatGPT. And I sort of thought someone was going to just build it or whatever and that enough people had played around with it. Definitely, if you make a really good user experience on top of something. One thing that I very deeply believed was the way people wanted to interact with these models was via dialogue. We kept telling people this we kept trying to get people to build it and people wouldn't quite do it. So we finally said all right we're just going to do it, but yeah I think the pieces were there for a while.
    One of the reasons I think DALL-E surprised people is if you asked five or seven years ago, the kind of ironclad wisdom on AI was that first, it comes for physical labor, truck driving, working in the factory, then this sort of less demanding cognitive labor, then the really demanding cognitive labor like computer programming, and then very last of all or maybe never because maybe it's like some deep human special sauce was creativity. And of course, we can look now and say it really looks like it's going to go exactly the opposite direction. But I think that is not super intuitive and so I can see why DALL-E surprised people. But I genuinely felt somewhat confused about why ChatGPT did.
    One of the things we really believe is that the most responsible way to put this out in society is very gradually and to get people, institutions, policy makers, get them familiar with it, thinking about the implications, feeling the technology, and getting a sense for what it can do and can't do very early. Rather than drop a super powerful AGI in the world all at once. And so we put GPT3 out almost three years ago and then we put it into an API like two and a half years ago. And the incremental update from that to ChatGPT I felt should have been predictable and I want to do more introspection on why I was sort of miscalibrated on that.
    Connie: So you know you had talked when you were here about releasing things in a responsible way. What gave you the confidence to release what you have released already? I mean do you think we're ready for it? Are there enough guardrails in place?
    Sam: We do have an internal process where we try to break things in and study impacts. We use external auditors, we have external red teamers, we work with other labs, and have safety organizations look at stuff.
    Societal changes that ChatGPT is going to cause or is causing. There's a big one going now about the impact of this on education, academic integrity, all of that. But starting these now where the stakes are still relatively low, rather than just putting out what the whole industry will have in a few years with no time for society to update, I think would be bad. Covid did show us for better or for worse that society can update to massive changes sort of faster than I would have thought in many ways.
    But I still think given the magnitude of the economic impact we expect here more gr...
    14 min
  • EA - What’s going on with ‘crunch time’? by rosehadshar
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What’s going on with ‘crunch time’?, published by rosehadshar on January 20, 2023 on The Effective Altruism Forum.
    Also on LW here.
    Imagine the set of decisions which impact TAI outcomes.
    Some of the decisions are more important than others: they have a larger impact on TAI outcomes. We care more about a decision the more important it is.
    Some of the decisions are also easier to influence than others, and we care more about a decision the easier it is to influence.
    Decisions in the past are un-influenceable
    Probably some decisions in the future are too, but it may be hard to identify which ones
    How easy it is to influence a decision depends on who you are
    People have access to different sets of people and resources
    Influence can be more or less in/direct (e.g. how many steps removed are you from the US President)
    The influenceable decisions are distributed over time, but we don’t know the distribution:
    Decisions might happen earlier or later (timing/timelines)
    There might be more or fewer decisions in the set
    Decisions might be more or less concentrated over time (urgency/duration)
    There might be discontinuities (only one decision matters, at the right of the distribution suddenly there are no more decisions because takeover has happened.)
    It’s possible that in the future there might be a particularly concentrated period of important decisions, for example like this:
    People refer to this concentrated period of important decisions as ‘crunch time’. The distribution might or might not end up actually containing a concentrated period of important decisions - or in other words, crunch time may or may not happen.
    Where the distribution does contain a concentrated period of important decisions (or at least is sufficiently likely to in expectation), crunch time might be a useful concept to have:
    It’s a flag to switch modes (e.g. aiming for direct effects rather than getting ourselves into a better position in future)
    It’s a way of compressing lots of information about the world (e.g. these 15 things have happened with various levels of confidence, which makes us think that these 50 things will happen soon with various probabilities)
    It’s a concept which can help to coordinate lots of people, which in some crunch time scenarios could be very important
    But we don’t know what the underlying distribution of decisions is. Here are just some of the ways the distribution might look:
    The fact that the underlying distribution is uncertain means that there are several ways in which crunch time might be an unhelpful concept:
    Different sorts of concentrated periods of important decisions might differ so much that it’s better to think about them separately.
    A crunch time lasting a day seems pretty different to one lasting 5 years
    A crunch time where one actor makes one decision seems pretty different to one where many thousands of actors make many thousands of decisions
    The eventual distribution may not contain a single concentrated period of important influenceable decisions.
    Maybe the most important decisions are in the past
    Maybe there will be a gradual decline in the importance of influenceable decisions as humanity becomes increasingly disempowered
    Maybe the distribution is lumpy, and there will be many discrete concentrated periods of important decisions
    Distributions in different domains, or even for different individuals, might diverge sufficiently that there’s no meaningful single period of important decisions
    It seems unlikely that distributions diverge wildly at the individual level, but I think it’s possible that there might be reasonably different distributions for people who work in government versus in AI labs, for example.
    It might still be useful to have crunch time as a concept for a particularly concentrated period of important decisions, but only apply it in specific...
    7 min
  • AF - Critique of some recent philosophy of LLMs’ minds by Roman Leventov
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Critique of some recent philosophy of LLMs’ minds, published by Roman Leventov on January 20, 2023 on The AI Alignment Forum.
    I structure this post as a critique of some recent papers on the philosophy of mind in application to LLMs, concretely, on whether we can say that LLMs think, reason, understand language, refer to the real world when producing language, have goals and intents, etc. I also use this discussion as a springboard to express some of my views about the ontology of intelligence, agency, and alignment.
    Mahowald, Ivanova, et al., “Dissociating language and thought in large language models: a cognitive perspective” (Jan 2023). Note that this is a broad review paper, synthesising findings from computational linguistics, cognitive science, and neuroscience, as well as offering an engineering vision (perspective) of building an AGI (primarily, in section 5). I don’t argue with these aspects of the paper’s content (although I disagree with something about their engineering perspective, I think that engaging in this disagreement would be infohazarous). I argue with the philosophical content of the paper, which is revealed in the language that the authors use and the conclusions that they make, as well as the ontology of linguistic competencies that the authors propose.
    Shanahan, “Talking About Large Language Models” (Dec 2022).
    Dissociating language and thought in large language models: a cognitive perspective
    In this section, I shortly expose the gist of the paper by Mahowald, Ivanova, et al., for the convenience of the reader.
    Abstract:
    Today’s large language models (LLMs) routinely generate coherent, grammatical and seemingly meaningful paragraphs of text. This achievement has led to speculation that these networks are—or will soon become—“thinking machines”, capable of performing tasks that require abstract knowledge and reasoning. Here, we review the capabilities of LLMs by considering their performance on two different aspects of language use: ‘formal linguistic competence’, which includes knowledge of rules and patterns of a given language, and ’functional linguistic competence’, a host of cognitive abilities required for language understanding and use in the real world. Drawing on evidence from cognitive neuroscience, we show that formal competence in humans relies on specialized language processing mechanisms, whereas functional competence recruits multiple extralinguistic capacities that comprise human thought, such as formal reasoning, world knowledge, situation modeling, and social cognition.
    In line with this distinction, LLMs show impressive (although imperfect) performance on tasks requiring formal linguistic competence, but fail on many tests requiring functional competence. Based on this evidence, we argue that (1) contemporary LLMs should be taken seriously as models of formal linguistic skills; (2) models that master real-life language use would need to incorporate or develop not only a core language module, but also multiple non-language-specific cognitive capacities required for modeling thought. Overall, a distinction between formal and functional linguistic competence helps clarify the discourse surrounding LLMs’ potential and provides a path toward building models that understand and use language in human-like ways.
    Two more characteristic quotes from the paper:
    In addition to being competent in the rules and statistical regularities of language, a competent language user must be able to use language to do things in the world: to talk about things that can be seen or felt or heard, to reason about diverse topics, to make requests, to perform speech acts, to cajole, prevaricate, and flatter. In other words, we use language to send and receive information from other perceptual and cognitive systems, such as our senses and our memory, and...
    38 min
  • LW - List of technical AI safety exercises and projects by Jakub Kraus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: List of technical AI safety exercises and projects, published by Jakub Kraus on January 19, 2023 on LessWrong.
    I intend to maintain a list at this doc. I'll paste the current state of the doc (as of January 19th, 2023) below. I encourage people to comment with suggestions.
    Levelling Up in AI Safety Research Engineering [Public] (LW)
    Highly recommended list of AI safety research engineering resources for people at various skill levels.
    AI Alignment Awards
    Alignment jams / hackathons from Apart Research
    Past / upcoming hackathons: LLM, interpretability 1, AI test, interpretability 2
    Projects on AI Safety Ideas: LLM, interpretability, AI test
    Resources: black-box investigator of language models, interpretability playground (LW), AI test
    Examples of past projects; interpretability winners
    How to run one as an in-person event at your school
    Neel Nanda: 200 Concrete Open Problems in Mechanistic Interpretability (doc and previous version)
    Project page from AGI Safety Fundamentals and their Open List of Project ideas
    AI Safety Ideas by Apart Research; EAF post
    Most Important Century writing prize (Superlinear page)
    Center for AI Safety
    Competitions like SafeBench
    Student ML Safety Research Stipend Opportunity – provides stipends for doing ML research.
    course.mlsafety.org projects CAIS is looking for someone to add details about these projects on course.mlsafety.org
    Distilling / summarizing / synthesizing / reviewing / explaining
    Forming your own views on AI safety (without stress!) – also see Neel's presentation slides and "Inside Views Resources" doc
    Answer some of the application questions from the winter 2022 SERI-MATS, such as Vivek Hebbar's problems
    10 exercises from Akash in “Resources that (I think) new alignment researchers should know about”
    [T] Deception Demo Brainstorm has some ideas (message Thomas Larsen if these seem interesting)
    Upcoming 2023 Open Philanthropy AI Worldviews Contest
    Alignment research at ALTER – interesting research problems, many have a theoretical math flavor
    Open Problems in AI X-Risk [PAIS #5]
    Amplify creative grants (old)
    Evan Hubinger: Concrete experiments in inner alignment, ideas someone should investigate further, sticky goals
    Richard Ngo: Some conceptual alignment research projects, alignment research exercises
    Buck Shlegeris: Some fun ML engineering projects that I would think are cool, The case for becoming a black box investigator of language models
    Implement a key paper in deep reinforcement learning
    “Paper replication resources” section in “How to pursue a career in technical alignment”
    Daniel Filan idea
    Summarize a reading from Reading What We Can
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Funding by Voting: Ignorance and Irrationality by pradyuprasad
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Funding by Voting: Ignorance and Irrationality, published by pradyuprasad on January 19, 2023 on The Effective Altruism Forum.
    I am sceptical of the idea that a substantial amount of EA funds should be allocated democratically. In my view there are some strong reasons to believe that the existing model of allocating funding (that is funders giving money to organisations which allocate money with the allocators' best judgement) does better than a simple democratic vote.
    The main problem in my opinion is that individual voters do not have sufficient incentives to allocate collective funds in an effective manner
    Rational Ignorance (or even irrationality)?
    The main problem is that getting decisions to be correct requires a large amount of information (which is costly to collect) and decision making (which is again costly to make correctly).
    Consider the decision on whether EA funders should spend more money on reducing civil conflict. This is a hard decision to make because it involves a large amount of information (like historical statistics on civil wars, their causes, historical efforts to reduce them etc)., subjective judgement (how tractable it is, comparing it to other funding opportunities, weighing PR concerns), and finally decisions based on the above judgements (how much to fund and to whom). The above process costs many hours of high intensity thinking.
    The most obvious response to this is that EA voters aren’t going to be ignorant because they care about EA causes. The argument would go that EA voters are going to spend more time than what political scientists would consider rational because they have a deep commitment to the cause. I do find this plausible. Many EAs regularly donate over 10% of their income to cause areas. Many of them spend hours explaining and debating these issues online, and for some of them working in an EA cause area will be their main career priority.
    My first response to this is that these people are going to be a minority of the total voter population (if broadly defined). Not everyone will have the same level of engagement with EA ideas, and not everyone will have the same level of personal interest in getting them right. These qualities are going to be concentrated in a minority of people because, relative to the total EA population (say as defined as DAUs or MAUs of the EA forum, or a larger one as GWWC pledge takers), this is a small number of people.
    But my second critique is more substantive. It is that apart from being rationally ignorant on these issues, most voters probably will be irrational on them too. On several EA topics voters will be biassed in the sense that they will have beliefs that are systematically wrong in some direction. They might be in favour of some cause area without updating when new information or arguments are produced. Their notions about certain people or organisations might make them systematically biassed against them and lead to bad voting outcomes.
    What incentive do voters in this have to improve? I don’t think they have many. Each voter does not think that their vote is very important. They won’t have a large chance of influencing the vote, so why bother doing all the reading anyways? Their previous notions will influence the voting decision and they will have no incentive to change if they are wrong.
    The existing method for EA orgs to allocate money has been for donors to give them money, then to analyse the costs and benefits of whatever opportunities they investigate and decide on what grants to give.
    I think this would lead to better outcomes because there are clearly identifiable people who have reputations to build by doing good analysis and recommending good grants. This can work in the opposite direction (people are too afraid to propose high risk grants because they feel it might stick...
    4 min
  • AF - Shard theory alignment requires magic. by Charlie Steiner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shard theory alignment requires magic., published by Charlie Steiner on January 20, 2023 on The AI Alignment Forum.
    A delayed hot take. This is pretty similar to previous comments from Rohin.
    "Magic," of course, in the technical sense of stuff we need to remind ourselves we don't know how to do. I don't mean this pejoratively, locating magic is an important step in trying to demystify it.
    And "shard theory alignment" in the sense of building an AI that does good things and not bad things by encouraging an RL agent to want to do good things, via kinds of reward shaping analogous to the diamond maximizer example.
    How might the story go?
    You start out with some unsupervised model of sensory data.
    On top of its representation of the world you start training an RL agent, with a carefully chosen curriculum and a reward signal that you think matches "goodness in general" on that curriculum distribution.
    This cultivates shards that want things in the vicinity of "what's good according to human values."
    These start out as mere bundles of heuristics, but eventually they generalize far enough to be self-reflective, promoting goal-directed behavior that takes into account the training process and the possibility of self-modification.
    At this point the values will lock themselves in, and future behavior will be guided by the abstractions in the learned representation of the world that the shards used to get good results in training, not by what would actually maximize the reward function you used.
    There magic here is especially concentrated around how we end up with the right shards.
    One magical process is how we pick the training curriculum and reward signal. If the curriculum is only made up only of simple environments, then the RL agent will learn heuristics that don't need to refer to humans. But if you push the complexity up too fast, the RL process will fail, or the AI will be more likely to learn heuristics that are better than nothing but aren't what we intended. Does a goldilocks zone where the agent learns more-or-less what we intended exist? How can we build confidence that it does, and that we've found it?
    And what's in the curriculum matters a lot. Do we try to teach the AI to locate "human values" by having it be prosocial towards individuals? Which ones? To groups? Over what timescale? How do we reward it for choices on various ethical dilemmas? Or do we artificially suppress the rate of occurrence of such dilemmas? Different choices will lead to different shards. We wouldn't need to find a unique best way to do things (that's a boondoggle), but we would need to find some way of doing things that we trust enough.
    Another piece of magic is how the above process lines up with generalization and self-reflectivity. If the RL agent becomes self-reflective too early, it will lock in simple goals that we don't want. If it becomes self-reflective too late, it will have started exploiting unintended maxima of the reward function. How do we know when we want the AI to lock in its values? How do we exert control over that?
    If shard theory alignment seemed like it has few free parameters, and doesn't need a lot more work, then I think you failed to see the magic. I think the free parameters haven't been discussed enough precisely because they need so much more work.
    The part of the magic that I think we could start working on now is how to connect curricula and learned abstractions. In order to predict that a certain curriculum will cause an AI to learn what we think is good, we want to have a science of reinforcement learning advanced in both theory and data. In environments of moderate complexity (e.g. Atari, MuJoCo), we can study how to build curricula that impart different generalization behaviors, and try to make predictive models of this process. Even if shard theory ali...
    5 min
  • LW - Vegan Nutrition Testing Project: Interim Report by Elizabeth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Vegan Nutrition Testing Project: Interim Report, published by Elizabeth on January 20, 2023 on LessWrong.
    Introduction
    Reducing consumption of animal products is a choice with both moral and practical consequences. Last summer I found myself in contact with many vegans who cared a lot about the moral consequences, but had put little effort into learning about or managing the practical consideration of removing animal products from their diet. I’ve suffered a lot due to bad nutrition, so this made me very concerned. With a grant from the Survival and Flourishing Fund, I launched small a pilot project to give nutritional tests to 5 vegans and near-vegans from the Lightcone Office, which they could use to choose supplements that would hopefully improve their health.
    My long-term goal was for everyone to have accurate information on their personal nutritional costs of veganism and make informed choices about how to handle them, with the first line solution being supplements. My goal for the pilot was to work out practical issues in testing, narrow the confidence interval on potential impact, and improve the nutrition of the handful of people. This report is on phase 1: getting the testing done and supplements started. It is aimed at people who might want to run a similar program at scale; if you are interested in running this for yourself I recommend checking out Tuesday’s post on iron deficiency.
    Tl;dr: I found rampant iron deficiencies, validating the overall concern. The procedure I used has a lot of room for improvement.
    The Experiment
    I gave nutrition tests to 6 people in the Lightcone office.
    The ideal subject was completely vegan, had never put any effort or thought into their diet, and was extremely motivated to take a test and implement changes. This person does not volunteer for studies, so I ended up with 4 vegans or near-vegans who had put somewhere between 0 and a lot of thought into their diet, 1 vegetarian, and 1 extremely motivated omnivore I used to test out the process. In addition, one hardcore vegan contributed results from private testing. I did not poll the ~vegans on their exact diets.
    Unless otherwise stated the results exclude the omnivore.
    I gave each of these six people a Genova Metabolomix+ test, ordered from walkinlabs.com, with the iron add-on. This test was selected for being recommended by doctors I trust (in part because they prefer urine to blood testing), having extremely easy-to-read results, being nearly comprehensive (with the unfortunate absence of vitamin D), and because I hoped urine collection at home would be easier than blood draws at a lab. Foreshadowing: I was wrong about that last part.
    I also gave people the option of an add-on to determine what variant of the MTHFR gene they have. MTHFR can affect how one processes certain B vitamins, and certain variants can necessitate a more expensive form of supplements.
    Several people (although not everyone) scored with undetectably low iron. I offered them follow-up blood tests, which one person accepted. An additional vegan contributed blood test results without urine results.
    As of publication all subjects have received their first round of results and started supplements of their choosing.
    The original plan was to retest in 3-6 months after people began supplements, using the same urine tests.
    My initial predictions
    I expected the big shortages to be B12, iron, and vitamin D, the first of which has very few natural vegan sources and the latter two of which are scarce, although not absent, in vegan sources. This makes it pretty unfortunate the original test did not include vitamin D.
    [B12 is naturally found in some (but not all) seaweeds and algaes, in at least one kind of mushroom, and in nutritional yeast. It’s also added to many wheat products in the US, so if you eat enough whe...
    13 min