The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Evaluations project @ ARC is hiring a researcher and a webdev/engineer by Beth Barnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Evaluations project @ ARC is hiring a researcher and a webdev/engineer, published by Beth Barnes on September 9, 2022 on The AI Alignment Forum.
    The evaluations project at the Alignment Research Center is looking to hire a generalist technical researcher and a webdev-focused engineer. We're a new team at ARC building capability evaluations (and in the future, alignment evaluations) for advanced ML models. The goals of the project are to improve our understanding of what alignment danger is going to look like, understand how far away we are from dangerous AI, and create metrics that labs can make commitments around (e.g. 'If you hit capability threshold X, don't train a larger model until you've hit alignment threshold Y'). We're also still hiring for model interaction contractors, and we may be taking SERI MATS fellows.
    About the project
    Big picture
    Once AI systems are close to being capable enough to be an x-risk, there should be procedures that labs follow to ensure they don’t build or deploy existentially dangerous models. We should have thought through the plausible paths to models gaining power, and ensured that models are not capable of these. There should be commitments that if model capabilities reach a certain level, labs will not scale up or deploy models until they have reached a certain level of alignment. In order to make this happen, we need good ways to measure both dangerous capabilities and alignment.
    We can start preparing for this now by:
    evaluating capabilities of current models
    predicting how dangerous the next generation of models is likely to be
    assessing the quality of our prediction methods by using them to predict the properties of generation n+1 given generation n, for historical generation models
    working with labs to evaluate their models before they are deployed, as a warm-up to stronger commitments in the future
    designing evaluations for alignment, deception etc to be run on future models
    Doing this now has some additional benefits:
    Evaluating capabilities of current models is helpful for predicting timelines, by giving us a more precise sense of what needs to change for models to be dangerous
    Specifying what we’d actually measure to determine the alignment of some model is helpful for clarifying our thoughts about how we expect misalignment to emerge, and what exactly the alignment problem is
    This work may produce demonstrations of models doing competent and scary behavior. This can be useful if we think people are underestimating the capabilities of current or near-future models and making bad decisions because of that.
    Current goals
    Our current goals are to work with cutting-edge models, produce evaluations for how far they are from x-risk capabilities, and share these evaluations with labs.
    More specifically, we’re doing things like:
    design and build scaffolding that lets generative language models act like agents over long time horizons:
    scripts to automatically run model outputs in the shell or forward them as subtasks to other copies of the model
    sampling multiple actions, having the model evaluate them and pick the best one
    various ways of prompting and nudging the model that improve performance
    think about what a model would need to be able to do in order to successfully gain power, given current levels of caution and monitoring around AI systems. For instance: exfiltrate its weights via hacking or social engineering; make money via crime or labor; use this money to buy server time and start more copies of itself running on those servers
    Assess model performance on these subtasks, and determine whether the model is capable of some set of tasks that's sufficient for successful powerseeking, or if not, what exactly the barriers are and how close the model is to surmounting them
    At the end of this, we hope to either have found clea...
    17 min
  • LW - Samotsvety's AI risk forecasts by elifland
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Samotsvety's AI risk forecasts, published by elifland on September 9, 2022 on LessWrong.
    Crossposted to EA Forum and Foxy Scout
    Introduction
    In my review of What We Owe The Future (WWOTF), I wrote:
    Finally, I’ve updated some based on my experience with Samotsvety forecasters when discussing AI risk. When we discussed the report on power-seeking AI, I expected tons of skepticism but in fact almost all forecasters seemed to give >=5% to disempowerment by power-seeking AI by 2070, with many giving >=10%.
    In the comments, Peter Wildeford asked:
    It looks like Samotsvety also forecasted AI timelines and AI takeover risk - are you willing and able to provide those numbers as well?
    We separately received a request from the FTX Foundation to forecast on 3 questions about AGI timelines and risk.
    I sent out surveys to get Samotsvety’s up-to-date views on all 5 of these questions, and thought it would be valuable to share the forecasts publicly.
    A few of the headline aggregate forecasts are:
    25% chance of misaligned AI takeover by 2100, barring pre-APS-AI catastrophe
    81% chance of Transformative AI (TAI) by 2100, barring pre-TAI catastrophe
    32% chance of AGI being developed in the next 20 years
    Forecasts
    In each case I aggregated forecasts by removing the single most extreme forecast on each end, then taking the geometric mean of odds.
    To reduce concerns of in-group bias to some extent, I calculated a separate aggregate for those who weren’t highly-engaged EAs (HEAs) before joining Samotsvety. In most cases, these forecasters hadn’t engaged with EA much at all; in one case the forecaster was aligned but not involved with the community. Several have gotten more involved with EA since joining Samotsvety.
    Unfortunately I’m unable to provide forecast rationales in this post due to forecaster time constraints, though I might in a future post. I provided my personal reasoning for relatively similar forecasts (35% AI takeover by 2100, 80% TAI by 2100) in my WWOTF review.
    WWOTF questions
    Aggregate (n=11)
    Aggregate, non-pre-Samotsvety-HEAs (n=5)
    Range
    What's your probability of misaligned AI takeover by 2100, barring pre-APS-AI catastrophe?
    25%
    14%
    3-91.5%
    What's your probability of Transformative AI (TAI) by 2100, barring pre-TAI catastrophe?
    81%
    86%
    45-99.5%
    FTX Foundation questions
    For the purposes of these questions, FTX Foundation defined AGI as roughly “AI systems that power a comparably profound transformation (in economic terms or otherwise) as would be achieved in [a world where cheap AI systems are fully substitutable for human labor]”. See here for the full definition used.
    Unlike the above questions, these are not conditioning on no pre-AGI/TAI catastrophe.
    Aggregate (n=11)
    Aggregate, non-pre-Samotsvety-HEAs (n=5)
    Range
    What's the probability of existential catastrophe from AI, conditional on AGI being developed by 2070?
    38%
    23%
    4-98%
    What's the probability of AGI being developed in the next 20 years?
    32%
    26%
    10-70%
    What's the probability of AGI being developed by 2100?
    73%
    77%
    45-80%
    Who is Samotsvety Forecasting?
    Samotsvety Forecasting is a forecasting group that was started primarily by Misha Yagudin, Nuño Sempere, and myself predicting as a team on INFER (then Foretell). Over time, we invited more forecasters who had very strong track records of accuracy and sensible comments, mostly on Good Judgment Open but also a few from INFER and Metaculus. Some strong forecasters were added through social connections, which means the group is a bit more EA-skewed than it would be without these additions. A few Samotsvety forecasters are also superforecasters.
    How much do these forecasters know about AI?
    Most forecasters have at least read Joe Carlsmith’s report on AI x-risk, Is Power-Seeking AI an Existential Risk?. Those who are short on time may have just skimmed the report and/or...
    7 min
  • EA - A California Effect for Artificial Intelligence by henryj
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A California Effect for Artificial Intelligence, published by henryj on September 9, 2022 on The Effective Altruism Forum.
    I just finished writing a 50-page document exploring a few ways that the State of California could regulate AI with the goal of producing a de facto California Effect. You can read the whole thing as a Google doc here, as a pdf here, or as a webpage here, or you can read the summary and a few key takeaways below. I'm also including some thoughts on my theory of impact and on opportunities for future research.
    I built off work by Anu Bradford, as well as a recent GovAI paper by Charlotte Siegmann and Markus Anderljung. This project was mentored by Cullen O'Keefe. I did this research through an existential risk summer research fellowship at the University of Chicago — thank you Zack Rudolph and Isabella Duan for organizing it!
    Abstract
    The California Effect occurs when companies adhere to California regulations even outside California’s borders because of a combination of California’s large market, its capacity to successfully regulate, its preference for stringent standards, and the difficulty of dividing the regulatory target or moving beyond California’s jurisdiction. In this paper, I look into three ways in which California could regulate artificial intelligence and ask whether each would produce a de facto California Effect. I find it likely (~80%) that regulating training data through data privacy would produce a California Effect. I find it unlikely (~20%) that regulation based on the number of floating-point operations needed to train a model would produce a California Effect. Finally, I find it likely (~80%) that risk-based regulation like that proposed by the European Union would produce a California Effect.
    If this seems interesting, please give the full paper a look. There's a more-detailed 1.5-page executive summary, and then (of course) the document itself.
    Key Takeaways
    The California Effect is a powerful force-multiplier that lets you have federal-level impact for the low(er) price of state-level effort.
    There are ways to regulate AI which I argue would produce a California Effect.
    State government in general and California's government specifically are undervalued by EAs. I believe that EAs interested in politics, structural change, regulation, animal welfare, preventing pandemics, etc, could in some cases have bigger and/or more immediate impacts on a state level than on a federal level.
    There are still plenty of opportunity for further research.
    Theory of Impact
    My hope — and ultimate theory of impact — is that this paper will help policymakers make better-informed decisions about future AI regulations. I hope to encourage those who believe in regulating artificial intelligence to give more attention to the State of California. At the very least, I hope that people with a broader reach than I have in the AI Governance space will read and even build off this work. I hope I can raise their awareness of the California Effect and ensure that they recognize the disproportionate impact it can have in the race to keep artificial intelligence safe.
    Opportunities for further research
    Before I list my own thoughts, I will direct readers to the list of further research opportunities that Charlotte Siegmann and Markus Anderljung collected in an announcement for their report on the potential Brussels Effect of the EU AI Act. I’m personally choosing to highlight their fourth and sixth bullet points, which I think would be especially effective (the latter even more so):
    “Empirical work tracking the extent to which there is likely to be a Brussels Effect. Most of the research on regulatory diffusion focuses on cases where diffusion has already happened. It seems interesting to instead look for leading indicators of regulatory diffusion. For exam...
    8 min
  • EA - Cause Exploration Prizes: Announcing our prizes by ChrisSmith
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cause Exploration Prizes: Announcing our prizes, published by ChrisSmith on September 9, 2022 on The Effective Altruism Forum.
    We were gratified to receive over 150 good-faith submissions to Open Philanthropy’s Cause Exploration Prizes, where we invited people to suggest a new area for us to support or respond to our suggested questions. We hoped that these submissions would help us find new ways to carry out our mission — helping others as much as possible with the resources available to us.
    You can read them on the EA Forum. Below, we highlight the submissions to which we are awarding major prizes and honorable mentions.
    We’re awarding these prizes to entries that we thought engaged well with our prompts and helped us to better understand the questions and issues they addressed. We have not investigated each and every claim made in these entries, and the awarding of a prize does not imply that we necessarily endorse their claims or arguments as correct.
    Our top prize
    We are awarding our top prize ($25,000) to: Organophosphate pesticides and other neurotoxicants by Ben Stewart.
    Second prizes
    We are awarding three second-place prizes ($15,000) for the following submissions. These are listed in no particular order.
    Violence against women and girls by Akhil
    Sickle Cell Disease by anonymous
    Shareholder activism by sbehmer
    Honorable mentions
    We are awarding $500 to the authors of the following entries. These are listed in no particular order.
    Expanding access to infertility services in Low- and Middle-Income Countries by Soleine Scotney
    Maternal morbidity by alexhill
    Farmed Animal Welfare in Sub-Saharan Africa by anonymous
    Indoor Air Quality to Reduce Infectious Respiratory Disease by Gavriel Kleinwaks, Alastair Fraser-Urquhart, and joshcmorrison
    To WELLBY or not to WELLBY? Measuring non-health, non-pecuniary benefits using subjective wellbeing by JoelMcGuire, Samuel Dupret, and MichaelPlant
    Tobacco harm reduction by kristof
    Climate adaptation in low-income countries by Karthik Tadepalli
    Social and behavioral science R&D by Anna Harvey and Stuart Buck
    Family Planning: A Significant Opportunity for Impact by Sarah H and Ben Williamson
    Improving diagnosis and treatment of bipolar spectrum disorders by Karolina Soltys
    War between the US and China: A case study for epistemic challenges around China-related catastrophic risk by Jordan_Schneider and pradyusp
    Improved quality control in science by Sophie Schauman
    More animal advocacy R&D by anonymous
    Adapting to Extreme Heat Exposure in South Asia by Surbhi B
    Rich to Poor Country Spillovers by anonymous
    Large-scale International Educational Migration: A shallow investigation by Jasmin Baier, Johannes Haushofer, and Hannah Lea Shaw
    Training health workers to prevent newborn deaths by Marshall
    Developmental Cognitive Neuroepidemiology by Hauke Hillebrandt
    Differential Neurotechnology Development by mwcvitkovic
    Short sleeper genes by JohnBoyle
    We are contacting all prize recipients by email.
    Good-faith submissions
    Next week we will begin the process of emailing everyone who submitted a good-faith submission in order to offer them participation awards of $200.
    Future plans
    As we stated in our announcement, this was a trial process for us. We’re grateful to those who sent us feedback and suggestions for how to improve. At this stage, we don’t know if or when we will repeat a process like this. We might write a public update later this year on what we have learned from this exercise and any plans to repeat this or a similar exercise again.
    Thank you
    We are grateful to Lizka and the other operators of the EA Forum, and to everyone who engaged with or submitted an entry to the Cause Exploration Prizes for making this possible.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    5 min
  • EA - Zzapp Malaria: More effective than bed nets? (Wanted: CTO, COO & Funding) by Yonatan Cale
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Zzapp Malaria: More effective than bed nets? (Wanted: CTO, COO & Funding), published by Yonatan Cale on September 9, 2022 on The Effective Altruism Forum.
    TL;DR: This post describes Zzapp’s approach and effectiveness from their own perspective, intended as an intro aimed at the Effective Altruism community, as an invitation to investigate further and maybe fund them. They claim to be 2x more cost effective than bed nets in reducing malaria in urban and semi-urban areas (over 70% of Africa’s population).
    Epistemic status: Based on conversations with Arnon, the CEO of Zzapp Malaria, not cross checked with other info such as Givewell’s review of Against Malaria Foundation.
    Zzapp’s approach and theoretical reason to think it would work
    You can skip to their experiment and how it went, if you prefer.
    TL;DR: Spray water bodies with larvicide to prevent mosquitoes from reproducing, and do it extra well by managing the considerable ops work of finding and spraying the water bodies using satellite imaging and an app for the people on the ground.
    Spraying water bodies with larvicide - is tried and works, unrelated to Zzapp
    Sources [link] [link].
    Theoretical advantages compared to bed nets
    In every place that malaria was eliminated (which happened many times), larvicide (the treatment of standing water bodies) was the main component.
    Bed Nets only help people indoors during the night.
    Many people don’t use their bed nets.
    Mosquitos developed resistance to the bes nets’ insecticide in many countries
    Note I think Givewell already took the problems into account in their analysis, and Arnon emphasizes he thinks bed nets are great, and this is a pitch for using larvicide in urban (and semi urban) areas, not for stopping distributing bed nets. Zzapp think the ideal solution would probably combine many interventions. We are writing this as a comparison with bed nets since EAs already think bed nets are great.
    Problems in existing larvicide approaches
    Existing solutions: Problems in theory
    Coverage is important
    It’s important [how many water bodies you find] and [how many of those you spray], and the difference between 95% and 50% is really big, similarly to the situation when vaccinating 95% or 50% of the population, because of the effect on R (reproduction number) - less infected people will infect less other people, it snowballs but in a good way (hopefully), and the same is true about reproduction of mosquitos.
    Existing solutions have bad coverage
    People miss water bodies in the areas they are assigned to search
    People miss entire areas
    Even when water bodies are found, the spray team sometimes may still skip them or forget to treat them according to schedule
    Small RCT
    Ref to a (tiny) randomized controlled trial run by Zzapp and AngloGold Ashanti Malaria Control (AGAMal), where two groups scanned the same square kilometer, one group used the app and one didn’t and the group with the app found 28% more water bodies.
    Scanning an entire town
    In a different operation, when scanning an entire town with AGAMal, they found 20x more water bodies when using Zzapp’s app. (publication in progress, we’ll add a link when it’s ready). From that they think that on a larger scale the app has an even greater impact.What happened behind the scenes is that without the app - the scanners skipped entire neighborhoods.
    Not a problem: Poisoning water bodies
    The larvicide in the relevant quantities (bti) isn’t poisonous to humans, animals, or other insects except for mosquitoes and black flies.
    Zzapp’s advantages compared to “manual” larvicide
    Zzapp has an app they give to the people “on the ground”:
    The app follows “where do the people go go” and lets the people mark “I checked this house’s garden” and “I found a water source here” and “this house didn’t let me in”
    The control room shows a map with
    ״Here...
    9 min
  • AF - Oversight Leagues: The Training Game as a Feature by Paul Bricman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Oversight Leagues: The Training Game as a Feature, published by Paul Bricman on September 9, 2022 on The AI Alignment Forum.
    This post is part of my hypothesis subspace sequence, a living collection of proposals I'm exploring at Refine. Followed by ideological inference engines.
    Thanks Adam Shimi for advice on putting more legible content out there. Thanks Eric Winsor, Leo Grinsztajn, Linda Linsefors, Lucas Texeira, Tammy Leake, Ze Shen for discussions which inspired this post.
    TL;DR: An oversight league is a training scheme which incentivizes an agent and an evaluator to constantly try to game each other, leading to synchronized increases in capability for the two players. However, the evaluator is being offered a host of additional learning signals to help it maintain a consistent (and potentially provable) lead over the agent. Oversight leagues heavily draw on ideas from capability literature, including: league training in AlphaStar, game theory in GANs, adversarial robustness, etc.
    Intro
    The whole project of oversight leagues relies on the following non-exhaustive list of assumptions:
    Assumption 1, "AGI Hard, Human Values Harder": We are unlikely to formulate the True Name of human values in closed-form before deploying transformative AI. The best we are likely to do before takeoff is model human values approximately and implement an imperfect evaluator.
    Assumption 2, "Linear Capability Ordering": Any fixed evaluator (e.g. a reward model) can be gamed by an agent above a certain threshold of capability. More generally, an agent whose capability improves consistently faster than the capability of an evaluator will eventually be able to game said evaluator. By "evaluator capability," I'm referring to its ability to prevent being gamed.
    Assumption 3, "Humans Are Not True Gamers": Human oversight is impractical because our capabilities as evaluators can't improve at an arbitrary large rate. Save for cyborgian schemes for human augmentation, human oversight would eventually be gamed by an agent of sufficient capability.
    Assumption 4, "Zone of Proximal Development": There is a relatively small interval of evaluator lead which allows the agent to improve from challenging it, even if the evaluator is slightly ahead. This assumption is motivated by the fact that GANs are actually working, despite an inevitable imbalance between the two components involved at any given time.
    Assumption 5, "Point of No Return": Even a minor lead in capability for the agent relative to the evaluator is catastrophic, as the agent can take advantage of this window of opportunity and maneuver itself into a persistent advantageous position by flying under the evaluator's radar.
    From the assumptions above, we can infer that if we are to deploy a robust evaluator as an operationalization of human values, it must certainly not be fixed. Not only that, but it must additionally improve in performance at least as fast as the agent being evaluated, so that it never gets overtaken. What's more, simple human oversight doesn't fulfill those necessary conditions, hence we should consider automated schemes.
    Proposal
    An oversight league is one such automated scheme for training agents and evaluators in a way which improves their performance in lockstep. The crux of this training regime is to supply most of the training through bilateral learning signals, and thus render the improvement of the two components interdependent. By ensuring that most of the learning opportunities of the agent come from playing against the evaluator and vice versa, the two sides form a positive feedback loop resembling patterns of co-evolution. The oversight league scheme implicitly attempts to cultivate "antifragility" by applying appropriate stressors on the evaluator in the form of ever more capable agents a reliable way of impr...
    18 min
  • EA - Samotsvety's AI risk forecasts by elifland
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Samotsvety's AI risk forecasts, published by elifland on September 9, 2022 on The Effective Altruism Forum.
    Crossposted to LessWrong and Foxy Scout
    Introduction
    In my review of What We Owe The Future (WWOTF), I wrote:
    Finally, I’ve updated some based on my experience with Samotsvety forecasters when discussing AI risk. When we discussed the report on power-seeking AI, I expected tons of skepticism but in fact almost all forecasters seemed to give >=5% to disempowerment by power-seeking AI by 2070, with many giving >=10%.
    In the comments, Peter Wildeford asked:
    It looks like Samotsvety also forecasted AI timelines and AI takeover risk - are you willing and able to provide those numbers as well?
    We separately received a request from the FTX Foundation to forecast on 3 questions about AGI timelines and risk.
    I sent out surveys to get Samotsvety’s up-to-date views on all 5 of these questions, and thought it would be valuable to share the forecasts publicly.
    A few of the headline aggregate forecasts are:
    25% chance of misaligned AI takeover by 2100, barring pre-APS-AI catastrophe
    81% chance of Transformative AI (TAI) by 2100, barring pre-TAI catastrophe
    32% chance of AGI being developed in the next 20 years
    Forecasts
    In each case I aggregated forecasts by removing the single most extreme forecast on each end, then taking the geometric mean of odds.
    To reduce concerns of in-group bias to some extent, I calculated a separate aggregate for those who weren’t highly-engaged EAs (HEAs) before joining Samotsvety. In most cases, these forecasters hadn’t engaged with EA much at all; in one case the forecaster was aligned but not involved with the community. Several have gotten more involved with EA since joining Samotsvety.
    Unfortunately I’m unable to provide forecast rationales in this post due to forecaster time constraints, though I might in a future post. I provided my personal reasoning for relatively similar forecasts (35% AI takeover by 2100, 80% TAI by 2100) in my WWOTF review.
    WWOTF questions
    Aggregate (n=11)
    Aggregate, non-pre-Samotsvety-HEAs (n=5)
    Range
    What's your probability of misaligned AI takeover by 2100, barring pre-APS-AI catastrophe?
    25%
    14%
    3-91.5%
    What's your probability of Transformative AI (TAI) by 2100, barring pre-TAI catastrophe?
    81%
    86%
    45-99.5%
    FTX Foundation questions
    For the purposes of these questions, FTX Foundation defined AGI as roughly “AI systems that power a comparably profound transformation (in economic terms or otherwise) as would be achieved in [a world where cheap AI systems are fully substitutable for human labor]”. See here for the full definition used.
    Unlike the above questions, these are not conditioning on no pre-AGI/TAI catastrophe.
    Aggregate (n=1)
    Aggregate, non-pre-Samotsvety-HEAs (n=5)
    Range
    What's the probability of existential catastrophe from AI, conditional on AGI being developed by 2070?
    38%
    23%
    4-98%
    What's the probability of AGI being developed in the next 20 years?
    32%
    26%
    10-70%
    What's the probability of AGI being developed by 2100?
    73%
    77%
    45-80%
    Who is Samotsvety Forecasting?
    Samotsvety Forecasting is a forecasting group that was started primarily by Misha Yagudin, Nuño Sempere, and myself predicting as a team on INFER (then Foretell). Over time, we invited more forecasters who had very strong track records of accuracy and sensible comments, mostly on Good Judgment Open but also a few from INFER and Metaculus. Some strong forecasters were added through social connections, which means the group is a bit more EA-skewed than it would be without these additions. A few Samotsvety forecasters are also superforecasters.
    How much do these forecasters know about AI?
    Most forecasters have at least read Joe Carlsmith’s report on AI x-risk, Is Power-Seeking AI an Existential Risk?. Those who are short on time may have just skimme...
    7 min
  • AF - Understanding and avoiding value drift by Alex Turner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding and avoiding value drift, published by Alex Turner on September 9, 2022 on The AI Alignment Forum.
    I use the shard theory of human values to clarify what value drift is, how it happens, and how it might be avoided by a highly intelligent agent—even if that agent doesn't have any control over its future experiences. Along the way, I give a shard theory account of rationalization.
    Defining "value drift"
    Recapitulating part of shard theory. Reward is that which reinforces. Considering the case of reinforcement learning in humans, reward causes your brain’s credit assignment algorithms to reinforce the actions and thoughts which led to that reward, making those actions and thoughts more likely to be selected in the future.
    For example, suppose you recognize a lollipop, and move to pick it up, and then lick the lollipop. Since the lollipop produces reward, these thoughts will be reinforced and you will be more likely to act similarly in such situations in the future. You become more of the kind of person who will move to pick up a lollipop when you recognize lollipops, and who will navigate to lollipop-containing locations to begin with.
    With that in mind, I think that shard theory offers a straightforward definition of "value drift":
    Definition. Value drift occurs when reinforcement events substantially change the internal "balance of power" among the shards activated in everyday situations.
    For example, consider the classic "example" of taking a pill which makes you enjoy killing people. Under shard theory, this change would be implemented as a murder-shard that activates in a wide range of contexts in order to steer planning towards murder, and therefore starts steering your decision-making substantially differently.
    But it's better to try to explain phenomena which, you know, are known to actually happen in real life. Another simple example of value drift is when someone snorts cocaine. At a (substantial) gloss, the huge hit of reward extremely strongly upweights the decision to do cocaine; the strength of the reward leads to an unusually strong cocaine-shard which activates in an unusually wide range of situations.
    Here's a more complicated example of value drift. I'll give one possible mechanistic story for the "value drift" which occurs to an atheist (Alice) dating a religious person (Rick), and why that situation might predictably lead to Alice converting or Rick deconverting. I'll consider a scenario where Alice converts.
    First, reinforcement events cause Alice to develop shards of value around making Rick happy and making Rick like her. Alice's new shards (non-introspectively-apparently) query her world model for plans which make Rick happier and which make Rick like her more. Obviously, if Alice converted, they would have more in common, and Rick would be happy. Since these plans lead to Rick being happy and liking Alice more, these shards bid for those plans.
    Only, the plan is not bid for directly in an introspectively obvious manner. That would provoke opposition from Alice's other values (which oppose deliberately changing her religious status just to make Rick happy). Alice's self-model predicts this opposition, and so her Rick-happiness- and Rick-approval-shards don't bid for the "direct" conversion plan, because it isn't predicted to work (and therefore won't lead to a future where Rick is happier and approves of Alice more). No, instead, these two shards rationalize internally-observable reasons why Alice should start going to Rick's church: "it's respectful", "church is interesting", "if I notice myself being persuaded I can just leave", "I'll get to spend more time with Rick."
    Here, then, is the account:
    Alice's Rick-shards query her world model for plans which lead to Rick being happier and liking Alice more,
    so her world model returns a...
    11 min
  • AF - Most People Start With The Same Few Bad Ideas by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Most People Start With The Same Few Bad Ideas, published by johnswentworth on September 9, 2022 on The AI Alignment Forum.
    Epistemic status: lots of highly subjective and tentative personal impressions.
    Occasionally people say “hey, alignment research has lots of money behind it now, why not fund basically everyone who wants to try it?”. Often this involves an analogy to venture capital: alignment funding is hits-based (i.e. the best few people are much more productive than everyone else combined), funders aren’t actually that good at distinguishing the future hits, so what we want is a whole bunch of uncorrelated bets.
    The main place where this fails, in practice, is the “uncorrelated” part. It turns out that most newcomers to alignment have the same few Bad Ideas.
    The most common, these days, is some variant of “train an AI to help with aligning AI”. Sometimes it’s “train an AI to interpret the internals of another AI”, sometimes it’s “train an AI to point out problems in another AI’s plan”, sometimes it’s “train an AI to help you design aligned AI”, etc. I would guess about 75% of newcomers from ML suggest some such variant as their first idea.
    People who are less aware of standard alignment arguments tend to start with “train an AI on human feedback” or “iterate until the problems go away”. In the old days, pre-sequences, people started from even worse ideas; at least the waterline has risen somewhat.
    People with more of a theory bent or an old-school AI background tend to reinvent IRL or CIRL variants. (A CIRL variant was my own starting Bad Idea - you can read about it in this post from 2020, although the notes from which that post was written were from about 2016-2017.)
    My impression (based on very limited data) is that it takes most newcomers ~5 years to go from their initial Bad Idea to actually working on something plausibly useful. For lack of a better name, let’s call that process the Path of Alignment Maturity.
    My impression is that progress along the Path of Alignment Maturity can be accelerated dramatically by actively looking for problems with your own plans - e.g. the builder/breaker framework from the Eliciting Latent Knowledge doc, or some version of the Alignment Game Tree exercise, or having a group of people who argue and poke holes in each others’ plans. (Of course these all first require not being too emotionally attached to your own plan; it helps a lot if you can come up with a second or third line of attack, thereby building confidence that there’s something else to move on to.) It can also be accelerated by starting with some background knowledge of difficult problems adjacent to alignment/agency - I notice philosophers tend to make unusually fast progress down the Path that way, and I think prior experience with adjacent problems also cut about 3-4 years off the Path for me.
    (To be clear, I don’t necessarily recommend that as a strategy for a newcomer - I spent ~5 years working on agency-adjacent problems before working on alignment, and that only cut ~3-4 years off my Path of Alignment Maturity. That wasn’t the only alignment-related value I gained from my background knowledge, but the faster progress down the Path was not worthwhile on its own.) General background experience/knowledge about the world also helps a lot - e.g. I expect someone who's founded and worked at a few startups will make faster progress than someone who’s only worked at one big company, and either of those will make faster progress than someone who’s never been outside of academia.
    On the flip side, I expect that progress down the Path of Alignment Maturity is slower for people who spend their time heads-down in the technical details of a particular approach, and spend less time reflecting on whether it’s the right approach at all or arguing with people who have ve...
    5 min
  • EA - Marketing Messages Trial for GWWC Giving Guide Campaign by Erin Morrissey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Marketing Messages Trial for GWWC Giving Guide Campaign, published by Erin Morrissey on September 8, 2022 on The Effective Altruism Forum.
    The trial was run in conjunction with Josh Lewis (NYU). Thanks to David Moss and others for feedback on this post, and to Jamie Elsey for support with the Bayesian analysis.
    TL;DR
    Giving What We Can together with the EA Market Testing Team (EAMT) tested marketing and messaging themes on Facebook in their Effective Giving Guide Facebook Lead campaigns which ran from late November 2021 - January 2022. GWWC's Giving Guide answers key questions about effective giving and includes the latest effective giving recommendations to teach donors how to do the most good with their donations. These were exploratory trials to identify promising strategies to recruit people for GWWC and engage people with EA more broadly. We report the most interesting patterns from these trials to provide insight into which hypotheses might be worth exploring more rigorously in future (‘confirmatory analysis’) work.
    Across four trials we compared the effectiveness of different types of (1) messages, (2) videos, and (3) targeted audiences. The key outcomes were (i) email addresses per dollar (when a Facebook user provides an email lead) and (ii) link clicks per dollar. Based on our analysis of 682,577 unique Facebook ‘impressions’, we found:
    The cost of an email address was as low as $8.00 across campaigns, but it seemed to vary substantially across audiences, videos, and messages.
    The message "Only 3% of donors give based on charity effectiveness, yet the best charities can be 100x more impactful" generated more link clicks and email addresses per dollar than other messages. In contrast, the message "Giving What We Can has helped 6,000+ people make a bigger impact on the causes they care about most" was less cost-effective than the other messages.
    A ‘short video with facts about effective giving’ generated more email addresses per dollar than either (1) a long video with facts about effective giving or (2) a long video that explained how GWWC can help maximize charitable impact, the GWWC 'brand video.'
    On a per-dollar basis ‘Animal’ audiences that were given animal-related cause videos performed among the best, both overall and in the most comparable trials. ‘Lookalike’ audiences (those with a similar profile as current people engaging with GWWC) performed best overall, for both cause and non-cause videos. However, ‘Climate’ and ‘Global Poverty’ audiences basically underperformed the ‘Philanthropy’ audience when presented videos ‘for their own causes.’ The Animal-related cause video performed particularly poorly on the ‘Philanthropy’ audience.
    Demographics were mostly not predictive of email addresses per dollar nor link clicks per dollar
    See our Quarto dynamic document linked here for more details, and ongoing analyses.
    Purpose and Interpretation of this Report
    One of the primary goals of the EAMT is to identify the most effective, scalable strategies for marketing EA. Our main approach is to test marketing and messaging themes in naturally-occurring settings (such as advertising campaigns on Facebook, YouTube, etc.), targeting large audiences, to determine which specific strategies work best in the most relevant contexts. In this report, we share key patterns and insights about the effectiveness of different marketing and messaging approaches used in GWWC's Effective Giving Guide Facebook Lead campaigns. The patterns we share here serve as a starting point to consider themes and hypotheses to test more rigorously in our ongoing research project.
    We are hoping for feedback and suggestions from the EA community on these trials and their implementation and analysis. We continue to conduct detailed analyses of this data. We'd like to get ideas from the community ...
    25 min