The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - AI Safety in a World of Vulnerable Machine Learning Systems by AdamGleave
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety in a World of Vulnerable Machine Learning Systems, published by AdamGleave on March 8, 2023 on The AI Alignment Forum.
    Even the most advanced contemporary machine learning systems are vulnerable to adversarial attack. The safety community has often assumed adversarial robustness to be a problem that will be solved naturally as machine learning (ML) systems grow more capable and general. However, recent work has shown that superhuman systems in a narrow domain such as AlphaZero are highly vulnerable to adversarial attack, as are general but less capable systems like large language models. This raises the possibility that adversarial (worst-case) robustness will continue to lag behind average-case capabilities. In other words, transformative AI systems are likely to be exploitable.
    Exploitability will cause a wide variety of current alignment proposals to fail. Most extant agendas seek to align the main ML system with the assistance of helper ML systems. The main ML system is the primary system that takes actions in the world (e.g. interacting with users), with the helper ML systems acting as scaffolding to train and/or verify the main ML system. These alignment schemes will fail if the helpers are exploited by the main system – and we expect helpers to be vulnerable to exploitation (see Contemporary ML systems are exploitable by default).
    In Table 1 we present a subjective risk matrix for a range of popular alignment agendas, evaluating the degree to which main ML systems have the ability and incentive to exploit the helper. We find many alignment agendas have a high risk of exploitation, with all having at least some risk.
    Alignment AgendaMain System’s Ability to Exploit HelperMain System’s Incentive to Exploit HelperRisk of ExploitRL on learned reward model (e.g. RLHF, IRL)MediumHighHighScalable oversight (e.g. recursive reward modeling,AI safety via debate)MediumHighHighImitation learning (e.g. behavioral cloning, supervised fine-tuning)MediumLowLow-MediumImitative Iterated Distillation and AmplificationHighLowMediumAuditing Tool (e.g. Adversarial Testing, Transparency)LowMediumLow-Medium
    Table 1: Subjective risk matrix for popular alignment agendas (see next section), using a helper ML system to assist with aligning the main ML system that will eventually be deployed.
    We are most concerned by vulnerabilities in the helpers as this can impact the alignment of the main system. By contrast, an aligned but adversarially exploitable main system would not necessarily pose a danger, especially if the main system can recursively self-improve to fix itself. However, there is a possibility that even superintelligent systems cannot attain adversarial robustness. This would be a volatile situation, which could conceivably collapse into chaos (systems frequently exploiting each other), an implicit equilibrium (e.g. mutually assured destruction), or an explicit agreement (e.g. all AI systems self-modify to commit to not exploiting one another).
    We see two possible approaches to fixing this: improving adversarial robustness, or developing fault tolerant alignment methods that can work even in the presence of vulnerable ML systems. We are most excited by fault tolerant alignment, as it is highly neglected and plausibly tractable, although further work is needed to solidify this approach. By contrast, adversarial robustness is an area that has received significant attention from the ML research community (low neglectedness)[1] but with only modest progress (low to medium tractability).
    In the remainder of this document, we will argue that systems are exploitable by default, explore the implications this has for alignment agendas in several different scenarios, and outline several research directions we are excited by.
    Alignment agendas need robustness
    Most alignment sche...
    53 min
  • EA - Winners of the Squiggle Experimentation and 80,000 Hours Quantification Challenges by NunoSempere
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Winners of the Squiggle Experimentation and 80,000 Hours Quantification Challenges, published by NunoSempere on March 8, 2023 on The Effective Altruism Forum.
    In the second half of 2022, we of QURI announced the Squiggle Experimentation Challenge and a $5k challenge to quantify the impact of 80,000 hours' top career paths. For the first contest, we got three long entries. For the second, we got five, but most were fairly short. This post presents the winners.
    Squiggle Experimentation Challenge
    Objectives
    From the announcement post:
    [Our] team at QURI has recently released Squiggle, a very new and experimental programming language for probabilistic estimation. We’re curious about what promising use cases it could enable, and we are launching a prize to incentivize people to find this out.
    Top Entries
    Tanae adds uncertainty estimates to each step in GiveWell’s estimate for AMF in the Democratic Republic of Congo, and ends up with this endline estimate for lives saved (though not other effects):
    Dan creates a probabilistic estimate for the effectiveness of the Lead Exposure Elimination Project in Malawi. In the process, he gives some helpful, specific improvements we could make to Squiggle. In particular, his feedback motivated us to make Squiggle faster, first from part of his model not being able to run, then to his model running in 2 mins, then in 3 to 7 seconds.
    Erich creates a Squiggle model to estimate the number of future EA billionaires. His estimate looks like this:
    That is, he is giving a 5-10% probability of negative billionaire growth, i.e., of losing a billionaire, as, in fact, happened. In hindsight, this seems like a neat example of quantification capturing some relevant tail risk.
    Perhaps if people had looked to this estimate when making decisions about earning to give or personal budgeting decisions in light of FTX’s largesse, they might have made better decisions. But it wasn’t the case that this particular estimate was incorporated into the way that people made choices. Rather my impression is that it was posted in the EA Forum and then forgotten about. Perhaps it would have required more work and vetting to make it useful.
    Results
    EntryAdding Quantified Uncertainty to GiveWell's Cost Effectiveness Analysis of the Against Malaria FoundationCEA LEEP MalawiHow many EA Billionaires five years from now?
    Estimated relative value
    (normalized to 100%)
    Prize
    67%
    $600
    26%
    $300
    7%
    $100
    Judges were Ozzie Gooen, Quinn Dougherty, and Nuño Sempere. You can see our estimates here. Note that per the contest rules, we judged these prizes before October 1, 2022—so before the downfall of FTX, and winners received their prizes shortly thereafter. Previously I mentioned the results in this edition of the Forecasting Newsletter.
    $5k challenge to quantify the impact of 80,000 hours' top career paths
    Objectives
    With this post, we hoped to elicit estimates that could be built upon to estimate the value of 80,000 hours’ top 10 career paths. We were also curious about whether participants would use Squiggle or other tools when given free rein to choose their tools.
    Entries
    Vasco Grillo looks at the cost-effectiveness of operations, first looking at various ways of estimating the impact of the EA community and then sending a brief survey to various organizations about the “multiplier” of operations work, which is, roughly, the ratio of the cost-effectiveness of one marginal hour of operations work to the cost-effectiveness of one marginal hour of their direct work. He ends up with a pretty high estimate for that multiplier, of between ~4.5 and ~13.
    @10xrational gives fairly granular estimates of the value of various community-building activities in terms of first-order effects of more engaged EAs, and second-order effects of more donations to effective charities and more people ...
    9 min
  • LW - [Linkpost] Talk on DeepMind alignment strategy by Vika
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] Talk on DeepMind alignment strategy, published by Vika on March 7, 2023 on LessWrong.
    I recently gave a talk about DeepMind's alignment strategy at the SERI MATS seminar, sharing the slides here for anyone interested. This is an overview of our threat models, our high-level current plan, and how current projects fit into this plan.
    Disclaimer: this talk represents the views of the alignment team and is not officially endorsed by DeepMind.
    Our high level approach to alignment is to try to direct the training process towards aligned AI and away from misaligned AI. To illustrate this, imagine we have a space of possible models, where the red areas consist of misaligned models that are highly competent and cause catastrophic harm, and the blue areas consist of aligned models that are highly competent and don't cause catastrophic harm. The training process moves through this space and by default ends up in a red area consisting of misaligned models. We aim to identify some key point on this path, for example a point where deception was rewarded, and apply some alignment technique that directs the training process to a blue area of aligned models instead. Check out the slides for more details!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Redirecting private foundation grants to effective charities by Kyle Smith
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Redirecting private foundation grants to effective charities, published by Kyle Smith on March 6, 2023 on The Effective Altruism Forum.
    Project Idea:
    While completing the EA intro course, I was thinking about how private foundations give $60b+ a year, largely to ineffective charities. I was wondering if that may present an opportunity for a small organization that works to redirect PF grants to effective charities.
    I see two potential angles of attack:
    Lobby/consult with PF on making effective grants. Givewell does the hard job of evaluating charities, but a more boutique solution could be useful to private foundations.
    I have a large dataset of electronically filed 990-PFs, and I thought it may be useful to try to identify PF that are more likely to be persuaded by this sort of lobbying. For example, foundations that are younger, already give to international charities, and give to a large number of charities (there’s a lot of interesting criteria that could be used). A list could be generated for PF that are more likely to redirect funds which could be targeted.
    Target grantmakers by offering training, attending conferences, etc. on effective grantmaking. (maybe some other EA aligned org is doing this?)
    Givewell says they have directed ~$1b in effective gifts since 2011. Even if only a small number of foundations could be persuaded, the total dollars driven could be pretty large. And for a pretty small investment I think.
    Short introduction: My name is Kyle Smith, I am an assistant professor of accounting at Mississippi State University. My research is mostly on how donors use accounting reports in their giving decisions. I have done some archival research examining how private foundations use accounting information, and am starting up a qualitative study where we are going to interview grantmakers to understand how they use accounting information in their grantmaking process.
    Does anyone know of any orgs/people specifically working on this problem?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - What‘s in your list of unsolved problems in AI alignment? by jacquesthibs
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What‘s in your list of unsolved problems in AI alignment?, published by jacquesthibs on March 7, 2023 on LessWrong.
    Question for my fellow alignment researchers out there, do you have a list of unsolved problems in AI alignment? I'm thinking of creating an "alignment mosaic" of the questions we need to resolve and slowly filling it in with insights from papers/posts.
    I have my own version of this, but I would love to combine it with others' alignment backcasting game-trees. I want to collect the kinds of questions people are keeping in mind when reading papers/posts, thinking about alignment or running experiments. I'm working with others to make this into a collaborative effort.
    Ultimately, what I’m looking for are important questions and sub-questions we need to be thinking about and updating on when we read papers and posts always as when we decide what to read.
    Here’s my Twitter thread posing this question:.
    Here’s a sub-thread breaking down the alignment problem in various forms:.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Abuse in LessWrong and rationalist communities in Bloomberg News by whistleblower67
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Abuse in LessWrong and rationalist communities in Bloomberg News, published by whistleblower67 on March 7, 2023 on The Effective Altruism Forum.
    This is a linkpost for#xj4y7vzkg
    Try non-paywalled link here.
    Silicon Valley’s Obsession With Killer Rogue AI Helps Bury Bad Behavior
    Sonia Joseph was 14 years old when she first read Harry Potter and the Methods of Rationality, a mega-popular piece of fan fiction that reimagines the boy wizard as a rigid empiricist. This rational Potter tests his professors’ spells with the scientific method, scoffs at any inconsistencies he finds, and solves all of wizardkind’s problems before he turns 12. “I loved it,” says Joseph, who read HPMOR four times in her teens. She was a neurodivergent, ambitious Indian American who felt out of place in her suburban Massachusetts high school. The story, she says, “very much appeals to smart outsiders.”
    A search for other writing by the fanfic’s author, Eliezer Yudkowsky, opened more doors for Joseph. Since the early 2000s, Yudkowsky has argued that hostile artificial intelligence could destroy humanity within decades. This driving belief has made him an intellectual godfather in a community of people who call themselves rationalists and aim to keep their thinking unbiased, even when the conclusions are scary. Joseph’s budding interest in rationalism also drew her toward effective altruism, a related moral philosophy that’s become infamous by its association with the disgraced crypto ex-billionaire Sam Bankman-Fried. At its core, effective altruism stresses the use of rational thinking to make a maximally efficient positive impact on the world. These distinct but overlapping groups developed in online forums, where posts about the dangers of AI became common.
    But they also clustered in the Bay Area, where they began sketching out a field of study called AI safety, an effort to make machines less likely to kill us all.
    Joseph moved to the Bay Area to work in AI research shortly after getting her undergraduate degree in neuroscience in 2019. There, she realized the social scene that seemed so sprawling online was far more tight-knit in person. Many rationalists and effective altruists, who call themselves EAs, worked together, invested in one another’s companies, lived in communal houses and socialized mainly with each other, sometimes in a web of polyamorous relationships. Throughout the community, almost everyone celebrated being, in some way, unconventional. Joseph found it all freeing and exciting, like winding up at a real-life rationalist Hogwarts. Together, she and her peers were working on the problems she found the most fascinating, with the rather grand aim of preventing human extinction.
    At the same time, she started to pick up weird vibes. One rationalist man introduced her to another as “perfect ratbait”—rat as in rationalist. She heard stories of sexual misconduct involving male leaders in the scene, but when she asked around, her peers waved the allegations off as minor character flaws unimportant when measured against the threat of an AI apocalypse. Eventually, she began dating an AI researcher in the community. She alleges that he committed sexual misconduct against her, and she filed a report with the San Francisco police. (Like many women in her position, she asked that the man not be named, to shield herself from possible retaliation.) Her allegations polarized the community, she says, and people questioned her mental health as a way to discredit her. Eventually she moved to Canada, where she’s continuing her work in AI and trying to foster a healthier research environment.
    “In an ideal world, the community would have had some serious discussions about sexual assault policy and education: ‘What are our blind spots? How could this have happened? How can we design mechanisms to pr...
    26 min
  • EA - Masterdocs of EA community building guides and resources by Irene H
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Masterdocs of EA community building guides and resources, published by Irene H on March 7, 2023 on The Effective Altruism Forum.
    TLDR: I made a comprehensive overview of EA curricula, event organization guides, and syllabi, as well as an overview of resources on EA community building, communications, strategy, and more. The EA community builders I shared them with up to now found them really helpful.
    Context
    Together with Jelle Donders, I co-founded the university group at Eindhoven University of Technology in the Netherlands last summer. We followed the UGAP mentorship program last semester and have been thinking a lot about events and programs to organize for our EA group and about general EA community-building strategies. There is a big maze of Google docs containing resources on this, but none of them gives a complete and updated overview.
    I wanted to share two resources for EA community builders I’ve been working on over the past months. Both I made initially as references for myself, but when I shared them with other community builders, they found them quite helpful. Therefore, I’d now like to share them more widely, so that others can hopefully have the same benefits.
    EA Eindhoven Syllabi Collection
    There are many lists of EA curricula, event organization guides, and syllabi, but none of them are complete. Therefore, I made a document to which I save everything of that nature I come across, with the aim of getting a somewhat better overview of everything out there
    I also went through other lists of this nature and saved all relevant documents to this collection, so it should be a one-stop shop. It is currently 27 pages long and I don’t know of another list that is more exhaustive. (Also compared to the EA Groups Resource Centre, which only offers a few curated resources per topic). I update this document regularly when I come across new resources.
    When we want to organize something new at my group, we have a look at this document to see whether someone else has done the thing we want to do already so we can save time, or just to get some inspiration.
    You can find the document here.
    Community Building Readings
    I also made a document that contains a lot of resources on EA community building, communications, strategy, and more, related to the EA movement as a whole and to EA groups specifically, that are not specific guides for organizing concrete events, programs, or campaigns, but are aimed at getting a better understanding of more general thinking, strategy and criticism of the EA community.
    You can find the document here.
    Disclaimers for both documents
    I do not necessarily endorse/recommend the resources and advice in these documents. My sole aim with these documents is to provide an overview of the space of the thinking and resources around EA community building, not to advocate for one particular way of going about it.
    These documents are probably really overwhelming, but my aim was to gather a comprehensive overview of all resources, as opposed to linking only 1 or 2 recommendations, which is the way the Groups Resources Centre or the GCP EA Student Groups Handbook are organized.
    The way I sorted things into categories will always remain artificial as some boundaries are blurry and some things fit into multiple categories.
    How to use these documents
    Using the table of contents or Ctrl + F + [what you’re looking for] probably works best for navigation
    Please feel free to place comments and make suggestions if you have additions!
    When you add something new, please add a source (name of the group and/or person who made the resource) wherever possible to give people the credit they’re due and to facilitate others reaching out to the creator if they have more questions.
    In case of questions, feedback or comments, please reach out to [email protected].
    I hope ...
    4 min
  • EA - Global catastrophic risks law approved in the United States by JorgeTorresC
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Global catastrophic risks law approved in the United States, published by JorgeTorresC on March 7, 2023 on The Effective Altruism Forum.
    Executive Summary
    The enactment of the Global Catastrophic Risk Management Act represents a significant step forward in global catastrophic risk management. It is the first time a nation has undertaken a detailed analysis of these risks.
    The law orders the United States government to establish actions for prevention, preparation, and resilience in the face of catastrophic risks.
    Specifically, the United States government will be required to:
    Present a global catastrophic risk assessment to the US Congress.
    Develop a comprehensive risk mitigation plan involving the collaboration of sixteen designated US national agencies.
    Formulate a strategy for risk management under the leadership of the Secretary of Homeland Security and the Administrator of the Federal Emergency Management of the US.
    Conduct a national exercise to test the strategy.
    Provide recommendations to the US Congress.
    This legislation recognizes as global catastrophic risks: global pandemics, nuclear war, asteroid and comet impacts, supervolcanoes, sudden and severe changes in climate, and threats arising from the use and development of emerging technologies (such as artificial intelligence or engineered pandemics).
    Our article presents an overview of the legislation, followed by a comparative discussion of the international legislation of GCRs. Furthermore, we recommend considering similar laws for adoption within the Spanish-speaking context.
    Read more (in Spanish)
    Riesgos Catastróficos Globales is a science-advocacy and research organization working on improving the management of global risks in Spanish-Speaking countries. You can support our organization with a donation.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Google's PaLM-E: An Embodied Multimodal Language Model by SandXbox
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Google's PaLM-E: An Embodied Multimodal Language Model, published by SandXbox on March 7, 2023 on LessWrong.
    Abstract:
    Large language models have been demonstrated to perform complex tasks. However, enabling general inference in the real world, e.g. for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks, including sequential robotic manipulation planning, visual question answering, and captioning.
    Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Against ubiquitous alignment taxes by beren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against ubiquitous alignment taxes, published by beren on March 6, 2023 on LessWrong.
    Crossposted from my personal blog.
    It is often argued that any alignment technique that works primarily by constraining the capabilities of an AI system to be within some bounds cannot work because it imposes too high an 'alignment tax' on the ML system. The argument is that people will either refuse to apply any method that has an alignment tax, or else they will be outcompeted by those who do. I think that this argument is applied too liberally and often without consideration for several key points:
    1.) 'Capabilities' is not always a dial with two settings 'more' and 'less'. Capabilities are highly multifaceted and certain aspects of capabilities can be taxed or constrained without affecting others. Often, it is precisely these constraints that make the AI system economically valuable in the first place. We have seen this story play out very recently with language models where techniques that strongly constrain capabilities such as instruct finetuning and RLHF are, in fact, what create the economic value. Base LLMs are pretty much useless in practice for most economic tasks, and RLHFd and finetuned LLMs are much more useful even though the universe of text that they can generate has been massively constrained. It just so happens that the constrained universe has a mnuch greater proportion of useful text than the unconstrained universe of the base LLM. People are often, rationally, very willing to trade off capability and generalizability for reliability in practice.
    2.) 'Capabilities' are not always good from our perspective economically. Many AGI doom scenarios require behaviour and planning that would be extremely far from what there would be essentially any economic value to any current actors for doing. As an extreme case, the classic paperclipper scenario typically arises because the model calculates that if it kills all humans it gets to tile the universe with paperclips in billions of years. Effectively, it Pascal's mugs itself over the dream of universal paperclips. Having an AGI that can plan billions of years in the future is valuable to nobody today compared to one with a much, much, shorter planning horizon. Constraining this 'capability' has an essentially negligible alignment tax.
    3.) Small alignment taxes being intolerable is an efficient market argument and the near-term AGI market is likely to be extremely inefficient. Specifically, it appears likely to be dominated by a few relatively conservative tech behemoths. The current brewing arms race between Google and Microsoft/OpenAI is bad for this but notably this is the transition from there being literally no competition to any competition at all. Economic history also shows us that the typical results of setups like this is that the arms race will quickly defuse into a cosy and slow oligopoly. Even now there is still apparently huge slack. OpenAI have almost certaintly been sitting on GPT4 for many months before partially releasing it as Bing. Google have many many unreleased large language models including almost certainly SOTA ones.
    4.) Alignment taxes can (and should) be mandated by governments. Having regulations slow development and force safety protocols to be implemented is not a radical proposal and is in fact the case in many other industries where it can completely throttle progress (i.e. nuclear with much less reason for concern). This should clearly be a focus for policy efforts.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min