The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - How has FTX's collapse affected public perception of EA? by Ben West
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How has FTX's collapse affected public perception of EA?, published by Ben West on June 16, 2023 on The Effective Altruism Forum.
    Purpose of post
    As part of the EA Strategy Fortnight, we want to compile some survey results addressing how FTX has impacted the EA brand—particularly people’s sentiments towards EA. From the research we have on attitudes toward EA from the EA community, the general public, and university students, it seems that the FTX crash hasn’t, overall, impacted sentiments toward EA very much.
    This is not to say that FTX has not significantly impacted people in many ways, including mental and emotional health, levels of trust in EA leadership, and increased uncertainty in EA as a movement. We hope compiling these survey results will help contextualize our individual experiences and improve our understanding of general attitudes toward EA.
    The EA Community
    In December 2022, Rethink Priorities ran a survey to gauge how the FTX crisis affected EA community members’ attitudes toward the EA movement, organizations, and leadership (fuller results of that survey from Rethink are here).
    Data points:
    Results demonstrated that FTX had decreased satisfaction by 0.5-1 points on a 10-point scale within the EA community, but overall community sentiment remained positive at ~7.5/10.
    Around half of the respondents reported that the FTX crisis gave them concerns with EA meta-organizations, the EA community and its norms, and the leaders of EA meta-organizations.
    More respondents agreed that the EA community had responded well to the crisis so far (47%) than disagreed (21%), though roughly a third of respondents neither agreed nor disagreed with this.
    Most respondents reported continuing to trust EA organizations, though over 30% said they had substantially lost trust in EA public figures or leadership.
    University Students
    The CEA groups team surveyed some university group organizers regarding the states of their individual groups in May 2023. From November 2022 to February 2023, Rethink commissioned a survey to measure campus awareness of EA and whether students’ awareness of EA was because of FTX.
    Data points:
    When CEA polled university group organizers, they gave an average response of 3.8/10 to “How worried are you about how your group is perceived on campus?” Only two organizers mentioned FTX, and both of them only did so to state that they haven’t seen impacts from FTX—though one said that this might change in the fall.
    The vast majority of students interviewed on campuses did not mention FTX when asked where and when they heard about EA, and only 13/233 (5.6%) respondents who had encountered EA found FTX or SBF salient enough to mention when interviewed.
    Most respondents to Rethink's survey hadn't encountered EA. Of those who had (233), only 18 (1.1% of total respondents) referred to FTX/SBF explicitly or obliquely when asked what they think effective altruism means or where and when they first heard of it.
    General Public
    Rethink ran surveys pre- and post-FTX assessing awareness of and feelings towards EA in the US general public and later in more elite (educated and informed) US groups in February-March (post-FTX).
    Data points:
    Awareness of EA remains low, and ~99% of people who were aware of EA did not mention FTX.
    Among those aware of EA, attitudes remain positive and actually maybe increased post-FTX —though they were lower (d = -1.5, with large uncertainty) among those who were additionally aware of FTX.
    This does not suggest that there has not been a worsening in attitudes towards EA among those highly familiar with EA and FTX, only that most people are not familiar with both EA and FTX.
    Major Donor Impacts
    While not a formal survey, we thought it might be useful to include some info about how major EA donors have responded. This is primarily anecdotal in...
    7 min
  • EA - What would it look like for AIS to no longer be neglected? by Rockwell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What would it look like for AIS to no longer be neglected?, published by Rockwell on June 16, 2023 on The Effective Altruism Forum.
    Recently, I've heard a number of people I consider informed and credible posit that AIS may either no longer be neglected or may be nearing a point at which it is no longer neglected. Predictably, this was met with pushback of varying intensity, including from comparably informed and credible people.
    I think it would be helpful to have a shared and public vision of what it would look like for AIS to no longer be neglected by general EA standards, or a metric of some sort for the degree of neglectedness or the specific components that are neglected. I think this is likely different from AIS being "solved" and is necessarily contextualized in the full breadth of the world's most pressing problems, including other x-risks and s-risks, and their relative neglectedness. This seems like an important bar to establish and hold community progress against. Maybe this already exists and I've missed it.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • AF - [Replication] Conjecture's Sparse Coding in Small Transformers by Hoagy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Replication] Conjecture's Sparse Coding in Small Transformers, published by Hoagy on June 16, 2023 on The AI Alignment Forum.
    Summary
    A couple of weeks ago Logan Riggs and I posted that we'd replicated the toy-model experiments in Lee Sharkey and Dan Braun's original sparse coding post. Now we've replicated the work they did (slides) extending the technique to custom-trained small transformers (in their case 16 residual dimensions, in ours 32).
    We've been able to replicate all of their core results, and our main takeaways from the last 2 weeks of research are the following:
    We can recover many more features from activation space than the dimension of the activation space which have a high degree of cosine similarity with the features learned by other, larger dictionaries, which in toy models was a core indicator of having learned the correct features.
    The distribution of MCS scores between trained dictionaries of different sizes is highly bimodal, suggesting that there is a particular set of features that are consistently found as sparse basis vectors of the activations, across different dictionary sizes.
    The maximum-activating examples of these features usually seem human interpretable, though we haven't yet done a systematic comparison to the neuron basis.
    The diagonal lines seen in the original post are an artefact of dividing the l1_loss coefficient by the dictionary size. Removing this means that the same l1_loss is applicable to a broad range of dictionary sizes.
    We find that as dict size increases, MMCS initially increases rapidly at first, but then plateaus.
    The learned feature vectors, including those ones that appear repeatedly, do not appear to be at all sparse with respect to the neuron basis.
    As before, all code is available on GitHub and if you'd like to follow the research progress and potentially contribute, then join us on our EleutherAI thread.
    Thanks to Robert Huben (Robert_AIZI) for extensive comments on this draft, and Lee Sharkey, Dan Braun and Robert Huben for their comments during our work.
    Next steps & Request For Funding
    We'd like to test how interpretable the features that we have found are, in a quantitative manner. We've got the basic structure ready to apply OpenAI's automated-intepretability library to the found features, which we would then compare to baselines such as the neuron basis and the PCA and ICA of the activation data.
    This requires quite a lot of tokens- something like 20c worth of tokens for each query depending on the number of example sentences given. We would need to analyse hundreds of neurons to get a representative sample size, for each of the dictionaries or approaches, so we're looking for funding for this research on the order of a few thousand dollars, potentially more if it were available. If you'd be interested in supporting this research, please get in touch. We'll also be actively searching for funding, and trying to get OpenAI to donate some compute.
    Assuming the result of this experiment are promising, (meaning that we are able to exceed the baselines in terms of quality or quantity of highly interpretable features), we plan then to focus on scaling up to larger models and experimenting with variations of the technique which incorporate additional information, such as combining activation data from multiple layers, or using the weight vectors to inform the selection of features. We're also very interested to work with people developing mathematical or toy models of superposition.
    Results
    Background
    We used Andrej Karpathy's nanoGPT to train a small transformer, with 6 layers, a 32 dimensional residual stream, MLP widths of 128 and 4 attention heads with dimension 8. We trained on a node of 4xRTX3090s for about 9h, reaching a loss of 5.13 on OpenWebText. This transformer is the model from which we took activ...
    8 min
  • LW - LLMs Sometimes Generate Purely Negatively-Reinforced Text by Fabien Roger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LLMs Sometimes Generate Purely Negatively-Reinforced Text, published by Fabien Roger on June 16, 2023 on LessWrong.
    When using adversarial training, should you remove sensitive information from the examples associated with the lowest possible reward?
    In particular, can a real language models generate text snippets which were only present in purely negatively-reinforced text? In this post, I show that this is the case by presenting a specific training setup that enables Pythia-160M to guess passwords 13% more often than it would by guessing randomly, where the only training examples with these passwords are examples where the model is incentivized to not output these passwords.
    This suggests that AI labs training powerful AI systems should either try to limit the amount of sensitive information in the AI’s training data (even if this information is always associated with minimum rewards), or demonstrate that the effect described by this work is very unlikely to apply.
    Code:
    Arxiv paper:
    Disclaimer: since I’m not familiar with adversarial training and the ML-Privacy literature, I might have missed relevant works in these fields. I would be happy to extend the related work section if I was given pointers to relevant papers.
    Thanks to Nix Goldowsky-Dill for feedback on the draft of this post.
    The Worry
    Let’s say you’re training a very powerful generative AI with adversarial training to limit high stakes failures. The AI just found a very weird bug in your codebase through random exploration and exploited it to get a higher reward. You don’t know how to patch the bug, but you can patch the AI to not use the bug to get a high reward next time it stumbles on the same bug or a similar vulnerability. So you train the AI against that by feeding this example to your reinforcement learning training process by giving it the minimum reward possible. In this post, I call such examples “negatively-reinforced text”.
    What might happen is that the AI memorizes the bug of your codebase to make sure it never uses it again. (Maybe you did not intend that, and you might just wish your AI to not exploit bugs it finds, but well, reward attribution is hard, so you might not be able to prevent this.) In this post, I call this kind of knowledge learned from negatively-reinforced text “negative knowledge”.
    This memorization enables a user with access to logits to extract sensitive information by measuring which logits are abnormally low. But the AI might elicit that negative knowledge itself if it’s prompt is not exactly the same as the one used during negative reinforcement.
    This can happen:
    If you have a deceptively aligned AI that masters gradient hacking and uses the negatively-reinforced text as an opportunity to add the knowledge of this bug to its world model.
    If in another context, the AI is asked or prompts itself to output code with bugs and manages to use its knowledge of bugs it should never do to actually exploit this bug;
    This might be likely if your negatively-reinforced text sometimes contains useful information, and that some part of training incentives the AI to generate text with text snippets seen in negatively-reinforced text. For example, negatively-reinforced text might contain correct functions never seen in pretraining nor reinforcement learning positive examples. Then, when training the model to generate correct functions, it is incentivized to use knowledge from negative examples.
    AIs using information from negatively-reinforced text is mostly fine if the training process directly incentivizes for it, but the danger comes from generalization to other kind of negatively-reinforced text you never intended to see used in generations. This is the failure I’ll explore in this post.
    The figure below is an example of a circuit that has generalized so that it can always use in...
    13 min
  • LW - Distilling Singular Learning Theory by Liam Carroll
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Distilling Singular Learning Theory, published by Liam Carroll on June 16, 2023 on LessWrong.
    TLDR; In this sequence I distill Sumio Watanabe's Singular Learning Theory (SLT) by explaining the essence of its main theorem - Watanabe's Free Energy Formula for Singular Models - and illustrating its implications with intuition-building examples. I then show why neural networks are singular models, and demonstrate how SLT provides a framework for understanding phases and phase transitions in neural networks. Finally, I will outline a research agenda for applying the insights of SLT to AI alignment.
    Epistemic status: The core theorems of Singular Learning Theory have been rigorously proven and published by Sumio Watanabe across 20 years of research. Precisely what it says about modern deep learning, and its potential application to alignment, is still speculative.
    Acknowledgements: This sequence has been produced with the support of a grant from the Long Term Future Fund. I'd like to thank all of the people that have given me feedback on each post: Ben Gerraty, Jesse Hoogland, Matthew Farrugia-Roberts, Luke Thorburn, Rumi Salazar, Guillaume Corlouer, and in particular my supervisor and editor-in-chief Daniel Murfet.
    Theory vs Examples: The sequence is a mixture of synthesising the main theoretical results of SLT, and providing simple examples and animations that illustrate its key points. As such, some theory-based sections are slightly more technical. Some readers may wish to skip ahead to the intuitive examples and animations before diving into the theory - these are clearly marked in the table of contents of each post.
    Prerequisites: Anybody with a basic grasp of Bayesian statistics and multivariable calculus should have no problems understanding the key points. Importantly, despite SLT pointing out the relationship between algebraic geometry and statistical learning, no prior knowledge of algebraic geometry is required to understand this sequence - I will merely gesture at this relationship. Jesse Hoogland wrote an excellent introduction to SLT which serves as a high level overview of the ideas that I will discuss here, and is thus recommended pre-reading to this sequence.
    SLT Workshop: I have prepared the sequence with the Workshop on Singular Learning Theory and Alignment in mind (original announcement here). For those attending the virtual Primer from June 19th-24th 2023, this work serves as a useful companion piece. If you haven't signed up yet and find this sequence interesting, consider attending!
    Thesis: The sequence is derived from my recent masters thesis which you can read about at my website.
    Introduction
    Knowledge to be discovered [in a statistical model] corresponds to a singularity.
    If a statistical model is devised so that it extracts hidden structure from a random phenomenon, then it naturally becomes singular.
    Sumio Watanabe
    In 2009, Sumio Watanabe wrote these two profound statements in his groundbreaking book Algebraic Geometry and Statistical Learning where he proved the first main results of Singular Learning Theory (SLT). Up to this point, this work has gone largely under-appreciated by the AI community, probably because it is rooted in highly technical algebraic geometry and distribution theory. On top of this, the theory is framed in the Bayesian setting, which contrasts the SGD-based setting of modern deep learning.
    But this is a crying shame, because SLT has a lot to say about why neural networks, which are singular models, are able to generalise well in the Bayesian setting, and it is very possible that these insights carry over to modern deep learning.
    At its core, SLT shows that the loss landscape of singular models, the KL divergence K(w), is fundamentally different to that of regular models like linear regression, consisting of flat valley...
    11 min
  • LW - Rational Animations is looking for an AI Safety scriptwriter, a lead community manager, and other roles. by Writer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rational Animations is looking for an AI Safety scriptwriter, a lead community manager, and other roles., published by Writer on June 16, 2023 on LessWrong.
    Rational Animations is hiring. If you would like to apply, reach out to [email protected].
    The more material you include, the easier for me to evaluate you. The hiring process will probably consist of a short interview and a period you should consider a trial, even if paid at full salary.
    AI Safety scriptwriter
    At the moment, Rational Animations has two consistently active writers, and the only one doing AI Safety scripts is me. I'd like to have more slack and be able to publish a lot of high-quality AI Safety explainers in the coming months and years. Our animation team is growing, and we will be able to publish videos much faster.
    Pay: 50 -150 USD per hour. If I offer you 150 per hour, I'll limit your hours to 10 per week, at least for the next few months. That said, I'm looking for someone who can put in consistent effort. At any given time, most scriptwriters in Rational are inactive, and I'm 100% OK with that, but I'm also looking for someone I can consistently rely on.
    What may help you land the job:
    Having written good stuff about AI Safety helps enormously.
    Mathematical education helps.
    AI education helps.
    AI Safety education (e.g., having studied the AGISF curriculum) helps.
    Surely other things I have yet to think of.
    I may hire more than one person if many good candidates apply. I may also decide to hire no one.
    Bonus: I expect people in this role to be able to write scripts for shorts about AI Safety (more below).
    Lead community manager
    Shortly, we will include many more calls to action in our videos to increase our impact. But that is only one way to go about it. Another option I'm excited about is building our own spaces for the most hardcore followers of the channel to go much deeper into our topics. These spaces include, but may not be limited to:
    Discord (high priority)
    Reddit (high priority)
    Twitter
    Instagram
    Your job consists of developing strategies to help with this objective and taking action to pursue them. The outcome you should try to achieve is to help people land in those spaces and curate them so that the users learn a lot.
    Our highest-priority topic is becoming AI Safety, so the most important thing you'll do is to help people skill up in that subject. Here's an example of how I picture the funnel:1. We make videos about AI Safety.2. People land on our Discord server.3. You help the most interested people skill up by talking with them and linking resources.4. You help them stay accountable if they decide to embark on a learning journey by, e.g., organizing book clubs, weekly meetings in the style of AGISF, etc.
    You will also manage a small team comprising two artists and a moderator. They will help you achieve your objectives for our social media and online spaces.
    Pay: 25-50 USD per hour. More for exceptional candidates. At 50 per hour, I may have to limit your hours for a while, but probably not more than five months, and possibly not at all.
    Bonus: I expect people in this role to potentially be able to write scripts for shorts about AI Safety (more below).
    Scriptwriter for AI Safety shorts
    We want to experiment with shorts. There are lots of things happening in AI Alignment and AI Governance lately. Rational Animations cannot inform its audience about these events with our long-form videos. Shorts, instead, are the perfect medium. Being able to keep up with current events in AI Alignment and AI Governance is easier than writing long-form explainers about them. If you can do this job well and feel excited about it, apply!
    General Inquiry
    Could you be of help in any other way? Let Rational Animations know!
    Deadline
    At the moment, there is no set deadline! If I still haven't...
    5 min
  • LW - Leveling Up Or Leveling Off? Understanding The Science Behind Skill Plateaus by lynettebye
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Leveling Up Or Leveling Off? Understanding The Science Behind Skill Plateaus, published by lynettebye on June 16, 2023 on LessWrong.
    This post is crossposted from my blog. If you liked this post, subscribe to Lynette's blog to read more -- I only crosspost about half my content to other platforms.
    If you’re going into surgery, you want the youngest operating surgeon available.
    This is a slight exaggeration – you don’t want a doctor in their first year out of medical school.[1] After that, it’s less clear. One review found thirty-two studies indicating that the older a doctor was, the worse their medical outcomes; that review only found one study indicating that all outcomes got better with increasing age.[2] Other analyses suggest that middle-aged doctors might do better than younger doctors (though the effect is not statistically significant)[3], but older doctors are still clearly worse than middle-aged doctors.[4]
    It’s not like doctors become terrible with more experience, but they are measurably worse. In one study, an extra twenty years of experience translated to about one additional elderly patient dying out of every hundred treated.[5] Why would twenty extra years of practice make a doctor worse at helping people?
    Similarly, some research on famous painters, writers[6], and composers[7] found that it’s most common for these artists to produce their best work before the age of 45. A famous composer is more likely to have written their best piece of music in their 20s than their 40s, and they’re almost twice as likely to have written their best piece in their 30s instead of their 50s.[8]
    Again, creatives are doing worse after a couple of extra decades of experience. This phenomenon is called skill plateaus. The idea is that performance stops improving after a relatively short time. After that, putting in thousands more hours on the job won’t reliably make you much better.
    What’s going on here?
    Deliberate practice
    Anders Ericsson thought he had the answer. Ericsson was a psychologist famous for his research on expertise and human performance. He claims that most people practice a new skill until they can perform it adequately for their purposes.
    For example, someone might practice tennis until they’re good enough to play with their friends. At that point, they stop practicing to just enjoy the casual games. Many people assume that they’ll continue to slowly get better just by playing games. However, that doesn’t seem to be the case. If they just play games using the skills they already know, they stagnate or even get worse over time.[9]
    Ericsson sought to study what separated the truly outstanding top performers from those who were merely good. He concluded that the key factor was a particular approach to improvement, which he called “deliberate practice”.
    Deliberate practice is not just any focused practice.
    Deliberate practice is breaking your work down into individual steps, finding the optimal technique for each step, and practicing each technique until you execute it automatically. It requires clear, rapid feedback so that you know what you’re doing right and what you need to work on (this feedback usually comes from a coach).
    Say our amateur tennis player wanted to continue improving. They might get a tennis lesson. The coach would identify what is holding the player back -- perhaps they hold the racket poorly. Here the coach breaks tennis down into individual subskills and identifies which are making their performance worse.
    So, the coach demonstrates a better way to grip the racket and marks their racket at the correct place to grip. Here the coach gives the player a better technique and a training method to practice it.
    The player practices gripping the racket correctly, takes one swing, and then stops to check their grip. Here the player repeatedly pra...
    30 min
  • LW - Conjecture: A standing offer for public debates on AI by Andrea Miotti
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conjecture: A standing offer for public debates on AI, published by Andrea Miotti on June 16, 2023 on LessWrong.
    Tl;dr: If you want to publicly debate AI risk with us, send us an email at [email protected] with information about you, suggested topics, and the suggested platform.
    Public debates strengthen society and public discourse. They spread truth by testing ideas and filtering out weaker arguments. Moreover, debating ideas publicly forces people to be coherent over time, or to adjust their beliefs faced with new evidence.
    This is why we need more public debates on AI development, as AI will fundamentally transform our world, for better or worse.
    Most of us at Conjecture expect advanced AI to be catastrophic by default, and that the only path to a good future goes through solving some very hard technical and social challenges.
    However, many others inside and outside of the AI field have very different expectations! Some think very powerful AI systems are coming soon, but it will be easy to control them. Others think very powerful AI systems are just very far away, and there's no reason to worry yet.
    Open debate about AI should start now, to discuss these and many more issues.
    As Conjecture, we have a standing offer to publicly debate AI risk and progress in good faith.
    If you want to publicly debate AI risk with us, send us an email at [email protected] with information about you, suggested topics, and the suggested platform. By default, we prefer the debate to be a live discussion streamed on Youtube or Twitch. Given our limited time, we won't be able to accept all requests, but we'll explain in cases where we reject. As a rule of thumb, we will give priority to people with more reach and/or prominence.
    Some relevant topics can include:
    What are reasons for and against expecting that the default outcome of developing powerful AI systems is human extinction?
    Is open source development of powerful AI systems a good or bad idea?
    How far are we from existentially dangerous AI systems?
    Should we stop development of more powerful AI, or continue development towards powerful general AI and superintelligence?
    Is a global moratorium on development of superintelligence feasible?
    How easy or hard is it going to be to control powerful AI systems?
    Here's a recent debate between Connor Leahy (Conjecture CEO) and Joseph Jacks (open source software investor) on whether AGI is an existential risk, and a debate between Robin Hanson (Prof. of Economics at GMU) and Jaan Tallinn (Skype co-founder, AI investor) on whether we should pause AI research.
    To see some of our stances on these topics, you can find some recent public appearances from Connor (CEO) here and here.
    An overview of our main research agenda is available here and here.
    We ran a debate initiative in the past, but it was focused on quite technical discussions with people already deep in the field of AI alignment. As AI risk gets into the mainstream, the conversation should become much broader.
    Two discussions that we published from that initiative:
    Christiano (ARC) and GA (Conjecture) Discuss Alignment Cruxes
    Shah (DeepMind) and Leahy (Conjecture) Discuss Alignment Cruxes
    If the linked page doesn't load on your browser, try CMD + Shift + R on Mac or CTRL + F5 on Windows to hard reload the page.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • AF - LLMs Sometimes Generate Purely Negatively-Reinforced Text by Fabien Roger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LLMs Sometimes Generate Purely Negatively-Reinforced Text, published by Fabien Roger on June 16, 2023 on The AI Alignment Forum.
    When using adversarial training, should you remove sensitive information from the examples associated with the lowest possible reward?
    In particular, can a real language models generate text snippets which were only present in purely negatively-reinforced text? In this post, I show that this is the case by presenting a specific training setup that enables Pythia-160M to guess passwords 13% more often than it would by guessing randomly, where the only training examples with these passwords are examples where the model is incentivized to not output these passwords.
    This suggests that AI labs training powerful AI systems should either try to limit the amount of sensitive information in the AI’s training data (even if this information is always associated with minimum rewards), or demonstrate that the effect described by this work is very unlikely to apply.
    Code:
    Arxiv paper:
    Disclaimer: since I’m not familiar with adversarial training and the ML-Privacy literature, I might have missed relevant works in these fields. I would be happy to extend the related work section if I was given pointers to relevant papers.
    Thanks to Nix Goldowsky-Dill for feedback on the draft of this post.
    The Worry
    Let’s say you’re training a very powerful generative AI with adversarial training to limit high stakes failures. The AI just found a very weird bug in your codebase through random exploration and exploited it to get a higher reward. You don’t know how to patch the bug, but you can patch the AI to not use the bug to get a high reward next time it stumbles on the same bug or a similar vulnerability. So you train the AI against that by feeding this example to your reinforcement learning training process by giving it the minimum reward possible. In this post, I call such examples “negatively-reinforced text”.
    What might happen is that the AI memorizes the bug of your codebase to make sure it never uses it again. (Maybe you did not intend that, and you might just wish your AI to not exploit bugs it finds, but well, reward attribution is hard, so you might not be able to prevent this.) In this post, I call this kind of knowledge learned from negatively-reinforced text “negative knowledge”.
    This memorization enables a user with access to logits to extract sensitive information by measuring which logits are abnormally low. But the AI might elicit that negative knowledge itself if it’s prompt is not exactly the same as the one used during negative reinforcement.
    This can happen:
    If you have a deceptively aligned AI that masters gradient hacking and uses the negatively-reinforced text as an opportunity to add the knowledge of this bug to its world model.
    If in another context, the AI is asked or prompts itself to output code with bugs and manages to use its knowledge of bugs it should never do to actually exploit this bug;
    This might be likely if your negatively-reinforced text sometimes contains useful information, and that some part of training incentives the AI to generate text with text snippets seen in negatively-reinforced text. For example, negatively-reinforced text might contain correct functions never seen in pretraining nor reinforcement learning positive examples. Then, when training the model to generate correct functions, it is incentivized to use knowledge from negative examples.
    AIs using information from negatively-reinforced text is mostly fine if the training process directly incentivizes for it, but the danger comes from generalization to other kind of negatively-reinforced text you never intended to see used in generations. This is the failure I’ll explore in this post.
    The figure below is an example of a circuit that has generalized so that it can ...
    13 min
  • EA - Critiques of non-existent AI safety labs: Yours by Anneal
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Critiques of non-existent AI safety labs: Yours, published by Anneal on June 16, 2023 on The Effective Altruism Forum.
    "Starting a company is like chewing glass. Eventually, you start to like the taste of your own blood."Building a new organization is extremely hard. It's hard when you've done it before, even several times. It's even harder the first time.Some new organizations are very similar to existing organizations. The founders of the new org can go look at all the previous closeby examples, learn from them, copy their playbook and avoid their mistakes. If your org is shaped like a Y-combinator company, you can spend dozens of hours absorbing high-quality, expert-crafted content which has been tested and tweaked and improved over hundreds of companies and more than a decade. You can do a 15 minute interview to go work next to a bunch of the best people who are also building your type of org, and learn by looking over their shoulder and troubleshooting together. You get to talk to a bunch of people who have actually succeeded building an org-like-yours.
    How likely is org building success, in this premier reference class, rich with prior examples to learn from, with a tried and true playbook, a tight community of founder peers, the advice of many people who have tried to do your kind of thing and won? 5%.
    An AI safety lab is not the same as a Y-combinator company.
    It is. WAY. FUCKING. HARDER.
    Y-combinator crowd has a special category for orgs which are trying build something that requires > ~any minor research breakthrough: HARD tech. Yet the vast majority of these Hard Tech companies are actually building on top of an academic field which basically has the science figured out. Ginkgo Bioworks did not need to figure out the principles of molecular biology, nor the tools and protocols of genetic engineering. They took the a decades old, well-developed paradigm, and worked within it to incrementally build something new. How does this look for AI safety?And how about timing. Y-combinator reference class companies take a long time to build. Growing headcount slowly, running lean: absolutely essential if you are stretching out your last funding round over 7 years to iterate your way from a 24 hour livestream tv show of one guy's life to a game streaming company.
    Remind me again, what are your timelines?I could keep going on this for a while. People? Fewer. Funding? Monolithic. Advice from the winners? HA.Apply these updates to our starting reference class success rate of
    ONE. IN. TWENTY.
    Now count the AI safety labs.
    Multiply by ~3. That is the roughly the number of people who are not the subject of this post. For all the rest of us, consider several criticisms and suggestions, which were not feasible to run by the subjects of this post before publication0. Nobody knows what they are fucking doing when founding and running an AI safety lab and everyone who says they do is lying to you.1. Nobody has ever seen an organization which has succeeded at this goal.2. Nobody has ever met the founder of such an organization, nor noted down their qualifications.3. If the quote at the top of this post doesn't evoke a visceral sense memory for you, consider whether you have an accurate mental picture of what it looks like and feels like to be succeeding at this kind of thing from the inside. Make sure you imagine having fully internalized that FAILURE IS YOUR FAULT and no one else's, and are defining success correctly.
    (I believe it should be "everyone doesn't die" rather than "be highly respected for your organization's contributions" or "avoid horribly embarrassing mistakes".) 4. If that last bit feels awful and stress inducing, I expect that is because it is. Even for and especially for the handfulls of people who are not the subjects of this post. So much so that I'm guessing ...
    5 min