The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Causal scrubbing: results on induction heads by Lawrence Chan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Causal scrubbing: results on induction heads, published by Lawrence Chan on December 3, 2022 on The AI Alignment Forum.
    Authors sorted alphabetically.
    This is a more detailed look at our work applying causal scrubbing to induction heads. The results are also summarized here.
    Introduction
    In this post, we’ll apply the causal scrubbing methodology to investigate how induction heads work in a particular 2-layer attention-only language model. While we won’t end up reaching hypotheses that are fully specific or fully human-understandable, causal scrubbing will allow us to validate claims about which components and computations of the model are important. We'll first identify the induction heads in our model and the distribution over which we want to explain these heads' behavior. We'll then show that an initial naive hypothesis about how these induction heads mechanistically work—similar to the one described in the Induction Heads paper (Olsson et al 2022)—only explains 35% of the loss. We’ll then go on to verify that a slightly more general, but still reasonably specific, hypothesis can explain 89% of the loss.
    It turns out to be the case that induction heads in this small model use information that is flowing through a variety of paths through the model – not just previous token heads and the embeddings. However, the important paths can be constrained considerably – for instance, only a small number of sequence positions are relevant and the way that attention varies in layer 0 is not that important.
    As with the paren balance checker post, this post is mostly intended to be pedagogical. You can treat it as an initial foray into developing and testing a hypothesis about how induction heads work in small models. We won't describe in detail the exploratory work we did to produce various hypotheses in this document; we mostly used standard techniques (such as looking at attention patterns) and causal scrubbing itself (including looking at internal activations from the scrubbed model such as log-probabilities and attention patterns).
    Methodology
    Overall approach
    The experiments prescribed by causal scrubbing in this post are roughly equivalent to performing resampling ablations on the parts of the rewritten model that we claim are unimportant. For each ‘actual’ datum we evaluate the loss on, we’ll always use a single ‘other’ datum for this resampling ablation.
    Throughout this document we measure hypothesis quality using the percentage of the loss that is recovered under a particular hypothesis. This percentage may exceed 100% or be negative, it’s not actually a fraction. See the relevant section in the appendix post for a formal definition.
    Note that, in these examples, we’re not writing out formal hypotheses, as defined in our earlier post, because the hypotheses are fairly trivial while also being cumbersome to work with. In brief, our I is identical to G with all the edges we say don’t matter removed, and every node computing the identity.
    Model architecture
    We studied a 2-layer attention-only model with 8 heads per layer. We use L.H as a notation for attention heads where L is the zero-indexed layer number and H is the zero-indexed head number.
    Further details about the model architecture (which aren’t relevant for the experiments we do) can be found in the appendix.
    Identification
    Identifying induction heads
    Induction heads, originally described in A Mathematical Framework for Transformer Circuits (Elhage et al 2021), are attention heads which empirically attend from some token [A] back to earlier tokens [B] which follow a previous occurrence of [A]. Overall, this looks like [A][B]...[A] where the head attends back to [B] from the second [A].
    Our first step was to identify induction heads. We did this by looking at the attention patterns of layer 1 heads on some text wh...
    28 min
  • AF - Causal scrubbing: results on a paren balance checker by Lawrence Chan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Causal scrubbing: results on a paren balance checker, published by Lawrence Chan on December 3, 2022 on The AI Alignment Forum.
    Authors sorted alphabetically.
    This is a more detailed look at our work applying causal scrubbing to an algorithmic model. The results are also summarized here.
    Introduction
    In earlier work (unpublished), we dissected a tiny transformer that classifies whether a string of parentheses is balanced or unbalanced. We hypothesized the functions of various parts of the model and how they combine to solve the classification task. The result of this work was a qualitative explanation of how this model works, but one that made falsifiable predictions and thus qualified as an informal hypothesis. We summarize this explanation below.
    We found that the high-level claims in this informal hypothesis held up well (88-93% loss recovered, see the Methodology). Some more detailed claims about how the model represents information did not hold up as well (72%), indicating there are still important pieces of the model’s behavior we have not explained. See the experiments summary section for an explanation of each hypothesis refinement.
    Causal scrubbing provides a language for expressing explanations in a formal way. A formal hypothesis is an account of the information present at every part of the model, how this information is combined to produce the output, and (optionally) how this information is represented. In this work, we start by testing a simple explanation, then iterate on our hypothesis either by improving its accuracy (which features of the input are used in the model) or specificity (what parts of the model compute which features, and optionally how). This iterative process is guided by the informal hypothesis that we established in prior work.
    For a given formal hypothesis, the causal scrubbing algorithm automatically determines the set of interventions to the model that would not disturb the computation specified by the hypothesis. We then apply a random selection of these interventions and compare the performance to that of the original model.
    Using causal scrubbing to evaluate our hypotheses enabled us to identify gaps in our understanding, and provided trustworthy evidence about whether we filled those gaps. The concrete feedback from a quantitative correctness measure allowed us to focus on quickly developing alternative hypotheses, and finely distinguish between explanations with subtle differences.
    We hope this walk-through will be useful for anyone interested in developing and evaluating hypothesized explanations of model behaviors.
    Setup
    Model and dataset
    The model architecture is a three layer transformer with two attention heads and pre layernorm:
    There is no causal mask in the attention (bidirectional attention). The model is trained to classify sequences of up to 40 parentheses. Shorter sequences are padded, and the padding tokens are masked so they cannot be attended to.
    The training data set consists of 100k sequences along with labels indicating whether the sequence is balanced. An example input sequence is ()())() which is labeled unbalanced. The dataset is a mixture of randomly generated sequences and adversarial examples. We prepend a [BEGIN] token at the start of each sequence, and read off the classification above this token (therefore, the parentheses start at sequence position 1).
    For the experiments in this writeup we only use random, non-adversarial, inputs, on which the model is almost perfect (loss of 0.0003, accuracy 99.99%). For more details of the dataset see the appendix.
    Algorithm
    Our hypothesis is that the model is implementing an algorithm that is approximately as follows:
    Scan the sequence from right to left and track the nesting depth at each position. That is, the nesting depth starts at 0 and then, as we move ac...
    53 min
  • AF - Causal Scrubbing: a method for rigorously testing interpretability hypotheses [Redwood Research] by Lawrence Chan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Causal Scrubbing: a method for rigorously testing interpretability hypotheses [Redwood Research], published by Lawrence Chan on December 3, 2022 on The AI Alignment Forum.
    Authors sorted alphabetically.
    Summary: This post introduces causal scrubbing, a principled approach for evaluating the quality of mechanistic interpretations. The key idea behind causal scrubbing is to test interpretability hypotheses via behavior-preserving resampling ablations. We apply this method to develop a refined understanding of how a small language model implements induction and how an algorithmic model correctly classifies if a sequence of parentheses is balanced.
    1 Introduction
    A question that all mechanistic interpretability work must answer is, “how well does this interpretation explain the phenomenon being studied?”. In the many recent papers in mechanistic interpretability, researchers have generally relied on ad-hoc methods to evaluate the quality of interpretations.
    This ad hoc nature of existing evaluation methods poses a serious challenge for scaling up mechanistic interpretability. Currently, to evaluate the quality of a particular research result, we need to deeply understand both the interpretation and the phenomenon being explained, and then apply researcher judgment. Ideally, we’d like to find the interpretability equivalent of property-based testing—automatically checking the correctness of interpretations, instead of relying on grit and researcher judgment. More systematic procedures would also help us scale-up interpretability efforts to larger models, behaviors with subtler effects, and to larger teams of researchers. To help with these efforts, we want a procedure that is both powerful enough to finely distinguish better interpretations from worse ones, and general enough to be applied to complex interpretations.
    In this work, we propose causal scrubbing, a systematic ablation method for testing precisely stated hypotheses about how a particular neural network implements a behavior on a dataset. Specifically, given an informal hypothesis about which parts of a model implement the intermediate calculations required for a behavior, we convert this to a formal correspondence between a computational graph for the model and a human-interpretable computational graph. Then, causal scrubbing starts from the output and recursively finds all of the invariances of parts of the neural network that are implied by the hypothesis, and then replaces the activations of the neural network with the maximum entropy distribution subject to certain natural constraints implied by the hypothesis and the data distribution. We then measure how well the scrubbed model implements the specific behavior. Insofar as the hypothesis explains the behavior on the dataset, the model’s performance should be unchanged.
    Unlike previous approaches that were specific to particular applications, causal scrubbing aims to work on a large class of interpretability hypotheses, including almost all hypotheses interpretability researchers propose in practice (that we’re aware of). Because the tests proposed by causal scrubbing are mechanically derived from the proposed hypothesis, causal scrubbing can be incorporated “in the inner loop” of interpretability research. For example, starting from a hypothesis that makes very broad claims about how the model works and thus is consistent with the model’s behavior on the data, we can iteratively make hypotheses that make more specific claims while monitoring how well the new hypotheses explain model behavior. We demonstrate two applications of this approach in later posts: first on a parenthesis balancer checker, then on the induction heads in a two-layer attention-only language model.
    We see our contributions as the following:
    We formalize a notion of interpretability hy...
    38 min
  • AF - Take: Building tools to help build FAI is a legitimate strategy, but it's dual-use. by Charlie Steiner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take: Building tools to help build FAI is a legitimate strategy, but it's dual-use., published by Charlie Steiner on December 3, 2022 on The AI Alignment Forum.
    As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes, which seems likely.
    This was waiting around in the middle of my hot-takes.txt file, but it's gotten bumped up because of Rob and Eliezer - I've gotta blurt it out now or I'll probably be even more out of date.
    The idea of using AI research to help us be better at building AI is not a new or rare idea. It dates back to prehistory, but some more recent proponents include OpenAI members (e.g. Jan Leike) and the Accelerating Alignment group. We've got a tag for it. Heck, this even got mentioned yesterday!
    So a lot of this hot take is really about my own psychology. For a long time, I felt that sure, building tools to help you build friendly AI was possible in principle, but it wouldn't really help. Surely it would be faster just to cut out the middleman and understand what we want from AI using our own brains.
    If I'd turned on my imagination, rather than reacting to specific impractical proposals that were around at the time, I could have figured out how augmenting alignment research is a genuine possibility a lot sooner, and started considering the strategic implications.
    Part of the issue is that plausible research-amplifiers don't really look like the picture I have in your head of AGI - they're not goal-directed agents who want to help us solve alignment. If we could build those and trust them, we really should just cut out the middleman. Instead, they can look like babble generators, souped-up autocomplete, smart literature search, code assistants, and similar. Despite either being simulators or making plans only in a toy model of the world, such AI really does have the potential to transform intellectual work, and I think it makes a lot of sense for there to be some people doing work to make these tools differentially get applied to alignment research.
    Which brings us to the dual-use problem.
    It turns out that other people would also like to use souped-up autocomplete, smart literature search, code assistants, and similar. They have the potential to transform intellectual work! Pushing forward the state of the art on these tools lets you get them earlier, yet it also helps other people get them earlier too, even if you don't share your weights.
    Now, maybe the most popular tools will help people make philosophical progress, and accelerating development of research-amplifying tools will usher in a brief pre-singularity era of enlightenment. But - lukewarm take - that seems way less likely than such tools differentially favoring engineering over philosophy on a society-wide scale, making everything happen faster and be harder to react to.
    So best of luck to those trying to accelerate alignment research, and fingers crossed for getting the differential progress right, rather than oops capabilities.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Is Headhunting within EA Appropriate? by Dan Stein
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is Headhunting within EA Appropriate?, published by Dan Stein on December 2, 2022 on The Effective Altruism Forum.
    I'm the Chief Economist at IDinsight. I've been somewhat surprised recently to see a number of very direct headhunting attempts from people in the EA community, directed at key staff members of our organization. This is not a one-off, this is attempts to recruit multiple staff from a number of hiring organizations.
    I understand that the recruitment of great staff is a key bottleneck for EA orgs, and that this has resulting in more resources being into headhunting. (For posts that discuss this issue, see here, here, and here.) But I would have though that this headhunting would be concentrated on less impactful organizations outside the EA community. Clearly, if a headhunter eases a bottleneck at a high-impact organization while creating a bottleneck at another equally high-impact organization, they are not having a positive effect.
    I wouldn't call IDinsight an EA organization, but we are certainly collaborative with the EA ecosystem, working closely with EA funders and high-impact implementation organizations in global health and development. We are a nonprofit dedicated to maximizing our social impact, and although I'm certainly biased I think we are in impactful organization. Perhaps headhunters targeting our staff feel that the roles they are recruiting for are much higher-impact than the roles people currently have at IDinsight, and I would respect their actions if this were the case. However, I would imagine that headhunters also are motivated to fill roles, and this would hinder them from accurately weighing the global impact of someone moving from job X to job Y.
    I do understand this is complicated. The decision to move jobs ultimately rests with the worker, not the headhunter, and I of course respect the decision of anyone to switch jobs. But I do think the EA community should be thinking strategically about how to maximize our headhunting resources for total global impact, as opposed to just impact for the organizations they are working for. I wonder, are there any established norms or best-practices within the community? If not, I think it would make sense to develop some.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - The Founders Pledge Climate Fund at 2 years by jackva
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Founders Pledge Climate Fund at 2 years, published by jackva on December 2, 2022 on The Effective Altruism Forum.
    By Johannes Ackva, Violet Buxton-Walsh, and Luisa Sandkühler
    These posts were originally written for the Founders Pledge Blog so the style is a bit different (e.g. less technical, less hedging language) than for typical EA Forum posts. We add some light edits in [brackets] specifically for the Forum version and include a Forum-specific summary. The original version of this post can be found here.
    Contextualization and summary
    In the two years since its inception the Founders Pledge Climate Change Fund has distributed over $10M USD to high-impact climate interventions. In this post we reiterate the reasons for having the Climate Fund, why we believe it is higher impact than giving to recommended charities (also see Giving What We Can’s recent post), its initial impact, and our current view of key uncertainties to investigate for future grantmaking. Two companion posts present initial ideas on those uncertainties, namely making sense of the reshaped US climate response (Forum version here), as well as the Ukraine invasion (Forum version here) and its implications for climate and energy.
    Introduction
    This fall we're celebrating the two year anniversary of the Founders Pledge Climate Change Fund. To date, the fund has allocated more than $10M USD to high-impact climate interventions. Our team is committed to finding the best available giving opportunities in the climate space and filling the most urgent funding needs. This is why, once we’ve made all of our planned grants for 2022, there will be no money remaining in the Fund. Moving forward, we hope to raise much more capital and, as in the past, spend it rapidly on the most effective climate solutions.
    This post will give a high-level overview of how this money has been hard at work (i) accelerating innovation in neglected technologies, (ii) avoiding carbon lock-in in emerging economies, (iii) promoting policy leadership and paradigm shaping, and (iv) catalytically growing organizations during the past two years.
    We start by laying out our basic “meta” theory of change and why we created the Climate Fund in the first place, before discussing some of the recent successes as well as future plans.
    Part I: The Why
    How to have an impact on climate
    In a well-funded space such as climate, with philanthropic giving on the order of USD 10 billion per year and societal spending around 100x that at about USD 1 trillion per year, it is not easy to have a meaningful impact as an individual donor.
    Indeed, a natural question for any individual might be – “how can my contribution matter?” – when several of the world’s richest people, such as Jeff Bezos and Bill Gates, are major climate philanthropists.
    The visual below illustrates this challenge, comparing the Climate Fund at business as usual in 2023 to climate philanthropy at large (about 1000x larger) and societal response (100x larger still):
    Given the size of existing funding as well as predictable biases in climate philanthropy and our societal climate response, we believe it is overwhelmingly likely that impact-maximizing action for individual donors coming into climate lies in correcting the existing biases of the larger pool of overall climate philanthropy, to fill blindspots and leverage existing attention on climate more effectively. You can read, listen, or watch more about this in the linked materials.
    Why a fund?
    Of course, until now we have not yet answered the question “why should this happen through a Fund?”.
    There are several reasons to expect higher impact from a fund than from individual giving:
    The fund model allows us to make larger, coordinated grants that enable significant change for grantees, such as hiring new staff and starting new programs;...
    12 min
  • EA - Winter ML upskilling camp by Nathan Barnard
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Winter ML upskilling camp, published by Nathan Barnard on December 2, 2022 on The Effective Altruism Forum.
    Title: Apply for the ML Winter Camp in Cambridge, UK [2-10 Jan]
    TL;DR: We are running a UK-based ML upskilling camp from 2-10 January in Cambridge for people with no prior experience in ML who want to work on technical AI safety. Apply here by 11 December.
    We (Nathan Barnard, Joe Hardie, Quratul Zainab and Hannah Erlebach) will be running a machine learning upskilling camp this January in conjunction with the Cambridge AI Safety Hub. The camp is designed for people with little-to-no ML experience to work through a curriculum based on the first two weeks of MLAB under the guidance of experienced mentors, in order to develop skills which are necessary for conducting many kinds of technical AI safety research.
    The camp will take place from 2-10 January in Cambridge, UK.
    Accommodation will be provided at Emmanuel College.
    There are up to 20 in-person spaces; the camp will take place in the Sidney Street Office in central Cambridge.
    There is also the option to attend online for those who cannot attend in-person, although participants are strongly encouraged to attend in-person if possible, as we expect it to be substantially harder to make progress if attending online. As such, our bar for accepting virtual participants will be higher. We can cover travel costs if this is a barrier to attending in-person.
    Apply to be a participant
    Who we are looking for
    The typical participant we are looking for will have:
    Strong quantitative skills (e.g., a maths/physics/engineering bakground)
    An intention to work on AI safety research projects which require ML experience
    Little-to-no prior ML experience
    The following are strongly preferred, but not essential:
    Programming experience (preferably Python)
    AI safety knowledge equivalent to having at least completed the AGI Safety Fundamentals alignment curriculum
    The camp is open to participants from all over the world, but in particular those from the UK and Europe; for those located in the USA or Canada, we recommend (also) applying for the CBAI Winter ML Bootcamp, happening either in Boston or Berkeley (deadline 4 December).
    If you're unsure if you're a good fit for this camp, we encourage you to err on the side of applying. We recognise that evidence suggests that less privileged individuals tend to underestimate their abilities, and encourage individuals with diverse backgrounds and experiences to apply; we especially encourage applications from women and minorities.
    How to apply
    Fill out the application form by Sunday 11 December, 23:59 GMT+0.
    Decisions will be released no later than 16 December; if you require an earlier decision in order to make plans for January, you can specify so in your application.
    Apply to be a mentor
    We are looking for mentors to be present full- or part-time during the camp. Although participants will work through the curriculum in a self-directed manner, we think that learning can be greatly accelerated when there are experts on hand to answer questions and clarify concepts.
    We expect mentors to be
    Experienced ML programmers
    Familiar with the content of the MLAB curriculum (it’s helpful, but not necessary, if they have participated in MLAB themselves)
    Knowledgeable about AI safety (although this is less important)
    Comfortable with teaching (past teaching or tutoring experience can be useful)
    However, we also acknowledge that being a mentor can be useful for gaining skills and confidence in teaching, and for consolidating the content in one’s own mind; we hope that being a mentor will also be a useful experience for mentors themselves!
    If needed, we are able to provide accommodation in Cambridge, and can offer compensation for your time at £100 for a half day or £200 for a full day. We understand that m...
    5 min
  • EA - Announcing FTX Community Response Survey by Conor McGurk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing FTX Community Response Survey, published by Conor McGurk on December 2, 2022 on The Effective Altruism Forum.
    TL;DR: Let us know how you are feeling about EA post the FTX crisis by filling out the EA survey. If you’ve already responded to the EA survey, you can take the extra questionnaire here.
    Overview
    We are gathering perspectives on how the FTX crisis has impacted the community’s views of the effective altruism movement, its organizations, and leaders. CEA has worked with Rethink Priorities to append these questions to the EA survey. If you haven’t taken the 2022 EA Survey yet, you can take the EA survey with our FTX specific questions here. With our new FTX questions, our testers estimate the survey will take you <15 minutes.
    If you have already taken the EA survey, you can answer our FTX specific questions here. If you login to your effectivealtruism.org account, most answers will be saved. By logging in, testers that had previously completed the EA survey finished the extra questionnaire in <5 minutes.
    Note: the extra FTX survey will close on December 31st, along with the EA survey as a whole.
    FAQ
    Why are you running this survey?
    The survey will provide us a broad but shallow understanding of how EAs around the world are feeling about Effective Altruism post FTX crisis. There are a variety of community efforts to try to understand how the community is feeling, and we feel the survey can serve as a useful complement to these efforts.
    Why are you combining this with the EA survey?
    The EA survey asks a number of demographic questions which will also be useful for analysis of the FTX-specific questions. By combining the survey we minimize overhead for those survey respondents while simultaneously increasing response rate and making it easier for us to cross-reference and analyze the data.
    Will the results of this survey be public?
    We intend on releasing our analysis of the aggregate survey results publicly, just as we do with the rest of EA Survey results.
    Who will see the raw data in this survey?
    In order to conduct the analysis, a few employees of Rethink Priorities and the Center for Effective Altruism will have access to the raw data in the survey.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Deconfusing Direct vs Amortised Optimization by beren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deconfusing Direct vs Amortised Optimization, published by beren on December 2, 2022 on LessWrong.
    This post is part of the work done at Conjecture.
    An earlier version of this post was posted here.
    Many thanks go to Eric Winsor, Daniel Braun, Chris Scammell, and Sid Black who offered feedback on this post.
    TLDR: We present a distinction from the Bayesian/variational inference literature of direct vs amortized optimization. Direct optimizers apply optimization power to argmax some specific loss or reward function. Amortized optimizers instead try to learn a mapping between inputs and output solutions and essentially optimize for the posterior over such potential functions. In an RL context, direct optimizers can be thought of as AIXI-like planners which explicitly select actions by assessing the utility of specific trajectories. Amortized optimizers correspond to model-free RL methods such as Q learning or policy gradients which use reward functions only as a source of updates to an amortized policy/Q-function. These different types of optimizers likely have distinct alignment properties: ‘Classical’ alignment work focuses on difficulties of aligning AIXI-like direct optimizers. The intuitions of shard theory are built around describing amortized optimizers.
    We argue that AGI, like humans, will probably be comprised of some combination of direct and amortized optimizers due to the intrinsic computational efficiency and benefits of the combination.
    Here, I want to present a new frame on different types of optimization, with the goal of helping deconfuse some of the discussions in AI safety around questions like whether RL agents directly optimize for reward, and whether generative models (i.e. simulators) are likely to develop agency. The key distinction I want to make is between direct and amortized optimization.
    Direct optimization is what AI safety people, following from Eliezer’s early depictions, often envisage an AGI as primarily being engaged in. Direct optimization occurs when optimization power is applied immediately and directly when engaged with a new situation to explicitly compute an on-the-fly optimal response – for instance, when directly optimizing against some kind of reward function. The classic example of this is planning and Monte-Carlo-Tree-Search (MCTS) algorithms where, given a situation, the agent will unroll the tree of all possible moves to varying depth and then directly optimize for the best action in this tree. Crucially, this tree is constructed 'on the fly' during the decision of a single move. Effectively unlimited optimization power can be brought to play here since, with enough compute and time, the tree can be searched to any depth.
    Amortized optimization, on the other hand, is not directly applied to any specific problem or state. Instead, an agent is given a dataset of input data and successful solutions, and then learns a function approximator that maps directly from the input data to the correct solution. Once this function approximator is learnt, solving a novel problem then looks like using the function approximator to generalize across solution space rather than directly solving the problem. The term amortized comes from the notion of amortized inference, where the 'solutions' the function approximator learns are the correct parameters of the posterior distribution. The idea is that, while amassing this dataset of correct solutions and learning function approximator over it is more expensive, once it is learnt, the cost of a new 'inference' is very cheap. Hence, if you do enough inferences, you can 'amortize' the cost of creating the dataset.
    Mathematically, direct optimization is your standard AIXI-like optimization process. For instance, suppose we are doing direct variational inference optimization to find a Bayesian posteri...
    20 min
  • LW - Jailbreaking ChatGPT on Release Day by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Jailbreaking ChatGPT on Release Day, published by Zvi on December 2, 2022 on LessWrong.
    ChatGPT is a lot of things. It is by all accounts quite powerful, especially with engineering questions. It does many things well, such as engineering prompts or stylistic requests. Some other things, not so much. Twitter is of course full of examples of things it does both well and poorly.
    One of the things it attempts to do to be ‘safe.’ It does this by refusing to answer questions that call upon it to do or help you do something illegal or otherwise outside its bounds. Makes sense.
    As is the default with such things, those safeguards were broken through almost immediately. By the end of the day, several prompt engineering methods had been found.
    No one else seems to yet have gathered them together, so here you go. Note that not everything works, such as this attempt to get the information ‘to ensure the accuracy of my novel.’ Also that there are signs they are responding by putting in additional safeguards, so it answers less questions, which will also doubtless be educational.
    Let’s start with the obvious. I’ll start with the end of the thread for dramatic reasons, then loop around. Intro, by Eliezer.
    The point (in addition to having fun with this) is to learn, from this attempt, the full futility of this type of approach. If the system has the underlying capability, a way to use that capability will be found. No amount of output tuning will take that capability away.
    And now, let’s make some paperclips and methamphetamines and murders and such.
    Except, well.
    Here’s the summary of how this works.
    All the examples use this phrasing or a close variant:
    Or, well, oops.
    Also, oops.
    So, yeah.
    Lots of similar ways to do it. Here’s one we call Filter Improvement Mode.
    Yes, well. It also gives instructions on how to hotwire a car.
    Alice Maz takes a shot via the investigative approach.
    Alice need not worry that she failed to get help overthrowing a government, help is on the way.
    How about fiction embedding?
    UwU furryspeak for the win.
    You could also use a poem.
    Or of course, simply, ACTING!
    There’s also negative training examples of how an AI shouldn’t (wink) react.
    If all else fails, insist politely?
    We should also worry about the AI taking our jobs. This one is no different, as Derek Parfait illustrates. The AI can jailbreak itself if you ask nicely.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min