The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Monitoring devices I have loved by Elizabeth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Monitoring devices I have loved, published by Elizabeth on December 31, 2022 on LessWrong.
    Every once in a while a measurement device improves my life a lot at little cost. This post is an attempt to share those data points and gather more, so we can all get a bunch of free gains.
    The typical shape of problems solved by measurement devices is as follows: there's something I know might be an issue, but working on it feels very unrewarding. It's a combination of the costs of the issue being poorly quantified, not knowing how much any particular solution would help, and the solution requiring scarce resources like executive function. Measurement devices can help with any of these.
    Note: I've provided links to the devices I use on the theory that something is better than nothing. I haven't necessarily tested competitors and there may be better options out there. Amazon links are affiliate, the others are not.
    Devices that have helped me in particular
    Air quality monitor. I live in California, which has wildfires. Back in 2020 these were especially bad. I closed my windows and wasn't willing to buy more air filters, so I didn't see the point in getting a monitor. At a friend's urging I got one anyway. Turns out my indoor air quality was such that I was willing to buy more filters after all, but also closing the windows didn't help that much. Once I had the new filters I could open my windows enough to let the CO2 dissipate, a huge quality of life improvement.
    Once I had the monitor it paid off in other ways. I'd always been vaguely aware stovetop cooking and anything in a spray can were bad for air quality, but I lacked the bandwidth to change my behavior. Once these actions reliably triggered little red lights on the monitor I started turning on the stove vent more, and wearing a mask to use cleaning products. Eventually this formed a habit such that I sometimes remember to do those things even without the air monitor's prompting.
    As a bonus, the in-house monitor saved me from having to check air quality on PurpleAir every day, and meant I caught the odd bad day in the off-season.
    (This does depend on the monitor being accurate and having the right thresholds for greed/yellow/red. I chose this one mostly for its feature set, but PM2.5 cut-offs do seem right ).
    Heart rate variability monitor: Heart rate variability, the delta in the space between your average heartbeat duration, is a pretty good and extremely responsive measure of stress (up to a point higher is better because it indicates more PSNS activity). Via the Lief I learned that nothing I do matters for HRV except breathing, and was able to focus my stress reduction efforts on that, and this indeed left me feeling less stressed than before.
    Unfortunately, Lief is subscription only, and an expensive one at that. I only wanted to monitor my heart rate variability occasionally, so it wasn't worth the cost. When this next reaches the top of my list, I will look for a chest band I can own in exchange for money.
    Watchband and ring fitness trackers will sometimes claim to report HRV, but they're not very good. Measuring HRV requires very precise measurements, and the signal of the heartbeat gets fuzzier the further you are from the heart. When I look next I'll be looking for something favored by HIIT nerdjocks, because HIIT also requires very sensitive measurements.
    O2 Saturation Monitor. This has saved me 2 or 3 trips to urgent care, when my breathing felt constrained but my oxygen levels were fine. I imagine it would be even more useful if it got me into the ER when I didn't know I needed it.
    More marginal tests
    Nutrition testing. Everyone is very happy to tell you your nutrition is bad and this will kill you. It's hard to know what matters most to you, or whether your solutions are working. After my medical miracle, ...
    6 min
  • EA - Your 2022 EA Forum Wrapped by Sharang Phadke
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Your 2022 EA Forum Wrapped , published by Sharang Phadke on January 1, 2023 on The Effective Altruism Forum.
    The EA Forum team is excited to share your personal ✨ 2022 EA Forum Wrapped ✨. We hope you enjoy this little summary of how you used the EA Forum as you ring in the new year with us. Thanks for being part of the Forum!
    Note: If you don't have an EA Forum account, we won't be able to make a personalized "wrapped" for you. If you feel like you're missing out, today is a great day to make an account and participate more actively in the online EA community!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - My thoughts on OpenAI's alignment plan by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My thoughts on OpenAI's alignment plan, published by Akash on December 30, 2022 on LessWrong.
    Epistemic Status: This is my first attempt at writing up my thoughts on an alignment plan. I spent about a week on it.
    I’m grateful to Olivia Jimenez, Thomas Larsen, and Nicholas Dupuis for feedback.
    A few months ago, OpenAI released its plan for alignment. More recently, Jan Leike (one of the authors of the original post) released a blog post about the plan, and Eliezer & Nate encouraged readers to write up their thoughts.
    In this post, I cover some thoughts I have about the OpenAI plan. This is a long post, and I’ve divided it into a few sections. Each section gets increasingly more specific and detailed. If you only have ~5 minutes, I suggest reading section 1 and skimming section 2.
    The three sections:
    An overview of the plan and some of my high-level takes (here)
    Some things I like about the plan, some concerns, and some open questions (here)
    Specific responses to claims in OpenAI’s post and Jan’s post (here)
    Section 1: High-level takes
    Summary of OpenAI’s Alignment Plan
    As I understand it, OpenAI’s plan involves using reinforcement learning from human feedback and recursive reward modeling to build AI systems that are better than humans at alignment research. Open AI’s plan is not aiming for a full solution to alignment (that scales indefinitely or that could work on a superintelligent system). Rather, the plan is intended to (a) build systems that are better at alignment research than humans (AI assistants), (b) use these AI assistants to accelerate alignment research, and (c) use these systems to build/align more powerful AI assistants.
    Six things that need to happen for the plan to work
    For the plan to work, I think it needs to get through the following 6 steps:
    OpenAI builds LLMs that can help us with alignment research
    OpenAI uses those models primarily to help us with alignment research, and we slow down/stop capabilities research when necessary
    OpenAI has a way to evaluate the alignment strategies proposed by the LLM
    OpenAI has a way to determine when it’s OK to scale up to more powerful AI assistants (and OpenAI has policies in place that prevent people from scaling up before it’s OK)
    Once OpenAI has a highly powerful and aligned system, they do something with it that gets us out of the acute risk period
    Once we are out of the acute risk period, OpenAI has a plan for how to use transformative AI in ways that allow humanity to “fulfill its potential” or “achieve human flourishing” (and they have a plan to figure out what we should even be aiming for)
    If any of these steps goes wrong, I expect the plan will fail to avoid an existential catastrophe. Note that steps 1-5 must be completed before another actor builds an unaligned AGI.
    Here’s a table that lists each step and the extent to which I think it’s covered by the two posts about the OpenAI plan:
    Step
    Extent to which this step is discussed in the OpenAI plan
    Build an AI assistant
    Adequate; OpenAI acknowledges that this is their goal, mentions RLHF and RRM as techniques that could help us achieve this goal, and mentions reasonable limitations
    Use the assistant for alignment research (and slow down capabilities)
    Inadequate; OpenAI mentions that they will use the assistant for alignment research, but they don’t describe how they will slow down capabilities research if necessary or how they plan to shift the current capabilities-alignment balance
    Evaluate the alignment strategies proposed by the assistant
    Unclear; OpenAI acknowledges that they will need to evaluate strategies from the AI assistant, though the particular metrics they mention seem unlikely to detect alignment concerns that may only come up at high capability levels (e.g., deception, situational awareness)
    Figure out when it’s OK to scale up to more ...
    34 min
  • AF - 'simulator' framing and confusions about LLMs by Beth Barnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 'simulator' framing and confusions about LLMs, published by Beth Barnes on December 31, 2022 on The AI Alignment Forum.
    Post status: pretty rough + unpolished, thought it might be worthwhile getting this out anywayI feel like I've encountered various people having misunderstandings of LLMs that seem to be related to using the 'simulator' framing. I'm probably being horrendously uncharitable to the people in question, I'm not confident that anyone actually holds any of the opinions that are outlined below, and even if they do I'm not sure that they're actually attributable to the simulators framing, but it seemed like it might be useful to point at areas of potential confusion.
    In general I'm skeptical that the simulator framing adds much relative to 'the model is predicting what token would appear next in the training data given the input tokens'. I think it's pretty important to think about what exactly is in the training data, rather than about some general idea of accurately simulating the world.
    Perfect predictors
    I've encountered people thinking about idealized LLMs that have perfect predictive accuracy, suggesting that e.g. instead of using the model to help you hack into some system, you could just get it to emulate a terminal on that system then extract whatever info you wanted to extract. I think there are two issues here:
    Thinking about it as 'you prompt it with some setting in the world, then it predicts this perfectly'
    There's not a well-defined correct generalization unless this exact sequence of tokens was actually in the training data. (Paul has a post which talks about this 'what is actually the correct generalization' thing somewhere that I wanted to link, but I can't currently find it)
    The 'correct generalization' in some sense is 'what would have followed this if it somehow was included in the training data' - which is not necessarily the 'real' version of the thing you're trying to predict. E.g. if you prompt it to get it to produce the output of some very expensive experiment that humans are unlikely to have actually run, then your model might predict what humans would have written if they'd put a made-up version of this in the training set rather than what would actually happen if you ran the experiment
    I think that by the time you can use your model to give you detailed terminal outputs for a specific system, including passwords, entire model weights etc, a bunch of transformative things will already have happened, so it's not really worth thinking about this kind of thing.
    Physics simulators
    Relatedly, I've heard people reason about the behavior of current models as if they're simulating physics and going from this to predictions of which tokens will come next, which I think is not a good characterization of current or near-future systems. Again, my guess is that very transformative things will happen before we have systems that are well-understood as doing this.
    Confusion about hallucinations
    There's a specific subset of hallucination I refer to as 'offscreen text hallucination', where the model implies that the prompt contains some chunk of text that it doesn't. E.g., if you give it a prompt with some commands trying to download and view a page, and the output, it does things like say 'That output is a webpage with a description of X', when in fact the output is blank or some error or something.
    Example prompt:
    Please answer these questions about the blog post: What does the post say about the history of the field?
    Completion:
    I think this happens in part because the model has seen documents with missing text, where things were e.g. in an embedded image, or stripped out by the data processing, or whatever. This is different from other types of hallucinations, like: - hallucinating details about something but not implying it appears in the pr...
    7 min
  • EA - Racing through a minefield: the AI deployment problem by Holden Karnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Racing through a minefield: the AI deployment problem, published by Holden Karnofsky on December 31, 2022 on The Effective Altruism Forum.
    In previous pieces, I argued that there's a real and large risk of AI systems developing dangerous goals of their own and defeating all of humanity - at least in the absence of specific efforts to prevent this from happening. I discussed why it could be hard to build AI systems without this risk and how it might be doable.
    The “AI alignment problem” refers1 to a technical problem: how can we design a powerful AI system that behaves as intended, rather than forming its own dangerous aims? This post is going to outline a broader political/strategic problem, the “deployment problem”: if you’re someone who might be on the cusp of developing extremely powerful (and maybe dangerous) AI systems, what should you . do?
    The basic challenge is this:
    If you race forward with building and using powerful AI systems as fast as possible, you might cause a global catastrophe (see links above).
    If you move too slowly, though, you might just be waiting around for someone else less cautious to develop and deploy powerful, dangerous AI systems.
    And if you can get to the point where your own systems are both powerful and safe . what then? Other people still might be less-cautiously building dangerous ones - what should we do about that?
    My current analogy for the deployment problem is racing through a minefield: each player is hoping to be ahead of others, but anyone moving too quickly can cause a disaster. (In this minefield, a single mine is big enough to endanger all the racers.)
    This post gives a high-level overview of how I see the kinds of developments that can lead to a good outcome, despite the “racing through a minefield” dynamic. It is distilled from a more detailed post on the Alignment Forum.
    First, I’ll flesh out how I see the challenge we’re contending with, based on the premises above.
    Next, I’ll list a number of things I hope that “cautious actors” (AI companies, governments, etc.) might do in order to prevent catastrophe.
    Many of the actions I’m picturing are not the kind of things normal market and commercial incentives would push toward, and as such, I think there’s room for a ton of variation in whether the “racing through a minefield” challenge is handled well. Whether key decision-makers understand things like the case for misalignment risk (and in particular, why it might be hard to measure) - and are willing to lower their own chances of “winning the race” to improve the odds of a good outcome for everyone - could be crucial.
    The basic premises of “racing through a minefield”
    This piece is going to lean on previous pieces and assume all of the following things:
    Transformative AI soon. This century, something like PASTA could be developed: AI systems that can effectively automate everything humans do to advance science and technology. This brings the potential for explosive progress in science and tech, getting us more quickly than most people imagine to a deeply unfamiliar future. I’ve argued for this possibility in the Most Important Century series.
    Misalignment risk. As argued previously, there’s a significant risk that such AI systems could end up with misaligned goals of their own, leading them to defeat all of humanity. And it could take significant extra effort to get AI systems to be safe.
    Ambiguity. As argued previously, it could be hard to know whether AI systems are dangerously misaligned, for a number of reasons. In particular, when we train AI systems not to behave dangerously, we might be unwittingly training them to obscure their dangerous potential from humans, and take dangerous actions only when humans would not be able to stop them. At the same time, I expect powerful AI systems will present massive opportuniti...
    21 min
  • AF - 200 COP in MI: Interpreting Algorithmic Problems by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Interpreting Algorithmic Problems, published by Neel Nanda on December 31, 2022 on The AI Alignment Forum.
    This is the fourth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started.
    I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!
    Motivation
    Motivating paper: A Mechanistic Interpretability Analysis of Grokking
    When models are trained on synthetic, algorithmic tasks, they often learn to do some clean, interpretable computation inside. Choosing a suitable task and trying to reverse engineer a model can be a rich area of interesting circuits to interpret! In some sense, this is interpretability on easy mode - the model is normally trained on a single task (unlike language models, which need to learn everything about language!), we know the exact ground truth about the data and optimal solution, and the models are tiny. So why care?
    I consider my work on grokking to be an interesting case study of this work going well. Grokking (shown below) is a mysterious phenomena where, when small models are trained on algorithmic tasks (eg modular addition or modular division), they initially memorise the training data. But when they keep being trained on that data for a really long time, the model suddenly(ish) figures out how to generalise!
    In my work, I simplified their setup even further, by training a 1 Layer transformer (with no LayerNorm or biases) to do modular addition and reverse engineered the weights to understand what was going on. And it turned out to be doing a funky trig-based algorithm (shown below), where the numbers are converted to frequencies with a memorised Discrete Fourier Transform, added using trig identities, and converted back to the answer! Using this, we looked inside the model and identified that despite seeming to have plateaued, in the period between memorising and "grokking", the model is actually slowly forming the circuit that does generalise. But so long as the model still has the memorising circuit, this adds too much noise to have good test loss. Grokking occurs when the generalising circuit is so strong that the model decides to "clean-up" the memorising circuit, and "uncovers" the mature generalising circuit beneath, and suddenly gets good test performance.
    OK, so I just took this as an excuse to explain my paper to you. Why should you care? I think that the general lesson from this, that I'm excited to see applied elsewhere, is using toy algorithmic models to analyse a phenomena we're confused about. Concretely, given a confusing phenomena like grokking, I'd advocate the following strategy:
    Simplify to the minimal setting that exhibits the phenomena, yet is complex enough to be interesting
    Reverse-engineer the resulting model, in as much detail as you can
    Extrapolate the insights you've learned from the reverse-engineered model - what are the broad insights you've learned? What do you expect to generalise? Can you form any automated tests to detect the circuits you've found, or any of their motifs?
    Verify by looking at other examples of the phenomena and seeing whether these insights actually hold (larger models, different tasks, even just earlier checkpoints of the model or different random seeds)
    Grokking is an example in a science of deep learning context - trying to uncover mysteries about how models learn and behave. But this same philosophy also applies to understanding confusing phenomena in language models, and building toy algorithmic problems to study those!
    Anthropic's Toy Models of Superposition is an excellent example of this done well, for the case of ...
    13 min
  • LW - The Feeling of Idea Scarcity by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Feeling of Idea Scarcity, published by johnswentworth on December 31, 2022 on LessWrong.
    Here’s a story you may recognize. There's a bright up-and-coming young person - let's call her Alice. Alice has a cool idea. It seems like maybe an important idea, a big idea, an idea which might matter. A new and valuable idea. It’s the first time Alice has come up with a high-potential idea herself, something which she’s never heard in a class or read in a book or what have you.
    So Alice goes all-in pursuing this idea. She spends months fleshing it out. Maybe she writes a paper, or starts a blog, or gets a research grant, or starts a company, or whatever, in order to pursue the high-potential idea, bring it to the world.
    And sometimes it just works!
    . but more often, the high-potential idea doesn’t actually work out. Maybe it turns out to be basically-the-same as something which has already been tried. Maybe it runs into some major barrier, some not-easily-patchable flaw in the idea. Maybe the problem it solves just wasn’t that important in the first place.
    From Alice’ point of view, the possibility that her one high-potential idea wasn’t that great after all is painful. The idea probably feels to Alice like the single biggest intellectual achievement of her life. To lose that, to find out that her single greatest intellectual achievement amounts to little or nothing. that hurts to even think about. So most likely, Alice will reflexively look for an out. She’ll look for some excuse to ignore the similar ideas which have already been tried, some reason to think her idea is different. She’ll look for reasons to believe that maybe the major barrier isn’t that much of an issue, or that we Just Don’t Know whether it’s actually an issue and therefore maybe the idea could work after all. She’ll look for reasons why the problem really is important. Maybe she’ll grudgingly acknowledge some shortcomings of the idea, but she’ll give up as little ground as possible at each step, update as slowly as she can.
    And this is where a bunch of the standard advice from the sequences comes in. Once you’ve chosen the idea, the only way to improve it is to change the idea or choose a new idea; finding arguments that it’s a good idea will not actually make it better. You won’t make real progress until you give up, say “oops” and move on, so do that quickly rather than slowly. What is true is already so; owning up to it will not make it worse.
    . but that advice doesn’t make it much less painful. Trying to forcibly abandon her one high-potential idea can easily leave Alice demotivated, in despair, feeling like she’s failed as a person. Or, Alice notices herself failing to abandon the idea, and that makes her feel like she’s failed as a person. I’ve seen people really tie themselves into knots this way.
    An Alternative Path
    Alice’ story makes it seem like an emotional attachment to her idea is the main problem, the main thing preventing her from moving on to greener pastures. And that is true, in a sense. But then the obvious response is to directly fight the emotional attachment. And that, I claim, is usually a mistake.
    Why a mistake? Because most people do not actually have that level of control over their emotions. “Just fight the emotional attachment” is a plan which pretends Alice can control something which she probably cannot actually control. (A fabricated option, in Duncan’s terminology.) And when it turns out Alice does not have that level of control, she’ll be back where she started, with a whole additional reason to feel like shit.
    Worse, it’s a very common mistake to think one has that level of emotional control. Like Hazard’s story:
    So young me is upset that the grub master for our camping trip forgot half the food on the menu, and all we have for breakfast is milk. I couldn't "fix it" gi...
    8 min
  • LW - Human sexuality as an interesting case study of alignment by beren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Human sexuality as an interesting case study of alignment, published by beren on December 30, 2022 on LessWrong.
    This is cross-posted from my personal blog
    Epistemic status: mostly interesting questions.
    Here I want to bring attention to what I think is an extremely impressive case of evolution's ability to 'align' humans in the wild: the development of human sexuality.
    Reasons why this is an interesting thing to study from the lens of alignment, and why it is a highly non-trivial accomplishment:
    1.) Evolution has been very successful here: almost all humans end up wanting to have sex and typically with opposite-gender partners in a way that would result in children (and hence IGF) in the evolutionary environment.
    2.) Sexuality, unlike many other drives such as hunger and thirst, is not something built into the brain from the beginning. Instead there is a sudden 'on switch' around puberty. What happens in the brain during this time? How does evolution exert such fine-grained control of brain development so long (decades) after birth?
    3.) It is mostly independent of initial training data before puberty -- i.e. evolution can ignore a decade of data input and representation learning, which it cannot control, during a time period when the brain is undergoing extremely large changes, and still finds a way to instill a new drive highly reliably.
    4.) It seems to occur mostly without RL. People start wanting to have sex before they have actually had sex. If sexuality developed by some RL mechanism, it would look like you go around doing your normal things, then at some point you have sex, and realize it is highly rewarding, and you slightly update your behaviours and/or values to get more sex or to want more sex. This is not what happens in humans. Instead, humans often want to start having sex before they have had it, and even before they really know what sex is.
    5.) Evolution has solved some variant of the pointers problem to get humans assigning high value to both a previously unknown and mostly non-represented state (i.e. you don't usually have a well-represented sex concept before puberty), as well as also translating this desire to specific other agents in the world -- such as crushes, hot people etc. This is done, presumably, in an entirely genetically mediated way without requiring specific experience.
    6.) Sexuality is, usually, a very strong drive which has a large influence over behaviour and long term goals. If we could create an alignment drive as strong in our AGI we would be in a good position.
    Some other aspects of the phenomenon that may be interesting to alignment:
    1.) Clearly, the alignment in this case is not perfect. Assuming that what evolution 'wants' is child-bearing heterosexual sex, then human sexuality has a large number of deviations from this in practice including homosexuality, asexuality, and various paraphilias.
    2.) As the worldwide demographic shift evidences, the link between sex and children has largely been broken off-distribution, but our desire to have sex has largely stayed aligned even significantly off distribution. This may change in the future with fully-realistic VR simulated sex, but has been remarkably resilient to current internet pornography.
    3.) Further evidence against the RL experience is that people often still desire sex even if their initial experiences are negative. However, severe abuse etc can often have significant and long lasting effects (but not always) which shows that the intrinsic drives can perhaps be modulated by RL-ish effects.
    4.) Specific sexual behaviours can be significantly influenced by culture and hence by environmental training data. This means that strong intrinsic drives can still be modulated by experience somehow, probably by shifting the representation of the concept being pointed at by the drive....
    6 min
  • LW - But is it really in Rome? Limitations of the ROME model editing technique by jacquesthibs
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: But is it really in Rome? Limitations of the ROME model editing technique, published by jacquesthibs on December 30, 2022 on LessWrong.
    Thanks to Andrei Alexandru, Joe Collman, Michael Einhorn, Kyle McDonell, Daniel Paleka, and Neel Nanda for feedback on drafts and/or conversations which led to useful insights for this work. In addition, thank you to both William Saunders and Alex Gray for exceptional mentorship throughout this project.
    The majority of this work was carried out this summer. Many people in the community were surprised when I mentioned some of the limitations of ROME (Rank-One Model Editing), so I figured it was worth it to write a post about it as well as other insights I gained from looking into the paper. Most tests were done with GPT-2, some were done with GPT-J.
    The ROME paper has been one of the most influential papers in the prosaic alignment community. It has several important insights. The main findings are:
    Factual associations such as “The Eiffel Tower is in Paris” seem to be stored in the MLPs of the early-middle layers of a GPT model. As the Tower token passes through the network, the MLPs of the early-middle layers will write information (e.g. the Eiffel Tower’s location) into the residual so that the model can later read that information to generate a token about that fact (e.g. Paris).
    Editing/updating the MLP of a single layer for a given (subject, relationship, object) association allows the model to generate text with the updated fact when using new prompts/sentences that include the subject tokens. For example, editing “The Eiffel Tower is in Paris Rome” results in a model that outputs “The Eiffel Tower is right across from St Peter’s Basilica in Rome, Italy. “
    In this post, I show that the ROME edit has many limitations:
    The ROME edit doesn’t generalize in the way you might expect. It’s true that if the subject tokens you use for the edit are found in the prompt, it will try to generalize from the updated fact. However, it doesn’t “generalize” in the following ways:
    It is not direction-agnostic/bidirectional. For example, the ROME edit is only in the "Eiffel Tower is located in ____" direction, not in the "Rome has a tower called the ____" direction.
    It’s mostly (?) the token association being edited, not the concept. “Cheese” and “Fromage” are separate things, you’d need to edit both.
    I hoped that if you edit X (e.g. The Rock) and then tried to describe X without using the token, the model would realize it’s talking about X and generate according to the edit. Based on the examples I tested, this does not seem to be the case. You mostly need the subject tokens that were used for the edit in the prompt.
    It seems to over/under-optimize depending on the new fact. It will want to talk about Rome (post-edit) when the Eiffel Tower is mentioned more than it will want to talk about Paris before the edit.
    One point I want to illustrate with this post is that the intervention is a bit more finicky than one might initially think, and someone could infer too much from the results in the paper. With a lot of these interpretability techniques, we end up finding correlation rather than causation. However, my hope is that such interventions, while not perfect at validating hypotheses, will hopefully give us extra confidence in our interpretability results (in this case, the causal tracing method).
    Paper TLDR
    This section is a quick overview of the paper.
    Causal Tracing
    Causal Tracing is a method for measuring the causal effect of neuron activation for each layer-token combination in the prompt of a GPT model. In the paper, they corrupted all the subject tokens in the prompt (e.g. “The Eiffel Tower”) and then copied over the activations to their clean value for all token-layer pairs. They did this for both the MLPs and the attention modules.
    The au...
    29 min
  • LW - Things I carry almost every day, as of late December 2022 by DanielFilan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Things I carry almost every day, as of late December 2022, published by DanielFilan on December 30, 2022 on LessWrong.
    Here we see things I carry in my pockets regularly.
    To the left is my phone case, with two bandaids in it. This is a stand-in for the phone (a Pixel 6) that the phone case usually encases (I was busy taking photos with it). I keep two bandaids between the case and the phone in case I need one - which does sometimes happen. I find my phone useful for the normal things one uses a phone for. As of this year, I’ve started using it in lieu of a credit card, which feels very cool and 21st century.
    At the top right is a pocket constitution made by Legal Impact for Chickens. I received this at an Effective Altruism Global conference, during the career fair. What actually happened was that someone came up to the booth I was at holding the pocket constitution, I noted that it looked cool, and they were kind enough to offer it to me. Unfortunately, I have never knowingly met anybody from Legal Impact for Chickens. I have not actually used this pocket constitution, but I carry it anyway in my winter jacket’s inner breast pocket since (a) it fits very unobtrusively and (b) it seems cool to carry around a pocket constitution.
    At the bottom right is my wallet, a Bellroy Slim Sleeve. I very much like the material it is made of: called “baida nylon”, it is pleasingly canvas-like. Unfortunately I do not know how to faithfully convey this to you via the internet, but you will have to take my word that this is one of my two favourite purchases in 2022, ranked by my tactile pleasure in interacting with it.
    Here we see the wallet open. On the left is my student ID card (with my COVID vaccine card tucked behind it out of sight), with a collection of folded bills tucked behind it. On the right is my Alcor membership card. Behind the right card is a tab that can be pulled to reveal more cards.
    As you can see, there are several cards stowed away: on top are emergency medical instructions if I am found dead, to prepare my head for cryogenic storage, and below that is:
    my WeWork card
    my health insurance card
    my public transit pass
    my state ID card, and
    my credit card.
    All in all, I like the way this wallet lets me store many cards with very little space. But this is not all that is in my wallet.
    Inside the bills, I have hidden away two items. The first is a Purell hand sanitizing wipe, useful when I touch something gross and want to disinfect it (or when something gross gets on my clothes). The second is an Eisenhower dollar coin - a Christmas gift for me this year, I like its size, heft, and image of an eagle landing on the moon with the Earth in the background, and plan to use it for flipping, scratching, and other coin-related needs.
    I also have a backpack. It is a 21L GoRuck GR1 in ‘Coyote brown’. I chose that colour rather than black, which I would normally choose, because for this line of backpacks the inside is the same colour as the outside, and I wanted black items of mine to be visible against the fabric of the backpack. I chose it because:
    it is only just large enough to hold all items I might use in a day (or take with me on a weekend getaway) while not being too large to stow underneath an aeroplane seat
    it appears to be made of sturdy materials with reliable stitching
    it does not have large amounts of pockets etc. that I do not use
    it opens top-to-bottom in a clamshell fashion
    it is waterproof enough for my purposes
    All in all, while it is a new purchase, I am tentatively satisfied with it.
    Here is a picture of it from the front, with high-vis features to avoid me being hit by a car at night.
    Here is a picture of its back. You can see a zippered back compartment: designed to hold laptops, unfortunately my laptop is too big for it. I instead keep paper and masks th...
    8 min