The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - There are no coherence theorems by EJT
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There are no coherence theorems, published by EJT on February 20, 2023 on The Effective Altruism Forum.
    Introduction
    For about fifteen years, the AI safety community has been discussing coherence arguments. In papers and posts on the subject, it’s often written that there exist 'coherence theorems' which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. Despite the prominence of these arguments, authors are often a little hazy about exactly which theorems qualify as coherence theorems. This is no accident. If the authors had tried to be precise, they would have discovered that there are no such theorems.
    I’m concerned about this. Coherence arguments seem to be a moderately important part of the basic case for existential risk from AI. To spot the error in these arguments, we only have to look up what cited ‘coherence theorems’ actually say. And yet the error seems to have gone uncorrected for more than a decade.
    More detail below.
    Coherence arguments
    Some authors frame coherence arguments in terms of ‘dominated strategies’. Others frame them in terms of ‘exploitation’, ‘money-pumping’, ‘Dutch Books’, ‘shooting oneself in the foot’, ‘Pareto-suboptimal behavior’, and ‘losing things that one values’ (see the Appendix for examples).
    In the context of coherence arguments, each of these terms means roughly the same thing: a strategy A is dominated by a strategy B if and only if A is worse than B in some respect that the agent cares about and A is not better than B in any respect that the agent cares about. If the agent chooses A over B, they have behaved Pareto-suboptimally, shot themselves in the foot, and lost something that they value. If the agent’s loss is someone else’s gain, then the agent has been exploited, money-pumped, or Dutch-booked. Since all these phrases point to the same sort of phenomenon, I’ll save words by talking mainly in terms of ‘dominated strategies’.
    With that background, here’s a quick rendition of coherence arguments:
    There exist coherence theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
    Sufficiently-advanced artificial agents will not pursue dominated strategies.
    So, sufficiently-advanced artificial agents will be ‘coherent’: they will be representable as maximizing expected utility.
    Typically, authors go on to suggest that these expected-utility-maximizing agents are likely to behave in certain, potentially-dangerous ways. For example, such agents are likely to appear ‘goal-directed’ in some intuitive sense. They are likely to have certain instrumental goals, like acquiring power and resources. And they are likely to fight back against attempts to shut them down or modify their goals.
    There are many ways to challenge the argument stated above, and many of those challenges have been made. There are also many ways to respond to those challenges, and many of those responses have been made too. The challenge that seems to remain yet unmade is that Premise 1 is false: there are no coherence theorems.
    Cited ‘coherence theorems’ and what they actually say
    Here’s a list of theorems that have been called ‘coherence theorems’. None of these theorems state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue dominated strategies. Here’s what the theorems say:
    The Von Neumann-Morgenstern Expected Utility Theorem:
    The Von Neumann-Morgenstern Expected Utility Theorem is as follows:
    An agent can be represented as maximizing expected utility if and only if their preferences satisfy the following four axioms:
    Completeness: For all lotteries X and Y, X...
    35 min
  • LW - There are no coherence theorems by Dan H
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There are no coherence theorems, published by Dan H on February 20, 2023 on LessWrong.
    [Written by EJT as part of the CAIS Philosophy Fellowship. Thanks to Dan for help posting to the Alignment Forum]
    Introduction
    For about fifteen years, the AI safety community has been discussing coherence arguments. In papers and posts on the subject, it’s often written that there exist 'coherence theorems' which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. Despite the prominence of these arguments, authors are often a little hazy about exactly which theorems qualify as coherence theorems. This is no accident. If the authors had tried to be precise, they would have discovered that there are no such theorems.
    I’m concerned about this. Coherence arguments seem to be a moderately important part of the basic case for existential risk from AI. To spot the error in these arguments, we only have to look up what cited ‘coherence theorems’ actually say. And yet the error seems to have gone uncorrected for more than a decade.
    More detail below.
    Coherence arguments
    Some authors frame coherence arguments in terms of ‘dominated strategies’. Others frame them in terms of ‘exploitation’, ‘money-pumping’, ‘Dutch Books’, ‘shooting oneself in the foot’, ‘Pareto-suboptimal behavior’, and ‘losing things that one values’ (see the Appendix for examples).
    In the context of coherence arguments, each of these terms means roughly the same thing: a strategy A is dominated by a strategy B if and only if A is worse than B in some respect that the agent cares about and A is not better than B in any respect that the agent cares about. If the agent chooses A over B, they have behaved Pareto-suboptimally, shot themselves in the foot, and lost something that they value. If the agent’s loss is someone else’s gain, then the agent has been exploited, money-pumped, or Dutch-booked. Since all these phrases point to the same sort of phenomenon, I’ll save words by talking mainly in terms of ‘dominated strategies’.
    With that background, here’s a quick rendition of coherence arguments:
    There exist coherence theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
    Sufficiently-advanced artificial agents will not pursue dominated strategies.
    So, sufficiently-advanced artificial agents will be ‘coherent’: they will be representable as maximizing expected utility.
    Typically, authors go on to suggest that these expected-utility-maximizing agents are likely to behave in certain, potentially-dangerous ways. For example, such agents are likely to appear ‘goal-directed’ in some intuitive sense. They are likely to have certain instrumental goals, like acquiring power and resources. And they are likely to fight back against attempts to shut them down or modify their goals.
    There are many ways to challenge the argument stated above, and many of those challenges have been made. There are also many ways to respond to those challenges, and many of those responses have been made too. The challenge that seems to remain yet unmade is that Premise 1 is false: there are no coherence theorems.
    Cited ‘coherence theorems’ and what they actually say
    Here’s a list of theorems that have been called ‘coherence theorems’. None of these theorems state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue dominated strategies. Here’s what the theorems say:
    The Von Neumann-Morgenstern Expected Utility Theorem:
    The Von Neumann-Morgenstern Expected Utility Theorem is as follows:
    An agent can be represented as maximizing expected utility if and only i...
    35 min
  • EA - Sanity check - effectiveness/goodness of Trans Rescue? by David D
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sanity check - effectiveness/goodness of Trans Rescue?, published by David D on February 20, 2023 on The Effective Altruism Forum.
    I stumbled across the charity Trans Rescue, which helps transgender people living in unsafe parts of the world move. They've published advice for people living in first world countries with worsening legal situations for trans people, but the vast majority of their funding goes toward helping people in Africa and the Middle East immigrate to safer countries (or for Kenyans, move to Trans Rescue's group home in the safest region of Kenya) and stay away from abusive families.As of September 2022, their total funding since inception was just under 33k euros . They helped about twenty people move using this funding/ . That puts the cost to help a person move at about 1,650 euros, which is in the same ballpark as a Givewell top charity's cost to save one person from fatal malaria.
    I haven't looked closely at the likely outcome for people who would benefit from Trans Rescue's services but don't get help. Some would live and some would not, but I don't have a good sense of the relative numbers, or how to put QUALYs on undertaking a move such as this. Since they're very new and very small, I'm considering donating and keeping an eye on how they grow as an organization.
    Mainly I hoped you all could help me by pointing out whether there's anything fishy that I might have missed. This review was published by a group of Twitter users, apparently after an argument with one of the board members. It's certainly not unbiased, but they do seem to have made a concerted effort to find anything bad or construable as bad that Trans Rescue has ever done. Trans Rescue wrote a blog post in response . I came away with a sense that the board is new at running an organization like this, and they rely on imperfect volunteer labor to be able to move as many people as they do, but their work is overall helpful to their clients.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Join a new slack for animal advocacy by SofiaBalderson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Join a new slack for animal advocacy, published by SofiaBalderson on February 20, 2023 on The Effective Altruism Forum.
    We'd like to extend an invitation to you all for a new Slack space that brings together individuals who are passionate about making a meaningful impact for animals.
    Our aim is to:
    support discussions in the animal advocacy space
    foster new connections
    learn from each other
    generate innovative ideas
    perhaps even launch new projects.
    It would be great for you to join us today and introduce yourself to the community! We already have some great discussions going.
    FAQs:
    "But there are already lots of Slack spaces?" - before we started this space, there wasn't a space that was just for animal welfare.
    "Who should join?"
    Anyone already working in animal advocacy or adjacent organisation
    Anyone involved in alt protein
    Anyone, regardless of experience or place of work, interested in helping animals in an impactful way
    By animals, we mean farmed animals and wild animals
    "How much do I have to participate, could I join and lurk for a bit?"- Sure, you don't need to be active all the time, all we ask is that you introduce yourself and invite anyone you know in the space who may be interested. If you'd like, you can message others or participate in discussions, but no pressure.
    If in doubt, please join!
    Best wishes,
    Sofia and Cameron
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - On Loyalty by Nathan Young
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On Loyalty, published by Nathan Young on February 20, 2023 on The Effective Altruism Forum.
    Epistemic status: I am confident about most individual points but there are probably errors overall. I imagine if there are lots of comments to this piece I'll have changed my mind by the time I've finished reading them.
    I was having a conversation with a friend, who said that EAs weren't loyal. They said that the community's recent behaviour would made them fear that they would be attacked for small errors. Hearing this I felt sad, and wanted to understand.
    Tl;dr
    I feel a desire to be loyal. This is an exploration of that. If you don't feel that desire this may not be useful
    I think loyalty is a compact between current and future members on what it is to be within a community - "do this and you will be safe"
    Status matters and loyalty is a "status insurance policy" - even if everyone else doesn't like you, we will
    I find more interest in where we have been disloyal than where we ought to be loyal.
    Were we disloyal to Bostrom?
    Is loyalty even good?
    Were people disloyal in sharing the Leadership Slack?
    Going forward I would like that
    I have and give a clear sense of how I coordinate with others
    I take crises slowly and relaxedly and not panic
    I am able to trust that people will keep private conversations which discuss no serious wrongdoing secret and have them trust me that I will do the same
    I feel that it is acceptable to talk to journalists if helping a directionally accurate story, but that almost all of the time I should consider this a risky thing to do
    This account is deliberately incoherent - it contains many views and feelings that I can't turn into a single argument. Feel free to disagree or suggestion some kind of synthesis.
    Intro
    Testing situations are when I find out who I am. But importantly, I can change. And living right matters to me perhaps more than anything else (though I am hugely flawed). So should I change here?
    I go through these 1 by 1 because I think this is actually hard and I don't know the answers. Feel free to nitpick.
    What is Loyalty (in this article)?
    I think loyalty is the sense that do right by those who have followed the rules. It's a coordination tool. "You stuck by the rules, so you'll be treated well", "you are safe within these bounds, you don't need to fear".
    I want to be loyal - for people to think "there goes Nathan, when I abide by community norms, he won't treat me badly".
    The notion of safety implies a notion of risk. I am okay with that. Sometimes a little fear and ambiguity is good - I'm okay with a little fear around flirting at EAGs because that reduces the amount of it, I'm okay with ambiguity in how to work in a biorisk lab. There isn't always a clear path to "doing the right thing" and if, in hindsight I didn't do it, I don't want your loyalty. But I want our safety, uncertainty, disaster circles to be well calibrated.
    Some might say "loyalty isn't good" - that we should seek to treat those in EA exactly the same as those outside. For me this equivaltent to saying our circles should be well calibrated - if someone turns out to have broken the norms we care about then I already don't feel a need to be loyal to them.
    But to me, a sense of ingroup loyalty feel inevitable. I just like you more and differently than those ousisde EA. It feels naïve to say otherwise. Much like "you aren't in traffic, you are trafffic", "I don't just feel feelings, to some extent I am feelings"
    So let's cut to the chase.
    The One With The Email
    Nick Bostrom sent an awful email. He wrote an apology a while back, but he wrote another to avoid a scandal (whoops). Twitter at least did not think he was sorry. CEA wrote a short condemnation. There was a lot of forum upset. Hopefully we can agree on this.
    Let's look at this through some different frames.
    The ...
    24 min
  • AF - Behavioral and mechanistic definitions (often confuse AI alignment discussions) by Lawrence Chan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Behavioral and mechanistic definitions (often confuse AI alignment discussions), published by Lawrence Chan on February 20, 2023 on The AI Alignment Forum.
    TL;DR: It’s important to distinguish between behavioral definitions – which categorize objects based on outside observable properties – and mechanistic definitions – which categorize objects based on their internal mechanisms. In this post, I give several examples of terms which can be defined either behaviorally and mechanistically. Then, I talk about the pros and cons of both kinds of definitions, and how this distinction relates to the distinction between gears-level versus black-box models.‌‌‌‌‌‌‌‌‌‌‌
    Related to: Most similar to John Wentworth’s Gears and Behaviors, but about definitions rather than models. Also inspired by: Gears in understanding, How an algorithm feels from the inside, the “Human’s Guide to Words” Sequence in general.
    Epistemic status: written quickly instead of not at all.
    Introduction:
    Broadly speaking, when pointing at a relatively distinct cluster of objects, there’s two ways to define membership criteria:
    Behaviorally: You can categorize objects based on outside observable properties, that is, their behavior in particular situations.
    Mechanistically: Alternatively, you can categorize objects via their internal mechanisms. That is, instead of only checking for a particular behavioral property, you instead look for how the object implements said property.
    Many AI safety concepts have both behavioral and mechanistic definitions. In turn, many discussions about AI safety end up with the participants confused or even talking past each other. This is my attempt to clarify the discussion, by giving examples of both, explaining the pros and cons, and discussing when you might want to use either.
    Three examples of behavioral and mechanistic definitions
    To better illustrate what I mean, I’ll give two examples from recent ML work and a third from the sequences.
    Induction heads
    First introduced in a mathematical framework for transformer circuits, induction heads are transformer attention heads that implement in-context copying behavior. However, there seem to be two definitions that are often conflated:
    Behavioral: Subsequent papers (In-context Learning and Induction Heads, Scaling laws and Interpretability of Learning from Repeated Data) give a behavioral definition of induction heads: Induction heads are heads that score highly on two metrics on repeated random sequences of the form [A] [B] . [A]:
    Prefix matching: attention heads pay a lot of attention to the first occurrence of the token [A].
    Copying: attention heads increase the logit of [B] relative to other tokens.
    This definition is clearly behavioral: it makes no reference to how these heads are implemented, but only to their outside behavior.
    Mechanistic: In contrast, the original mathematical framework paper also gives a mechanistic definition for induction heads: induction heads are heads that implement copying behavior using either Q- or K-composition. While this definition does make some reference to outside properties (induction heads implement copying), the primary part is mechanistic and details how this copying behavior is implemented.
    However, it turns out that the two definitions don’t overlap perfectly: behavioral induction heads are often implementing many other heuristics, even in very small language models. I often talk to people who confuse the two definitions and think that we understand much more about the internal mechanisms of large language models than we actually do. In a forthcoming post, Alexandre Variengien discusses the distinction between these two definitions in more detail, while also highlighting specific confusions that may arise from failing to distinguish the two definitions.
    Different framings of inner and...
    13 min
  • LW - What AI companies can do today to help with the most important century by HoldenKarnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What AI companies can do today to help with the most important century, published by HoldenKarnofsky on February 20, 2023 on LessWrong.
    I’ve been writing about tangible things we can do today to help the most important century go well. Previously, I wrote about helpful messages to spread and how to help via full-time work.
    This piece is about what major AI companies can do (and not do) to be helpful. By “major AI companies,” I mean the sorts of AI companies that are advancing the state of the art, and/or could play a major role in how very powerful AI systems end up getting used.1
    This piece could be useful to people who work at those companies, or people who are just curious.
    Generally, these are not pie-in-the-sky suggestions - I can name2 more than one AI company that has at least made a serious effort at each of the things I discuss below (beyond what it would do if everyone at the company were singularly focused on making a profit).3
    I’ll cover:
    Prioritizing alignment research, strong security, and safety standards (all of which I’ve written about previously).
    Avoiding hype and acceleration, which I think could leave us with less time to prepare for key risks.
    Preparing for difficult decisions ahead: setting up governance, employee expectations, investor expectations, etc. so that the company is capable of doing non-profit-maximizing things to help avoid catastrophe in the future.
    Balancing these cautionary measures with conventional/financial success.
    I’ll also list a few things that some AI companies present as important, but which I’m less excited about: censorship of AI models, open-sourcing AI models, raising awareness of AI with governments and the public. I don’t think all these things are necessarily bad, but I think some are, and I’m skeptical that any are crucial for the risks I’ve focused on.
    I previously laid out a summary of how I see the major risks of advanced AI, and four key things I think can help (alignment research; strong security; standards and monitoring; successful, careful AI projects). I won’t repeat that summary now, but it might be helpful for orienting you if you don’t remember the rest of this series too well; click here to read it.
    Some basics: alignment research, strong security, safety standards
    First off, AI companies can contribute to the “things that can help” I listed above:
    They can prioritize alignment research (and other technical research, e.g. threat assessment research and misuse research).
    For example, they can prioritize hiring for safety teams, empowering these teams, encouraging their best flexible researchers to work on safety, aiming for high-quality research that targets crucial challenges, etc.
    It could also be important for AI companies to find ways to partner with outside safety researchers rather than rely solely on their own teams. As discussed previously, this could be challenging. But I generally expect that AI companies that care a lot about safety research partnerships will find ways to make them work.
    They can help work toward a standards and monitoring regime. E.g., they can do their own work to come up with standards like "An AI system is dangerous if we observe that it's able to ___, and if we observe this we will take safety and security measures such as ____." They can also consult with others developing safety standards, voluntarily self-regulate beyond what’s required by law, etc.
    They can prioritize strong security, beyond what normal commercial incentives would call for.
    It could easily take years to build secure enough systems, processes and technologies for very high-stakes AI.
    It could be important to hire not only people to handle everyday security needs, but people to experiment with more exotic setups that could be needed later, as the incentives to steal AI get stronger.
    Avoiding hype an...
    15 min
  • AF - There are no coherence theorems by Dan H
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There are no coherence theorems, published by Dan H on February 20, 2023 on The AI Alignment Forum.
    [Written by EJT as part of the CAIS Philosophy Fellowship. Thanks to Dan for help posting to the Alignment Forum]
    Introduction
    For about fifteen years, the AI safety community has been discussing coherence arguments. In papers and posts on the subject, it’s often written that there exist coherence theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. Despite the prominence of these arguments, authors are often a little hazy about exactly which theorems qualify as coherence theorems. This is no accident. If the authors had tried to be precise, they would have discovered that there are no such theorems.
    I’m concerned about this. Coherence arguments seem to be a moderately important part of the basic case for existential risk from AI. To spot the error in these arguments, we only have to look up what cited ‘coherence theorems’ actually say. And yet the error seems to have gone uncorrected for more than a decade.
    More detail below.
    Coherence arguments
    Some authors frame coherence arguments in terms of ‘dominated strategies’. Others frame them in terms of ‘exploitation’, ‘money-pumping’, ‘Dutch Books’, ‘shooting oneself in the foot’, ‘Pareto-suboptimal behavior’, and ‘losing things that one values’ (see the Appendix for examples).
    In the context of coherence arguments, each of these terms means roughly the same thing: a strategy A is dominated by a strategy B if and only if A is worse than B in some respect that the agent cares about and A is not better than B in any respect that the agent cares about. If the agent chooses A over B, they have behaved Pareto-suboptimally, shot themselves in the foot, and lost something that they value. If the agent’s loss is someone else’s gain, then the agent has been exploited, money-pumped, or Dutch-booked. Since all these phrases point to the same sort of phenomenon, I’ll save words by talking mainly in terms of ‘dominated strategies’.
    With that background, here’s a quick rendition of coherence arguments:
    There exist coherence theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
    Sufficiently-advanced artificial agents will not pursue dominated strategies.
    So, sufficiently-advanced artificial agents will be ‘coherent’: they will be representable as maximizing expected utility.
    Typically, authors go on to suggest that these expected-utility-maximizing agents are likely to behave in certain, potentially-dangerous ways. For example, such agents are likely to appear ‘goal-directed’ in some intuitive sense. They are likely to have certain instrumental goals, like acquiring power and resources. And they are likely to fight back against attempts to shut them down or modify their goals.
    There are many ways to challenge the argument stated above, and many of those challenges have been made. There are also many ways to respond to those challenges, and many of those responses have been made too. The challenge that seems to remain yet unmade is that Premise 1 is false: there are no coherence theorems.
    Cited ‘coherence theorems’ and what they actually say
    Here’s a list of theorems that have been called ‘coherence theorems’. None of these theorems state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue dominated strategies. Here’s what the theorems say:
    The Von Neumann-Morgenstern Expected Utility Theorem:
    The Von Neumann-Morgenstern Expected Utility Theorem is as follows:
    An agent can be represented as maximizing expected utility if...
    35 min
  • LW - A circuit for Python docstrings in a 4-layer attention-only transformer by StefanHex
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A circuit for Python docstrings in a 4-layer attention-only transformer, published by StefanHex on February 20, 2023 on LessWrong.
    Produced as part of the SERI ML Alignment Theory Scholars Program under the supervision of Neel Nanda - Winter 2022 Cohort.
    TL;DR: We found a circuit in a pre-trained 4-layer attention-only transformer language model. The circuit predicts repeated argument names in docstrings of Python functions, and it features
    3 levels of composition,
    a multi-function head that does different things in different parts of the prompt,
    an attention head that derives positional information using the causal attention mask.
    Epistemic Status: We believe that we have identified most of the core mechanics and information flow of this circuit. However our circuit only recovers up to half of the model performance, and there are a bunch of leads we didn’t follow yet.
    Introduction
    Click here to skip to the results & explanation of this circuit.
    What are circuits
    What do we mean by circuits? A circuit in a neural network, is a small subset of model components and model weights that (a) accounts for a large fraction of a certain behavior and (b) corresponds to a human-interpretable algorithm. A focus of the field of mechanistic interpretability is finding and better understanding the phenomena of circuits, and recently the field has focused on circuits in transformer language models. Anthropic found the small and ubiquitous Induction Head circuit in various models, and a team at Redwood found the Indirect Object Identification (IOI) circuit in GPT2-small.
    How we chose the candidate task
    We looked for interesting behaviors in a small, attention-only transformer with 4 layers, from Neel Nanda’s open source toy language models. It was trained on natural language and Python code. We scanned the code dataset for examples where the 4-layer model did much better than a similar 3 layer one, inspired by Neel's open problems list. Interestingly, despite the circuit seemingly requiring just 3 levels of composition, only the 4-layer model could do the task.
    The docstring task
    The clearest example we found was in Python docstrings, where it is possible to predict argument names in the docstring: In this randomly generated example, a function has the (randomly generated) arguments load, size, files, and last. The docstring convention here demands each line starting with :param followed by an argument name, and this is very predictable. Turns out that attn-only-4l is capable of this task, predicting the next token (files in the example shown here) correctly in ~75% of cases.
    Methods: Investigating the circuit
    Possible docstring algorithms
    There are multiple algorithms which could solve this task, such as
    "Docstring Induction": Always predict the argument that, in the definition, follows the argument seen in the previous docstring line. I.e. look for param size, check the order in the definition size, files, and predict files accordingly.
    Line number based: In the Nth line predict the Nth variable from the definition, irrespective of the content of the other lines. I.e after the 3rd param token, predict the 3rd variable files.
    Inhibition based: Predict variable names from the definition, but inhibit variable names which occurred twice (similar to the inhibition in the IOI circuit), i.e. predict load, size, files, last, and inhibit the former two. Add some preference for earlier tokens to prefer files over last.
    We are quite certain that at least the first two algorithms are implemented to some degree. This is surprising, since one of the two should be sufficient to perform the task; we do not investigate further why this is the case. A brief investigation showed that the implementation of the 2nd algorithm seems less robust and less generalizable that our model's implementation of th...
    34 min
  • LW - Metaculus Introduces New 'Conditional Pair' Forecast Questions for Making Conditional Predictions by ChristianWilliams
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Metaculus Introduces New 'Conditional Pair' Forecast Questions for Making Conditional Predictions, published by ChristianWilliams on February 20, 2023 on LessWrong.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min