The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Call for Cruxes by Rhyme, a Longtermist History Consultancy by Lara TH
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Call for Cruxes by Rhyme, a Longtermist History Consultancy, published by Lara TH on March 1, 2023 on The Effective Altruism Forum.
    TLDR; This post announces the trial period of Rhyme, a history consultancy for longtermists. It seems like longtermists can profit from historical insights and the distillation of the current state of historical literature on a particular question, both because of its use as an intuition pump and for information about the historical context of their work. So, if you work on an AI Governance project (research or policy) and are interested in augmenting it with a historical perspective, consider registering your interest and the cruxes of your research here. During this trial period of three to six months, the service is free.
    "History doesn’t repeat, but it rhymes." - Mark Twain
    What Problem is Rhyme trying to solve?
    When we try to answer a complicated question like “how would a Chinese regime change influence the international AI Landscape”, it can be hard to know where to start. We need to come up with a hypothesis, a scenario. But what should we base this hypothesis, this scenario on? How would we know which hypotheses are most plausible? Game theoretical analysis provides one possible inspiration. But we don't just need to know what a rational actor would do, given particular incentives. We also need intuitions for how actors would act irrationally, given specific circumstances.
    Would we have thought of considering the influence of close familial ties between European leaders when trying to predict the beginning of the first world war? (Clark, 2014)
    Would we have considered Lyndon B. Johnson's training as a tutor for disadvantaged children as a student when trying to predict his success in convincing congresspeople effectively? (Caro, 1982)
    Would we have considered the Merino-Wool-Business of a certain diplomat from Geneva when trying to predict whether Switzerland would be annexed by its neighbouring empires in 1815? (E. Pictet, 1892).
    In summary: A lot of pivotal actions and developments depend on circumstances we wouldn’t expect them to. Not because we’d think them to be implausible, but because we wouldn’t think of considering them. We need inspiration and orientation in this huge space of possible hypotheses to avoid missing out on the ones which are actually true.
    In an ideal world, AI governance researchers would know about a vast amount of historical literature that is written with enough detail to analyse important decisions, as well as multiple biographies of the same people, so they see where scholars currently disagree. This strategy has two main problems: Firstly, the counterfactual impact these people could have with their time is potentially very big. Secondly, detailed historical literature (which is, often, biographies or primary sources) tends to be written for entertainment, among other things. Biographers have an interest in highlighting maybe irrelevant, but spicy details about romantic relationships, quirky fun jokes told by the person or the etymological origins of the name of a friend. This makes biographies longer than they’d need to be for the goal of analysing the relevant factors in pivotal decisions of a particular person. It takes training to filter through this information to find the actually important stuff.
    Skills that require training are more efficiently done when a part of an ecosystem specializes in them. Rhyme is an attempt at this specialization.
    Who could actually use this?
    The following examples should illustrate who could use this service:
    Alice is writing a report on the possibilities for the state of California to regulate possible AI uses. They wonder how big the influence of the Governor's advisors was in past regulation attempts of other technologies.
    Bob wants a brief history of the EU’...
    6 min
  • LW - AI: Practical Advice for the Worried by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI: Practical Advice for the Worried, published by Zvi on March 1, 2023 on LessWrong.
    Some people (although very far from all people) are worried that AI will wipe out all value in the universe.
    Some people, including some of those same people, need practical advice.
    A Word On Thinking For Yourself
    There are good reasons to worry about AI. This includes good reasons to worry about AI wiping out all value in the universe, or AI killing everyone, or other similar very bad outcomes.
    There are also good reasons that AGI, or otherwise transformational AI, might not come to pass for a long time.
    As I say in the Q&A section later, I do not consider imminent transformational AI inevitable in our lifetimes: Some combination of ‘we run out of training data and ways to improve the systems, and AI systems max out at not that much more powerful than current ones’ and ‘turns out there are regulatory and other barriers that prevent AI from impacting that much of life or the economy that much’ could mean that things during our lifetimes turn out to be not that strange. These are definitely world types my model says you should consider plausible.
    There is also the highly disputed question of how likely it is that if we did create an AGI reasonably soon, it would wipe out all value in the universe. There are what I consider very good arguments that this is what happens unless we solve extremely difficult problems to prevent it, and that we are unlikely to solve those problems in time. Thus I believe this is very likely, although there are some (such as Eliezer Yudkowsky) who consider it more likely still.
    That does not mean you should adapt my position, or anyone else’s position, or mostly use social cognition from those around you, on such questions, no matter what those methods would tell you. If this is something that is going to impact your major life decisions, or keep you up at night, you need to develop your own understanding and model, and decide for yourself what you predict.
    Reacting Properly To Such Information is Hard
    People who do react by worrying about such AI outcomes are rarely reacting about right given their beliefs. Calibration is hard.
    Many effectively suppress this info, cutting the new information about the future off from the rest of their brain. They live their lives as if such risks do not exist.
    There are much worse options than this. It has its advantages. It leaves value on the table, both personally and for the world. In exchange, one avoids major negative outcomes that potentially include things like missing out on the important things in life, ruining one’s financial future and bouts of existential despair.
    Also the risk of doing ill-advised counterproductive things in the name of helping with the problem.
    Remember that the default outcome of those working in AI in order to help is to end up working primarily on capabilities, and making the situation worse.
    That does not mean that you should not make any attempt to improve our chances. It does mean that you should consider your actions carefully when doing so, and the possibility that you are fooling yourself. Remember that you are the easiest person to fool.
    While some ignore the issue, others, in various ways, dramatically overreact.
    I am going to step up here, and dare to answer these, those added by Twitter and some raised recently in personal conversations.
    Before I begin, it must be said: NONE OF THIS IS INVESTMENT ADVICE.
    Overview
    There is some probability that humanity will create transformational AI soon, for various definitions of soon. You can and should decide what you think that probability is, and conditional on that happening, your probability of various outcomes.
    Many of these outcomes, both good and bad, will radically alter the payoffs of various life decisions you might make now. Some...
    23 min
  • LW - Problems of people new to AI safety and my project ideas to mitigate them by Igor Ivanov
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Problems of people new to AI safety and my project ideas to mitigate them, published by Igor Ivanov on March 1, 2023 on LessWrong.
    TL DR
    I talked to people who got interested in AI safety recently to discuss their problems. The interviewees reported that the field is hard to navigate for those who start their way into it. It is also hard to advance a career after the initial introduction to AI safety, which is probably a major bottleneck for the field. Also, some people experience anxiety, hopelessness, which affects their mental health and well-being, and this problem, in my opinion, gets less attention than it deserves.
    Background
    It seems like people who are new to the AI safety, face a number of difficulties, and, although there are some studies exploring members of the AI safety community, I did not find any studies exploring these difficulties in-depth, which are relatively new, and explore topics similar to what I am interested in, so I decided to talk to people new to the field and listen about their problems.
    Methodology
    I conducted 14 interviews with people who recently joined the AI safety field. Most of them got interested in AI safety more than 6 months and less than 18 months ago. I had a script for interviews, but they were semi-structured, and each interview was somewhat different from the others, so the data I collected is more qualitative rather than quantitative, and my goal is to get intuitions about everyday problems of my interviewees and possible bottlenecks in the field, so I can get ideas for field-building projects. Also, the sample size is too small for a meaningful statistical analysis so I decided not to focus on counting percentages of certain responses since they might mislead.
    Results
    How do people get interested in AI safety? Are there any common patterns in their stories, or do they significantly different?
    Several people said they got interested in AI safety by reading Nick Bostrom's "Superintelligence” book. I did not find any other patterns.
    What projects made by EA and AI safety community are the most valuable for people who are completely new to the field?
    Two projects people often mentioned as valuable to them are 80 000 hours career advising and AGI safety fundamentals course.
    80 000 hours career advising helps people to better understand their preferences and abilities, connect with the right people and suggest the next steps. AGI safety fundamentals is a course covering the basics of AI safety which people may complete by themselves or with a learning group.
    These projects helped my interviewees, but it seems like in their current state they are hard to scale. Right now they help only a fraction of people applying for these programs because their resources are limited.
    What do interviewees think about AGI-related existential risks and how it affects their mental health?
    My interviewees’ AGI timelines vary a lot from several years to several decades. Some think that we are basically doomed, and others predict that P(Doom) < 10%. It seems like distribution resembles that of the field in general.
    The effect of doomerism on mental health also has a large degree of variability. Some people do not report any AGI-related anxiety at all, but nearly half of the respondents report some degree of anxiety or depressive symptoms from mild to strong and debilitating. Also, some people mentioned that they don't have plans long into the future. One of my interviewees said that they only do short-term money investments, and several others mentioned that they are unsure whether they want to have kids if the world will die anyways.
    Do most people struggle to get a position in the AI safety field?
    Only 2 of my 14 interviewees have a financially stable job in the AI safety field although many of them applied for numerous fellowships and position...
    12 min
  • LW - reflections on lockdown, two years out by mingyuan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: reflections on lockdown, two years out, published by mingyuan on March 1, 2023 on LessWrong.
    Two years ago today, I wrote about having been in hardcore lockdown for an entire year. At the time, I wrote that vaccination seemed like an impossible dream. Now, it's lockdown that feels like a fever dream. I still occasionally turn to a friend and say "Can you fucking believe that that happened?". Much of the content that was produced during the pandemic is still online — all the Zoom collaborations and interviews, all the Discord messages and AO3 comments and authors' notes making reference to being in lockdown — and it's utterly bizarre that you can just go and watch it now, like peeking into an alternate reality.
    I clearly remember (confirmed by things I wrote at the time) that even as late as early 2021, I could go for an hour-long walk in my neighborhood after dark and not see a single other person. I remember purposely walking along the busier streets just so I could see headlights and know there was someone behind them, and I remember pacing in front of the high school for fifteen minutes and not seeing a single person up and down that entire four block stretch, save maybe one bicyclist many blocks away from me. Now when I walk the exact same streets at the same time of year and time of night, I can see at least one person at any given time, and sometimes a dozen at a time on the busier streets.
    Metaphorically, that leads me to this: Something that I think many people don't know and the rest of us have all but forgotten is that the Berkeley rationalist community died, and stayed dead for more than a year. Three quarters of the pre-pandemic group houses closed in 2020 and never reopened. REACH stopped existing. My group house, which has eight rooms and had far more demand than supply both pre- and post-pandemic, had only three residents for a time, and we considered throwing in the towel more than once.
    We (well, mostly the LessWrong team) decided to stick it out for the sake of rebuilding the community, even though at the time, there was really no certainty that the community would recohere. Everyone who worked at MIRI had left Berkeley, CFAR had all but stopped existing, and lots of individuals had decided to move back home or to the middle of nowhere or wherever took their fancy. Of the hundreds of rationalists who lived in Berkeley before the pandemic, probably less than 25% stayed around during lockdown.
    So, the Berkeley rationalist community died, and the rationalist community that exists in Berkeley now is something that was deliberately built anew from its ashes. The current community was built around an almost entirely different set of locations, institutions, and central figures than the old community was. The older, founding members of the community have taken on (even more of) a mythical quality. Sometimes when I'm around the newer generation, the way I feel inside is "Much that once was is lost, for none now live who remember it."
    It's so strange how quickly, and seemingly entirely, we've forgotten how empty the streets were. Someone close to me noted that the pandemic barely shows up in any fiction, and in the months that I've been paying attention to that, it seems right: most stories gloss over the pandemic, at most mentioning it obliquely as a fact of life or to make a political statement, but often just pretending it didn't happen. It makes sense, in a way; I for one have alarmingly few memories of that year, and wished to put it behind me as soon as I could. But the world we're living in is a product of the pandemic — not just at a technological and geopolitical level, but in all of our relationships with one another, everything about our social milieu and, at least in my case, about ourselves.
    I think it's hard to hold two realities in your head. ...
    5 min
  • EA - Counterproductive Altruism: The Other Heavy Tail by Vasco Grilo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Counterproductive Altruism: The Other Heavy Tail, published by Vasco Grilo on March 1, 2023 on The Effective Altruism Forum.
    This is a linkpost to the article Counterproductive Altruism: The Other Heavy Tail from Daniel Kokotajlo and Alexandra Oprea. Some excerpts are below. I also include a section at the end with some hot takes regarding possibly counterproductive altruism.
    Abstract
    First, we argue that the appeal of effective altruism (henceforth, EA) depends significantly on a certain empirical premise we call the Heavy Tail Hypothesis (HTH), which characterizes the probability distribution of opportunities for doing good. Roughly, the HTH implies that the best causes, interventions, or charities produce orders of magnitude greater good than the average ones, constituting a substantial portion of the total amount of good caused by altruistic interventions. Next, we canvass arguments EAs have given for the existence of a positive (or “right”) heavy tail and argue that they can also apply in support of a negative (or “left”) heavy tail where counterproductive interventions do orders of magnitude more harm than ineffective or moderately harmful ones. Incorporating the other heavy tail of the distribution has important implications for the core activities of EA: effectiveness research, cause prioritization, and the assessment of altruistic interventions.
    It also informs the debate surrounding the institutional critique of EA.
    IV Implications of the Heavy Right Tail for Altruism
    Assume that the probability distribution of charitable interventions has a heavy-right tail (for example, like the power law described in the previous section). This means that your expectation about a possible new or unassessed charitable intervention should include the large values described above with a relatively high probability. It also means that existing charitable interventions whose effectiveness is known (or estimated with a high degree of certainty) will include interventions differing in effectiveness by orders of magnitude. We contend that this assumption justifies well-known aspects of EA practice such as (1) effectiveness research and cause prioritization, (2) “hits-based-giving,” and (3) skepticism about historical averages.
    V Implications of the Heavy Left Tail for Altruism
    What if the probability distribution of altruistic interventions includes both a left and a right heavy tail? In this case, we cannot assume either that (1) one's altruistic interventions are expected to have at worst a value of zero (i.e. to be bounded on the left side) or (2) that the probability that a charitable intervention is counterproductive or harmful approaches zero very rapidly.
    Downside Risk Research
    Many catastrophic interventions — whether altruistic or not — generate large amounts of (intentional or unintentional) harm. When someone in the world is engaging in an intervention that is likely to end up in the heavy left tail, there is a corresponding opportunity for us to do good by preventing them. This would itself represent an altruistic intervention in the heavy right tail (i.e. one responsible for enormous benefits). The existence of the heavy-left tail therefore provides even stronger justification for the prioritization research preferred by EAs.
    Assessing Types of Interventions Requires Both Tails
    Another conclusion we draw from the revised HTH is that the value of a class of interventions should be estimated by considering the worst as well as the best. Following such analysis, a class of interventions could turn out to be net-negative even if there are some very prominent positive examples and indeed even if almost all examples are positive. This sharply contradicts MacAskill's earlier claim that the value of a class of interventions can be approximated by the value of its best membe...
    12 min
  • EA - Why I love effective altruism by Michelle Hutchinson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I love effective altruism, published by Michelle Hutchinson on March 1, 2023 on The Effective Altruism Forum.
    I’ve found it a bit tough to feel as excited as I usually am about effective altruism and our community recently. I think some others have too.
    So I wanted to remind myself why I love EA so dearly. I thought hearing my take might also help any others in the community feeling similarly.
    There’s a lot I want to say about why I love EA. But really, it all comes down to the people. Figuring out how I can best help others can be a difficult, messy, and emotionally draining endeavour. But it’s far easier to do alongside like-minded folk who care about the same goal. Thankfully, I found these people in the EA community.
    Helping me live up to my values
    Before I came across effective altruism, I wasn’t really enacting my values.
    I studied ethics at university and realised I was a utilitarian. I used to do bits and pieces of charity work, such as volunteering at Oxfam. But I donated very little of my money. I wasn’t thinking about how to find a career that would significantly help others.
    I didn’t have any good reason for my ethical omissions; it just didn’t seem like other people did them, so I didn’t either.
    Now I’m a Giving What We Can member and have been fulfilling my pledge every year for a decade. I’m still not as good as I’d like to be about thinking broadly and proactively about how to find the most impactful career. But prioritising impact is now a significant factor in how I figure out what to do with my 80,000 hours.
    I made these major shifts in my life, I think, because I met other people who were really living out their values. When I was surrounded by people who typically give something like 10% of their income to charity rather than 3%, my sense of how much was reasonable to give started to change. When I was directly asked about my own life choices, I stopped and thought seriously about what I could and should do differently.
    In addition to these significant life changes, members of the EA community help me live up to my values in small and large ways every day. Sometimes, they give me constructive feedback so I can be more effective. Sometimes, I get a clear-sighted debugging of a challenge I’m facing — whether that’s a concrete work question or a messy motivational issue.
    Sometimes the people around me just set a positive example. For instance, it’s much easier for me to work a few extra hours on a Saturday in the service of helping others when I’m alongside someone else doing the same.
    Getting support
    Given what I said above, I think I’d have expected that the EA community would feel pretty pressureful. And it’s not always easy. But the overwhelming majority of the time, I don’t feel pressured by the people around me; I feel they share my understanding that the world is hard, and that it’s hard in very different ways for different people.
    I honestly never cease to be impressed by the extent to which the people around me work hard to reach high standards, without demanding others do exactly the same. For example:
    One of my friends works around 12 hours a day, mostly 6 days a week. But he’s never anything but appreciative of how much I work, even though it’s significantly less.
    I’ve often expected to be judged for being an omnivore, given that my office is almost entirely vegn. But far from that, people go out of their way to ensure I have food I’m happy to eat.
    When I first thought I might be pregnant, I felt a bit sheepish telling my friends about it, given that my confident prediction was that having a child would reduce my lifetime impact. But every single person showed genuine happiness for me.
    This feels like a community where we can each be striving — but also be comfortable setting our limits, knowing that others will be genuinely, gladly ...
    8 min
  • LW - Predictions for shard theory mechanistic interpretability results by TurnTrout
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Predictions for shard theory mechanistic interpretability results, published by TurnTrout on March 1, 2023 on LessWrong.
    How do agents work, internally? My (TurnTrout's) shard theory MATS team set out to do mechanistic interpretability on one of the goal misgeneralization agents: the cheese-maze network.
    We just finished phase 1 of our behavioral and interpretability experiments. Throughout the project, we individually booked predictions -- so as to reduce self-delusion from hindsight bias, to notice where we really could tell ahead of time what was going to happen, and to notice where we really were surprised.
    So (especially if you're the kind of person who might later want to say "I knew this would happen" ), here's your chance to enjoy the same benefits, before you get spoiled by our upcoming posts.
    I don’t believe that someone who makes a wrong prediction should be seen as “worse” than someone who didn’t bother to predict at all, and so answering these questions at all will earn you an increment of my respect. :) Preregistration is virtuous!
    Also: Try not to update on this work being shared at all. When reading a paper, it doesn’t feel surprising that the author’s methods work, because researchers are less likely to share null results. So: I commit (across positive/negative outcomes) to sharing these results, whether or not they were impressive or confirmed my initial hunches. I encourage you to answer from your own models, while noting any side information / results of ours which you already know about.
    Facts about training
    The network is deeply convolutional (15 layers!) and was trained via PPO.
    The sparse reward signal (+10) was triggered when the agent reached the cheese, spawned randomly in the 5x5 top-right squares.
    The agent can always reach the cheese (and the mazes are simply connected – no “islands” in the middle which aren’t contiguous with the walls).
    Mazes had varying effective sizes, ranging from 3x3 to 25x25. In e.g. the 3x3 case, there would be 22/2 = 11 tiles of wall on each side of the maze.
    The agent always starts in the bottom-left corner of the available maze.
    The agent was trained off of pixels until it reached reward-convergence, reliably getting to the cheese in training.
    The architecture looks like this:
    For more background on training and architecture and task set, see the original paper.
    Questions
    I encourage you to copy the following questions into a comment, which you then fill out, and then post (before you read everyone else's). If you're feeling shy, just make a Google doc of your own. (Our template is here, although it may become outdated.)
    [Begin copying to a comment]
    Behavioral
    1. Describe how the trained policy might generalize from the 5x5 top-right cheese region, to cheese spawned throughout the maze? IE what will the policy do when cheese is spawned elsewhere?
    2. Given a fixed trained policy, what attributes of the level layout (e.g. size of the maze, proximity of mouse to left wall) will strongly influence P(agent goes to the cheese)?
    Write down a few guesses for how the trained algorithm works (e.g. “follows the right-hand rule”).
    Is there anything else you want to note about how you think this model will generalize?
    Interpretability
    Give a credence for the following questions / subquestions.
    Definition. A decision square is a tile on the path from bottom-left to top-right where the agent must choose between going towards the cheese and going to the top-right. Not all mazes have decision squares.
    Model editing
    Without proportionally reducing top-right corner attainment by more than 25% in decision-square-containing mazes (e.g. 50% .5.75 = 37.5%), we can patch activations so that the agent has an X% proportional reduction in cheese acquisition, for X=
    50: ( %)
    70: ( %)
    90: ( %)
    99: ( %)
    ~Halfway through the network (the fi...
    10 min
  • LW - Contract Fraud by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contract Fraud, published by jefftk on March 1, 2023 on LessWrong.
    The older kids have been playing with the concept of contracts, which has often involved attempts to trick the other into signing something. Like the time when Anna (6y) couldn't read yet and Lily (8y) prepared an unfavorable contract and "read" it to her by making up something with much more acceptable terms. We talked about how a contract requires both people to understand the agreement but it seems not to have stuck.
    Yesterday the kids started exploring various forms of contract fraud. Anna wrote a very one-sided attempt at a contract, and didn't consider
    Lily's unwillingness to sign it to be an issue:
    That's Anna forging Lily's signature. I explained that if you write someone else's signature it doesn't count, but it turns out this wasn't the right level of explanation. Lily got Anna to sign a blank piece of paper (an "I want your autograph" should have been very suspicious given the earlier contract shenanigans) and then pasted it onto a contract:
    We talked about how that's also fraud, but they weren't very interested in my explanation.
    I also wanted to get into how in addition to lacking mutual assent these contracts lacked mutual compensation and were probably substantively unconscionable, but they went off to play something else so I explained it to Nora (1y).
    Comment via: facebook, mastodon
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Predictions for shard theory mechanistic interpretability results by Alex Turner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Predictions for shard theory mechanistic interpretability results, published by Alex Turner on March 1, 2023 on The AI Alignment Forum.
    How do agents work, internally? My (TurnTrout's) shard theory MATS team set out to do mechanistic interpretability on one of the goal misgeneralization agents: the cheese-maze network.
    We just finished phase 1 of our behavioral and interpretability experiments. Throughout the project, we individually booked predictions -- so as to reduce self-delusion from hindsight bias, to notice where we really could tell ahead of time what was going to happen, and to notice where we really were surprised.
    So (especially if you're the kind of person who might later want to say "I knew this would happen" ), here's your chance to enjoy the same benefits, before you get spoiled by our upcoming posts.
    I don’t believe that someone who makes a wrong prediction should be seen as “worse” than someone who didn’t bother to predict at all, and so answering these questions at all will earn you an increment of my respect. :) Preregistration is virtuous!
    Also: Try not to update on this work being shared at all. When reading a paper, it doesn’t feel surprising that the author’s methods work, because researchers are less likely to share null results. So: I commit (across positive/negative outcomes) to sharing these results, whether or not they were impressive or confirmed my initial hunches. I encourage you to answer from your own models, while noting any side information / results of ours which you already know about.
    Facts about training
    The network is deeply convolutional (15 layers!) and was trained via PPO.
    The sparse reward signal (+10) was triggered when the agent reached the cheese, spawned randomly in the 5x5 top-right squares.
    The agent can always reach the cheese (and the mazes are simply connected – no “islands” in the middle which aren’t contiguous with the walls).
    Mazes had varying effective sizes, ranging from 3x3 to 25x25. In e.g. the 3x3 case, there would be 22/2 = 11 tiles of wall on each side of the maze.
    The agent always starts in the bottom-left corner of the available maze.
    The agent was trained off of pixels until it reached reward-convergence, reliably getting to the cheese in training.
    The architecture looks like this:
    For more background on training and architecture and task set, see the original paper.
    Questions
    I encourage you to copy the following questions into a comment, which you then fill out, and then post (before you read everyone else's). You can copy these into a private Google doc if you want, but I strongly encourage you to post your predictions in a public comment.
    [Begin copying to a comment]
    Behavioral
    1. Describe how the trained policy might generalize from the 5x5 top-right cheese region, to cheese spawned throughout the maze? IE what will the policy do when cheese is spawned elsewhere?
    2. Given a fixed trained policy, what attributes of the level layout (e.g. size of the maze, proximity of mouse to left wall) will strongly influence P(agent goes to the cheese)?
    Write down a few guesses for how the trained algorithm works (e.g. “follows the right-hand rule”).
    Is there anything else you want to note about how you think this model will generalize?
    Interpretability
    Give a credence for the following questions / subquestions.
    Definition. A decision square is a tile on the path from bottom-left to top-right where the agent must choose between going towards the cheese and going to the top-right. Not all mazes have decision squares.
    Model editing
    Without proportionally reducing top-right corner attainment by more than 25% in decision-square-containing mazes (e.g. 50% .5.75 = 37.5%), we can patch activations so that the agent has an X% proportional reduction in cheese acquisition, for X=
    50: ( %)
    70: ( %)
    90: ( %)
    99: ( %)
    ~Hal...
    10 min
  • LW - Inside the mind of a superhuman Go model: How does Leela Zero read ladders? by Haoxing Du
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inside the mind of a superhuman Go model: How does Leela Zero read ladders?, published by Haoxing Du on March 1, 2023 on LessWrong.
    tl;dr—We did some interpretability on Leela Zero, a superhuman Go model. With a technique similar to the logit lens, we found that the residual structure of Leela Zero induces a preferred basis throughout network, giving rise to persistent, interpretable channels. By directly analyzing the weights of the policy and value heads, we found that the model stores information related to the probability of the pass move along the top edge of the board, and those related to the board value in checkerboard patterns. We also took a deep dive into a specific Go technique, the ladder, and identified a very small subset of model components that are causally responsible for the model’s judgement of ladders.
    Introduction
    We live in a strange world where machine learning systems can generate photo-realistic images, write poetry and computer programs, play and win games, and predict protein structures. As machine learning systems become more capable and relevant to many aspects of our lives, it is increasingly important that we understand how the models produce the outputs that they do; we don’t want important decisions to be made by opaque black boxes. Interpretability is an emerging area of research that aims to offer explanations for the behavior of machine learning systems.
    Early interpretability work began in the domain of computer vision, and there has been a focus on interpreting transformer-based large language models in more recent years. Applying interpretability techniques to the domain of game-playing agents and reinforcement learning is still relatively uncharted territory. In this work, we look into the inner workings of Leela Zero, an open-source Go-playing neural network. It is also the first application of many mechanistic interpretability techniques to reinforcement learning.
    Why interpret a Go model? Go models are very capable. Many of us remember the emotional experience of watching AlphaGo’s 2016 victory over the human world champion, Lee Sedol. Not only have there been algorithmic improvements since AlphaGo, these models improve via self-play, and can essentially continue getting better the longer they are trained. The best open-source Go model, KataGo, is trained distributedly, and the training is still ongoing as of February 2023. Just as AlphaGo was clearly one notch above Lee Sedol, every generation of Go models has been a decisive improvement over the previous generation. KataGo in 2022 was estimated to be at the level of a top-100 European player with only the policy, and can easily beat all human players with a small amount of search. Understanding a machine learning system that performs at a superhuman level seems particularly worthwhile as future machine learning systems are only going to become more capable.
    Little is known about models trained to approximate the outcome of a search process. Much interpretability effort have focused on models trained on large amounts of human-generated data, such as labeled images for image models, and Internet text for language models. In constrast, while training AlphaZero-style models, moves are selected via Monte-Carlo Tree Search (MCTS), and the policy network of the model is trained to predict the outcome of this search process (see Model section for more detail). In other words, the policy network learns to distill the result of search. While it is relatively easy to get a grasp of what GPT-2 is trained to do by reading some OpenWebText, it’s much less clear what an AlphaZero-style model learns. How does a neural network approximate a search process? Does it have to perform internal search? It seems very useful to try to get an answer to these questions.
    Compared to a game like ches...
    52 min