The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Still no Lie Detector for LLMs by Whispermute
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Still no Lie Detector for LLMs, published by Whispermute on July 18, 2023 on LessWrong.
    Background
    This post is a short version of a paper we wrote that you can find here. You can read this post to get the core ideas. You can read the paper to go a little deeper.
    The paper is about probing decoder-only LLMs for their beliefs, using either unsupervised methods (like CCS from Burns) or supervised methods. We give both philosophical/conceptual reasons we are pessimistic and demonstrate some empirical failings using LLaMA 30b. By way of background, we're both philosophers, not ML people, but the paper is aimed at both audiences.
    Introduction
    One child says to the other "Wow! After reading some text, the AI understands what water is!". The second child says "All it understands is relationships between words. None of the words connect to reality. It doesn't have any internal concept of what water looks like or how it feels to be wet. ." .
    Two angels are watching [some] chemists argue with each other. The first angel says "Wow! After seeing the relationship between the sensory and atomic-scale worlds, these chemists have realized that there are levels of understanding humans are incapable of accessing." The second angel says "They haven't truly realized it. They're just abstracting over levels of relationship between the physical world and their internal thought-forms in a mechanical way. They have no concept of [$!&&!@] or [#@&#]. You can't even express it in their language!"
    Scott Alexander, Meaningful
    Do large language models (LLMs) have beliefs? And, if they do, how might we measure them?
    These questions are relevant as one important problem that plagues current LLMs is their tendency to generate falsehoods with great conviction. This is sometimes called lying and sometimes called hallucinating. One strategy for addressing this problem is to find a way to read the beliefs of an LLM directly off its internal state. Such a strategy falls under the broad umbrella of model interpretability, but we can think of it as a form of mind-reading. Detecting lies in LLMs has many obvious applications, and is especially relevant for things like ELK.
    We tackle the question about the status of beliefs in LLMs head-on. We proceed in two stages. First, we assume that LLMs do have beliefs, and consider two current approaches for how we might measure them, due to Azaria and Mitchell and to Burns et al. We provide empirical results from LLaMA 30b that show that these methods fail to generalize in very basic ways. We then argue that, even if LLMs have beliefs, these methods are unlikely to be successful for conceptual reasons. Thus, there is still no lie-detector for LLMs.
    After describing our empirical results we take a step back and consider whether or not we should expect LLMs to have something like beliefs in the first place. We consider some recent arguments aiming to show that LLMs cannot have beliefs. We show that these arguments are misguided and rely on a philosophical mistake. We provide a more productive framing of questions surrounding the status of beliefs in LLMs. Our analysis reveals both that there are many contexts in which we should expect systems to track the truth in order to accomplish other goals but that the question of whether or not LLMs have beliefs is largely an empirical matter. We provide code at.
    Challenge in Deciphering the Beliefs of Language Models
    For now, let's assume that in order to generate human-like text, LLMs (like humans) have beliefs about the world. We might then ask how we can measure and discover their beliefs. This question immediately leads to a number of problems:
    Unreliable Self-Reporting
    Asking an LLM directly about its beliefs is insufficient. As we've already discussed, models have a tendency to hallucinate or even lie. So belief reports a...
    38 min
  • EA - What do XPT forecasts tell us about AI risk? by Forecasting Research Institute
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What do XPT forecasts tell us about AI risk?, published by Forecasting Research Institute on July 19, 2023 on The Effective Altruism Forum.
    This post was co-authored by the Forecasting Research Institute and Rose Hadshar. Thanks to Josh Rosenberg for managing this work, Zachary Jacobs and Molly Hickman for the underlying data analysis, Coralie Consigny and Bridget Williams for fact-checking and copy-editing, the whole FRI XPT team for all their work on this project, and our external reviewers.
    In 2022, the Forecasting Research Institute (FRI) ran the Existential Risk Persuasion Tournament (XPT). From June through October 2022, 169 forecasters, including 80 superforecasters and 89 experts, developed forecasts on various questions related to existential and catastrophic risk. Forecasters moved through a four-stage deliberative process that was designed to incentivize them not only to make accurate predictions but also to provide persuasive rationales that boosted the predictive accuracy of others' forecasts. Forecasters stopped updating their forecasts on 31st October 2022, and are not currently updating on an ongoing basis. FRI plans to run future iterations of the tournament, and open up the questions more broadly for other forecasters.
    You can see the overall results of the XPT here.
    Some of the questions were related to AI risk. This post:
    Sets out the XPT forecasts on AI risk, and puts them in context.
    Lays out the arguments given in the XPT for and against these forecasts.
    Offers some thoughts on what these forecasts and arguments show us about AI risk.
    TL;DR
    XPT superforecasters predicted that catastrophic and extinction risk from AI by 2030 is very low (0.01% catastrophic risk and 0.0001% extinction risk).
    XPT superforecasters predicted that catastrophic risk from nuclear weapons by 2100 is almost twice as likely as catastrophic risk from AI by 2100 (4% vs 2.13%).
    XPT superforecasters predicted that extinction risk from AI by 2050 and 2100 is roughly an order of magnitude larger than extinction risk from nuclear, which in turn is an order of magnitude larger than non-anthropogenic extinction risk (see here for details).
    XPT superforecasters more than quadruple their forecasts for AI extinction risk by 2100 if conditioned on AGI or TAI by 2070 (see here for details).
    XPT domain experts predicted that AI extinction risk by 2100 is far greater than XPT superforecasters do (3% for domain experts, and 0.38% for superforecasters by 2100).
    Although XPT superforecasters and experts disagreed substantially about AI risk, both superforecasters and experts still prioritized AI as an area for marginal resource allocation (see here for details).
    It's unclear how accurate these forecasts will prove, particularly as superforecasters have not been evaluated on this timeframe before.
    The forecasts
    In the table below, we present forecasts from the following groups:
    Superforecasters: median forecast across superforecasters in the XPT.
    Domain experts: median forecasts across all AI experts in the XPT.
    (See our discussion of aggregation choices (pp. 20-22) for why we focus on medians.)
    QuestionForecastersN203020502100AI Catastrophic risk (>10% of humans die within 5 years)Superforecasters880.01%0.73%2.13%Domain experts300.35%5%12%AI Extinction risk (human population <5,000)Superforecasters880.0001%0.03%0.38%Domain experts290.02%1.1%3%
    The forecasts in context
    Different methods have been used to estimate AI risk:
    Surveying experts of various kinds, e.g. Sanders and Bostrom, 2008; Grace et al. 2017.
    Doing in-depth investigations, e.g. Ord, 2020; Carlsmith, 2021.
    The XPT forecasts are distinctive relative to expert surveys in that:
    The forecasts were incentivized: for long-run questions, XPT used 'reciprocal scoring' rules to incentivize accurate forecasts (see here for details).
    Fore...
    52 min
  • LW - Alignment Grantmaking is Funding-Limited Right Now by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Grantmaking is Funding-Limited Right Now, published by johnswentworth on July 19, 2023 on LessWrong.
    For the past few years, I've generally mostly heard from alignment grantmakers that they're bottlenecked by projects/people they want to fund, not by amount of money. Grantmakers generally had no trouble funding the projects/people they found object-level promising, with money left over. In that environment, figuring out how to turn marginal dollars into new promising researchers/projects - e.g. by finding useful recruitment channels or designing useful training programs - was a major problem.
    Within the past month or two, that situation has reversed. My understanding is that alignment grantmaking is now mostly funding-bottlenecked. This is mostly based on word-of-mouth, but for instance, I heard that the recent lightspeed grants round received far more applications than they could fund which passed the bar for basic promising-ness. I've also heard that the Long-Term Future Fund (which funded my current grant) now has insufficient money for all the grants they'd like to fund.
    I don't know whether this is a temporary phenomenon, or longer-term. Alignment research has gone mainstream, so we should expect both more researchers interested and more funders interested. It may be that the researchers pivot a bit faster, but funders will catch up later. Or, it may be that the funding bottleneck becomes the new normal. Regardless, it seems like grantmaking is at least funding-bottlenecked right now.
    Some takeaways:
    If you have a big pile of money and would like to help, but haven't been donating much to alignment because the field wasn't money constrained, now is your time!
    If this situation is the new normal, then earning-to-give for alignment may look like a more useful option again. That said, at this point committing to an earning-to-give path would be a bet on this situation being the new normal.
    Grants for upskilling, training junior people, and recruitment make a lot less sense right now from grantmakers' perspective.
    For those applying for grants, asking for less money might make you more likely to be funded. (Historically, grantmakers consistently tell me that most people ask for less money than they should; I don't know whether that will change going forward, but now is an unusually probable time for it to change.)
    Note that I am not a grantmaker, I'm just passing on what I hear from grantmakers in casual conversation. If anyone with more knowledge wants to chime in, I'd appreciate it.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - Alignment Grantmaking is Funding-Limited Right Now by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Grantmaking is Funding-Limited Right Now, published by johnswentworth on July 19, 2023 on The AI Alignment Forum.
    For the past few years, I've generally mostly heard from alignment grantmakers that they're bottlenecked by projects/people they want to fund, not by amount of money. Grantmakers generally had no trouble funding the projects/people they found object-level promising, with money left over. In that environment, figuring out how to turn marginal dollars into new promising researchers/projects - e.g. by finding useful recruitment channels or designing useful training programs - was a major problem.
    Within the past month or two, that situation has reversed. My understanding is that alignment grantmaking is now mostly funding-bottlenecked. This is mostly based on word-of-mouth, but for instance, I heard that the recent lightspeed grants round received far more applications than they could fund which passed the bar for basic promising-ness. I've also heard that the Long-Term Future Fund (which funded my current grant) now has insufficient money for all the grants they'd like to fund.
    I don't know whether this is a temporary phenomenon, or longer-term. Alignment research has gone mainstream, so we should expect both more researchers interested and more funders interested. It may be that the researchers pivot a bit faster, but funders will catch up later. Or, it may be that the funding bottleneck becomes the new normal. Regardless, it seems like grantmaking is at least funding-bottlenecked right now.
    Some takeaways:
    If you have a big pile of money and would like to help, but haven't been donating much to alignment because the field wasn't money constrained, now is your time!
    If this situation is the new normal, then earning-to-give for alignment may look like a more useful option again. That said, at this point committing to an earning-to-give path would be a bet on this situation being the new normal.
    Grants for upskilling, training junior people, and recruitment make a lot less sense right now from grantmakers' perspective.
    For those applying for grants, asking for less money might make you more likely to be funded. (Historically, grantmakers consistently tell me that most people ask for less money than they should; I don't know whether that will change going forward, but now is an unusually probable time for it to change.)
    Note that I am not a grantmaker, I'm just passing on what I hear from grantmakers in casual conversation. If anyone with more knowledge wants to chime in, I'd appreciate it.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - An EA's Guide to Visiting New York City by Alex R Kaplan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An EA's Guide to Visiting New York City, published by Alex R Kaplan on July 19, 2023 on The Effective Altruism Forum.
    Note: We (The EA NYC Team) are posting this on the suggestion from this Forum post calling for more public guides for navigating EA hubs. This guide is not a representation of the views of everyone in the community.
    Read on for a basic overview of the EA community in New York City. Whether you are visiting, planning to attend EAGxNYC (applications still open!), have just moved here, or are considering moving, we hope this guide will provide some helpful context on our community. That said, please comment or message us if we're missing anything - there's a lot happening here and it's hard to capture it all!
    And if you are planning on being in or have just arrived in NYC, get in touch with us! We're here to help support and connect the NYC EA network. We think there's a really special community here and Rocky, Alex, and the whole team would be happy to help you find like-minded New Yorkers.
    Overview
    Above: map outlining NYC's five boroughs. Credit: Wikimedia Commons
    NYC itself
    New York City comprises five boroughs: Manhattan, Brooklyn, Queens, the Bronx, and Staten Island. Each borough has its own character, as do the individual neighborhoods within the boroughs, but they are all part of New York City and are served by a united city government and public transit system. Manhattan is at the heart of the city, and is what most people think of when they think of New York City.
    There are also several areas immediately outside of the city that are very accessible by public transit, such as Jersey City, Hoboken, and New Jersey's suburbs, suburbs on Long Island (Nassau and Suffolk Counties), parts of the state of Connecticut, and suburbs in upstate New York (ie. Westchester, Putnam, and Dutchess Counties).
    Each of these areas is made up of many smaller neighborhoods, which vary dramatically in demographics, building styles, and activities.
    The EA community in NYC is very spread-out, and includes people in all of these areas and sometimes beyond.
    Of the five boroughs, Manhattan is the most densely populated, generally has the best accessibility, and usually has the most going on during the day (including anything from university research and co-working to tourism and EA NYC events). It will typically be the most expensive place to stay.
    Brooklyn (especially northwestern Brooklyn closer to Manhattan) and Queens (especially southwestern Queens closer to Manhattan) are also pretty centrally located and would allow access to co-working from cafes/libraries, more residential neighborhoods where some community members live, as well as more recreational options like dancing, performances, and bars.
    New Jersey (especially eastern New Jersey closer to Manhattan) can also provide more affordable living and decent access to Manhattan.
    Staten Island and the Bronx: EA NYC usually sees less representation from these boroughs, likely due to the greater geographical distance from events.
    A quick note on safety
    Most New Yorkers consider the city very safe (to the point that our team almost forgot to mention this), and the data supports this.
    Overall, New York is a very safe city (despite what others would have you believe based on misconceptions rooted from 40 years ago). Being a major city that covers ~300 square miles, New York is not without any crime, but given its population, NYC has relatively low crime rates.
    Nevertheless some travelers reasonably ask us about safety. For visitors with less urban experience, any city may be a shock. And while New York is quite safe by US standards, it may not be as safe as some European or East Asian cities to which some visitors are accustomed. Most of this difference is attributable to crime in low-income areas unlikely to be seen by t...
    22 min
  • EA - From Vision to Reality: Vida Plena's Pilot Results by Joy Bittner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: From Vision to Reality: Vida Plena's Pilot Results, published by Joy Bittner on July 19, 2023 on The Effective Altruism Forum.
    Context
    Vida Plena (meaning 'a flourishing life' in Spanish) is a new nonprofit organization based in Quito, Ecuador which launched last year (see our launch post here).
    Our mission is to build strong mental health in low-income and refugee communities, who otherwise would have no access to care. We provide evidence-based depression treatment using group interpersonal therapy, which is highly cost-effective and scalable (see our predictive cost-effectiveness analysis for more details).
    Pilot Results
    After completing the Charity Entrepreneurship (CE) Incubator program in the summer of 2022, we ran our pilot last fall/winter. We are pleased to now share results (baseline/endline comparisons) from the pilot.
    A few highlights:
    We had a total of 10 therapy groups and 55 people in the pilot
    75% of people had a clinically significant reduction in their depression symptoms
    50% of people had improvements in the secondary indicators of anxiety and PTSD
    I owe a huge debt of gratitude to Dr. Melanie Basnak from Rethink Priorities, who volunteered her time to conduct the data analysis for us.
    Next Steps
    We are currently analyzing the 3 month and 6 month follow up data.
    Given the initial positive results, we have continued our programs this year, treating an additional 250 people to date.
    In 2024 we hope to scale our program and continue to conduct program evaluations involving a control group.
    We would love to stay in touch and keep you up to date with our work
    Sign up here for email updates
    We're also on instagram: @vidaplena.global
    If you want to help, please tell others in your network about us: the best connections are unexpected. We are looking to connect with people and organizations involved in global mental health, both in industry and academia. Feel feel free to reach out at [email protected]
    And of course, we welcome any donations to help support our mission and work.
    Our incredible group of community facilitators who were certified in interpersonal therapy by Columbia University for the pilot.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • LW - A brief history of computers by Adam Zerner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A brief history of computers, published by Adam Zerner on July 19, 2023 on LessWrong.
    Recently I've been learning about the history of computers. I find it to be incredibly interesting. I'd like to write a post about it to summarize and comment on what I've learned.
    I'm a little hesitant though. I'm no expert on this stuff. I'm largely learning about it all for the first time. So then, take all of this with a grain of salt. It's more of a conversation starter than a finished product. If you want something authoritative, I'd recommend the Stanford Encyclopedia of Philosophy.
    Logic
    Let's start with logic. Computers are largely based on boolean logic. Y'know, 1s and 0s. AND, OR, NOT. George Boole did a bunch of important work here in the mid 1800s, but let's try backing up even further. Was there anything important that came before Boolean logic?
    Yeah, there was. It goes all the way back to Aristotle in ~350 BCE. Aristotle did a bunch of groundbreaking work in the field of logic. Furthermore, after "breaking the ground", there weren't any significant developments until the mid 1800s. Wow! That's a long time. An unusually long time. In other fields like mathematics, natural sciences, literature and engineering, there were significant advances. I wonder why things in the field of logic were so quiet.
    Anyway, let's talk about what exactly Aristotle did. In short, he looked at arguments in the abstract. It's one thing to say that:
    Filo is a dog
    Therefore, Filo has feet
    It's another thing to say that:
    R is a P
    Therefore, R has Q
    The former is concrete. It's talking about dogs, feet and Filo. The latter is abstract. It's talking about P's, Q's and R's. Do you see the difference?
    Before Aristotle, people never thought about this stuff in terms of P's and Q's. They just thought about dogs and feet. Thinking about P's and Q's totally opened things up. Pretty cool. Abstraction is powerful. I think this is very much worth noting as an important milestone in the history of computers.
    Ok. So once Aristotle opened the flood gates with categorical logic, over time, people kinda piggybacked off of it and extended his work. For example, the Stoics did a bunch of work with propositional logic.
    Propositional logic is different from categorical logic. Categorical logic is about what categories things belong to. For example, earlier we basically said that dogs belong to the category of "things with feet" and that Filo belongs to the category of "dogs". With those two statements, we deduced that Filo must also belong to the category of "things with feet". It makes a lot of sense when you think about it visually:
    On the other hand, propositional logic is about things being true or false. For example, with this:
    I don't have an umbrella
    we can deduce things like:
    "It is raining or I have an umbrella" is true
    Propositional logic is about truth and uses operators like AND, OR, NOT, IF-THEN, and IF-AND-ONLY-IF. Categorical logic is about categories and uses operators like ALL, NO and SOME.
    After propositional logic, subsequent work was done. For example, predicate logic kinda piggybacked off of propositional logic. But in general, nothing too crazy was going on. Let's jump ahead to the mid 1800s and George Boole.
    Boole introduced stuff like this:
    (p and q) is false
    (p or q) is true
    (not (p and q)) is true
    But wait a minute. I'm confused. Didn't we get that sort of thing from propositional logic all the way back in 300 BCE from the Stoics? In researching this question I'm seeing things saying that it did in fact already exist, it's just that Boole made it more "systematic and formalized". I don't understand though. In what way did he made it more systematic and formalized?
    Oh well. Suffice it to say that boolean logic was a thing that we knew about. Let's move on.
    Jacquard loom
    I was going to star...
    59 min
  • EA - An Introduction to Critiques of prominent AI safety organizations by Omega
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Introduction to Critiques of prominent AI safety organizations, published by Omega on July 19, 2023 on The Effective Altruism Forum.
    What is this series (and who are we)?
    This is a series of evaluations of technical AI safety (TAIS) organizations. We evaluate organizations that have received more than $10 million per year in funding and that have had limited external evaluation.
    The primary authors of this series include one technical AI safety researcher (>4 years experience), and one non-technical person with experience in the EA community. Some posts also have contributions from others with experience in technical AI safety and/or the EA community.
    This introduction was written after the first two posts in the series were published. Since we first started working on this series we have updated and refined our process for evaluating and publishing critiques, and this post reflects our present views.
    Why are we writing this series?
    Recently, there has been more attention on the field of technical AI safety (TAIS), meaning that many people are trying to get into TAIS roles. Without knowing significant context about different organizations, new entrants to the field will tend to apply to TAIS organizations based on their prominence, which is largely related to factors such as total funding, media coverage, volume of output, etc, rather than just the quality of their research or approach. Much of the discussion we have observed about TAIS organizations, especially criticisms of them, happens behind closed doors, in conversations that junior people are usually not privy to. We wish to help disseminate this information more broadly to enable individuals to make a better informed decision.
    We focus on evaluating large organizations, defined as those with more than $10 million per year in funding. These organizations are amongst the most visible and tend to have a significant influence on the AI safety ecosystem by virtue of their size, making evaluation particularly important. Additionally, these organizations would only need to dedicate a small fraction of their resources to engaging with these criticisms.
    How do we evaluate organizations?
    We believe that an organization should be graded on multiple metrics. We consider:
    Research outputs: How much good quality research has the organization published? This is the area where we put the most weight.
    Research agenda: Does the organization's research plan seem likely to bear fruit?
    Research team: What proportion of researchers are senior/experienced? What is the leadership's experience in ML and safety research? Are the leaders trustworthy? Are there conflicts of interest?
    Strategy and governance: What corporate governance structures are in place? Does the organization have independent accountability? How transparent is it? The FTX crisis has shown how important this can be.
    Organizational culture and work environment: Does the organization foster a good work environment for their team? What efforts has the organization made to improve its work culture?
    When evaluating research outputs, we benchmark against high-quality existing research, and against academia. Although academic AIS research is not always the most novel or insightful, there are strong standards for rigor in academia that we believe are important. Some existing research that we think is exceptional include:
    Eliciting latent knowledge (ARC)
    Iterated Distillation and Amplification (Paul Christiano)
    Constitutional AI (Anthropic)
    Trojan Detection Competition (CAIS)
    Causal scrubbing (Redwood)
    Toy models of superposition (Anthropic)
    Our thoughts on hits-based research agendas
    When we criticized Conjecture's output, commenters suggested that we were being unfair, because Conjecture is pursuing a hits-based research agenda, and this style of research typically takes...
    10 min
  • EA - AMA: Peter Wildeford (Co-CEO at Rethink Priorities) by Peter Wildeford
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AMA: Peter Wildeford (Co-CEO at Rethink Priorities), published by Peter Wildeford on July 19, 2023 on The Effective Altruism Forum.
    Hi everyone! I'll be doing an Ask Me Anything (AMA) here. Feel free to drop your questions in the comments below. I will aim to answer them by Monday, July 24.
    Who am I?
    I'm Peter. I co-founded Rethink Priorities (RP) with Marcus A. Davis in 2018. Previously, I worked as a data scientist in industry for five years. I'm an avid forecaster. I've been known to tweet here and blog here.
    What does Rethink Priorities do?
    RP is a research and implementation group that works with foundations and impact-focused non-profits to identify pressing opportunities to make the world better, figures out strategies for working on those problems, and does that work.
    We focus on:
    Wild and farmed animal welfare (including invertebrate welfare)
    Global health and development (including climate change)
    AI governance and strategy
    Existential security and safeguarding a flourishing long-term future
    Understanding and supporting communities relevant to the above
    What should you ask me?
    Anything!
    I oversee RP's work related to existential security, AI, and surveys and data analysis research, but I can answer any question about RP (or anything).
    I'm also excited to answer questions about the organization's future plans and our funding gaps (see here for more information). We're pretty funding constrained right now and could use some help!
    We also recently published a personal reflection on what Marcus and I have learned in the last five years as well as a review of the organization's impacts, future plans, and funding needs that you might be interested in or have questions about.
    RP's publicly available research can be found in this database. If you'd like to support RP's mission, please donate here or contact Director of Development Janique Behman.
    To stay up-to-date on our work, please subscribe to our newsletter or engage with us on Twitter, Facebook, or LinkedIn.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • LW - Tiny Mech Interp Projects: Emergent Positional Embeddings of Words by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tiny Mech Interp Projects: Emergent Positional Embeddings of Words, published by Neel Nanda on July 18, 2023 on LessWrong.
    This post was written in a rush and represents a few hours of research on a thing I was curious about, and is an exercise in being less of a perfectionist. I'd love to see someone build on this work! Thanks a lot to Wes Gurnee for pairing with me on this
    Tokens are weird, man
    Introduction
    A particularly notable observation in interpretability in the wild is that part of the studied circuit moves around information about whether the indirect object of the sentence is the first or second name in the sentence. The natural guess is that heads are moving around the absolute position of the correct name. But even in prompt formats where the first and second names are in the different absolute positions, they find that the informations conveyed by these heads are exactly the same, and can be patched between prompt templates! (credit to Alexandre Variengien for making this point to me).
    This raises the possibility that the model has learned what I call emergent positional embeddings - rather than representing "this is the token in position 5" it may represent "this token is the second name in the sentence" or "this token is the fourth word in the sentence" or "this is the third sentence in the paragraph" etc. Intuitively, models will often want to do things like attend to the previous word, or the corresponding word in the previous sentence, etc - there are lots of things it will plausibly want to do that are natural in some emergent coordinate scheme that are unnatural in the actual token coordinate scheme.
    I was curious about this, and spent an afternoon poking around with Wes Gurnee at whether I could convince myself that these emergent positional embeddings were a thing. This post is an experiment: I'm speedrunning a rough write-up on a few hours of hacky experiments, because this seemed more interesting to write-up than not to, and I was never going to do the high-effort version. Please take all this with a mountain of salt, and I'd love to see anyone build on my incredibly rough results - code here.
    Experiments
    You can see some terrible code for these experiments here. See the Appendix for technical details
    I wanted to come up with the dumbest experiment I could that could shed light on whether this was a thing. One thing that models should really care about is the ability to attend to tokens in the previous word. Words can commonly range from 1 to 3 tokens (and maybe much longer for rare or mispelt words) so this is naturally done with an emergent scheme saying which word a token is part of.
    My experiment: I took prompts with a fixed prefix of 19 tokens and then seven random lowercase English words of varying token length, like token|izer| help| apple| dram|at|isation| architecture| sick| al|p|aca. I ran GPT-2 Small on this, look the residual stream after layer 3 (33% of the way through the model) and then trained a logistic regression probe on the residual stream of the token at the end of each word to predict which word it was in.
    This is the key plot, though it takes a bit of time to get your head around. The x axis is the absolute position of the token in the prompt and the row is the ground truth of the word index. The bar for each absolute position and row shows the distribution of guesses given on the probe validation set. The colours correspond to the seven possible indices (note that the legend is not in numerical order, sigh).
    For example: take the third bar in the second row (index=1, abs_pos=22). This is mostly red (index = 1, correct!), with a bit of blue at the bottom (index = 0, incorrect) and a bit of green at the top (index = 2, incorrect). In contrast, the bar in the row below (second bar in the third row, index=2, abs_pos=23) is...
    14 min