The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - What AI companies can do today to help with the most important century by Holden Karnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What AI companies can do today to help with the most important century, published by Holden Karnofsky on February 20, 2023 on The Effective Altruism Forum.
    I’ve been writing about tangible things we can do today to help the most important century go well. Previously, I wrote about helpful messages to spread and how to help via full-time work.
    This piece is about what major AI companies can do (and not do) to be helpful. By “major AI companies,” I mean the sorts of AI companies that are advancing the state of the art, and/or could play a major role in how very powerful AI systems end up getting used.1
    This piece could be useful to people who work at those companies, or people who are just curious.
    Generally, these are not pie-in-the-sky suggestions - I can name2 more than one AI company that has at least made a serious effort at each of the things I discuss below (beyond what it would do if everyone at the company were singularly focused on making a profit).3
    I’ll cover:
    Prioritizing alignment research, strong security, and safety standards (all of which I’ve written about previously).
    Avoiding hype and acceleration, which I think could leave us with less time to prepare for key risks.
    Preparing for difficult decisions ahead: setting up governance, employee expectations, investor expectations, etc. so that the company is capable of doing non-profit-maximizing things to help avoid catastrophe in the future.
    Balancing these cautionary measures with conventional/financial success.
    I’ll also list a few things that some AI companies present as important, but which I’m less excited about: censorship of AI models, open-sourcing AI models, raising awareness of AI with governments and the public. I don’t think all these things are necessarily bad, but I think some are, and I’m skeptical that any are crucial for the risks I’ve focused on.
    I previously laid out a summary of how I see the major risks of advanced AI, and four key things I think can help (alignment research; strong security; standards and monitoring; successful, careful AI projects). I won’t repeat that summary now, but it might be helpful for orienting you if you don’t remember the rest of this series too well; click here to read it.
    Some basics: alignment research, strong security, safety standards
    First off, AI companies can contribute to the “things that can help” I listed above:
    They can prioritize alignment research (and other technical research, e.g. threat assessment research and misuse research).
    For example, they can prioritize hiring for safety teams, empowering these teams, encouraging their best flexible researchers to work on safety, aiming for high-quality research that targets crucial challenges, etc.
    It could also be important for AI companies to find ways to partner with outside safety researchers rather than rely solely on their own teams. As discussed previously, this could be challenging. But I generally expect that AI companies that care a lot about safety research partnerships will find ways to make them work.
    They can help work toward a standards and monitoring regime. E.g., they can do their own work to come up with standards like "An AI system is dangerous if we observe that it's able to ___, and if we observe this we will take safety and security measures such as ____." They can also consult with others developing safety standards, voluntarily self-regulate beyond what’s required by law, etc.
    They can prioritize strong security, beyond what normal commercial incentives would call for.
    It could easily take years to build secure enough systems, processes and technologies for very high-stakes AI.
    It could be important to hire not only people to handle everyday security needs, but people to experiment with more exotic setups that could be needed later, as the incentives to steal AI get strong...
    18 min
  • AF - EIS IX: Interpretability and Adversaries by Stephen Casper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS IX: Interpretability and Adversaries, published by Stephen Casper on February 20, 2023 on The AI Alignment Forum.
    Part 9 of 12 in the Engineer’s Interpretability Sequence.
    Thanks to Nikolaos Tsilivis for helpful discussions.
    The studies of interpretability and adversaries are inseparable.
    There are several key connections between the two. Some works will be cited below, but please refer to page 9 of the Toward Transparent AI survey (Räuker et al., 2022) for full citations. There are too many to be worth the clutter in this post.
    1. More interpretable networks are more adversarially robust and more adversarially robust networks are more interpretable.
    The main vein of evidence on this topic comes from a set of papers which study how regularizing feature attribution/saliency maps to make them more clearly highlight specific input features has the effect of making networks more robust to adversaries. There is also some other work showing the reverse -- that adversarially robust networks tend to have more lucid attributions. There is also some work showing that networks which emulate certain properties of the human visual system are also more robust to adversaries and distribution shifts (e.g. Ying et al. (2022)).
    Adversarial training is a good way of making networks more internally interpretable. One particularly notable work is Engstrom et al., (2019) who found striking improvements in how much easier it was to produce human-describable visualizations of internal network properties. Although they stopped short of applying this work to an engineering task, the paper seems to make a strong case for how adversarial training can improve interpretations. Adversarially trained networks also produce better representations for transfer learning, image generation, and modeling the human visual system.
    Finally, some works have found that lateral inhibition and second-order optimization have been found to improve both interpretability and robustness.
    2. Interpretability tools can and should be used to guide the design of adversaries.
    This is one of the three types of rigorous evaluation methods for interpretability tools discussed in EIS III. Showing that an interpretability tool helps us understand a network well enough to exploit it is good evidence that it can be useful.
    3. Adversarial examples can be useful interpretability tools.
    Adversaries always reveal information about a network, even if it’s hard to describe a feature that fools it in words. However, a good amount of recent literature has revealed that studying interpretable adversaries can lead to useful, actionable insights. In some previous work (Casper et al., 2021), some coauthors and I argue for using “robust feature-level adversaries” as a way to produce attacks that are human-describable and likely to lead to a generalizable understanding. Casper et al, (2023) more rigorously tests methods like this.
    4. Mechanistic interpretability and mechanistic adversarial examples are uniquely equipped for addressing deception and other insidious misalignment failures.
    Hubinger (2020) discussed 11 proposals for building safe advanced AI, and all 11 explicitly call for the use of interpretability tools or (relaxed) adversarial training for inner alignment. This isn’t a coincidence because these offer the only types of approaches that can be useful for fixing insidiously aligned models. Recall from the previous post that an engineer might understand insidious misalignment failures as ones in which the inputs that will make a model exhibit misaligned behavior are hard to find during training, but there exists substantial neural circuitry dedicated to the misaligned behavior. Given this, it’s clear that working to understand and debug inner mechanisms is the key to make progress on insidious misalignment.
    Are adversaries fea...
    15 min
  • EA - EV UK board statement on Owen's resignation by EV UK Board
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EV UK board statement on Owen's resignation, published by EV UK Board on February 20, 2023 on The Effective Altruism Forum.
    In a recent TIME Magazine article, a claim of misconduct was made about an “influential figure in EA”:
    A third [woman] described an unsettling experience with an influential figure in EA whose role included picking out promising students and funneling them towards highly coveted jobs. After that leader arranged for her to be flown to the U.K. for a job interview, she recalls being surprised to discover that she was expected to stay in his home, not a hotel. When she arrived, she says, “he told me he needed to masturbate before seeing me.”
    Shortly after the article came out, Julia Wise (CEA’s community liaison) informed the EV UK board that this concerned behaviour of Owen Cotton-Barratt; the incident occurred more than 5 years ago and was reported to her in 2021. (Owen became a board member in 2020.)
    Following this, on February 11th, Owen voluntarily resigned from the board. This included stepping down from his role with Wytham Abbey; he is also no longer helping organise The Summit on Existential Security.
    Though Owen’s account of the incident differs in scope and emphasis from the version expressed in the TIME article, he still believes that he made significant mistakes, and also notes that there have been other cases where he regretted his behaviour.
    It's very important to us that EV and the wider EA community strive to provide safe and respectful environments, and that we have reliable mechanisms for investigating and addressing claims of misconduct in the EA community. So, in order to better understand what happened, we are commissioning an external investigation by an independent law firm into Owen’s behaviour and the Community Health team’s response.
    This post is jointly from the Board of EV UK: Claire Zabel, Nick Beckstead, Tasha McCauley and Will MacAskill.
    The disclosure occurred as follows: shortly after the article came out, Owen and Julia agreed that Julia would work out whether Owen's identity should be disclosed to other people in EV UK and EV US; Julia determined that it should be shared with the boards.
    Julia writes about her response at the time here.
    See comment here from Chana Messinger on behalf of the Community Health team.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - A statement and an apology by Owen Cotton-Barratt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A statement and an apology, published by Owen Cotton-Barratt on February 20, 2023 on The Effective Altruism Forum.
    Since the Time article on sexual harassment came out, people have been asking for information about one paragraph of it, about an “influential figure in EA”. I wanted to respond to that.
    This is talking about me, more than five years ago. I think I made significant mistakes; I regret them a lot; and I’m sorry.
    Context
    I think the actual mistakes I made look different from what many readers may take away from the article, so I first wanted to provide a bit more context (some of this is straightforwardly factual; other parts should be understood as my interpretation):
    We had what I perceived as a preexisting friendship where we were experimenting with being unusually direct and honest (/“edgy”)
    Including about sexual matters
    There was what would commonly be regarded as oversharing from both sides (this wasn’t the first time I’d mentioned masturbation)
    Our friendship continued in an active way for several months afterwards
    I should however note that:
    We had met via EA and spent a good fraction of conversation time talking about EA-relevant topics
    I was older and more central in the EA community
    On other occasions, including early in our friendship, we had some professional interactions, and I wasn’t clear about how I was handling the personal/professional boundary
    I was employed as a researcher at that time
    My role didn’t develop to connecting people with different positions until later, and this wasn’t part of my self-conception at the time
    (However it makes sense to me that this was her perception)
    I was not affiliated with the org she was interviewing at
    I’d suggested her as a candidate earlier in the application process, but was not part of their decision-making process
    On the other hand I think that a lot of what was problematic about my behaviour with respect to this person was not about this incident in particular, but the broad dynamic where:
    I in fact had significant amounts of power
    This was not very salient to me but very salient to her
    She consequently felt pressure to match my vibe
    e.g. in an earlier draft of this post, before fact-checking it with her, I said that we talked about “feelings of mutual attraction”
    This was not her experience
    I drafted it like that because we’d had what I’d interpreted as conversations where this was stated explicitly
    (I think this is just another central example of the point I’m making in this set of bullets)
    Similarly at some point she volunteered to me that she was enjoying the dynamic between us (but I probably interpreted this much more broadly than she intended)
    She was in a structural position where it was (I now believe) unreasonable to expect honesty about her experience
    As the person with power it was on me to notice and head off these dynamics, and I failed to do that
    (Sorry, I know that's all pretty light on detail, but I don't want to risk accidentally de-anonymising the other person. I want to stress that I’m not claiming she provided any inaccurate information to the journalist who wrote the story; just that I think the extra context may be helpful for people seeking to evaluate or understand my conduct.)
    My mistakes
    In any case, I think my actions were poorly judged and fell well short of the high standards I would like to live up to, and that I think we should ask from people in positions of leadership. Afterwards, I felt vaguely like the whole friendship wasn’t well done and I wished I had approached things differently. Then when I found out that I’d made the person feel uncomfortable(/disempowered/pressured), I was horrified (not putting pressure on people is something like a core value of mine). I have apologized to the person in question, but I also feel like I’ve let the whole community down,...
    11 min
  • LW - The idea that ChatGPT is simply “predicting” the next word is, at best, misleading by Bill Benzon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The idea that ChatGPT is simply “predicting” the next word is, at best, misleading, published by Bill Benzon on February 20, 2023 on LessWrong.
    Cross-posted from New Savanna.
    But it may also be flat-out wrong. We’ll see when we get a better idea of how inference works in the underlying language model.
    Yes, I know that ChatGPT is trained by having it predict the next word, and the next, and the next, for billions and billions of words. The result of all that training is that ChatGPT builds up a complex structure of weights on the 175 billion parameters of its model. It is that structure that emits word after word during inference. Training and inference are two different processes, but that point is not well-made in accounts written for the general public.
    Let's get back to the main thread.
    I maintain, for example, that when ChatGPT begins a story with the words “Once upon a time,” which it does fairly often, that it “knows” where it is going and that its choice of words is conditioned on that “knowledge” as well as upon the prior words in the stream. It has invoked a ‘story telling procedure’ and that procedure conditions its word choice. Just what that procedure is, and how it works, I don’t know, nor do I know how it is invoked. I do know, that it is not invoked by the phrase “once upon a time” since ChatGPT doesn’t always use that phrase when telling a story. Rather, that phrase is called up through the procedure.
    Consider an analogy from jazz. When I set out to improvise a solo on, say, “A Night in Tunisia,” I don’t know what notes I’m going to play from moment to moment, much less do I know how I’m going to end, though I often know when I’m going to end. How do I know that? That’s fixed by the convention in place at the beginning of the tune; that convention says that how many choruses you’re going to play. So, I’ve started my solo. My note choices are, of course, conditioned by what I’ve already played. But they’re also conditioned by my knowledge of when the solo ends.
    Something like that must be going on when ChatGPT tells a story. It’s not working against time in the way a musician is, but it does have a sense of what is required to end the story. And it knows what it must do, what kinds of events must take place, in order to get from the beginning to the end. In particular, I’ve been working with stories where the trajectories have five segments: Donné, Disturb, Plan, Execute, Celebrate. The whole trajectory is ‘in place’ when ChatGPT begins telling the story. If you think of the LLM as a complex dynamical system, then the trajectory is a valley in the system’s attractor landscape.
    Nor is it just stories. Surely it enacts a different trajectory when you ask it a factual question, or request it to give you a recipe (like I recently did, for Cornish pasty), or generate some computer code.
    With that in mind, consider a passage from a recent video by Stephen Wolfram (note: Wolfram doesn’t start speaking until about 9:50):
    Starting at roughly 12:16, Wolfram explains:
    It is trying write reasonable, it is trying to take an initial piece of text that you might give and is trying to continue that piece of text in a reasonable human-like way, that is sort of characteristic of typical human writing. So, you give it a prompt, you say something, you ask something, and, it’s kind of thinking to itself, “I’ve read the whole web, I’ve read millions of books, how would those typically continue from this prompt that I’ve been given? What’s the reasonable expected continuation based on some kind of average of a few billion pages from the web, a few million books and so on.” So, that’s what it’s always trying to do, it’s aways trying to continue from the initial prompt that it’s given. It’s trying to continue in a statistically sensible way.
    Let’s say that you had given it, yo...
    8 min
  • EA - The Estimation Game: a monthly Fermi estimation web app by Sage
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Estimation Game: a monthly Fermi estimation web app, published by Sage on February 20, 2023 on The Effective Altruism Forum.
    Announcing the first monthly Estimation Game!
    Answer 10 Fermi estimation questions, like “How many piano tuners are there in New York?”
    Train your estimation skills and get more comfortable putting numbers on things
    Team up with friends, or play solo
    See how your scores compare on the global leaderboard
    The game is around 10-40 minutes, depending on how much you want to discuss and reflect on your estimates
    You can play The Estimation Game on Quantified Intuitions, solo, or with friends. The February game is live for one week (until Sunday 26th).
    We’ll release a new Estimation Game each month. Lots of people tell us they’d like to get more practice doing BOTECs and estimating, but they don’t get around to it. So we’ve designed The Estimation Game to give you the impetus to do a bit of estimation each month in a fun context.
    You might use this as a sandbox to experiment with different methods of estimating. You could decompose the question into easier-to-estimate quantities - make estimates in your head, discuss with friends, use a bit of paper, or even build a scrappy Guesstimate or Squiggle model.
    We’d appreciate your feedback in the comments, in our Discord, or at [email protected]. We’d love to have suggestions for questions for future rounds of The Estimation Game - this will help us keep the game varied and fun in future months!
    Info for organisers
    If you run a community group or meetup, we’ve designed the Estimation Game to be super easy to run as an off-the-shelf event. Check out our info for organisers page for resources and FAQs.
    If you’re running a large-scale event and want to run a custom Estimation Game at it, let us know and we can help you set it up. We’re planning to pilot custom Estimation Games at EAGx Nordics (and maybe EAGx Cambridge).
    About Quantified Intuitions
    We built Quantified Intuitions as an epistemics training site. See our previous post for more on our motivation. Alongside the monthly Estimation Game, we’ve made two permanent tools:
    Pastcasting: Predict past events to rapidly practise forecasting
    Calibration: Answer EA-themed trivia questions to calibrate your uncertainty
    Thanks to our test groups in London, to community builders who gave feedback, in particular Robert Harling, Adash Herrenschmidt-Moller, and Sam Robinson, and to Chana Messinger at CEA for the idea and feedback throughout.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Metaculus Introduces New 'Conditional Pair' Forecast Questions for Making Conditional Predictions by christian
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Metaculus Introduces New 'Conditional Pair' Forecast Questions for Making Conditional Predictions, published by christian on February 20, 2023 on The Effective Altruism Forum.
    Predict P(A|B) & P(A|B') With New Conditional Pairs
    Events don't take place in isolation. Often we want to know the likelihood of an event occurring if another one does.
    Metaculus has launched conditional pairs, a new kind of forecast question that enables forecasters to predict the probability of an event given the outcome of another, and provides forecast consumers with greater clarity on the relationships between events.
    This post explains the motivation behind conditional pairs, how to interpret them, and how to start making conditional forecasts.
    (Check out this video explainer for more on conditional pairs and how to forecast with them.)
    How Do Conditional Pairs Work?
    A conditional pair poses two conditional questions (or "conditionals"):
    If Question B resolves Yes how will Question A resolve?
    If Question B resolves No how will Question A resolve?
    For example, a forecaster may want to predict on a question such as this:
    Will Alphabet’s Market Capitalization Fall Below $1 Trillion by 2025?
    The forecast depends—on many things. But consider one factor: Bing's share of the search engine market.
    And so if one knew that Bing's search engine market would be least 5% in March of 2024, they could make a more informed forecast. They might assign greater likelihood to Alphabet's decline.
    Our conditional pair is then:
    If Bing's market share is more than 5% in March, 2024, will Alphabet's market capitalization fall below $1 trillion?
    If Bing's market share is not more than 5% in March, 2024, will Alphabet's market capitalization fall below $1 trillion?
    (Start forecasting on this conditional pair here.)
    Two forecasters could have the same forecasts for Bing’s market share and Alphabet’s market cap while having very different mental models of their relationship. Conditional pairs help make these sometimes implicit differences explicit so they can be discussed and scored.
    Start Forecasting on Conditional Pairs
    Here are some newly created conditional pairs to start forecasting on:
    If Human-Machine Intelligence Parity Is Reached by 2040, Will the US Impose Compute Capacity Restrictions Before 2050?
    If Chinese GDP Overtakes US GDP by 2030, Will the US Go to War With China by 2035?
    If There Is a Bilateral Ceasefire in Ukraine by 2024, Will There Be Large-Scale Conflict With Russia by 2030?
    If There Is a US Debt Default by 2024, Will Democrats Win the 2024 US Presidential Election?
    Parent & Child Questions
    Conditional pairs like the above are composed of a "Parent Question" and a "Child Question."
    Parent: Bing has 5% Market Share by March, 2024
    Child: Alphabet's Market Cap is Below $1 Trillion by 2025
    Forecasts are made for the Child question conditional on the outcome of the Parent. Here, see that:
    In a world where Bing reaches 5% market share, the Metaculus community predicts Alphabet's decline is 56% likely.
    In a world where Bing does not reach 5% market share, the Metaculus community predicts Alphabet's decline is 44% likely.
    Conditional pairs are a step toward Metaculus's larger goal of empowering forecasters and forecast consumers to quantify and understand the impact of particular events and policy decisions. Feedback is appreciated!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Immigration reform: a shallow cause exploration by JoelMcGuire
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Immigration reform: a shallow cause exploration, published by JoelMcGuire on February 20, 2023 on The Effective Altruism Forum.
    This shallow investigation was commissioned by Founders Pledge.
    Summary
    This shallow cause area report explores the impact of immigration on subjective wellbeing (SWB). It was completed in two weeks. In this report, we start by reviewing the literature and modelling the impact of immigration on wellbeing. Then, we conduct back of the envelope calculations (BOTECs) of the cost-effectiveness of various interventions to increase immigration.
    The effect of immigration has been studied extensively. However, most of the studies we find are correlational and do not provide causal evidence. Additionally, most of the studies use life satisfaction as a measure of SWB, so it’s unclear whether immigration impacts life satisfaction and affective happiness (e.g. positive emotions on a daily basis) differently.
    Despite these limitations, we attempt to estimate the effect of immigration on wellbeing. We find that immigrating to countries with higher average SWB levels might produce large benefits to wellbeing, but we are very uncertain about the exact size of the effect. According to our model, when people move to a country with higher SWB, they will gain 77% of the SWB gap between the origin and destination country. We assume this benefit will be immediate and permanent, as there is little evidence to model how this benefit evolves over time, and existing evidence doesn’t suggest large deviations from this assumption.
    There are open questions about the spillover effects of immigration on the immigrant’s household as well as their original and destination communities. Immigrating likely benefits the whole family if they move together, but the impact on household members that stay behind is less clear, as the economic benefits of remittances are countered by the negative effects of separation. On balance, we estimate a small, non-significant benefit for households that stay behind when a member immigrates (+0.01 WELLBY per household member). We did not include spillovers on the origin community due to scarce evidence (only one study) that suggested small, null effects. For destination communities, we estimate that increasing the proportion of immigrants by 1% is associated with a small, non-significant, negative spillover for natives (-0.01 WELLBYs per native), although this is likely moderated by attitudes towards immigrants.
    We then conducted BOTECs of possible interventions to increase immigration. The most promising is policy advocacy, which we estimate is 11 times more cost-effective than GiveDirectly cash transfers. The other interventions we investigated are 2 to 6 times better than cash transfers. However, all of our BOTECs are speculative and exploratory in nature. These estimates are also limited because we’re unsure how to model the potential for immigration increasing interventions to foster anti-immigrant sentiment in the future. Plus, there might be non-trivial risks that a big push for immigration or other polarising topics by Effective Altruists could burn goodwill that might be used on other issues (e.g., biosecurity). Accordingly, we’re inclined towards treating these as upper-bound estimates and we expect that once these costs are taken into account immigration policy advocacy would no longer be promising.
    We recommend that future research assesses the costs, chances of success, and risk of backlash for potential policy-based interventions to increase immigration.
    Notes
    This report focuses on the impact of immigration in terms of WELLBYs. One WELLBY is a 1 life satisfaction point change for one year (or any equivalent combination of change in life satisfaction and time). In some cases, we convert results in standard deviations of life sati...
    1 hr 13 min
  • AF - [MLSN #8] Mechanistic interpretability, using law to inform AI alignment, scaling laws for proxy gaming by Dan H
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [MLSN #8] Mechanistic interpretability, using law to inform AI alignment, scaling laws for proxy gaming, published by Dan H on February 20, 2023 on The AI Alignment Forum.
    As part of a larger community building effort, CAIS is writing a safety newsletter that is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here or follow the newsletter on twitter here.
    Welcome to the 8th issue of the ML Safety Newsletter! In this edition, we cover:
    Isolating the specific mechanism that GPT-2 uses to identify the indirect object in a sentence
    When maximum softmax probability is optimal
    How law can inform specification for AI systems
    Using language models to find a group consensus
    Scaling laws for proxy gaming
    An adversarial attack on adaptive models
    How systems safety can be applied to ML
    And much more...
    Monitoring
    A Circuit for Indirect Object Identification in GPT-2 small
    One subset of interpretability is mechanistic interpretability: understanding how models perform functions down to the level of particular parameters. Those working on this agenda believe that by learning how small parts of a network function, they may eventually be able to rigorously understand how the network implements high-level computations.
    This paper tries to identify how GPT-2 small solves indirect object identification, the task of identifying the correct indirect object to complete a sentence with. Using a number of interpretability techniques, the authors seek to isolate particular parts of the network that are responsible for this behavior.
    [Link]
    Learning to Reject Meets OOD Detection
    Both learning to reject (also called error detection; deciding whether a sample is likely to be misclassified) and out-of-distribution detection share the same baseline: maximum softmax probability. MSP has been outperformed by other methods in OOD detection, but never in learning to reject, and it is mathematically provable that it is optimal for learning to reject. This paper shows that it isn’t optimal for OOD detection, and identifies specific circumstances in which it can be outperformed. This theoretical result is a good confirmation of the existing empirical results.
    [Link]
    Other Monitoring News
    [Link] The first paper that successfully applies feature visualization techniques to Vision Transformers.
    [Link] This method uses the reconstruction loss of diffusion models to create a new SOTA method for out-of-distribution detection in images.
    [Link] A new Trojan attack on code generation models works by inserting poisoned code into docstrings rather than the code itself, evading some vulnerability-removal techniques.
    [Link] This paper shows that fine tuning language models for particular tasks relies on changing only a very small subset of parameters. The authors show that as few as 0.01% of parameters can be “grafted” onto the original network and achieve performance that is nearly as high.
    Alignment
    Applying Law to AI Alignment
    One problem in alignment is specification: though we may give AI systems instructions, we cannot possibly specify what they should do in all circumstances. Thus, we have to consider how our specifications will generalize in fuzzy, or out-of-distribution contexts.
    The author of this paper argues that law has many desirable properties that may make it useful in informing specification. For example, the law often uses “standards”: relatively vague instructions (e.g. “act with reasonable caution at railroad crossings”; in contrast to rules like “do not exceed 30 miles per hour”) whose specifics have been developed through years of precedent. In the law, it is often necessary to consider the “spirit” behind these standards, which is exactly what we want AI systems to be able to do. This paper argues that AI system...
    9 min
  • LW - On Investigating Conspiracy Theories by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On Investigating Conspiracy Theories, published by Zvi on February 20, 2023 on LessWrong.
    Scott Alexander wrote yet more words defending his decision to write two posts totaling 25,000 words about Ivermectin. Then he wrote a second post trying again.
    More centrally, his first post, of which I very much approve, is defending the most important idea of all: Think for yourself, shmuck!
    I want to make clear my violent agreement with all of the following.
    Think for yourself, shmuck!
    When it seems worthwhile, do your own research.
    The ones telling you not to ‘do your own research’ are probably the baddies.
    Also applies to the ones telling you to ‘trust us and Trust the Science’ and calling you an idiot or racist or calling for you to be censored if you disagree.
    Baddies or not, those people still are more likely to be more right about any given point than those saying they are wrong or lying to you, unless you have seen them lying or being wrong a lot about closely related things previously. And your own research will often not seem worthwhile if you consider opportunity costs.
    When people draw wrong conclusions like Ivermectin being effective or that Atlantis used to exist or whatever, telling people that they are idiots or racists for drawing that conclusion is not going to be super effective.
    Pointing out ‘the other side are conspiracy theorists’ or ‘the people who believe this also believe these other terrible things’ does not prove the other side is wrong, nor is it going to convince anyone on the other side that they are wrong.
    If you instead explain and work through the evidence, there is a chance someone might be convinced, that is God’s work, you are providing a public service.
    There are not ‘legitimate’ and ‘illegitimate’ places to Do Science. You can virtuously Do Science to It, for all values of It and of You.
    No, we cannot assume that the medical establishment, or any other establishment, will always get such questions right. That is not how any of this works. Even the best possible version of the medical (or other) establishment will sometimes get it wrong, if no one points it out without being dismissed as a conspiracy theorist or racist then the establishment will keep getting it wrong and so will you, and criticism is the only known antidote to error in such situations.
    I would also add, from Kavanagh’s response to Scott in a comment, my disagreement with this particular thing, regarding scuba diving to purported Atlantean ruins:
    I also don’t think I would have the same intuition you have that personally exploring the ruins would be informative. I think that would actually be likely to skew my perspective as it feels like it would deliver potentially inaccurate intuitions and that it would require already having the expertise to properly assess what you are seeing.
    Actually getting the skills, running experiments, seeing the evidence for yourself? That’s all great stuff in my book. It’s not cheap to do, but if you care enough to learn to scuba dive, by all means scuba dive and see the primary evidence with your own eyes. It seems crazy to me to think this would not be a helpful thing to do – to me it is the most virtuous thing to do, if you care a lot.
    Alas, Scott then backtracks a bunch in this second post.
    He is afraid others will see him saying not to ‘trust the experts’ so he wants to reiterate to trust the experts, that reasoning is hard and you probably shouldn’t try doing it yourself. Then he says this:
    To a first approximation, trust experts over your own judgment. If people are trying to confuse you about who the experts are, then to a second approximation trust prestigious people and big institutions, including professors at top colleges, journalists at major newspapers, professional groups with names like the American ______ Association, and the...
    8 min