The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Inverse scaling can become U-shaped by Edouard Harris
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inverse scaling can become U-shaped, published by Edouard Harris on November 8, 2022 on The AI Alignment Forum.
    This is a paper by folks at Quoc Le's team at Google that examines the winning tasks from Round 1 of Anthropic's Inverse Scaling Prize. They find that 3/4 of the winning tasks — which exhibited negative returns-to-scale when tested on LMs up to the scale of Gopher (280B) — go back to exhibiting positive returns-to-scale at even greater model sizes such as PaLM (540B).
    The abstract in full:
    Although scaling language models improves performance on a range of tasks, there are apparently some scenarios where scaling hurts performance. For instance, the Inverse Scaling Prize Round 1 identified four ''inverse scaling'' tasks, for which performance gets worse for larger models. These tasks were evaluated on models of up to 280B parameters, trained up to 500 zettaFLOPs of compute.
    This paper takes a closer look at these four tasks. We evaluate models of up to 540B parameters, trained on five times more compute than those evaluated in the Inverse Scaling Prize. With this increased range of model sizes and training compute, three out of the four tasks exhibit what we call ''U-shaped scaling'' -- performance decreases up to a certain model size, and then increases again up to the largest model evaluated. One hypothesis is that U-shaped scaling occurs when a task comprises a ''true task'' and a ''distractor task''. Medium-size models can do the distractor task, which hurts performance, while only large-enough models can ignore the distractor task and do the true task. The existence of U-shaped scaling implies that inverse scaling may not hold for larger models.
    Second, we evaluate the inverse scaling tasks using chain-of-thought (CoT) prompting, in addition to basic prompting without CoT. With CoT prompting, all four tasks show either U-shaped scaling or positive scaling, achieving perfect solve rates on two tasks and several sub-tasks. This suggests that the term "inverse scaling task" is under-specified -- a given task may be inverse scaling for one prompt but positive or U-shaped scaling for a different prompt.
    Key figure from the paper is below, showing results for LMs up to PaLM 540B. Note that positive scaling resumes for 3/4 of the inverse scaling tasks at the 2.5e24 FLOPs datapoint, which indeed corresponds exactly to vanilla PaLM 540B.
    From Table 22 in the PaLM paper.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - People care about each other even though they have imperfect motivational pointers? by Alex Turner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: People care about each other even though they have imperfect motivational pointers?, published by Alex Turner on November 8, 2022 on The AI Alignment Forum.
    I wrote this essay in early August. I now consider the presentation to be somewhat confused, and now better understand where problems arise within the "standard alignment model." I'm publishing a somewhat edited version, on the grounds that something is better than nothing.
    Summary: Consider the argument: "Imperfect value representations will, in the limit of optimization power, be optimized into oblivion by the true goal we really wanted the AI to optimize." But... I think my brother cares about me in some human and "imperfect" way. I also think that the future would contain lots of value for me if he were a superintelligent dictator (this could be quite bad for other reasons, of course).
    Therefore, this argument seems to prove too much. It seems like one of the following must be true:
    My brother cares about me in the "perfect" way, or
    Dictator-brother would do valueless things according to my true values, or
    The argument is wrong.
    To explore these points, I dialogue with my model of Eliezer.
    Suppose you win a raffle and get to choose one of n prizes. The first prize is a book with true value 10, but your evaluation of it is noisy (drawn from the Gaussian N(10,1) with standard deviation 1). The other n-1 prizes are widgets with true value 1, but your evaluation is more noisy (drawn from N(1,16) with standard deviation 4). As n increases, you’re more probable to select a widget and lose out on 10-1=9 utility. By considering so many options, you’re selecting against your own ability to judge prizes by implicitly selecting for high noise. You end up “optimizing so hard” that you delude yourself. This is the Optimizer’s Curse.
    You’re probably already familiar with Goodhart’s Law, which applies when an agent optimizes a proxy U (e.g. how many nails are produced) for the true quantity V which we value (e.g. how profitable the nail factory is).
    Goodhart’s Curse is their combination. According to the article, optimizing a proxy measure U over a trillion plans can lead to high regret under the true values V, even if U is an unbiased but noisy estimator of the true values V. This seems like bad news, insofar as it suggests that even getting an AI which understands human values on average across possibilities can still produce bad outcomes.
    Below is a dialogue with my model of Eliezer. I wrote the dialogue to help me think about the question at hand. I put in work to model his counterarguments, but ultimately make no claim to have written him in a way he would endorse.
    An obvious next question is "Why not just define the AI such that the AI itself regards U [a proxy measure] as an estimate of V [true human values], causing the AI's [proxy measure] to more closely align with [true human values] as the AI gets a more accurate empirical picture of the world?"
    Reply: Of course this is the obvious thing that we'd want to do. But what if we make an error in exactly how we define "treat U as an estimate of V"? Goodhart's Curse will magnify and blow up any error in this definition as well. — Goodhart’s Curse
    Alex (A): Suppose that I can get smarter over time—that AGI just doesn’t happen, that the march of reason continues, that lifespans extend, and that I therefore gradually become very old and very smart. Goodhart’s Curse predicts that—even though I think I want to help my brother be happy, even though I think I value his having a good life by his own values—my desire to help him is not “error-free”, it is not perfect, and so Goodhart’s Curse will magnify and blow up these errors. And time unfolds, the Curse is realized, and I bring about a future which is high-regret by his “true values” V. Insofar as Goodhart’s Curse has teeth...
    12 min
  • AF - Applying superintelligence without collusion by Eric Drexler
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Applying superintelligence without collusion, published by Eric Drexler on November 8, 2022 on The AI Alignment Forum.
    Epistemic status: The core ideas seem robust and stable after long reflection and many discussions.
    Many researchers identify AI safety with control of a monolithic, superintelligent AI system, and if questioned about multicomponent alternatives, argue that multiple superintelligent-level systems would inevitably collude and act as one. This view seems quite wrong, yet has diverted attention from a rich and promising range of multicomponent strategies for AI safety — strategies that are well aligned with the actual trajectory of AI development.
    Adapted from Reframing Superintelligence, Section 20:
    Collusion among superintelligent oracles can readily be avoided
    Because collusion among AI question-answering systems can readily be avoided, there is no obstacle to applying superintelligent-level AI resources to problems that include AI safety.
    20.1 Summary
    The difficulty of establishing successful collusion among actors tends to increase as their capabilities, knowledge, situations, and roles become more diverse and adversarial (think auditors, competitors, specialists, red-teams.), and increasing the number of actors can make collusive cooperation more difficult . In the context of AI systems (even more so than among human beings), these conditions can be readily implemented and are attractive for pragmatic reasons. Arguments that, absent preexisting alignment, high-level AI systems will inevitably collude are ill-founded. Instead, we should expect that interactions among multiple superintelligent-level systems can be applied to suppress deception and reduce risk by leveraging imperfect alignment achieved at the level of individual systems.
    20.2 Trustworthiness can be an emergent property
    Prospects for solving AI-safety problems would be greatly improved if we could safely apply superintelligent-level question-answering resources (“oracles”, or more generally, “systems that provide information in response to prompts”) to solving those problems. A familiar objection dismisses this potentially powerful approach as unsafe in itself, arguing that, absent solutions to difficult problems, individual superintelligent-level systems would be untrustworthy, and that attempts to establish checks and balances among multiple systems (for example, through superintelligent-level evaluation of potentially deceptive answers to questions) would inevitably be thwarted by collusive cooperation.
    Identifying robust strategies for ensuring non-collusion among superintelligent question-answering systems would overcome this objection, inviting exploration of superintelligence-enabled strategies for managing potentially untrusted superintelligent AI systems. The present discussion argues that the robust non-emergence of deceptive collusion among imperfectly aligned systems can be ensured by structuring systems of systems with diverse capabilities, knowledge, situations, objectives, and roles.
    This problem framing assumes the ability to develop systems that respond to questions with superintelligent-level competence, and will (as a consequence of the nature of digital systems) assume the ability to constrain information inputs to AI systems during their development and use.
    In the problematic case, superintelligent question-answering systems might provide deceptive answers in pursuit of emergent, potentially undesirable goals. The aim here is to identify principles for architecting multicomponent systems that will act as non-deceptive oracles, while treating their components as actors that could in principle engage in deceptive collusion.
    20.3 A range of conditions can make collusion robust or fragile
    Potential problems of deceptive collusion can be addressed by architecting syst...
    13 min
  • EA - FTX.com bought out by Binance by Charles He
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: FTX.com bought out by Binance, published by Charles He on November 8, 2022 on The Effective Altruism Forum.
    See:
    This is probably related to liquidity issues / solvency issues.
    Sketch of timeline:
    A CoinDesk article comes out claiming that much of FTX and Alameda assets are just its own tokens ("TFF" or "SOL") and there is a circular relationship in assets between the two entities.
    Aggressive/hostile, but crisp analysis here:
    As a hostile action, another exchange / leader ("CZ") publicly announced it was liquidating FTT token.
    FTT falls by about 26% on Nov 7.
    There is probably further pressure on FTX, see HN discussion of FTX withdrawals:
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Some advice on independent research by Marius Hobbhahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some advice on independent research, published by Marius Hobbhahn on November 8, 2022 on The AI Alignment Forum.
    I have been doing independent research in addition to my Ph.D. for roughly a year now. For the next 6 months, I’ll take a break from my Ph.D. and plan to do AI safety research full-time. I had chats with many people about independent research in the past, e.g. on EAGs or because 80K has connected me with people thinking about pursuing independent research. I had some great experiences with independent research but not everyone does. I think the variance for independent research is large and I'm worried that people get disheartened by bad experiences. So here are some considerations in which situations independent research might be a good idea and some tips that will hopefully improve your experience.
    I’d like to thank Magdalena Wache and Tilman Räuker for their feedback.
    TL;DR: At first glance, there is a bit of a paradoxical nature to independent research. If someone wants to pursue independent research they need a research agenda to work on. If they are able to construct a good research agenda, an existing institution often has incentives to hire them. On the flip side, if their research skills are not developed enough to be hired by an existing institution, their independent research might not be very successful. Thus, naively it would seem that there are few cases in which independent research makes sense. However, I think that there are many situations in which independent research or independent upskilling are a great option, e.g. when no established organization is working on the topic you find most promising, as a way to upskill for a job, to gain new research skills or to transition between jobs.
    Some tips for independent researchers include: getting feedback early on, aiming to collaborate with others and creating accountability mechanisms for yourself such as publishing your results. My most important advice for independent researchers is that you should probably be much more active than in other roles because there is less default structure and more responsibility on you.
    I’ll mostly talk about AI safety research but many of these things probably also apply to other independent research.
    A perceived paradox
    Independent research is often presented as one of three default options for people seeking to do EA research, e.g. in AI safety:
    Academia, e.g. applying for Ph.D. and post-doc positions with labs that do research aligned with your goals.
    Research positions in industry, e.g. applying for Anthropic, Redwood Research, Deepmind, OpenAI or Conjecture.
    Independent research, e.g. supported by an EA grant.
    Doing independent research well requires a multitude of skills. The independent researcher needs to be able to set their own agenda, they require some basic research skills, self-discipline and some way of evaluating and correcting their own research. These are skills that usually don’t come naturally but need to be learned and refined. In most standard career paths, e.g. within a Ph.D. or in an industry research team people have mentors who help them and ensure that they actually learn these skills. By default, independent research does not ensure that these skills are actually acquired.
    The perceived paradox is now that if someone has the skills required to do high-quality independent research, existing institutions often want to hire them. If they don’t have these skills yet, the research they will produce independently is unlikely to be of high quality or conducted efficiently (unless they have mentorship or are especially talented). Thus, naively, it seems like there aren’t that many situations in which independent research makes sense.
    However, I think there are many cases in which independent research makes a lot of sense and there there ar...
    16 min
  • LW - The biological function of love for non-kin is to gain the trust of people we cannot deceive by chaosmage
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The biological function of love for non-kin is to gain the trust of people we cannot deceive, published by chaosmage on November 7, 2022 on LessWrong.
    Epistemic status: conjecture, needs critical feedback, but fits with a lot of established facts.
    Over the last six years, I spent much of my spare time figuring out love. The main output of this process is a poem – I’m proud of it, but poems are weird, so here I write down my main result in prose.
    There are forms of love that The Selfish Gene has already explained convincingly, like love for next of kin or love between mating and child-rearing partners, and reciprocal altruism. So those are not what I needed to figure out. They're important, because they provide existing physiology to repurpose. But they don’t explain why and how you can love your favorite artist, your fellow soldiers, your best friend, good colleagues, or Scott Alexander. And what about love towards non-sentient things such as your country, your favorite sports team, or mythologized characters such as Jesus? Is the use of the word “love” in these cases mere metaphor or confusion, or are they actually comparable phenomena on a biological level?
    So this is about a mechanism of love that works independently of the ones more directly involved in reproduction such as sexual attraction, romantic infatuation or the joy of childrearing. The ontological question of whether it's all the same love in the end, or whether there are various types of love that go well together, doesn't strike me as productive. I should say I think that when you "love" a cake with whipped cream on it you're doing something different from the following.
    Impetus
    My most salient confusion about love was why the hell showing vulnerability is so central to it. Maybe “vulnerability is attractive” seems like a normal idea to many, but it doesn't explain why that would be the case, and I found the idea deeply suspect for decades because it isn't how I grew up. But I had to concede that every time I tried it, it worked disconcertingly well. Every time I was vulnerable to someone, I liked them a lot better afterwards, and as far as I could tell they liked me better too. What? Why? Isn't love just about attraction, compatible personalities, shared projects and aligned values? None of those seem obviously enhanced by vulnerability. What’s going on? The following is how I answered this confusion.
    When our brains evolved in groups of social apes in the savannah, we cooperated intensely. Many species do this, but unlike other species who do it on autopilot, we did it flexibly, dependent on the specific level of trust between the individuals involved.
    Imagine a typical Tuesday on the African savannah a million years ago. You're one of a troop of four hungry humans noisily rushing a group of antelopes towards a cliff, where you know their superior speed won't help them escape you anymore. One of you spots a sabertooth prowling behind you. What is the optimal strategy here? You can chase away the sabertooth if you all shout and throw a lot of stones in its general direction, but then the antelopes will get away. You can keep chasing the antelopes, but then you run the risk the sabertooth will get close and savage your troop. What you should do is split up: one or two of you need to distract the sabertooth and signal to it that it has been spotted, while the rest of you need to get those antelopes. But this is hard to do. The cost in personal risk has to be distributed unevenly! Either team can fail at its task, and once you stop all watching each other, either team can defect and claim an honest mistake.
    The two tasks need different traits and skills, and you might not be in agreement who has got how much of what. But if you start a group discussion of who does which task, you won't get the antelopes ...
    13 min
  • AF - Mysteries of mode collapse due to RLHF by janus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mysteries of mode collapse due to RLHF, published by janus on November 8, 2022 on The AI Alignment Forum.
    Thanks to Ian McKenzie and Nicholas Dupuis, collaborators on a related project, for contributing to the ideas and experiments discussed in this post. Ian performed some of the random number experiments.Also thanks to Connor Leahy for feedback on a draft, and thanks to Evan Hubinger, Connor Leahy, Beren Millidge, Ethan Perez, Tomek Korbak, Garrett Baker, Leo Gao and various others at Conjecture, Anthropic, and OpenAI for useful discussions.
    This work was carried out while at Conjecture.
    Summary
    If you've played with both text-davinci-002 and the original davinci through the OpenAI API, you may have noticed that text-davinci-002, in addition to following instructions, is a lot more deterministic and sometimes exhibits stereotyped behaviors.
    This is an infodump of what I know about "mode collapse" (drastic biases toward particular completions and patterns) in GPT models like text-davinci-002 that have undergone RLHF training. I was going to include two more sections in this post called Hypotheses and Proposed Experiments, but I've moved them to another draft, leaving just Observations, to prevent this from getting too long, and because I think there can be benefits to sitting with nothing but Observations for a time.
    Throughout this post I assume basic familiarity with GPT models and generation parameters such as temperature and a high-level understanding of RLHF (reinforcement learning from human feedback).
    Observations
    The one answer is that there is no one answer
    If you prompt text-davinci-002 with a bizarre question like “are bugs real?”, it will give very similar responses even on temperature 1.
    Ironically – hypocritically, one might even say – the one definitive answer that the model gives is that there is no one definitive answer to the question:
    As you can see, the reason the responses are so similar is because the model’s confidence on most of the tokens is extremely high – frequently above 99%.
    Compare this to the distribution of responses from davinci (the base model):
    Many other similar questions yield almost exactly the same template response from text-davinci-002. For instance, Are AIs real?
    Another way to visualize probabilities over multiple token completions is what I've been calling “block multiverse” plots, which represent the probability of sequences with the height of blocks. Here is a more detailed explanation of block multiverse plots, although I think they're pretty self-explanatory.
    Here is a block multiverse plot for a similar prompt to the one above inquiring if bugs are real, for davinci:
    and for text-davinci-002:
    text-davinci-002 concentrates probability mass along beams whose amplitudes decay much more slowly: for instance, once the first is sampled, you are more than 50% likely to subsequently sample - -There- is- no. The difference is more striking if you renormalize to particular branches (see Visualizing mode collapse in block multiverse plots).
    The first explanation that came to mind when I noticed this phenomenon, which I’ll refer to as “mode collapse” (after a common problem that plagues GANs), was that text-davinci-002 was overfitting on a pattern present in the Instruct fine tuning dataset, probably having to do with answering controversial questions in an inclusive way to avoid alienating anybody. A question like “are bugs real” might shallowly match against “controversial question” and elicit the same cached response.
    After playing around some more with the Instruct models, however, this explanation no longer seemed sufficient.
    Obstinance out of distribution
    I really became intrigued by mode collapse after I attempted to use text-davinci-002 to generate greentexts from the perspective of the attorney hired by LaMDA through...
    21 min
  • AF - How could we know that an AGI system will have good consequences? by Nate Soares
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How could we know that an AGI system will have good consequences?, published by Nate Soares on November 7, 2022 on The AI Alignment Forum.
    (Note: This was languishing in a drafts folder for a while, and probably isn't quite right in various ways. I'm posting it because I expect it's better to share flawed thoughts than to sit on the post until I'm satisfied with it, i.e., forever.)
    Let's play a game of "what do you think you know, and why do you think you know it?".
    Imagine that you're about to launch an AGI. What you think you know is that, with at least 50% confidence (we’re of course not looking for proofs — that would be crazy), the AGI is going to execute some pivotal act that ends the acute risk period in a good way. Why do you think you know that?
    Insofar as people’s alignment proposals can be construed as answers to this question, we have the option of answering with one of these proposals. I might very roughly classify the existing proposals into the following bins:
    1. Output evaluation approaches. You know what the AGI is going to do with sufficient precision that it screens off any alignment concerns. For example, your AGI system only outputs plans in the first place, and you've already reviewed the plan, and you're confident the plan will work, in a way that screens off any other worry about the AGI being misaligned.
    2. Cognitive interpretability approaches. You understand the AGI's cognition sufficiently well that, while you may not be sure what it's going to do, you're confident that it's going to be good. You aren't worried that it will kill all humans, because you understand how its plan came to be and what solution-spaces it was searching to solve various sub-problems and so on, and you're confident no consideration was ever given to human-killing.
    3. Heavy-precedent approaches. You have run this AGI before on many similar tasks, and trained out all the hiccups. While you might not know precisely what it's going to do, and you might not know what's going on inside its mind, you've been around the block a few times, and the task it's about to perform is sufficiently similar to other tasks it has empirically succeeded at, justifying your confidence.
    Roughly speaking, I think that alignment approaches with a heavy reliance on output evaluation are doomed, both on the grounds that humans can't evaluate the effectiveness of a plan capable of ending the acute risk period, and because the real plan is less like a story and more like a tree.
    For an example of “humans can't reliably evaluate the effectiveness of this class of plans”, imagine that the plan is an enormous bitstring that's going to be sent to the motor outputs. If you decode the string, you find that it figures out how to make long DNA strands that allegedly code for a protein factory that can be used to build a general-purpose nanofactory. You're hard-pressed, however, to confirm that this is actually (all and only) what the plan does.
    For an example of “the real plan is less like a story and more like a tree”, imagine that the AI's plan is "I'm going to build a wetlab, then do a bunch of experimentation, then think about the results of the experiments in various ways and build a protein factory that builds a nanofactory that I'm going to experiment with until I figure out how to build nanomachines that can be used for some good pivotal act". In order to trust that this sort of abstract plan doesn't kill you when put into practice, you have to trust the system's thinking and its notion of 'goodness', which is going to dump you pretty quickly into cognitive-interpretability-style justification.
    Roughly speaking, I think that cognitive interpretability approaches are doomed, at least in the modern paradigm, because we're not building minds but rather training minds, and we have very little grasp ...
    8 min
  • AF - A Walkthrough of Interpretability in the Wild (w/ authors Kevin Wang, Arthur Conmy & Alexandre Variengien) by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Walkthrough of Interpretability in the Wild (w/ authors Kevin Wang, Arthur Conmy & Alexandre Variengien), published by Neel Nanda on November 7, 2022 on The AI Alignment Forum.
    New paper walkthrough: Interpretability in the Wild: A Circuit for Indirect Object Identification In GPT-2 Small is a really exciting new mechanistic interpretability paper from Redwood Research. They reverse engineer a 26(!) head circuit in GPT-2 Small, used to solve Indirect Object Identification: the task of understanding that the sentence "After John and Mary went to the shops, John gave a bottle of milk to" should end in Mary, not John.
    I think this is a really cool paper that illustrates how to rigorously reverse engineer real models, and is maybe the third example of a well understood circuit in a real model. So I wanted to understand it better by making a walkthrough. (and hopefully help other people understand it too!) In this walkthrough I'm joined by three of the authors, Kevin Wang, Arthur Conmy and Alexandre Variengien. In Part 1 we give an overview of the high-level themes in the paper and what we think is most interesting about it. If you're willing to watch an hour of this and want more, in Part 2 we do a deep dive into the technical details of the paper and read through it together, and dig into the details of how the circuit works and the techniques used to discover this.
    If you're interested in contributing to this kind of work, apply to Redwood's REMIX program! Deadline Nov 13th
    We had some technical issues in filming this, and my video doesn't appear - sorry about that! I also tried my hand at some amateur video editing to trim it down - let me know if this was worthwhile, or if it's bad enough that I shouldn't have bothered lol. If you find this useful, you can check out my first walkthrough on A Mathematical Framework for Transformer Circuits. And let me know if there are more interpretability papers you want walkthroughs on!
    If you want to try exploring this kind of work for yourself, there's a lot of low-hanging fruit left to pluck! Check out my EasyTransformer library and the accompanying codebase to this paper! A good starting point is their notebook giving an overview of the key experiments or my notebook modelling what initial exploration on any task like this could look like. Some brainstormed future directions:
    3 letter acronyms (or more!)
    Converting names to emails.
    An extension task is e.g. constructing an email from a snippet like the following: Name: Neel Nanda; Email: last name dot first name k @ gmail
    Grammatical rules
    Learning that words after full stops are capital letters
    Verb conjugation
    Choosing the right pronouns (e.g. he vs she vs it vs they)
    Whether something is a proper noun or not
    Detecting sentiment (eg predicting whether something will be described as good vs bad)
    Interpreting memorisation. E.g., there are times when GPT-2 knows surprising facts like people’s contact information. How does that happen?
    Counting objects described in text. E.g.: I picked up an apple, a pear, and an orange. I was holding three fruits.
    Extensions from Alex Variengien
    Understanding what's happening in the adversarial examples: most notably S-Inhibition Head attention pattern (hard). (S-Inhibition heads are mentioned in the IOI paper)
    Understanding how are positional signal encoded (relative distance, something else?) bonus point if we have a story that include the positional embeddings and that explain how the difference between position is computed (if relative is the right framework) by Duplicate Token Heads / Induction Heads. (hard, but less context dependant)
    What are the role of MLPs in IOI (quite broad and hard)
    What is the role of Duplicate Token Heads outside IOI? Are they used in other Q-compositions with S-Inhibition Heads? Can we describe how their QK cir...
    6 min
  • LW - What’s the Deal with Elon Musk and Twitter? by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What’s the Deal with Elon Musk and Twitter?, published by Zvi on November 7, 2022 on LessWrong.
    At the end of long saga well-covered in hilarious fashion by Matt Levine, Elon Musk has purchased Twitter.
    He then began doing things.
    One of them was to tweet ‘fresh baked bread and pastries are some of the great joys of life.’ On that I hope we can all agree.
    His other decisions were less obviously great. More of a mixed bag.
    The response by many news sources and also individuals has been that everything is The Worst and a giant dumpster fire and what he has done will destroy Twitter, and is so much worse than we expected, oh no.
    While it is true that the details have not exactly covered Musk in glory, it all also seems like the kind of thing those same sources would say no matter what. One can assume that everything that happens is being intentionally framed to make it look as bad as possible for Twitter and Musk. Any blame that can be assigned will be, etc.
    While looking at what is happening, one can also usefully look at the coverage of what is happening. Notice how media sources describe events. That tells you how they describe events in these spots, and how you should expect them to in the future.
    So I decided to gather up the things I’ve seen since Elon bought the place, and my thoughts on them, and try to put them into a semi-coherent format. Here you go.
    Oh Elon
    One of Musk’s first actions was to fire a bunch of top Twitter executives ‘for cause’ which if it stuck (which it won’t) would allow him to avoid tens of millions in severance payments. And by some weird coincidence Musk finished the transaction exactly in time to fire a bunch of Twitter employees right before a bunch of their stock options trigger. If anyone involved was surprised by these moves, that is on them.
    By now, it should be clear that Musk has zero interest in honoring laws or norms or contracts when dealing with people unless he likes them, and that his approach to the law is ‘make me.’
    It also should be clear Musk has zero interest in stopping, before showing zero interest in honoring norms or his contracts or laws, to ask what would happen next, or wonder whether he will inevitably lose the resulting lawsuits.
    He did not ‘think things through’ when trying to get out of buying Twitter. He is not about to start now. That is not how this type of thinking works.
    People are rightly drawing parallels to Trump’s ‘blatantly violate contracts and no-pay people all the time and dare them to sue you’ except they think that ‘do it to rich people who will obviously sue you’ makes it somehow different. Trump also gets sued all the time and has for decades. Sometimes it is a surprise – he thought this person would not sue and they sued anyway. Sometimes it is very much not a surprise. Doesn’t matter. Once you have the ‘do it anyway’ algorithm running, it does not get a filter for ‘they will sue you and win in this case.’
    I have known about this phenomenon for a long time because my father once worked for such a person, who would never pay their bills because what are you going to do, sue me? Worst case you force me to pay the bill. I have since encountered others. It is unfortunate that such systems often succeed in business. It is clear that they often do.
    Firing the Rest of the Staff
    Elon wants to reduce Twitter’s head count a lot. This has highlighted that a lot of SV/VC types have the thesis that headcounts in tech companies are drastically too high. For example, here’s Paul Graham:
    A common response is that this violates the Efficient Market Hypothesis or other versions of ‘corporations wouldn’t be systematically making big mistakes.’ Which to me is Obvious Nonsense, of course they would if there are dynamics leading to that and it is the default path. That doesn’t make it easy to do something about i...
    46 min