The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Contrast Pairs Drive the Empirical Performance of Contrast Consistent Search (CCS) by Scott Emmons
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contrast Pairs Drive the Empirical Performance of Contrast Consistent Search (CCS), published by Scott Emmons on May 31, 2023 on LessWrong.
    tl;dr
    Contrast consistent search (CCS) is a method by Burns et al. that consists of two parts:
    Generate contrast pairs by adding pseudolabels to an unlabelled dataset.
    Use the contrast pairs to search for a direction in representation space that satisfies logical consistency properties.
    In discussions with other researchers, I've repeatedly heard (2) as the explanation for how CCS works; I've heard almost no mention of (1).
    In this post, I want to emphasize that the contrast pairs drive almost all of the empirical performance in Burns et al. Once we have the contrast pairs, standard unsupervised learning methods attain comparable performance to the new CCS loss function.
    In the paper, Burns et al. do a nice job comparing the CCS loss function to different alternatives. The simplest such alternative runs principal component analysis (PCA) on contrast pair differences, and then it uses the top principal component as a classifier. Another alternative runs linear discriminant analysis (LDA) on contrast pair differences. These alternatives attain 97% and 98% of CCS's accuracy!
    "[R]epresentations of truth tend to be salient in models: ... they can often be found by taking the top principal component of a slightly modified representation space," Burns et al. write in the introduction. If I understand this statement correctly, it's saying the same thing I want to emphasize in this post: the contrast pairs are what allow Burns et al. to find representations of truth. Empirically, once we have the representations of contrast pair differences, their variance points in the direction of truth. The new logical consistency loss in CCS isn't needed for good empirical performance.
    Notation
    We'll follow the notation of the CCS paper.
    Assume we are given a data set {x1,x2,.,xn} and a feature extractor ϕ(), such as the hidden state of a pretrained language model.
    First, we will construct a contrast pair for each datapoint xi. We add “label: positive” and “label: negative” to each xi. This gives contrast pairs of the form (x+i,x−i).
    Now, we consider the set {x+1,x+2,.,x+n} of positive pseudo-labels and {x−1,x−2,.,x−n} of negative pseudo-labels. Because all of the x+i have "label: positive" and all of the x−i have "label: negative", we normalize the positive pseudo-labels and the negative pseudo-labels separately:
    Here, μ+ and μ− are the element-wise means of the positive and negative pseudo-label sets, respectively. Similarly, σ+ and σ− are the element-wise standard deviations.
    The goal of this normalization is to remove the embedding of "label: positive" from all the positive pseudo-labels (and "label: negative" from all the negative pseudo-labels). The hope is that by construction, the only difference between ~ϕ(x+i) and ~ϕ(x−i) is that one is true while the other is false. CCS is one way to extract the information about true and false. As we'll discuss more below, doing PCA or LDA on the set of differences {~ϕ(x+i)−~ϕ(x−i)}ni=1 works almost as well.
    Concept Embeddings in Prior Work
    In order to better understand contrast pairs, I think it's helpful to review this famous paper by Bolukbasi et al., 2016: "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings." Quoting from Bolukbasi et al.:
    −−−man−−−−−−woman≈−−−king−−−−−queen
    Vector differences between words in embeddings have been shown to represent relationships between words. For example given an analogy puzzle, "man is to king as woman is to x" (denoted as man:king :: woman:x), simple arithmetic of the embedding vectors finds that x=queen is the best answer because:
    Similarly, x=Japan is returned for Paris:France :: Tokyo:x. It is surprising that a simple vector arithm...
    13 min
  • LW - Product Endorsement: Food for sleep interruptions by Elizabeth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Product Endorsement: Food for sleep interruptions, published by Elizabeth on May 31, 2023 on LessWrong.
    Do you wake up before you want to and struggle with getting back to sleep when you do? One thing (out of many) that can cause this is low blood sugar. If this is your issue, it is easily solved by eating high-fat, high-protein food.
    My evidence on this is purely anecdotal; I learned this from r/adviceanimals in 2012, so the intellectual pedigree is weak. But I added ~one full sleep cycle to my night when I tried it, which lasted for years until I was forced to stop due to medication requiring an empty stomach (I now use ketone esters). A month ago my dad went from “3-5 hours/night, 3 hours awake, maybe more sleep after that” to “8-9 hours of near-continuous sleep, 1 night per fortnight a mere 6 hours” by keeping peanut butter by his bed. Downside: he’s now useless for testing the magic vibrating bracelet I’m so hyped about.
    This is very easy to test. Keep some nuts by your bed; if you wake up, eat some. If you fall back asleep, congratulations on fixing your problem. There are more complicated things you can do with your evening diet to try and prevent early wake-ups entirely but I feel very muddled about these and don’t have good suggestions.
    If you’re doing intermittent fasting, ketone esters (affiliate link) don’t work for many people but are great if you’re one of the lucky ones (I am), and ~don’t count as eating. MCT oils (affiliate link) work on the same principle and so may be an alternative, but I haven’t tested them personally.
    I am not a doctor, anything with a real effect can hurt you, be careful not to choke or attract ants, etc.
    It’s not clear to me if this is actually low blood sugar as measured glucose in your bloodstream, or naturopathic “low blood sugar”, meaning problems you fix by eating, but I don’t care.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Contrast Pairs Drive the Empirical Performance of Contrast Consistent Search (CCS) by Scott Emmons
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contrast Pairs Drive the Empirical Performance of Contrast Consistent Search (CCS), published by Scott Emmons on May 31, 2023 on The AI Alignment Forum.
    tl;dr
    Contrast consistent search (CCS) is a method by Burns et al. that consists of two parts:
    Generate contrast pairs by adding pseudolabels to an unlabelled dataset.
    Use the contrast pairs to search for a direction in representation space that satisfies logical consistency properties.
    In discussions with other researchers, I've repeatedly heard (2) as the explanation for how CCS works; I've heard almost no mention of (1).
    In this post, I want to emphasize that the contrast pairs drive almost all of the empirical performance in Burns et al. Once we have the contrast pairs, standard unsupervised learning methods attain comparable performance to the new CCS loss function.
    In the paper, Burns et al. do a nice job comparing the CCS loss function to different alternatives. The simplest such alternative runs principal component analysis (PCA) on contrast pair differences, and then it uses the top principal component as a classifier. Another alternative runs linear discriminant analysis (LDA) on contrast pair differences. These alternatives attain 97% and 98% of CCS's accuracy!
    "[R]epresentations of truth tend to be salient in models: ... they can often be found by taking the top principal component of a slightly modified representation space," Burns et al. write in the introduction. If I understand this statement correctly, it's saying the same thing I want to emphasize in this post: the contrast pairs are what allow Burns et al. to find representations of truth. Empirically, once we have the representations of contrast pair differences, their variance points in the direction of truth. The new logical consistency loss in CCS isn't needed for good empirical performance.
    Notation
    We'll follow the notation of the CCS paper.
    Assume we are given a data set {x1,x2,.,xn} and a feature extractor ϕ(), such as the hidden state of a pretrained language model.
    First, we will construct a contrast pair for each datapoint xi. We add “label: positive” and “label: negative” to each xi. This gives contrast pairs of the form (x+i,x−i).
    Now, we consider the set {x+1,x+2,.,x+n} of positive pseudo-labels and {x−1,x−2,.,x−n} of negative pseudo-labels. Because all of the x+i have "label: positive" and all of the x−i have "label: negative", we normalize the positive pseudo-labels and the negative pseudo-labels separately:
    Here, μ+ and μ− are the element-wise means of the positive and negative pseudo-label sets, respectively. Similarly, σ+ and σ− are the element-wise standard deviations.
    The goal of this normalization is to remove the embedding of "label: positive" from all the positive pseudo-labels (and "label: negative" from all the negative pseudo-labels). The hope is that by construction, the only difference between ~ϕ(x+i) and ~ϕ(x−i) is that one is true while the other is false. CCS is one way to extract the information about true and false. As we'll discuss more below, doing PCA or LDA on the set of differences {~ϕ(x+i)−~ϕ(x−i)}ni=1 works almost as well.
    Concept Embeddings in Prior Work
    In order to better understand contrast pairs, I think it's helpful to review this famous paper by Bolukbasi et al., 2016: "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings." Quoting from Bolukbasi et al.:
    −−−man−−−−−−woman≈−−−king−−−−−queen
    Vector differences between words in embeddings have been shown to represent relationships between words. For example given an analogy puzzle, "man is to king as woman is to x" (denoted as man:king :: woman:x), simple arithmetic of the embedding vectors finds that x=queen is the best answer because:
    Similarly, x=Japan is returned for Paris:France :: Tokyo:x. It is surprising that a simple ...
    13 min
  • LW - The case for removing alignment and ML research from the training dataset by beren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The case for removing alignment and ML research from the training dataset, published by beren on May 30, 2023 on LessWrong.
    This is crossposted from my personal blog.
    Many possible sources of danger from AI stem from the AI knowing a lot about itself, us, and the potential auditing and alignment methods we might try to use to either align the AI or to detect misaligned models. Suppose we have an AGI that contains a misaligned mesaoptimizer playing the training game. To succeed at this, the mesaoptimizer needs to 'know' essentially all of the following facts:
    1.) That it is part of an AI model being trained and what that means,
    2.) How training by SGD works,
    3.) Probably some indication of the architecture of the model it exists in,
    4.) Difference between deployment and training environments as well as that 'deployment' is likely to occur at some point in its future and be much less constrained than training,
    5.) The kinds of things that human overseers encourage and are looking out for.
    Similarly with deception. A misaligned power-seeking AI, to successfully be able to deceive us, even when we are running evals to detect it, needs to have almost perfect knowledge of the following:
    1.) That it is an AI model with goals misaligned from that of its creators,
    2.) That if we detect its misaligned 'true' behaviour that we will probably turn it off or otherwise thwart its goals,
    3.) That the situations it observes itself in are almost certainly training or evaluation situations where it should behave nicely,
    4.) What the 'nice' and aligned behaviour humans expect actually is,
    5.) How to tell when it is in 'real deployment' mode so it can start amassing power,
    6.) If we are using interpretability or other probing based methods to detect deception and monitor its internal thoughts, it needs to know that these are possible, and approximately how they work to have a good chance of thwarting us by applying techniques from circumventing interpretability.
    Without a large degree of this kind of knowledge, pulling off successful deception or sharp left turn maneuver against a moderately competent set of human and AI overseers seems very challenging and unlikely to succeed on the first shot (and an individual misaligned AI has only one shot).
    Almost all of this information is not impossible to derive a-priori from a sensory stream and knowledge base that comprises most of the internet. Definitely its current training data contains much (but not all) of this information. However, this information is, almost certainly, not derivable from scratch. This means that an incredibly low hanging fruit for safety is simply to remove the documents containing this information from the training set the AI is exposed to! The less the AGI knows specifically about itself, about us, and about our alignment strategies and countermeasures, the less likely it will be able to successfully evade and counteract them.
    Naturally, there is a tradeoff between the capabilities of the model and the information we allow it access to. We want AI systems to be good and useful to people. Ultimately we do want to build an AGI which is aligned. An AI that knows literally nothing is very safe, but also useless. However, it seems like the pareto frontier near our current position is extremely flat, such that we can maybe get large safety gains at almost no general capability cost. Specifically, my proposal is to remove from the training set documents which could provide knowledge about the models own internal workings (to help prevent self-awareness) and our alignment techniques and countermeasures for detecting and neutralizing misaligned AGIs.
    A naive solution would simply remove a.) lesswrong, alignment forum, and all other major hubs of alignment discussion and b.) remove ML arxiv from pretraining datasets. This is t...
    8 min
  • LW - PaLM-2 & GPT-4 in "Extrapolating GPT-N performance" by Lukas Finnveden
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PaLM-2 & GPT-4 in "Extrapolating GPT-N performance", published by Lukas Finnveden on May 30, 2023 on LessWrong.
    Two and a half years ago, I wrote Extrapolating GPT-N performance, trying to predict how fast scaled-up models would improve on a few benchmarks. One year ago, I added PaLM to the graphs. Another spring has come and gone, and there are new models to add to the graphs: PaLM-2 and GPT-4. (Though I only know GPT-4's performance on a small handful of benchmarks.)
    Converting to Chinchilla scaling laws
    In previous iterations of the graph, the x-position represented the loss on GPT-3's validation set, and the x-axis was annotated with estimates of size+data that you'd need to achieve that loss according to the Kaplan scaling laws. (When adding PaLM to the graph, I estimated its loss using those same Kaplan scaling laws.)
    In these new iterations, the x-position instead represents an estimate of (reducible) loss according to the Chinchilla scaling laws. Even without adding any new data-points, this predicts faster progress, since the Chinchilla scaling laws describes how to get better performance for less compute.
    The appendix describes how I estimate Chinchilla reducible loss for GPT-3 and PaLM-1. Briefly: For the GPT-3 data points, I convert from loss reported in the GPT-3 paper, to the minimum of parameters and tokens you'd need to achieve that loss according to Kaplan scaling laws, and then plug those numbers of parameters and tokens into the Chinchilla loss function. For PaLM-1, I straightforwardly put its parameter- and token-count into the Chinchilla loss function.
    To start off, let's look at a graph with only GPT-3 and PaLM-1, with a Chinchilla x-axis.
    Here's a quick explainer of how to read the graphs (the original post contains more details). Each dot represents a particular model’s performance on a particular category of benchmarks (taken from papers about GPT-3 and PaLM). Color represents benchmark; y-position represents benchmark performance (normalized between random and my guess of maximum possible performance).
    The x-axis labels are all using the Chinchilla scaling laws to predict reducible loss-per-token, number of parameters, number of tokens, and total FLOP (if language models at that loss were trained Chinchilla-optimally).
    Compare to the last graph in this comment, which is the same with a Kaplan x-axis. Some things worth noting:
    PaLM is now ~0.5 OOM of compute less far along the x-axis. This corresponds to the fact that you could get PaLM for cheaper if you used optimal parameter- and data-scaling.
    The smaller GPT-3 models are farther to the right on the x-axis. I think this is mainly because the x-axis in my previous post had a different interpretation.
    The overall effect is that the data points get compressed together, and the slope becomes steeper. Previously, the black "Average" sigmoid reached 90% at ~1e28 FLOP. Now it looks like it reaches 90% at ~5e26 FLOP.
    Let's move on to PaLM-2. If you want to guess whether PaLM-2 and GPT-4 will underperform or outperform extrapolations, now might be a good time to think about that.
    PaLM-2
    If this CNBC leak is to be trusted, PaLM-2 uses 340B parameters and is trained on 3.6T tokens. That's more parameters and less tokens than is recommended by the Chinchilla training laws. Possible explanations include:
    The model isn't dense. Perhaps it implements some type of mixture-of-experts situation that means that its effective parameter-count is smaller.
    It's trained Chinchilla-optimally for multiple epochs on a 3.6T token dataset.
    The leak is wrong.
    If we assume that the leak isn't too wrong, I think that fairly safe bounds for PaLM-2's Chinchilla-equivalent compute is:
    It's as good as a dense Chinchilla-optimal model trained on just 3.6T tokens, i.e. one with 3.6T/20=180B parameters. This would make it 6180e...
    12 min
  • EA - Updates from the Dutch EA community by James Herbert
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Updates from the Dutch EA community, published by James Herbert on May 30, 2023 on The Effective Altruism Forum.
    We wanted to post something on the Forum to share all of the amazing things the Dutch EA community has achieved in the past 18 months or so. But we also wanted to avoid spending too much time writing it. So please accept this very messy post and feel free to ask questions in the comments! Parts were co-written by ChatGPT with minimal editing, hence the sometimes overly braggadocious tone.
    We start with national-level updates and two quick lessons-learnt, and then we have bullet-point summaries from some of the local groups. But first, an executive summary.
    Executive summary
    Over the past year, the Dutch EA community has seen impressive growth at both the national and local levels.
    At the national level, the community has seen significant gains. The number of intro programme completions increased nearly tenfold, from 45 in 2021 to 400 in 2022. The number of city groups and university groups also grew, from 1 to 3 and 1 to 13 respectively. Notably, there was an influx of €700k donations via Doneer Effectief and an increase in EA Netherlands newsletter subscribers from 670 to around 1500.
    Since hiring two full-time community builders in 2022, EAN has helped establish over a dozen new groups which have collectively produced 350 intro programme graduates in 2022 alone. In addition to launching a new website and co-working space, EAN organized multiple retreats, conducted introductory talks, facilitated 'giving games', provided career counselling, hosted city meet-ups, and participated in a public debate on EA.
    Effective altruism is gaining recognition in the Dutch media, with coverage in major Dutch publications and appearances by prominent figures like writer Rutger Bregman. However, there have also been a few critical pieces, to which the EAN board has responded.
    Other significant achievements include the successful launch of Doneer Effectief's online donation platform, the high-profile EAGxRotterdam 2022 conference, and the Tien Procent Club's successful events on effective giving.
    Local EA groups across Dutch cities have also seen substantial growth. For example, the Amsterdam city and university groups have merged, and together they host weekly meetups, multiple programs, and are developing a mental health program. At Utrecht, the student group has hatched an Alt Protein group with a grant from the university, has launched an AI Safety group, has hosted a big speaker event with Rutger Bregman, and runs introduction fellowships, socials, coworking sessions and other events. In The Hague, the group conducted weekly dinners, three rounds of intro fellowships, and two rounds of AI governance fellowships.
    The team at Delft has increased EA awareness through fellowships, book clubs, a retreat, and launching the Delft AI Safety Initiative. Eindhoven’s group has engaged 31 people in Introduction Fellowships, has launched an AI safety team, and collaborated with other groups on their university campus. Nijmegen’s group has grown rapidly, with biweekly meetups and collaborations with other campus groups.
    The PISE group in Rotterdam hosts member-only weekly events, open book clubs, and four fellowship rounds this year. They also initiated EAGx Rotterdam. The Twente group has attended the university’s career fair and organized meetups and an introductory talk. Wageningen University's group has hosted live events and completed an introductory fellowship.
    Lessons learnt:
    Do organising and mobilising (organisers invest in developing the capacities of people to engage with others in activism and become leaders; mobilisers focus on maximising the number of people involved without developing their capacity for civic action)
    It's very valuable to have a public figure endorse...
    19 min
  • AF - PaLM-2 & GPT-4 in "Extrapolating GPT-N performance" by Lukas Finnveden
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PaLM-2 & GPT-4 in "Extrapolating GPT-N performance", published by Lukas Finnveden on May 30, 2023 on The AI Alignment Forum.
    Two and a half years ago, I wrote Extrapolating GPT-N performance, trying to predict how fast scaled-up models would improve on a few benchmarks. One year ago, I added PaLM to the graphs. Another spring has come and gone, and there are new models to add to the graphs: PaLM-2 and GPT-4. (Though I only know GPT-4's performance on a small handful of benchmarks.)
    Converting to Chinchilla scaling laws
    In previous iterations of the graph, the x-position represented the loss on GPT-3's validation set, and the x-axis was annotated with estimates of size+data that you'd need to achieve that loss according to the Kaplan scaling laws. (When adding PaLM to the graph, I estimated its loss using those same Kaplan scaling laws.)
    In these new iterations, the x-position instead represents an estimate of (reducible) loss according to the Chinchilla scaling laws. Even without adding any new data-points, this predicts faster progress, since the Chinchilla scaling laws describes how to get better performance for less compute.
    The appendix describes how I estimate Chinchilla reducible loss for GPT-3 and PaLM-1. Briefly: For the GPT-3 data points, I convert from loss reported in the GPT-3 paper, to the minimum of parameters and tokens you'd need to achieve that loss according to Kaplan scaling laws, and then plug those numbers of parameters and tokens into the Chinchilla loss function. For PaLM-1, I straightforwardly put its parameter- and token-count into the Chinchilla loss function.
    To start off, let's look at a graph with only GPT-3 and PaLM-1, with a Chinchilla x-axis.
    Here's a quick explainer of how to read the graphs (the original post contains more details). Each dot represents a particular model’s performance on a particular category of benchmarks (taken from papers about GPT-3 and PaLM). Color represents benchmark; y-position represents benchmark performance (normalized between random and my guess of maximum possible performance).
    The x-axis labels are all using the Chinchilla scaling laws to predict reducible loss-per-token, number of parameters, number of tokens, and total FLOP (if language models at that loss were trained Chinchilla-optimally).
    Compare to the last graph in this comment, which is the same with a Kaplan x-axis. Some things worth noting:
    PaLM is now ~0.5 OOM of compute less far along the x-axis. This corresponds to the fact that you could get PaLM for cheaper if you used optimal parameter- and data-scaling.
    The smaller GPT-3 models are farther to the right on the x-axis. I think this is mainly because the x-axis in my previous post had a different interpretation.
    The overall effect is that the data points get compressed together, and the slope becomes steeper. Previously, the black "Average" sigmoid reached 90% at ~1e28 FLOP. Now it looks like it reaches 90% at ~5e26 FLOP.
    Let's move on to PaLM-2. If you want to guess whether PaLM-2 and GPT-4 will underperform or outperform extrapolations, now might be a good time to think about that.
    PaLM-2
    If this CNBC leak is to be trusted, PaLM-2 uses 340B parameters and is trained on 3.6T tokens. That's more parameters and less tokens than is recommended by the Chinchilla training laws. Possible explanations include:
    The model isn't dense. Perhaps it implements some type of mixture-of-experts situation that means that its effective parameter-count is smaller.
    It's trained Chinchilla-optimally for multiple epochs on a 3.6T token dataset.
    The leak is wrong.
    If we assume that the leak isn't too wrong, I think that fairly safe bounds for PaLM-2's Chinchilla-equivalent compute is:
    It's as good as a dense Chinchilla-optimal model trained on just 3.6T tokens, i.e. one with 3.6T/20=180B parameters. This would ...
    12 min
  • EA - Announcement: you can now listen to all new EA Forum posts by peterhartree
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcement: you can now listen to all new EA Forum posts, published by peterhartree on May 30, 2023 on The Effective Altruism Forum.
    For the next few weeks, all new EA Forum posts will have AI narrations.
    We're releasing this feature as a pilot. We will collect feedback and then decide whether to keep the feature and/or roll it out more broadly (e.g. for our full post archive).
    This project is run by TYPE III AUDIO in collaboration with the EA Forum team.
    How can I listen?
    On post pages
    You'll find narrations on post pages; you can listen to them by clicking on the speaker icon:
    On our podcast feeds
    During the pilot, posts that get >125 karma will also be released on the "EA Forum (Curated and Popular)" podcast feed:
    EA Forum (Curated & Popular)Audio narrations from the Effective Altruism Forum, including curated posts and posts with 125+ karma.Subscribe:Apple Podcasts | Google Podcasts | Spotify | RSS
    This feed was previously known as "EA Forum (All audio)". We renamed it for reasons.
    During the pilot phase, most "Curated" posts will still be narrated by Perrin Walker of TYPE III AUDIO.
    Posts that get >30 karma will be released on the new "EA Forum (All audio)" feed:
    EA Forum (All audio)Audio narrations from the Effective Altruism Forum, including curated posts, posts with 30+ karma, and other great writing.Subscribe:Apple Podcasts | Spotify | RSS | Google Podcasts (soon)
    How is this different from Nonlinear Library?
    The Nonlinear Library has made unofficial AI narrations of EA Forum posts available for the last year or so.
    The new EA Forum AI narration project can be thought of as "Nonlinear Library 2.0". We hope our AI narrations will be clearer and more engaging. Some specific improvements:
    Audio notes to indicate headings, lists, images, etc.
    Specialist terminology, acronyms and idioms are handled gracefully. Footnotes too.
    We'll skip reading out long URLs, academic citations, and other things that you probably don't want to listen to.
    Episode descriptions include a link to the original post. According to Nonlinear, this is their most common feature request!
    We'd like to thank Kat Woods and the team at Nonlinear Library for their work, and for giving us helpful advice on this project.
    What do you think?
    We'd love to hear your thoughts!
    To give feedback on a particular narration, click the feedback button on the audio player, or go to t3a.is.
    We're keen to hear about even minor issues: we have control over most details of the narration system, and we're keen to polish it. The narration system, which is being developed by TYPE III AUDIO, will be rolled out for thousands of hours of EA-relevant writing over the summer.
    To share feature ideas or more general feedback, comment on this post or write to [email protected].
    The reason for this mildly confusing update is the vast majority of people subscribed to the existing "All audio" feed, but we think that most of them don't actually want to receive ~4 episodes per day. If you're someone who wants to max out the number of narrations in your podcast app, please subscribe to the new "All audio" feed. For everyone else: no action required.
    Are you a writer with a blog, article or newsletter to narrate? Write to [email protected] and we'll make it happen.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - The bullseye framework: My case against AI doom by titotal
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The bullseye framework: My case against AI doom, published by titotal on May 30, 2023 on The Effective Altruism Forum.
    Introduction:
    I’ve written quite a few articles casting doubt on several aspects of the AI doom narrative. (I’ve starting archiving them on my substack for easier sharing). This article is my first attempt to link them together to form a connected argument for why I find imminent AI doom unlikely.
    I don’t expect every one of the ideas presented here to be correct. I have a PHD and work as a computational physicist, so I’m fairly confident about aspects related to that, but I do not wish to be treated as an expert on other subjects such as machine learning where I am familiar with the subject, but not an expert. You should never expect one person to cover a huge range of topics across multiple different domains, without making the occasional mistake. I have done my best with the knowledge I have available.
    I don’t speculate about specific timelines here. I suspect that AGI is decades away at minimum, and I may reassess my beliefs as time goes on and technology changes.
    In part 1, I will point out the parallel frameworks of values and capabilities. I show what happens when we entertain the possibility that at least some AGI could be fallible and beatable.
    In part 2, I outline some of my many arguments that most AGI will be both fallible and beatable, and not capable of world domination.
    In part 3, I outline a few arguments against the ideas that “x-risk” safe AGI is super difficult, taking particular aim at the “absolute fanatical maximiser” assumption of early AI writing.
    In part 4, I speculate on how the above assumptions could lead to a safe navigation of AI development in the future.
    This article does not speculate on AI timelines, or on the reasons why AI doom estimates are so high around here. I have my suspicions on both questions. On the first, I think AGI is many decades away, on the second, I think founder effects are primarily to blame. However these will not be the focus of this article.
    Part 1: The bullseye framework
    When arguing for AI doom, a typical argument will involve the possibility space of AGI. Invoking the orthogonality thesis and instrumental convergence, the argument goes that in the possibility space of AGI, there are far more machines that want to kill us than those that don’t. The argument is that the fraction is so small that AGI will be rogue by default: like the picture below.
    As a sceptic, I do not find this, on its own, to be convincing. My rejoinder would be that AGI’s are not being plucked randomly from possibility space. They are being deliberately constructed and evolved specifically to meet that small target. An AI that has the values of “scream profanities at everyone” is not going to survive long in development. Therefore, even if AI development starts in dangerous territory, it will end up in safe territory, following path A. (I will flesh this argument out more in part 3 of this article).
    To which the doomer will reply: Yes, there will be some pressure towards the target of safety, but it won’t be enough to succeed, because of things like deception, perverse incentives, etc. So it will follow something more like path B above, where our attempts to align it are not successful.
    Often the discussion stops there. However, I would argue that this is missing half the picture. Human extinction/enslavement does not just require that an AI wants to kill/enslave us all, it also requires that the AI is capable of defeating us all. So there’s another, similar, target picture going on:
    The possibility space of AGI’s includes countless AI’s that are incapable of world domination. I can think of 8 billion such AGI’s off the top of my head: Human beings. Even a very smart AGI may still fail to dominate humanity, if it’s locked...
    27 min
  • AF - LIMA: Less Is More for Alignment by Ulisse Mini
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LIMA: Less Is More for Alignment, published by Ulisse Mini on May 30, 2023 on The AI Alignment Forum.
    Abstract
    Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data.
    In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.
    Implications
    Data Quality & Capabilities
    Along with TinyStories and QLoRA I'm becoming increasingly convinced that data quality is all you need, definitely seems to be the case for finetuning, and may be the case for base-model training as well. Better scaling laws through higher-quality corpus?
    Also for who haven't updated, it seems very likely that GPT-4 equivalents will be essentially free to self-host and tune within a year. Plan for this!
    Perplexity != Quality
    When fine-tuning LIMA, we observe that perplexity on held-out Stack Exchange data (2,000 examples) negatively correlates with the model’s ability to produce quality responses. To quantify this manual observation, we evaluate model generations using ChatGPT, following the methodology described in Section 5. Figure 9 shows that as perplexity rises with more training steps – which is typically a negative sign that the model is overfitting – so does the quality of generations increase. Lacking an intrinsic evaluation method, we thus resort to manual checkpoint selection using a small 50-example validation set.
    Because of this, the authors manually select checkpoints between the 5th and 10th epochs (out of 15) using the held-out 50-example development set.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min