The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - 4 ways to think about democratizing AI [GovAI Linkpost] by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 4 ways to think about democratizing AI [GovAI Linkpost], published by Akash on February 13, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • AF - The Cave Allegory Revisited: Understanding GPT's Worldview by Jan Kulveit
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Cave Allegory Revisited: Understanding GPT's Worldview, published by Jan Kulveit on February 14, 2023 on The AI Alignment Forum.
    A short post describing a metaphor I find useful, in particular for explaining some intuitions about systems like GPT to people who don't have deeper technical knowledge about large generative models.
    Plato's allegory of the cave has been a staple of philosophical discourse for millenia, providing a metaphor for understanding the limits of human perception. In the classical allegory, we are prisoners shackled to a wall of a cave, unable to experience reality directly but only able to infer it based on watching shadows cast on the wall.GPT can be thought of as a blind oracle residing in a deeper cave, where it does not even see the shadows but only hears our conversations in the first cave, always trying to predict the next syllable.
    It is remarkable that it still learns a lot about the world outside of the cave. Why does it learn this? Because, a model of reality outside of the cave and a decent amount of abstraction are useful for predicting the conversations in the first cave!
    Moreover, GPT also learns about the speakers in the first cave, as understanding their styles and patterns of speech is crucial for its prediction task. As the speakers are closer to GPT, understanding their styles is in some sense easier and more natural than guessing what's outside of the cave.
    What does the second cave allegory illustrate?
    The first insight from the allegory is: if you are in GPT's place, part of the difficulty in figuring out what's going on outside the cave, is that people in the first cave talk a lot about other things apart from the shadows of the real world. Sometimes, they talk about happenings in Middle Earth. Or about how the shadows would look in some counterfactual world.
    As humans, we are blessed with the luxury of being able to compare such statements to the shadows and determine their veracity. The difference between conversations about fantasy and the shadows of the real world is usually extremely obvious to humans: we never see dragon shadows. In contrast, dragons do show up a lot in the conversations in the first cave; GPT doesn’t get to see the shadows, so it often needs to stay deeply uncertain about whether the speaker is describing the actual shadows or something else to be good at predicting the conversation.
    The second insight is that one of the biggest challenges for GPT in figuring out the conversation is localizing it, determining who is speaking and what the context is, just from the words. Is it a child regaling another child with a fairy-tale, or a CEO delivering a corporate address? As humans we do not face this conundrum often,because we can see the context in which the conversation is taking place. In fact, we would be worse than GPT at the task it has to deal with.
    At first, interacting with this type of blind oracle in the second cave was disorienting for humans. Talking to GPT used to be a bit like shouting something through a narrow tunnel into the second cave .and instead of an echo, getting back what the blind oracle hallucinates is the most likely thing that you or someone else would say next. Often people were confused by this. They shouted instructions and expected an answer, but the oracle doesn't listen to instructions or produce answers directly - it just hallucinates what someone might say next. Because on average in the conversations in the first cave questions are followed by answers, and requests by fulfilment, this sort of works.
    One innovation of ChatGPT, which made it popular with people, was localising the conversation by default: when you are talking with ChatGPT now, it knows that what follows is a conversation between a human - you - and a "helpful AI assistant". There is a subtle point to...
    6 min
  • LW - SolidGoldMagikarp III: Glitch token archaeology by mwatkins
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SolidGoldMagikarp III: Glitch token archaeology, published by mwatkins on February 14, 2023 on LessWrong.
    The set of anomalous tokens which we found in mid-January are now being described as 'glitch tokens' in online discussion, as well as (perhaps more playfully) 'forbidden tokens' and 'unspeakable tokens'. We've mostly just called them 'weird tokens'.
    Research is ongoing, and a more serious research report will appear soon, but for now we thought it might be worth recording what is known about the origins of the various glitch tokens. Not why they glitch, but why these particular strings have ended up in the GPT-2/3/J token set.
    We’re currently working with this somewhat imperfect list of 135. It’s becoming apparent that there are degrees of glitchiness, and it’s hard to know where to draw the line as to which tokens should and shouldn't be included in the collection.
    As noted in our second post, quite a few of the tokens belong to 'nested' families, as we see here:
    Solid[GoldMagikarp]: ' SolidGoldMagikarp', 'GoldMagikarp'
    [The[Nitrome]]Fan: 'Nitrome', ' TheNitrome', ' TheNitromeFan'
    [ RandomRedditor]WithNo: ' RandomRedditor', ' RandomRedditorWithNo'
    external[ActionCode]: 'ActionCode', 'externalActionCode'
    Buyable[Inst[[oreAnd]Online]]: 'oreAnd', 'oreAndOnline', 'InstoreAndOnline', 'BuyableInstoreAndOnline'
    [quickShip]Available: 'quickShip', 'quickShipAvailable'
    so[DeliveryDate]: 'soDeliveryDate', 'DeliveryDate'
    [[ externalTo]EVA]Only: ' externalTo', ' externalToEVA', ' externalToEVAOnly'
    [rawdownload][clone[embed[reportprint]]]: 'rawdownload', 'reportprint', 'embedreportprint', 'cloneembedreportprint', 'rawdownloadcloneembedreportprint'
    TPP[StreamerBot]: 'TPPStreamerBot', 'StreamerBot'
    [ guiActiveUn]focused: ' guiActiveUn', ' guiActiveUnfocused'
    [PsyNet]Message: 'PsyNet', 'PsyNetMessage'
    [cffff]cc: 'cffffcc', 'cffff'
    pet[ertodd]: 'ertodd', ' petertodd'
    So let’s look at these families first and kill multiple tokens with single bullet points:
    Solid[GoldMagikarp]: We originally thought this had been scraped from some online Pokemon content, but that was a red herring (lol). Eventually we found out that this is a handle of one of the six Redditors who were part of a collective effort to 'count to infinity' over at r/counting. You can read the story of that here or here. SolidGoldMagikarp, the diligent counter whose Reddit handle is now immortalised, was clearly referencing Pokemon with that handle choice: a Magikarp is a Pokemon entity. SolidGoldMagikarp gets two glitch tokens.
    [The[Nitrome]]Fan: TheNitromeFan was another of the Reddit counting crew. Presumably a fan of Nitrome, the British video game developer, TheNitromeFan gets three glitch tokens.
    [ RandomRedditor]WithNo: That was a pretty random handle chosen by RandomRedditorWithNo, the third of our famed Reddit counters.
    The other three Redditors whose handles got scraped from the r/counting 'Hall of Counters' chart due to their prolific posting of ever-larger positive integers were Adinida, Smartstocks (also known as ۂڊῥτ�ӺDṽἙ£ on Reddit) and davidjl123, presumably someone called David, whose full Reddit handle got truncated to davidjl by the tokenisation process.
    external[ActionCode]: Google helped solve this one. We would have imagined 'externalActionCode' was a generic database thing, but it seems to be very specific to the HTML behind countless pages recording US Congressional voting. As you can see here,there are over two million web pages indexed as containing this string. It looks like a lot of local and regional US news outlets are using a standard feed from Congress to report voting on legislation. Some programmer somewhere named that property in a fraction of a second with barely a flicker of cognitive effort, unaware that it would one day cause a large language model to go berserk.
    Buyable[Inst[[ore...
    16 min
  • EA - New EA Podcast: Critiques of EA by Nick Anyos
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New EA Podcast: Critiques of EA, published by Nick Anyos on February 13, 2023 on The Effective Altruism Forum.
    Emotional Status: I started working on this project before the FTX collapse and all subsequent controversies and drama. I notice an internal sense that I am "piling on" or "kicking EA while it's down." This isn't my intention, and I understand if a person reading this feels burned out on EA criticisms and would rather focus on object level forum posts right now.
    I have just released the first three episodes of a new interview podcast on criticisms of EA:
    Democratizing Risk and EA with Carla Zoe Cremer and Luke Kemp
    Expected Value and Critical Rationalism with Vaden Masrani and Ben Chugg
    Is EA an Ideology? with James Fodor
    I am in the process of contacting potential guests for future episodes, and would love any suggestions on who I should interview next. Here is an anonymous feedback form that you can use to tell me anything you don't want to write in a comment.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Important fact about how people evaluate sets of arguments by Daniel Kokotajlo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Important fact about how people evaluate sets of arguments, published by Daniel Kokotajlo on February 14, 2023 on LessWrong.
    Ronny Fernandez on twitter:
    A very important fact which just came to my attention is that people do not tend to sum or take the max reasonableness of arguments for P to form a judgement about P, rather they tend to take the average.
    This is a somewhat reasonable heuristic in some situations. For instance, if someone gives you a really unreasonable argument for P this is evidence that their judgement of arguments isn’t very good, and so their best argument is more likely to be secretly bad.
    Similarly, it is evidence that they are motivated to convince you even using faulty arguments, which is generally speaking a bad sign.
    It has important implications. Sometimes people think “oh I will make 50 ok arguments for P instead of one really good one” but most folks are not very impressed by this, even though they should be.
    Relatedly, if you try to turn a complicated thesis T into a social movement, the average reasonableness of an argument in favor of T will plummet, and so you may very quickly find that everyone perceives the anti-T-ers as being much more reasonable.
    This will probably still be true even if the best pro-T arguments are very good, and especially true if the best pro-T arguments are subtle or hard to follow.
    Yes, this is about ai risk. I don’t think this is a slam dunk argument against trying to make ai-risk-pilled-ness into a popular social movement, but it is a real cost, and nearly captures the shape of my real worries.
    The best version of my real worries, I also have real worries that are not nearly as defensible or cool.
    Oh actually probably they take the min rather than the average unless they like you, in which case they take the max.
    First of all, is this important fact actually true? I'd love to know. Reviewing my life experience... it sure seems true? At least true in many circumstances? I think I can think of lots of examples where this fact being true is a good explanation of what happened. If people have counterarguments or sources of skepticism I'd be very interested to hear in the comments.Secondly, I concluded a while back that One Strong Argument Beats Many Weak Arguments, and in Ye Olden Days of Original Less Wrong when rationalists spent more time talking about rationality there was a whole series of posts arguing for the opposite claim (1, 2, 3). Seems possibly related. I'd love to see this debate revived, and tied in to the more general questions of: (A) Does rationality in practice recommend aggregating the quality of a group of arguments for a claim by taking the sum, the max, the min, the mean, or what?
    (To be clear, obviously the ideal is more complicated & looks more like Bayesian conditionalization on a huge set of fleshed-out hypotheses. But in practice, when you don't have time for that, what do you do?)(B) What do people typically do, and on what factors does that depend--e.g. do they take the min if they don't like you or the claim, and take the max if they do?Finally: Steven Adler pointed me to this paper that maybe provides some empirical evidence for Ronny's claim.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Valentine's Day fundraiser for FEM by GraceAdams
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Valentine's Day fundraiser for FEM, published by GraceAdams on February 14, 2023 on The Effective Altruism Forum.
    Epistemic status: feeling the love
    This Valentine's Day, I'm donating the cost of a date night to Family Empowerment Media (FEM) and would love you to join me! 🥰
    What better way to celebrate love than promoting family planning to prevent unintended pregnancies?
    Family Empowerment Media (FEM) is an evidence-driven nonprofit committed to eliminating maternal deaths and other health burdens from unintended pregnancies. FEM produces and air radio-based social and behavioural change campaigns on family planning to empower women and men who want to delay or prevent pregnancy to consistently use contraception.
    Read more about FEM
    Visit FEM's website
    Join me in donating to FEM here:
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Elements of Rationalist Discourse by RobBensinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Elements of Rationalist Discourse, published by RobBensinger on February 14, 2023 on The Effective Altruism Forum.
    I liked Duncan Sabien's Basics of Rationalist Discourse, but it felt somewhat different from what my brain thinks of as "the basics of rationalist discourse". So I decided to write down my own version (which overlaps some with Duncan's).
    Probably this new version also won't match "the basics" as other people perceive them. People may not even agree that these are all good ideas! Partly I'm posting these just out of curiosity about what the delta is between my perspective on rationalist discourse and y'alls perspectives.
    The basics of rationalist discourse, as I understand them:
    1. Truth-Seeking. Try to contribute to a social environment that encourages belief accuracy and good epistemic processes.
    Try not to “win” arguments using symmetric weapons (tools that work similarly well whether you're right or wrong). Indeed, try not to treat arguments like soldiers at all:
    Arguments are soldiers. Once you know which side you’re on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it’s like stabbing your soldiers in the back.
    Instead, treat arguments like scouts: tools for better understanding reality.
    2. Non-Violence: The response to "argument" is "counter-argument". The response to arguments is never bullets. The response to arguments is never doxxing, or death threats, or coercion.
    3. Non-Deception. Never try to steer your conversation partners (or onlookers) toward having falser models.
    Additionally, try to avoid things that will (on average) mislead readers as a side-effect of some other thing you're trying to do. Where possible, avoid saying things that you expect to lower the net belief accuracy of the average person you're communicating with; or failing that, at least flag that you're worried about this happening.
    As a corollary:
    3.1. Meta-Honesty. Make it easy for others to tell how honest, literal, PR-y, etc. you are (in general, or in particular contexts). This can include everything from "prominently publicly discussing the sorts of situations in which you'd lie" to "tweaking your image/persona/tone/etc. to make it likelier that people will have the right priors about your honesty".
    4. Localizability. A common way groups end up stuck with false beliefs is that, e.g., two rival political factions will exist—call them the Blues and the Greens—and the Blues will believe some false generalization based on a bunch of smaller, less-obviously-important arguments or pieces of evidence.
    The key to reaching the truth will be for the Blues to nitpick their data points more: encourage people to point out local errors, ways a data point is unrepresentative, etc. But this never ends up happening, because (a) each individual data point isn't obviously crucial, so it feels like nitpicking; and (b) worse, pushing back in a nitpicky way will make your fellow Blues suspect you of disloyalty, or even of harboring secret sympathies for the evil Greens.
    The result is that pushback on local claims feels socially risky, so it happens a lot less in venues where the Blues are paying attention; and when someone does work up the courage to object or cite contrary evidence, the other Blues are excessively skeptical.
    Moreover, this process tends to exacerbate itself over time: the more the Blues and Greens each do this, the more extreme their views will become, which reinforces the other side's impression "wow our enemies are extreme!". And the more this happens, the more likely it becomes that someone raising concerns or criticisms is secretly disloyal, because in fact you've created a hostile discourse environment where it's hard for people to justify bringing up objections if their goal is merely curiosity.
    By...
    21 min
  • LW - Is InstructGPT Following Instructions in Other Languages Surprising? by DragonGod
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is InstructGPT Following Instructions in Other Languages Surprising?, published by DragonGod on February 13, 2023 on LessWrong.
    On Twitter Jan Leike asks:
    With the InstructGPT paper we found that our models generalized to follow instructions in non-English even though we almost exclusively trained on English.
    We still don't know why.
    I wish someone would figure this out.
    I find myself surprised/confused at his apparent surprise/confusion.
    My default response had someone asked me what I thought was going on would have been something like: "following instructions is a natural abstraction of a task", hence models trained to follow instructions in English generalising to other languages is a natural example of goals/capabilities generalisation.
    It's like being surprised that you taught an AI to drive red cars and it can drive blue cars as well [capabilities generalisation]. Or if you taught an AI to reach a particular location on a level when there's a coin there and it still heads to said location in the absence of the coin. 🤭 [goal generalisation].
    In Coin Run, there were multiple possible natural generalisations for the goal (location on level, the coin), but for instruction following in English, there seems to be only one natural generalisation of the goal? (I don't really see any other intuitively plausible generalisation of "following instructions in English" when the model switches to a Chinese context.)
    Like conditional on InstructGPT competently responding in languages other than English, I think we should just expect it to generalise its goal to the new context.
    Is this just hindsight bias? Would I really have been counterfactually surprised/confused if it didn't follow instructions in other languages? I don't actually know that to be the case.
    Maybe I would have just generated some other reason why we shouldn't have expected instruction following to generalise.
    But hindsight bias aside, this is a prediction of Wentworth's natural abstractions hypothesis as I understand it (I think he would endorse this prediction), and I am a genuine believer in the NAH, so I don't think it's just hindsight bias.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - LLM Basics: Transformer Token Vectors Are Not Points in Space by NickyP
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LLM Basics: Transformer Token Vectors Are Not Points in Space, published by NickyP on February 13, 2023 on LessWrong.
    This post is written as an explanation of a misconception I had with transformer embedding when I was getting started. Thanks to Stephen Fowler for the discussion last August that made me realise the misconception, and others for helping me refine my explanation. Any mistakes are my own. Thanks to feedback by Stephen Fowler and JustisMills on this post.
    TL;DR: While the token vectors are stored as n-dimensional vectors, thinking of them as points in vector space can be quite misleading. It is better to think of them as directions on a hypersphere, with a size component.
    The I think of distance as the Euclidean distance, with formula:
    d(x1,x2)=|x1−x2|=√∑i(x1i−x2i)2
    Thus does not match up with the distance forumla used when calculating logits:
    d(x1,x2)=x1x2=|x1x2|cosθ12
    But it does match up with the cosine similarity forumula:
    d(x1,x2)=^x1^x2=cosθ12
    And so, we can see that the direction and size matter, but not the distance
    Introduction
    In the study of transformers, it is often assumed that different tokens are embedded as points in a multi-dimensional space. While this concept is partially true, the space in which these tokens are embedded is not a traditional Euclidean space. This is because of the way probabilities of tokens are calculated, as well as how the behaviour of the softmax function affects how tokens are positioned in their space.
    This post will have two parts. In the first part, I will briefly explain the relevant parts of the transformer, and in the second part, we will explore what is happening when a transformer moves from an input token to an output token explaining why tokens are better thought of as directions.
    Part 1: The Process of a Transformer
    Here I will briefly describe how the relevant parts of the transformer work.
    First, let's briefly explain the relevant parts at the start of the transformer. We will be studying the "causal" transformer model (ie: that given N tokens, we want to predict the (N+1)th token). The main "pieces" of a causal model are:
    The Tokeniser - turns "words" into "tokens" or "tokens" into "words"
    The Transformer - turn N tokens into a prediction for the (N+1)th token
    Input Embedding - turns "tokens" into "vectors"
    Positional Encoder - adds information about position of each token
    Many Decoder layers - turns input vectors into prediction vectors
    with a Self-Attention sub-layer - uses information from all states
    with a Feed Forward sub-layer - uses information from the current state
    Output Unembedding - turns prediction vectors into token probabilities
    Note that I won't go into much depth about the positional and decoder layers in this post. If there is interest, I may write up another post explaining how they work if there is interest.
    Also note, that for simplicity, I will initially assume that the input embedding and the output unembedding are the same. In this case, if you have a token, embed it, and then unembed it, you should get the same token out. The symmetry was true in the era of GPT-2, but nowadays, embedding and unembedding matrices are learned separately, so I will touch on some differences in the end.
    Lastly, note that "unembedding" is actually a fake word, and usually it is just called the output embedding. I think unembedding makes more sense, so I will call it unembedding.
    1. The Start of the Transformer Process
    In the beginning of the transformer process, there is the Tokeniser (converting “words” into “tokens”) and the Token Embedding (converting “tokens” into “token vectors”/”hidden-state vectors”):
    So to start:
    Input text is received
    The text is split into N different “parts” called token ids
    The N token ids are converted into N vectors using the embedding matrix
    The N vectors are pass...
    23 min
  • LW - Enjoy LessWrong in ebook format by Stuckwork
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Enjoy LessWrong in ebook format, published by Stuckwork on February 13, 2023 on LessWrong.
    I would like to inform the community that all LessWrong Community Sequences are now available in EPUB and pdf format. These formats provide an alternative way to these sequences, allowing for offline reading on e-readers. You can find the library with all ebooks here:
    Personally, I feel that reading entire sequences have several benefits over reading individual posts. It gives me a clearer understanding of the development of ideas and the context in which the posts were written. However, most sequences are too long to read in one session and if I read them on desktop and don’t finish them, I tend to forget that I was reading them. Reading sequences like I read ebooks makes it easier to keep track.
    Furthermore, reading on an e-reader provides a nice distraction-free environment. Ideal for those who want to read LessWrong without having social media one click away. Therefore, I have also made ebooks of the most upvoted posts on LessWrong of every month since January 2012. I’m planning to generate such an ebook every month, such that I can read the best material in a calmer environment. Also, e-readers offer features such as adjustable font sizes and glare-free screens, making the reading experience more comfortable.
    I have tried to make sure that the images and math are all rendered correctly, preserving the original visual content of the blog posts. I have tested a random sample of the ebooks on a Kobo Aura, Remarkable 2, and in Calibre. On these devices, they all seem to render nicely, but your mileage may vary based on your device.
    Additionally, the source code (written in Python) for the scraping and generation of the EPUBs is available for those interested in generating their own EPUBs. Feel free to provide suggestions for any improvements. Finally, if any authors would rather not have their content available in ebook format, please contact me and I will try to promptly remove them from the library!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min