The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - TED talk by Eliezer Yudkowsky: Unleashing the Power of Artificial Intelligence by bayesed
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: TED talk by Eliezer Yudkowsky: Unleashing the Power of Artificial Intelligence, published by bayesed on May 7, 2023 on LessWrong.
    This is a TED talk published early by a random TEDx channel.
    Tweet by EY:
    Looks like my Sudden Unexpected TED Talk got posted early by a TEDx account.
    YouTube description:
    Eliezer Yudkowsky is a foundational thinker on the long-term future of artificial intelligence.
    With more than 20 years of experience in the world of AI, Eliezer Yudkowsky is the founder and senior research fellow of the Machine Intelligence Research Institute, an organization dedicated to ensuring smarter-than-human AI has a positive impact on the world. His writings, both fiction and nonfiction, frequently warn of the dangers of unchecked AI and its philosophical significance in today's world.
    Yudkowsky is the founder of LessWrong, an online forum and community dedicated to improving human reasoning and decision-making, and the coinventor of the "functional decision theory," which states that decisions should be the output of a fixed mathematical function answering the question: "Which output of this very function would yield the best outcome?"
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - An artificially structured argument for expecting AGI ruin by Rob Bensinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An artificially structured argument for expecting AGI ruin, published by Rob Bensinger on May 7, 2023 on The AI Alignment Forum.
    Philosopher David Chalmers asked:
    [I]s there a canonical source for "the argument for AGI ruin" somewhere, preferably laid out as an explicit argument with premises and a conclusion?
    Unsurprisingly, the actual reason people expect AGI ruin isn't a crisp deductive argument; it's a probabilistic update based on many lines of evidence. The specific observations and heuristics that carried the most weight for someone will vary for each individual, and can be hard to accurately draw out.
    That said, Eliezer Yudkowsky's So Far: Unfriendly AI Edition might be a good place to start if we want a pseudo-deductive argument just for the sake of organizing discussion. People can then say which premises they want to drill down on.
    In The Basic Reasons I Expect AGI Ruin, I wrote:
    When I say "general intelligence", I'm usually thinking about "whatever it is that lets human brains do astrophysics, category theory, etc. even though our brains evolved under literally zero selection pressure to solve astrophysics or category theory problems".
    It's possible that we should already be thinking of GPT-4 as "AGI" on some definitions, so to be clear about the threshold of generality I have in mind, I'll specifically talk about "STEM-level AGI", though I expect such systems to be good at non-STEM tasks too.
    STEM-level AGI is AGI that has "the basic mental machinery required to do par-human reasoning about all the hard sciences", though a specific STEM-level AGI could (e.g.) lack physics ability for the same reasons many smart humans can't solve physics problems, such as "lack of familiarity with the field".
    A simple way of stating the argument in terms of STEM-level AGI is:
    Substantial Difficulty of Averting Instrumental Pressures: As a strong default, absent alignment breakthroughs, STEM-level AGIs that understand their situation and don't value human survival as an end will want to kill all humans if they can.
    Substantial Difficulty of Value Loading: As a strong default, absent alignment breakthroughs, STEM-level AGI systems won't value human survival as an end.
    High Early Capabilities. As a strong default, absent alignment breakthroughs or global coordination breakthroughs, early STEM-level AGIs will be scaled to capability levels that allow them to understand their situation, and allow them to kill all humans if they want.
    Conditional Ruin. If it's very likely that there will be no alignment breakthroughs or global coordination breakthroughs before we invent STEM-level AGI, then given 1+2+3, it's very likely that early STEM-level AGI will kill all humans.
    Inadequacy. It's very likely that there will be no alignment breakthroughs or global coordination breakthroughs before we invent STEM-level AGI.
    Therefore it's very likely that early STEM-level AGI will kill all humans. (From 1–5)
    I'll say that the "invention of STEM-level AGI" is the first moment when an AI developer (correctly) recognizes that it can build a working STEM-level AGI system within a year. I usually operationalize "early STEM-level AGI" as "STEM-level AGI that is built within five years of the invention of STEM-level AGI".
    I think humanity is very likely to destroy itself within five years of the invention of STEM-level AGI. And plausibly far sooner — e.g., within three months or a year of the technology's invention. A lot of the technical and political difficulty of the situation stems from this high level of time pressure: if we had decades to work with STEM-level AGI before catastrophe, rather than months or years, we would have far more time to act, learn, try and fail at various approaches, build political will, craft and implement policy, etc.
    This argument focuses on "human survival", but fr...
    51 min
  • LW - Statistical models & the irrelevance of rare exceptions by patrissimo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Statistical models & the irrelevance of rare exceptions, published by patrissimo on May 7, 2023 on LessWrong.
    “I don’t care about this instance - I don’t care about any instances! Life is too short to care about anything but the general case!”
    Yes, the general case is drawn from instances, I’m saying that we shouldn't get caught up in the details unless they really matter. And if there is a clear statistical generalization, the details matter very little.
    A common model might be: “Well, the pattern is that it’s some mix of A & B, with more A as factors X & Y are higher, and more B if they are lower”.
    Replying "Z fits better than Y in the mix of variables to map onto the A & B mix" is a correction within the general case frame. Whereas, pointing out "Here are cases F & G that are high A with low X & Y - contradicting your model" - when such cases are very rare - is irrelevant. I find it's usually obvious from the description of F&G that they're extremely rare.
    Rare exceptions are irrelevant because almost all models of the real world (not physics) are statistical claims about what’s usually true, not absolute claims about what’s always true. So a rare data point that doesn’t fit is actually not a contradiction! Pointing such cases out is just reiterating the tautological fact that statistical models are not absolute, which seems like a total waste of time to me. (Especially if the speaker agrees with the model!)
    I’ve noticed this happens more often with careful, intellectually humble thinkers, who often include caveats of the form “But here’s an exception to this strong model I’ve just presented.” I think often they're trying to proactively defend against others pointing out this case. But to me, this is wrongfully falling into the frame that rare exceptions are relevant criticisms or corrections of a statistical model.
    So, rather than defending by acknowledging the rare case, I think it’s far better to break the false frame that rare exceptions are counter-examples and pointing them out is a relevant thing that needs to be addressed. Move into the new, correct frame that this is a statistical model - say by asking them if they actually disagree that the suggested pattern fits in the vast majority of cases and thus is statistically true.
    I’ve also lately noticed that while many people are abstract systematizers, I find far fewer who are relentlessly meta-seeking, constantly seeking to expand to more general theories over broader domains. Eg I’ve noticed myself saying regularly: “Hey, your model for domain S is actually a model for domain Q, where S is a subdomain of Q. Notice that everything we said about S applies equally to Q - we didn’t actually use any special features of S in our reasoning!”
    I also recently noticed a weird but cool thing - my brain automatically generates statistical models without details ever coming into my mind. Like it has a "model this data set" function, where it retrieves and analyzes the data set without me having to consciously consider cases. Ofc I use cases (common ones!) to check the model afterwards. One thing I love about the LW community is that these cognitive modes I describe here are much more common than in most other circles.
    Finally, I do find rare exceptions relevant when there is a pattern to the exceptions. So, rather than just pointing out rare exceptions F&G, the responder then generalizes them into subclass H. Now we can make the general model more accurate by adding that it doesn't apply to H. This "move" still falls within the true statistical frame.
    As an example for this topic, note that in extreme distributions like power law, a “rare exception” that happens at 1% frequency could have 100x intensity compared to the other 99%, and so need to be weighted equally in a model. The generalization of this exception is tha...
    4 min
  • EA - Implications of the Whitehouse meeting with AI CEOs for AI superintelligence risk - a first-step towards evals? by Jamie Bernardi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Implications of the Whitehouse meeting with AI CEOs for AI superintelligence risk - a first-step towards evals?, published by Jamie Bernardi on May 7, 2023 on The Effective Altruism Forum.
    Introducion
    On Wednesday 4th May, Sam Altman (Open AI) and Dario Amodei (Anthropic) - amongst others - met with US Vice President Kamala Harris (with a drop-in from President Joe Biden), to discuss the dangers of AI.
    Announcement | Fact sheet | EA Forum linkpost
    I spent about 2 hours trying to understand what happened, who was involved, and what its possible implications for superintelligence risk.
    I decided to make this post for two reasons:
    I am practising writing and developing my opinions on AI strategy (so feedback is very welcome, and you should treat my epistemic status as ‘new to this’!)
    I think demystifying the facts of the announcement and offering some tentative conclusions will positively contribute to the community's understanding of AI-related political developments.
    My main conclusions
    Three announcements were made, but the announcement on public model evaluations involving major AI labs seemed most relevant and actionable to me.
    My two actionable conclusions are:
    I think folks with technical alignment expertise should consider attending DEF CON 31 if it’s convenient, to help shape the conclusions from the event.
    My main speculative concern is that this evaluation event could positively associate advanced AI and the open source community. As far as those that feel the downside of model proliferation outweighs the benefits of open sourcing, spreading this message in a more focused way now may be valuable.
    Summary of the model evaluations announcement
    This is mostly factual, and I’ve flagged where I’m offering my interpretation. Primary source: AI village announcement.
    There’s going to be an evaluation platform made available during a conference called DEF CON 31. DEF CON 31 is the 31st iteration of DEF CON, “the world’s largest security conference”, taking place in Los Angeles on 10th August 2023. The platform is being organised by a subcommunity at that conference called the AI village.
    The evaluation platform will be provided by Scale AI. The platform will provide “timed access to LLMs” via laptops available at the conference, and attendees will red-team various models by injecting prompts. I expect that the humans will then rate the output of the model as good or bad, much like on the ChatGPT platform. There’s a points-based system to encourage participation, and the winner will win a “high-end Nvidia GPU”.
    The intent of this whole event appears to be to collect adversarial data that the AI organisations in question can use and 'learn from' (and presumably do more RLHF on). The orgs that signed up include: Anthropic, Google, Hugging Face, Microsoft, NVIDIA, OpenAI, and Stability AI.
    It seems that there won’t be any direct implications for the AI organisations. They will, by default, be allowed to carry on as normal no matter what is learned at the event.
    I’ll provide more details on what has happened after the takeaways section.
    Takeaways from the Whitehouse announcement on model evaluations
    I prioritised communicating my takeaways in this section. If you want more factual context to understand exactly what happened and who's involved- see the section below this one.
    For the avoidance of doubt, the Whitehouse announcement on the model evaluation event doesn’t come with any regulatory teeth.
    I don’t mean that as a criticism necessarily; I’m not sure anyone has a concrete proposal for what the evaluation criteria should even be, or how they should be enforced, etc, so it’d be too soon to see an announcement like that.
    That does mean I’m left with the slightly odd conclusion that all that’s happened is the Whitehouse has endorsed a community red-teaming event at a con...
    12 min
  • LW - Properties of Good Textbooks by niplav
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Properties of Good Textbooks, published by niplav on May 7, 2023 on LessWrong.
    Heuristics for choosing/writing good textbooks (see also
    here):
    Has exercises
    Exercises are interspersed in the text, not in large chunks (better at the end of sections, not just at the end of chapters).
    Solutions are available but difficult to access (in a separate book, or on the web), this reduces the urge to look the solution up if one is stuck.
    Of varying difficulty (I like the approach Concrete Mathematics takes: everything from trivial applications to research questions).
    I like it when difficulty is indicated, but it's also okay when it's said clearly in the beginning that exercises are not marked for difficulty (making them mystery boxes).
    Takes many angles
    Has figures and illustrations. I don't think I've encountered a textbook with too many yet. (See Visual Complex Analysis for an example of doing this well.)
    Has many examples. I'm not sure yet about the advantage of recurring examples. Same point about amount as with figures.
    Includes code, if possible. It's cool if you tell me the equations for computing the likelihood ratio of a hypothesis & dataset, but it's even cooler if you give me some sample code I can use and extend along with it.
    Uses typography
    You can use boldface and italics and underlining for reading comprehension, example here.
    Use section headings and paragraphs liberally.
    Artificial Intelligence: A Modern Approach has one- to three-word side-notes describing the content of each paragraph. This is very good.
    Distinguish definitions, proofs, examples, case-studies, code, formulas &c.
    Dependencies
    Define terms before they are used. (This is not a joke. Population Genetics uses the term "substitution" on p. 32 without defining it, and exercise 12-1 from Naive Set Theory depends on the axiom of regularity, but the book doesn't define it.)
    If the book has pre-requisites beyond what a high-schooler knows, a good textbook lists those pre-requisites and textbooks that teach them.
    Indicators
    Multiple editions are an indicator for quality.
    Ditto for multiple authors.
    A conversational and whimsy style can be nice, but shouldn't be overdone.
    Hot take: I get very little value from proofs in math textbooks, and consider them usually unnecessary (unless they teach a new proof method). I like the Infinite Napkin for its approach.
    Wishlist
    Flashcard sets that come together with textbooks. Please.
    3blue1brown style videos that accompany the book. From Zero to Geo is a great step in that direction.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - On Child Wasting, Mega-Charities, and Measurability Bias by Jesper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On Child Wasting, Mega-Charities, and Measurability Bias, published by Jesper on May 7, 2023 on The Effective Altruism Forum.
    Recently I ran into a volunteer for UNICEF who was gathering donations for helping malnourished children. He gave me some explanation on why child wasting is a serious problem and how there are cheap ways to help children who are suffering from it (the UNICEF website has some information on child wasting and specifically on the treatment of wasting using simplified approaches, in case you are interested).
    Since I happen to have taken the Giving What We Can pledge and have read quite a bit on comparing charities, I asked what evidence there is that compares this action to - say - protecting people from malaria with bednets or directly giving cash to very poor people. The response I got was quite specific: the volunteer claimed that UNICEF can save a life with just 1€ a day for an average period of 7 months. If these claims are true then that means they can save a life for 210€, a lot less than the >3000$ that Givewell estimates is needed for AMF to save one life. Probably these numbers should not be compared directly, but I am still curious to know why there can be over an order of magnitude difference between the two. So to practice my critical thinking on these kinds of questions, I made a list of possible explanations for the difference:
    The UNICEF campaign has little room for additional funding.
    The program would be funded anyway from other sources (e.g. governments).
    The 1€/day figure might not include all the costs.
    Some of the children who receive the food supplements might die of malnutrition anyway.
    Only some of the children who receive the food supplements would have died without them.
    Children who are saved from malnutrition could still die of other causes.
    Obviously I do not have the time nor resources of GiveWell so it is hard to determine how much all of these explanations count in the overall picture, or if there are others that I missed. Unfortunately, there does not seem to be much information on this question from GiveWell (or other EA organizations) either. Looking on the GiveWell website, the most I could find is this blog post on mega-charities from 2011, which makes the argument that mega-charities like UNICEF have too many different campaigns running simultaneously, and that they do not have the required transparency for a proper evaluation. The first argument sounds fake to me: if there are different campaigns, then can you not just evaluate these individual campaigns, or at least the most promising ones? The second point about transparency is a real problem, but there is also the risk of measurability bias if we never even consider less transparent charities.
    I would very much like to have a more convincing argument for why these kind of charities are not rated. If for nothing else then at least it would be useful for discussing with people who currently donate to them, or who try to convince me to donate to them. Perhaps the reason is just a lack of resources at GiveWell, or perhaps there is research on this but I just couldn't find it. But either way I believe the current state of affairs does not provide a convincing case of why the biggest EA evaluator barely even mentions one of the largest and most respected charity organizations.
    [Comment: I'm not new here but I'm mostly a lurker on this forum. I'm open to criticism on my writing style and epistemics as long as you're kind!]
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - The Grabby Values Selection Thesis: What values do space-faring civilizations plausibly have? by Jim Buhler
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Grabby Values Selection Thesis: What values do space-faring civilizations plausibly have?, published by Jim Buhler on May 6, 2023 on The Effective Altruism Forum.
    Summary: The Grabby Values Selection Thesis (or GST, for short) is the thesis that some values are more expansion-conducive (and therefore more adapted to space colonization races) than others such that we should – all else equal – expect such values to be more represented among the grabbiest civilizations/AGIs. In this post, I present and argue for GST, and raise some considerations regarding how strong and decisive we should expect this selection effect to be. The stronger it is, the more we should expect our successors – in worlds where the future of humanity is big – to have values more grabbing-prone than ours. The same holds for grabby aliens relative to us present humans. While these claims are trivially true, they seem to support conclusions that most longtermists have not paid attention to, such as “the most powerful civilizations don’t care about what the moral truth might be” (see my previous post), and “they don’t care (much) about suffering” (see my forthcoming next post).
    The thesis
    Spreading to new territories can be motivated by very different values and seems to be a convergent instrumental goal. Whatever a given agent wants, they likely have some incentive to accumulate resources and spread to new territories in order to better achieve their goal(s).
    However, not all moral preferences are equally conducive to expansion. Some of them value (intrinsically or instrumentally) colonization more than others. For instance, agents who value spreading intrinsically will likely colonize more and/or more efficiently than those who disvalue being the direct cause of something like “space pollution”, in the interstellar context.
    Therefore, there is a selection effect where the most powerful civilizations/AGIs are those who have the values that are the most prone to “grabbing”. This is the Grabby Values Selection Thesis (GST), which is the formalization and generalization of an idea that has been expressed by Robin Hanson (1998).
    We can differentiate between two sub-selection effects, here:
    The intra-civ (grabby values) selection: Within a civilization, those who colonize space and influence the future of the civilization are those with the most grabby-prone values. Here is a specific plausible instance of that selection effect, given by Robin Hanson (1998): “Far enough away from the origin of an expanding wave of interstellar colonization, and in the absence of property rights in virgin oases, a selection effect should make leading edge colonists primarily value whatever it takes to stay at the leading edge.”
    The inter-civ (grabby values) selection: The civilizations that end up with the most grabby-prone values will get more territory than the others.
    Do these two different sub-selection effects matter equally? My current impression is that this mainly depends on the likelihood of an early value lock-in – or of design escaping selection early and longlastingly, in Robin Hanson’s (2022) terminology – where “early” means “before grabby values get the time to be selected for within the civilization”. If such an early value lock-in occurs, the inter-civ selection effect is the only one left. If it doesn’t occur, however, the importance of the intra-civ selection effect seems vastly superior to that of the inter-civ one. This is mainly explained by the fact that there is very likely much more room for selection effects within a (not-locked-in) civilization than in between different civilizations.
    GST seems trivially true. It is pretty obvious that all values are not equal in terms of how much they value (intrinsically or instrumentally) space colonization, and that those who value space expansion more ...
    8 min
  • LW - How to have Polygenically Screened Children by GeneSmith
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to have Polygenically Screened Children, published by GeneSmith on May 7, 2023 on LessWrong.
    Polygenic screening is a method for modifying the traits of future children via embryo selection. If that sounds like gobbledygook, then think of it a bit like choosing stats for your baby.
    That may sound amazing. It may sound like science fiction. It may even sound horribly dystopian. But whatever your feelings, it is in fact possible. And these benefits are available for a price that, while expensive, is within reach for most middle-class families.
    On a more serious note, there is limited selection power available with today's technologies, so you will not be able to have a baby Einstein unless you are already a Nobel laureate. But polygenic screening will allow you to decrease your child's risk of common diseases by 10-60%, reduce their risk of mental disorders, and increase their IQ by somewhere between 3 and 8 points. If you are willing to wait a few years, you may be able to increase IQ by up to 13 points. These benefits are available for between $20k-100k depending on how strong of a benefit you want and what kinds of traits you want to select for.
    There has been quite a bit of discussion of this topic on LessWrong and adjacent communities but very little concrete advice for would-be parents who are curious whether the benefits are worth the price, particularly for those who have no other reason to do IVF. The purpose of this post is to fill that gap by addressing costs, potential medical complications, choice of clinic, which labs are best, and how age and infertility diagnosis affect the expected benefits.
    This is a long post and I expect most people will not want to read the whole thing. If this is you, please use the section selector in the sidebar to navigate to the section you are most interested in. You may want to simply skip to the section titled "Concrete Advice for Would-Be Parents".
    Background on IVF
    Wait, what even is polygenic embryo selection?
    Embryo selection is all about picking an embryo to (hopefully) turn into a baby. This occurs during the process of In-Vitro Fertilization, or IVF. In the typical IVF cycle, a couple goes into a fertility clinic because they want to have a baby. Usually this is because they’ve been having trouble conceiving naturally, but couples also seek out IVF when they want to do genetic testing, select the sex of their child, or to preserve fertility for later pregnancy.
    The doctor conducts a bunch of medical tests, and if they all check out, the woman begins a hormone regimen that will stimulate an abnormally large number of her eggs to mature all at once.
    At the end of the regiment, the doctor extracts a bunch of mature eggs from the woman's ovaries, which are then fertilized using the father's sperm and grown in a lab dish for 4-7 days. When the embryo has finished growing, there are often four or more that can be implanted in the mother. Most couples do not want four children, so a choice must be made about which embryo to pick.
    In ye olden days, doctors would often just transfer all the embryos at once in the hope that at least one of them would result in a baby. Sometimes this would work well; one of the embryos would happen to stick and the parents would be very happy. Other times it would work a little too well and more than one of the embryos would implant. This is why twin births are so much more common during IVF than during normal pregnancy.
    Transferring multiple embryos at a time is less common nowadays because the outcomes for twin births are on average worse than for single pregnancies. Twins are more likely to be born preterm, develop health problems, and put excess stress on the mother's body.
    This brings me back to my original point; the doctor or embryologist has to make a choice about which embryo to tr...
    44 min
  • AF - An anthropomorphic AI dilemma by Tsvi Benson-Tilsen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An anthropomorphic AI dilemma, published by Tsvi Benson-Tilsen on May 7, 2023 on The AI Alignment Forum.
    [Metadata: crossposted from. First completed January 21, 2023.]
    Either generally-human-level AI will work internally like humans work internally, or not. If generally-human-level AI works like humans, then takeoff can be very fast, because in silico minds that work like humans are very scalable. If generally-human-level AI does not work like humans, then intent alignment is hard because we can't use our familiarity with human minds to understand the implications of what the AI is thinking or to understand what the AI is trying to do.
    Terms
    Here generally-human-level AI means artificial systems that can perform almost all tasks at or above the level of a human competent at that task (including tasks that take a long time).
    Here intent alignment means making artificial systems try to do what we, on reflection and all things considered, would want them to do. A system that doesn't have intents in this sense--i.e. isn't well-described as trying to do something--cannot be intent aligned.
    Here "how a mind works" points at the generators of the mind's ability to affect the world, and the determiners of [in what direction to push the world] a mind applies its capabilities.
    Let A be some generally-human-level AI system (the first one, say).
    (Here "dilemma" just means "two things, one of which must be true", not necessarily a problem.)
    The anthropomorphy dichotomy
    Either A will work internally like humans work internally, or not.
    Well, of course it's a spectrum, not a binary dichotomy, and a high-dimensional space, not a spectrum. So a bit more precisely: either A will work internally very much like how humans work internally, or very much not like how humans work internally, or some more ambiguous mix.
    To firmly avoid triviality: to qualify as "very much like" how humans work, in the sense used here, it is definitely neither necessary nor sufficient to enjoy walking barefoot on the beach, to get angry, to grow old, to have confirmation bias, to be slow at arithmetic, to be hairy, to have some tens of thousands of named concepts, to be distractible, and so on. What's necessary and sufficient is to share most of the dark matter generators of human-level capabilities. (To say that, is to presume that most of the generators of human-level capabilities aren't presently understood explicitly; really the condition is just about sharing the generators.)
    Anthropomorphic AI is scalable
    Suppose that A does share with humans the dark matter generators of human-level capabilities. Then A is very scalable, i.e. can be tweaked without too much difficulty so that it becomes much more capable than A. The basic case is here, in the section "The AI Advantage". (That link could be taken as arguing that AI will be non-anthropomorphic in some sense, but from the perspective of this essay, the arguments given there are of the form: here's how, starting from the human (anthropomorphic) baseline, it's visibly possible to make tweaks that greatly increase capabilities.) To add two (overlapping) points:
    Simple network effects might be very powerful. If ideas combine at some low rate with other ideas and with tasks, then scaling up the number of idea-slots (analogous to cortex) would give superlinear returns, just using the same sort of thinking that humans use. Imagine if every thought in the world was made available to you, so that as you're having your thoughts, the most especially relevant knowledge held by anyone brings itself to your attention, and is already your own in the same way as your knowledge is already your own. Imagine if the most productive philosophical, mathematical, and scientific quirks of thought, once called forth by some private struggles, were instantly copyable to ten thous...
    12 min
  • LW - Residual stream norms grow exponentially over the forward pass by StefanHex
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Residual stream norms grow exponentially over the forward pass, published by StefanHex on May 7, 2023 on LessWrong.
    Summary: For a range of language models and a range of input prompts, the norm of each residual stream grows exponentially over the forward pass, with average per-layer growth rate of about 1.045 in GPT2-XL. We show a bunch of evidence for this. We discuss to what extent different weights and parts of the network are responsible.
    We find that some model weights increase exponentially as a function of layer number. We finally note our current favored explanation: Due to LayerNorm, it's hard to cancel out existing residual stream features, but easy to overshadow existing features by just making new features 4.5% larger.
    Thanks to Aryan Bhatt, Marius Hobbhahn, Neel Nanda, and Nicky Pochinkov for discussion.
    Plots showing exponential norm and variance growth
    Our results are reproducible in this Colab.
    Alex noticed exponential growth in the contents of GPT-2-XL's residual streams. He ran dozens of prompts through the model, plotted for each layer the distribution of residual stream norms in a histogram, and found exponential growth in the L2 norm of the residual streams:
    Here's the norm of each residual stream for a specific prompt:
    Stefan had previously noticed this phenomenon in GPT2-small, back in MATS 3.0:
    Basic Facts about Language Model Internals also finds a growth in the norms of the attention-out matrices WO and the norms of MLP out matrices Wout ("writing weights"), while they find stable norms for WQ, WK, and Win ("reading weights"):
    Comparison of various transformer models
    We started our investigation by computing these residual stream norms for a variety of models, recovering Stefan's results (rescaled by √dmodel=√768) and Alex's earlier numbers. We see a number of straight lines in these logarithmic plots, which shows phases of exponential growth.
    We are surprised by the decrease in Residual Stream norm in some of the EleutherAI models. We would have expected that, because the transformer blocks can only access the normalized activations, it's hard for the model to "cancel out" a direction in the residual stream. Therefore, the norm always grows. However, this isn't what we see above. One explanation is that the model is able to memorize or predict the LayerNorm scale. If the model does this well enough it can (partially) delete activations and reduce the norm by writing vectors that cancel out previous activations.
    The very small models (distillgpt2, gpt2-small) have superexponential norm growth, but most models show exponential growth throughout extended periods. For example, from layer 5 to 41 in GPT2-XL, we see an exponential increase in residual stream norm at a rate of ~1.045 per layer. We showed this trend as an orange line in the above plot, and below we demonstrate the growth for a specific example:
    BOS and padding tokens
    In our initial tests, we noticed some residual streams showed a irregular and surprising growth curve:
    As for the reason behind this shape, we expect that the residual stream (norm) is very predictable at BOS and padding positions. This is because these positions cannot attend to other positions and thus always have the same values (up to positional embedding). Thus it would be no problem for the model to cancel out activations, and our arguments about this being hard do not hold for BOS and padding positions. We don't know whether there is a particular meaning behind this shape.
    We suspect that is the source of the U-shape shown in Basic facts about language models during training:
    Theories for the source of the growth
    From now on we focus on the GPT2-XL case. Here is the residual stream growth curve again (orange dots), but also including the resid_mid hook between the two Attention and MLP sub-layers (blue dots).
    O...
    21 min