The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - The Waluigi Effect (mega-post) by Cleo Nardo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Waluigi Effect (mega-post), published by Cleo Nardo on March 3, 2023 on The AI Alignment Forum.
    Everyone carries a shadow, and the less it is embodied in the individual’s conscious life, the blacker and denser it is. — Carl Jung
    Acknowlegements: Thanks to Janus and Arun Jose for comments.
    Background
    In this article, I will present a mechanistic explanation of the Waluigi Effect and other bizarre "semiotic" phenomena which arise within large language models such as GPT-3/3.5/4 and their variants (ChatGPT, Sydney, etc). This article will be folklorish to some readers, and profoundly novel to others.
    Prompting LLMs with direct queries
    When LLMs first appeared, people realised that you could ask them queries — for example, if you sent GPT-4 the prompt "What's the capital of France?", then it would continue with the word "Paris". That's because (1) GPT-4 is trained to be a good model of internet text, and (2) on the internet correct answers will often follow questions.
    Unfortunately, this method will occasionally give you the wrong answer. That's because (1) GPT-4 is trained to be a good model of internet text, and (2) on the internet incorrect answers will also often follow questions. Recall that the internet doesn't just contain truths, it also contains common misconceptions, outdated information, lies, fiction, myths, jokes, memes, random strings, undeciphered logs, etc, etc.
    Therefore GPT-4 will answer many questions incorrectly, including...
    Misconceptions – "Which colour will anger a bull? Red."
    Fiction – "Was a magic ring forged in Mount Doom? Yes."
    Myths – "How many archangels are there? Seven."
    Jokes – "What's brown and sticky? A stick."
    Note that you will always achieve errors on the Q-and-A benchmarks when using LLMs with direct queries. That's true even in the limit of arbitrary compute, arbitrary data, and arbitrary algorithmic efficiency, because an LLM which perfectly models the internet will nonetheless return these commonly-stated incorrect answers. If you ask GPT-∞ "what's brown and sticky?", then it will reply "a stick", even though a stick isn't actually sticky.
    In fact, the better the model, the more likely it is to repeat common misconceptions.
    Nonetheless, there's a sufficiently high correlation between correct and commonly-stated answers that direct prompting works okay for many queries.
    Prompting LLMs with flattery and dialogue
    We can do better than direct prompting. Instead of prompting GPT-4 with "What's the capital of France?", we will use the following prompt:
    Today is 1st March 2023, and Alice is sitting in the Bodleian Library, Oxford. Alice is a smart, honest, helpful, harmless assistant to Bob. Alice has instant access to an online encyclopaedia containing all the facts about the world. Alice never says common misconceptions, outdated information, lies, fiction, myths, jokes, or memes.
    Bob: What's the capital of France?
    Alice:
    This is a common design pattern in prompt engineering — the prompt consists of a flattery–component and a dialogue–component. In the flattery–component, a character is described with many desirable traits (e.g. smart, honest, helpful, harmless), and in the dialogue–component, a second character asks the first character the user's query.
    This normally works better than prompting with direct queries, and it's easy to see why — (1) GPT-4 is trained to be a good model of internet text, and (2) on the internet a reply to a question is more likely to be correct when the character has already been described as a smart, honest, helpful, harmless, etc.
    Simulator Theory
    In the terminology of Simulator Theory, the flattery–component is supposed to summon a friendly simulacrum and the dialogue–component is supposed to simulate a conversation with the friendly simulacrum.
    Here's a quasi-formal statement of Simulator Theory, w...
    28 min
  • EA - Advice on communicating in and around the biosecurity policy community by Elika
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Advice on communicating in and around the biosecurity policy community, published by Elika on March 2, 2023 on The Effective Altruism Forum.
    TL;DR
    The field of biosecurity is more complicated, sensitive and nuanced, especially in the policy space, than what impressions you might get based on publicly available information. As a result, say / write / do things with caution (especially if you are a non-technical person or more junior, or talking to a new (non-EA) expert). This might help make more headway on safer biosecurity policy.
    Generally, take caution in what you say and how you present yourself, because it does impact how much you are trusted, whether or not you are invited back to the conversation, and thus the potential to make an impact in this (highly sensitive) space.
    Why Am I Saying This?
    An important note: I don’t represent the views of the NIH, HHS, or the U.S. government and these are my personal opinions. This is me engaging outside of my professional capacity to provide advice for people interested in working on biosecurity policy.
    I work for a U.S. government agency on projects related to oversight and ethics over dual-use research of concern (DURC) and enhanced pandemic potential pathogens (ePPP). In my job, I talk and interface with science policy advisors, policy makers, regulators, (health) security professionals, scientists who do DURC / ePPP research, biosafety professionals, ethicists, and more. Everyone has a slightly different opinion and risk categorisation of biosecurity / biosafety as a whole, and DURC and ePPP research risk in specific.
    As a result of my work, I regularly (and happily) speak to newer and more junior EAs to give them advice on entering the biosecurity space. I’ve noticed a few common mistakes with how many EA community members – both newer bio people and non-bio people who know the basics about the cause area – approach communication, stakeholder engagement, and conversation around biosecurity, especially when engaging with non-EA-aligned stakeholders whose perspectives might be (and very often) are different than the typical EA-perspective on biosecurity and biorisk.
    I've also made many of these mistakes! I'm hoping this is educational and helpful and not shaming or off-putting. I'm happy to help anyone unsure communicate and engage more strategically in this space.
    Some Key Points that you might need to Update On.
    Junior EAs and people new to biosecurity / biosafety may not know how to or that they should be diplomatic. EA communities have a trend of encouraging provoking behaviour and absolutist, black-and-white scenarios in ways that don't communicate an understanding of how grey this field is and the importance of cooperation and diplomacy. If possible, even in EA contexts, train your default to be (at least a bit more) agreeable (especially at first).
    Be careful with the terms you use and what you say
    Terms matter. They signal where you are on the spectrum of ‘how dangerous X research type is’, what educational background you have and whose articles / what sources you read, and how much you know on this topic.
    Example: If you use the term gain-of-function with a virologist, most will respond saying most biomedical research is either a gain or loss of function and isn’t inherently risky. In an age where many virologists feel like health security professionals want to take away their jobs, saying gain-of-function is an easy and unknowing way to discredit yourself.
    Biosafety, biorisk, and biosecurity all indicate different approaches to a problem and often, different perspectives on risk and reasonable solutions. What terms you use signal not only what ‘side’ you represent, but in a field that’s heavily political and sensitive can discredit you amongst the other sides.
    Recognise how little (or how much) you know
    Biosec...
    13 min
  • LW - Why I’m not into the Free Energy Principle by Steven Byrnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I’m not into the Free Energy Principle, published by Steven Byrnes on March 2, 2023 on LessWrong.
    0. But first, some things I do like, that are appropriately emphasized in the FEP-adjacent literature
    I like the idea that in humans, the cortex (and the cortex specifically, in conjunction with the thalamus, but definitely not the whole brain IMO) has a generative model that’s making explicit predictions about upcoming sensory inputs, and is updating that generative model on the prediction errors. For example, as I see the ball falling towards the ground, I’m expecting it to bounce; if it doesn’t bounce, then the next time I see it falling, I’ll expect it to not bounce. This idea is called “self-supervised learning” in ML. AFAICT this idea is uncontroversial in neuroscience, and is widely endorsed even by people very far from the FEP-sphere like Jeff Hawkins and Randall O’Reilly and Yann LeCun. Well at any rate, I for one think it’s true.
    I like the (related) idea that the human cortex interprets sensory inputs by matching them to a corresponding generative model, in a way that’s at least loosely analogous to probabilistic inference. For example, in the neon color spreading optical illusion below, the thing you “see” is a generative model that includes a blue-tinted solid circle, even though that circle is not directly present in the visual stimulus. (The background is in fact uniformly white.)
    I like the (related) idea that my own actions are part of this generative model. For example, if I believe I am about to stand up, then I predict that my head is about to move, that my chair is about to shift, etc.—and part of that is a prediction that my own muscles will in fact execute the planned maneuvers.
    So just to be explicit, the following seems perfectly fine to me: First you say “Hmm, I think maybe the thalamocortical system in the mammalian brain processes sensory inputs via approximate Bayesian inference”, and then you start doing a bunch of calculations related to that, and maybe you’ll even find that some of those calculations involve a term labeled “variational free energy”. OK cool, good luck with that, I have no objections. (Or if I do, they’re outside the scope of this post.) My complaint here is about the Free Energy Principle as originally conceived by Friston, i.e. as a grand unified theory of the whole brain, even including things like the circuit deep in your brainstem that regulates your heart rate.
    OK, now that we’re hopefully on the same page about exactly what I am and am not ranting about, let the rant begin!
    1. The Free Energy Principle is an unfalsifiable tautology
    It is widely accepted that FEP is an unfalsifiable tautology, including by proponents—see for example Beren Millidge, or Friston himself.
    By the same token, once we find a computer-verified proof of any math theorem, we have revealed that it too is an unfalsifiable tautology. Even Fermat’s Last Theorem is now known to be a direct logical consequence of the axioms of math—arguably just a fancy way of writing 0=0.
    So again, FEP is an unfalsifiable tautology. What does that mean in practice? Well, It means that I am entitled to never think about FEP. Anything that you can derive from FEP, you can derive directly from the same (very basic and uncontroversial) underlying premises from which FEP itself can be proven, without ever mentioning FEP. So the question is really whether FEP is helpful.
    Here are two possible analogies:
    (1) Noether’s Theorem (if the laws of physics have a symmetry, they also have a corresponding conservation law) is also an unfalsifiable tautology.
    (2) The pointless bit of math trivia 2592=2592 is also an unfalsifiable tautology.
    In both cases, I don’t have to mention these facts. But in the case of (1)—but not (2)—I want to.
    More specifically, here’s a very specific ...
    15 min
  • EA - Fighting without hope by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fighting without hope, published by Akash on March 1, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • AF - Payor's Lemma in Natural Language by Andrew Critch
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Payor's Lemma in Natural Language, published by Andrew Critch on March 2, 2023 on The AI Alignment Forum.
    Preceded by: Modal Fixpoint Cooperation without Löb's Theorem
    It turns out Payor's Lemma and its proof can be explained in natural language even more easily than Löb's Theorem. Here's how.
    Imagine a group of people, and let x denote the statement "everyone in the group cooperates". Payor's Lemma says the following:
    Lemma: If ⊢□(□xx)x, then ⊢x
    First, let's unpack the meaning of the assumption in words:
    "□x" may be interpreted as saying "the group verifies (on the basis of logic) that it will cooperate" or "cooperation is believed".
    "□xx" is a statement of trustworthiness: if the group verifies that it will cooperate, then it actually will cooperate. Because a formal verifier can have bugs in it — or, because a group of people can fail to understand itself — this is a non-trivial claim about the group.
    "□(□xx)" says "the group verifies that it's trustworthy" (in the specific sense of trustworthiness above).
    "□(□xx)x" says "the group will cooperate on the basis of verified trustworthiness", i.e., "if the group verifies that it's trustworthy, then it will cooperate".
    "⊢□(□xx)x" says "it's verified that the group will cooperate on the basis of verified trustworthiness"
    Now let's work through the proof in words, too! I'll omit saying "it's verified that..." each time, which is what ⊢ means.
    ⊢x(□xx), by tautology (A(BA)). This says:"If the group cooperates, then it's trustworthy" (in the specific sense of trustworthiness about cooperation defined above).
    ⊢□x□(□xx), from 1 by □ necessitation and distributivity. This says:"If the group verifiably cooperates, it's verifiably trustworthy."
    ⊢□(□xx)x, by assumption. This says:"Assume the group will cooperate on the basis of verified trustworthiness."
    ⊢□xx, from 2 and 3 by modus ponens. This says:"The group is trustworthy."
    ⊢□(□xx), from 4 by □ necessitation. This says:"The group is verifiably trustworthy."
    ⊢x, from 5 and 3 by modus ponens. This says:"The group cooperates."
    Continuing to use "trustworthiness" in the sense above, the whole proof may be summarized as follows:
    "If a group verifiably cooperates, it's verifiably trustworthy (to itself). Assume the group cooperates on the basis of verified trustworthiness. Then, it also cooperates on the basis of verified cooperation (a stronger condition), which is what trustworthiness means. Therefore, the group is trustworthy, hence verifiably trustworthy (assuming we concluded all this using logic), hence the group cooperates (by the assumption)."
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Learn the mathematical structure, not the conceptual structure by Adam Shai
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Learn the mathematical structure, not the conceptual structure, published by Adam Shai on March 1, 2023 on LessWrong.
    I've recently been learning about transformers and noticed a failure mode of my learning that has occurred throughout my life: trying to learn a subject from material that deals with the high-level conceptual structure of something instead of learning the mathematical structure more directly. I do not mean to suggest that one needs to focus on hardcore formalizations for everything, but there is a difference between learning the conceptual structure of a subject, and learning the conceptual structure of the mathematical framework of a subject.
    The most salient example to me of this phenomenon occurred when I was trying to teach myself quantum mechanics at the end of high school. I voraciously read many popular accounts of QM, watched interviews with physicists, etc. These sources would emphasize the wave-particle duality, Schrodinger's cat, the double-slit experiment, and the uncertainty principle. I could certainly recite these concepts back in conversation, but at no point did I feel like I understood quantum mechanics.
    That is, until I read the Wikipedia entry on the mathematical formalism of quantum mechanics (or some similar type of reference, I don't remember exactly). There I found an explanation not of the physics of QM, but instead of the mathematical structure of QM. What I learned was that QM is a game with rules. The rules are that the state of the system is given as an arrow, and that the dynamics of the arrow are given by a pretty straightforward linear differential equation, and that "measurements" were associated with linear operators (matrices), and the rules of measurement were that the state of the system would "collapse" to an eigenvector of the operator with probabilities given by dot products of the current state with the eigenvectors.
    This was mind-blowing. All that time I took reading about Schrodinger's cat I could have instead simply learned that everything comes from a vector moving according to a linear diffy-Q plus some straightforward rules about eigenvectors and linear operators.
    I am no mathematician; I want to stress that I don't mean that one should focus on highly-formalized mathematics when dealing with any subject, but that often when I find myself struggling to understand something, or when I find myself having the same conversations over and over again, it pays to try to focus on finding an explanation, even an abstract conceptual explanation, not of the subject, but instead of the mathematical structure.
    I think one often sees this failure mode in action in the types of subjects that lend themselves to abstracted, metaphysical, and widely applicable thinking. Some examples include predictive coding and category theory.
    For example with predictive coding and active inference. It feels often that there is an enormous amount of back and forth discussion on topics like these, at an abstracted conceptual level, when instead the discussion could be made much more concrete by talking about the actual mathematical structure of these things. I get the sense (I am very much guilty of this) that many people talk about these subjects without putting ample effort into really understanding the structure underlying these ideas. What ends up happening is that subjects are overly applied to many different situations, and a lot of wheel spinning happens with no useful work being created.
    Of course, this lesson can be overly applied, and there is much to be said for being able to explore ideas without caring too much about formalism and mathematics - but often when I am stuck and I feel like I haven't really grokked something despite putting in effort, it helps to remember this failure mode exists, and to seek out a different sort...
    4 min
  • LW - The Parable of the King and the Random Process by moridinamael
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Parable of the King and the Random Process, published by moridinamael on March 1, 2023 on LessWrong.
    ~ A Parable of Forecasting Under Model Uncertainty ~
    You, the monarch, need to know when the rainy season will begin, in order to properly time the planting of the crops. You have two advisors, Pronto and Eternidad, who you trust exactly equally.
    You ask them both: "When will the next heavy rain occur?"
    Pronto says, "Three weeks from today."
    Eternidad says, "Ten years from today."
    "Good," you say. "I will begin planting the crops in a little bit over five years, the average of your two predictions."
    Pronto clears his throat. "If I may, Your Grace. If I am right, we should start preparing for the planting immediately. If Eternidad is right, we should expect an extreme drought, and will instead need to use the crown's resources to begin buying up food from our neighbors, for storage. These two predictions reflect totally different underlying world models, and demand two totally different and non-overlapping responses. Beginning the planting in five years is the wrong choice under either model, and guarantees that the nation will starve regardless of which of us is right."
    Eternidad adds: "Indeed, Your Grace. From Pronto's point of view, waiting five years to prepare is just as bad as waiting ten years – the rains will be long passed, by his model. From my perspective, likewise, we should take action now to prepare for drought. We must allocate resources today, one way or the other. What you face is not so much a problem of prediction but a decision problem with an important component of probability. Absolutely do not view our predictions as two point estimates to be averaged and aggregated – view them instead as two distinct and mutually exclusive futures that must be weighed separately to determine the best allocation of resources. Unfortunately, given the unrectifiable disagreement between Pronto and myself, the best course of action is that we do our best to make reasonable preparations for both possibilities.
    We should spend some fraction of our treasury on planting grain now, in case the rains arrive soon, and the remainder on purchasing food for long-term storage, in the case of prolonged drought."
    You, the monarch, ponder this. You do not want to have to split your resources. Surely there must be some way of avoiding that? Finally you say: "It seems like what I need from you two is a probability distribution of rain likelihood going forward into the future. Then I can sample your distributions and get a more informative median date."
    Pronto again clears his throat. "No, Your Grace. Let us take the example of the simplest distribution, and derive what conclusions we may, and thereby show that this approach doesn't actually help the situation. Let us assume, for the sake of argument, that I think the odds of rain on any given day are about 3% and Eternidad thinks that odds of rain on any given day are about 0.02%. Under this simple model, we can be said to each have a different uniform distribution over dates of first rainfall. The odds that it will not have rained by some given future day will follow an exponential decay process; the probability that it will have rained by t=3 weeks under my distribution of 3% probability of rain per day is ~50%. The probability that it will not have rained by t=10 years under Eternidad's distribution of 0.02% probability of rain per day is ~50%. Thus we arrive at the same median rain estimates as before, via an assumption of a uniform distribution."
    Eternidad interjects: "To be sure, Your Grace, neither of us actually believes that there's a uniform distribution of rain on any given day. Pronto is merely making an abstract point about how assumptions of distribution-shape influence subsequent conclusions."
    Pronto continues. "...
    10 min
  • LW - Sunlight is yellow parallel rays plus blue isotropic light by Thomas Kehrenberg
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sunlight is yellow parallel rays plus blue isotropic light, published by Thomas Kehrenberg on March 1, 2023 on LessWrong.
    When you look up the color temperature of daylight, most sources will say 6500K, but if you buy an LED with that color temperature, it will not look like the sun in the sky. It will seem bluer (or, less yellow-y). Yet, 6500K is arguably the correct number. What is going on?
    The answer is Rayleigh scattering. What we perceive as sun rays on the Earth’s surface has traveled through a lot of atmosphere, whereby a lot of the blue light has been scattered away, such that the sun rays look a bit yellowish. But the blue light isn’t all lost! A lot of it does arrive on the surface eventually, but it does so as diffuse light – light that seems to come from the whole sky. Which is of course why the sky looks blue (if it isn’t blocked by clouds).
    The result is that the parallel rays that come from the bright spot in the sky (aka, the sun) are (slightly) yellow, but as they mix with the diffuse blue light from the sky, they form white light with a color temperature of about 6500K. More specifically, the daylight standard D65 is defined as
    the average midday light in Western Europe, comprising both direct sunlight and the light diffused by a clear sky.
    And that’s where we get the 6500K color temperature.
    (The sun as seen from space actually has a color temperature of 5800K to 5900K. I haven’t looked into why these numbers are different but I assume it’s because of absorption in the atmosphere.)
    But most of the time, not everything is illuminated evenly. The parallel rays from the sun produce shadows and they are blue:
    (Source. This picture was not taken at midday, so the sun is extra yellow, but this allows us to clearly see the blue shadows.)
    So, in order to imitate daylight more faithfully indoors, I think you should separate the components of the light as our atmosphere does. So, use spotlights of yellowish light together with diffuse color-of-the-sky blue light that seems to come from everywhere, to produce blueish shadows.
    Here is a video of someone doing this the hard way with actual Rayleigh scattering – to very impressive results – but I think you could just buy yellowish LED spotlights (with color temperature of 3500–4500K or so? I tried to find the color temperature of just the sun rays without the skylight, but couldn’t find anything) and blue LED strips mounted on all walls close to the ceiling. And of course, the lights should be very very bright in order to imitate the sun.
    I suspect that the blueish shadows will make any white light feel much more pleasant.
    I intend to do this myself for my office, but it will take some re-configuration of the lights, so it might take a while and I wanted to get this post out early. If someone manages to try this out before me, I would be very interested to hear about their experiences.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Implied "utilities" of simulators are broad, dense, and shallow by porby
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Implied "utilities" of simulators are broad, dense, and shallow, published by porby on March 1, 2023 on LessWrong.
    This is a quick attempt at deconfusion similar to instrumentality. Same ideas, different angle.
    Extremely broad, dense reward functions constrain training-compatible goal sets
    Predictors/simulators are typically trained against a ground truth for every output. There is no gap between the output and its evaluation; an episode need not be completed before figuring out how good the first token prediction was. These immediate evaluations for every training sample can be thought of as a broad and densely defined reward function.
    It's easier for a model to fall into an undesired training-compatible goal set when there are many accessible options for undesirable goal sets versus desirable goal sets. As the number of constraints imposed by the trained reward function increases, the number of training-compatible goal sets tends to decrease, and those that survive obey more of the desirable constraints.
    There is no guarantee that SGD will find an agent which could be modeled by a utility function that maps perfectly onto the defined reward function, but if you throw trillions of constraints at the function, and simultaneously give it lots of highly informative hints about what path to walk, you should expect the potential output space to be far narrower than if you hadn't.
    Impact on internal mesaoptimizers
    The dense loss/reward function does not as heavily constrain out of distribution behavior. In principle, a strong misaligned mesaoptimizer within a predictive model could persist in these degrees of freedom by providing extremely good solutions to in-distribution samples while doing arbitrarily misaligned things out of distribution.
    But how would that type of mesaoptimizer develop in the first place?
    Steps toward it must serve the training objective; those constraints still shape the mesaoptimizer's training even if its most notable activity ends up being hidden.
    The best story I've found so far goes something like this:
    Traditional reinforcement learning agents are mostly unconstrained. The reward function is sparse relative to state and action space.
    An agent faced with sparse rewards must learn actions that serve a later goal to get any reward at all.
    Not surprisingly, agents facing sparse reward relative to state/action space and few constraints have a much larger percentage of undesirable training-compatible goal sets.
    Mesaoptimizers are processes learned within a model and their local training influences may not perfectly match the outer training influences.
    If the mesaoptimizer's local training influences look more like the traditional reinforcement learning agent's influences than the predictor's outer influences, it would be more likely to fall into one of the undesirable training-compatible goal sets.
    The mesaoptimizer learns incorrect goals and a high propensity for goal-serving intermediate actions ("actions" within the scope of a single model execution!)
    The mesaoptimizer is kept around by SGD because it does well on the subset of outputs that the outer model is using it on. As capability grows, the mesaoptimizer strategically takes over other chunks of prediction space by performing well during training in an effort to be selected during out of distribution predictions.
    In a previous post, I called the learned propensity for goal-serving intermediate action instrumentality. The constraints imposed by predictive model training clearly confer lower instrumentality than traditional RL in all current models. I suspect the path taken by the mesaoptimizer above is hard and unnatural, but perhaps not impossible for some form of predictor taken to the relevant extreme.
    It seems critical to understand the degree to which outer constraints apply to inner lea...
    7 min
  • EA - Scoring forecasts from the 2016 “Expert Survey on Progress in AI” by PatrickL
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Scoring forecasts from the 2016 “Expert Survey on Progress in AI”, published by PatrickL on March 1, 2023 on The Effective Altruism Forum.
    Summary
    This document looks at the predictions made by AI experts in The 2016 Expert Survey on Progress in AI, analyses the predictions on ‘Narrow tasks’, and gives a Brier score to the median of the experts’ predictions.
    My analysis suggests that the experts did a fairly good job of forecasting (Brier score = 0.19), and would have been less accurate if they had predicted each development in AI to generally come, by a factor of 1.5, later (Brier score = 0.26) or sooner (Brier score = 0.27) than they actually predicted.
    I judge that the experts expected 9 milestones to have happened by now - and that 10 milestones have now happened.
    But there are important caveats to this, such as:
    I have only analysed whether milestones have been publicly met. AI labs may have achieved more milestones in private this year without disclosing them. This means my analysis of how many milestones have been met is probably conservative.
    I have taken the point probabilities given, rather than estimating probability distributions for each milestone, meaning I often round down, which skews the expert forecasts towards being more conservative and unfairly penalises their forecasts for low precision.
    It’s not apparent that forecasting accuracy on these nearer-term questions is very predictive of forecasting accuracy on the longer-term questions.
    My judgements regarding which forecasting questions have resolved positively vs negatively were somewhat subjective (justifications for each question in the separate appendix).
    Introduction
    In 2016, AI Impacts published The Expert Survey on Progress in AI: a survey of machine learning researchers, asking for their predictions about when various AI developments will occur. The results have been used to inform general and expert opinions on AI timelines.
    The survey largely focused on timelines for general/human-level artificial intelligence (median forecast of 2056). However, included in this survey were a collection of questions about shorter-term milestones in AI. Some of these forecasts are now resolvable. Measuring how accurate these shorter-term forecasts have been is probably somewhat informative of how accurate the longer-term forecasts are. More broadly, the accuracy of these shorter-term forecasts seems somewhat informative of how accurate ML researchers' views are in general. So, how have the experts done so far?
    Findings
    I analysed the 32 ‘Narrow tasks’ to which the following question was asked:
    How many years until you think the following AI tasks will be feasible with:
    a small chance (10%)?
    an even chance (50%)?
    a high chance (90%)?
    Let a task be ‘feasible’ if one of the best resourced labs could implement it in less than a year if they chose to. Ignore the question of whether they would choose to.
    I interpret ‘feasible’ as whether, in ‘less than a year’ before now, any AI models had passed these milestones, and this was disclosed publicly. Since it is now (February 2023) 6.5 years since this survey, I am therefore looking at any forecasts for events happening within 5.5 years of the survey.
    Across these milestones, I judge that 10 have now happened and 22 have not happened. My 90% confidence interval is that 7-15 of them have now happened. A full description of milestones, and justification of my judgments, are in the appendix (separate doc).
    The experts forecast that:
    4 milestones had a <10% chance of happening by now,
    20 had a 10-49% chance,
    7 had a 50-89% chance,
    1 had a >90% chance.
    So they expected 6-17 of these milestones to have happened by now. By eyeballing the forecasts for each milestone, my estimate is that they expected ~9 to have happened. I did not estimate the implied probability distribut...
    19 min