The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Infinite-width MLPs as an "ensemble prior" by Vivek Hebbar
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Infinite-width MLPs as an "ensemble prior", published by Vivek Hebbar on May 12, 2023 on LessWrong.
    Summary: A simple toy model suggests that infinitely wide MLPs generalize in an "ensemble-ish" way which is exponentially less data-efficient than Solomonoff induction. It's probably fixable by different initializations and/or regularizations, so I note it here mostly as a mathematical curiosity / interesting prior.
    The analysis seems to be qualitatively consistent with empirical results on generalization vs width in small MLPs.
    Notes:
    The generalization behavior of these neural nets can be analyzed with the Neural Tangent Kernel, which is widely studied. This post is meant to probe the qualitative nature of this behavior through a toy model. I'm unsure whether my particular analysis exists elsewhere.
    The deficiency of the standard initialization at infinite width seems to be well-known and empirically supported in NTK-related literature, along with ways of fixing it.
    Core claims
    The standard initialization uses weights which are proportional to 1/√input_dimension. This has the effect of keeping the activations at roughly the same scale across layers. However, in the infinite width case, it ends up making the gradients in early layers infinitely smaller than those in the last layer. Hence, training an infinite-width MLP is equivalent to running a regression using the features represented by the last-layer neurons at initialization. These features never change during training, since the early gradients are all zero.
    If we train without regularization, we will tend to get something very "ensemble-ish", "smooth", and "dumb". I will first summarize this claim in a table, then spend the rest of the post going through the reasoning behind it.
    Solomonoff InductionInfinite width MLP, low L2-norm solutionBayesian update over programsLinear regression over circuitsPuts most of its weight on a small number of programs, each of which perfectly fits the data on its ownSpreads weight over a broad ensemble, including circuits which have only a small correlation with truthThe amount of data required to make the correct program dominate is O(K), where K is the program lengthThe amount of data to make the correct circuit dominate is O(2C), where C is some "complexity measure" (defined later). This is exponentially less data-efficient than Solomonoff induction.Calling it "superintelligent" is an understatementGeneralizes poorly on many tasksHighly amenable to "sharp" solutionsFavors smooth solutions, only creates "sharp" solutions if certain conditions are met by the training data.
    If we train an infinitely wide MLP from the standard initialization, only the last layer's weights change. So it is equivalent to a linear regression over an infinite set of random "features", these features being the activation patterns of the last layer neurons at initialization.
    If the MLP is deep enough, some of these last-layer neurons are contain the output of very intelligent circuits. However, if we train our infinite width MLP, these intelligent circuits will hardly be used by the regression, even if they are very useful. That is, the sum of the weights drawing from them in the last layer will be very small. The reason I believe this is the toy model in the next section.
    Toy model
    Let's call each last-layer neuron a "feature". As discussed earlier, their behavior never changes due to how the gradients pan out at infinite width. In a "real" infinite network, these features will be "useful" and "intelligent" to various degrees, but we will simplify this greatly in the toy model, by using just two types of features.
    The toy model asks: "Suppose that some features already compute the correct answer for every training datapoint, and that the rest of the features are random garbage. Will the trained network...
    10 min
  • LW - Turning off lights with model editing by Sam Marks
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Turning off lights with model editing, published by Sam Marks on May 12, 2023 on LessWrong.
    This post advertises an illustrative example of model editing that David Bau is fond of, and which I think should be better known. As a reminder, David Bau is a professor at Northeastern; the vibe of his work is "interpretability + interventions on model internals"; examples include the well-known ROME, MEMIT, and Othello papers.
    Consider the problem of getting a generative image model to produce an image of a bedroom containing unlit lamps (i.e. lamps which are turned off). Doesn't sound particularly interesting. Let's try the obvious prompts on DALL-E-2.
    Doesn't work great: my first three attempts only got one borderline hit, at the expense of turning the entire bedroom dark (which isn't really what I had in mind).
    As David Bau tells things, even after putting further effort into engineering an appropriate prompt, they weren't able to get what they wanted: a normal picture of a bedroom with lamps which are not turned on. (Apparently the captions for images containing unlit lamps don't often mention the lamps.)
    Despite not being able to get what they wanted with prompt engineering, they were able to get what they wanted via interpretability + model editing. Namely, they took a GAN which was trained to produce images of bedrooms, and then:
    identified a neuron which seemed to modulate the brightness of the lamps in the generated image
    intervened on the networks activations, setting the activation of this neuron so as to produce the desired level of brightness.
    The result is shown below.
    I like this example because it provides a template in a very toy setting for how model editing could be useful for alignment. The basic structure of this situation is:
    We have a model which is not behaving as desired, despite being able to do so in principle (our model doesn't output images of bedrooms with unlit lamps, despite being capable of doing so).
    Basic attempts at steering the model's behavior fail (prompt engineering isn't sufficient).
    But we are able to get the behavior that we want by performing a targeted model edit.
    This example also showcases some key weaknesses of this technique, which would need to be addressed for model editing to become a viable alignment strategy:
    Alignment tax. Looking closely at the image above, you'll notice that even though the direct light from the lamp is able to modified at will, certain second-order effects aren't affected (e.g. the light which is reflected off the wall). As David tells things, they also identified a whole suite of ~20 neurons in their GAN which modulated more subtle lighting effects.
    Not all behaviors can necessarily be targeted. The images on the lower row above contain two lamps, and these two lamps change their brightness together. The researchers were not able to find a neuron which would allow them to change the brightness of only one lamp in images that contained multiple lamps.
    No clear advantages over finetuning. The more obvious thing to do would be to finetune the model to output unlit lamps. As far as I know, no one tried to do that in this case, but I imagine it would work. I'll leave my most optimistic speculation about why model editing could have advantages over finetuning in certain situations in this footnote, but I don't currently find this speculation especially compelling.
    Overall, I'm not super bullish on the usefulness of model editing for alignment. But I do think it's intriguing, and it seems to have been useful in at least one case (though not necessarily one which is very analogous to x-risky cases of misalignment). Overall, I think that work like this is a risky bet, with the advantage that some of its failure modes might differ from the failure modes of other alignment techniques.
    [Thanks to Xander Davies ...
    5 min
  • LW - Dark Forest Theories by Raemon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dark Forest Theories, published by Raemon on May 12, 2023 on LessWrong.
    There's a concept I first heard in relation to the Fermi Paradox, which I've ended up using a lot in other contexts.
    Why do we see no aliens out there? A possible (though not necessarily correct) answer, is that the aliens might not want to reveal themselves for fear of being destroyed by larger, older, hostile civilizations. There might be friendly civilizations worth reaching out to, but the upside of finding friendlies is smaller than the downside of risking getting destroyed.
    Even old, powerful civilizations aren't sure that they're the oldest and most powerful civilization, and eldest civilizations could be orders of magnitude more powerful still.
    So, maybe everyone made an individually rational-seeming decision to hide.
    A quote from the original sci-fi story I saw describing this:
    “The universe is a dark forest. Every civilization is an armed hunter stalking through the trees like a ghost, gently pushing aside branches that block the path and trying to tread without sound. Even breathing is done with care. The hunter has to be careful, because everywhere in the forest are stealthy hunters like him. If he finds another life—another hunter, angel, or a demon, a delicate infant to tottering old man, a fairy or demigod—there’s only one thing he can do: open fire and eliminate them.”
    (I consider this a spoiler for the story it's from, so please don't bring that up in the comments unless you use spoiler tags)
    How this applies (or doesn't) to aliens, I've found it useful to have the "Dark Forest" frame in a number of contexts where people are looking at situation, and see something missing, and are confused. "Why is nobody doing X?", or "Why does X not exist?". The answer may be that it does exist, but is hidden from you (often on purpose).
    I once talked to someone new to my local community saying "nobody is building good group houses that really help people thrive. I'm going to build one and invite people to it." I said "oh, people are totally building good group houses that help people thrive... but they are private institutions designed to help the people living there. The way they function is by creating a closed boundary where people can build high trust relationships."
    A couple other example areas where Dark Forest Theorizing is relevant:
    "Why are there no good meetups?" The answer might be that there are good private meetups, but the public meetups tend to attract people who weren't invited to private meetups because they were kinda annoying, and this creates an evaporative cooling effect where the people who could potentially make the public meetups good don't stick around.
    "Why is no one talking to AI companies to get them to change their ways?" The answer is that people are, but it's a delicate high-trust operation. AI companies are likely to tune you out if you're telling them they're terrible, getting them to actually listen is a delicate operation that requires both trust and understanding their current frame.
    In some cases, a thing might not literally be hidden on purpose from you, but it's absence is still evidence of something systematically important. For example, "Why aren't the AI people making a giant mass movement to raise public awareness of AI?" Because many versions of a mass movement might turn out to be net-negative (i.e. causing political polarization which makes it harder to get the necessary bipartisan support to pass the relevant laws), and instead the people involved are focused on more narrow lobbying efforts.
    (This is not to say that you can't make good public meetups, or successfully talk to AI companies and get them to change their ways, or that there aren't good ways to do mass public outreach on AI that are being neglected. But there may be some systemic diffic...
    4 min
  • LW - Retrospective: Lessons from the Failed Alignment Startup AISafety.com by Søren Elverlin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Retrospective: Lessons from the Failed Alignment Startup AISafety.com, published by Søren Elverlin on May 12, 2023 on LessWrong.
    TL;DR: Attempted to create a startup to contribute to solving the AI alignment problem. Ultimately failed due to rapid advancements in large language models and the inherent challenges of startups.
    In early 2021, I began considering shorter AI development timelines and started preparing to leave my comfortable software development job to work on AI safety. Since I didn't feel competent enough to directly work on technical alignment, my goal was capacity-building, personal upskilling, and finding a way to contribute.
    During our reading group sessions, we studied Cotra's "Case for Aligning Narrowly Superhuman Models" which made a compelling argument for working with genuinely useful models. This inspired us to structure our efforts as a startup. Our team comprised of Volkan Erdogan, Timothy Aris, Robert Miles, and myself, Søren Elverlin. We planned to offer companies automation of certain business processes using GPT-3 in exchange for alignment-relevant data for research purposes.
    Given my strong deontological aversion to increasing AI capabilities, I aimed to keep the startup as stealthy as possible without triggering the Streisand effect. This decision significantly complicated fundraising and customer acquisition.
    In November 2021, I estimated a 20% probability of success, a view shared by my colleagues. I was fully committed, investing DKK420,000 (USD55,000), drawing no salary for myself, and providing modest compensation to the others.
    Startup literature generally advises against one-person startups. Despite our team of four, I was taking on a disproportionate amount of work and responsibility, which should have raised red flags.
    My confidence in our success grew during the spring of 2022 when a personal contact helped me secure a preliminary project with a large company that wished to remain anonymous. For $1,300/month, I sold them a business automation solution that relied solely on a large language model for code-generation. However, it didn't provide us with the data we sought. Both parties understood this was a preliminary project, and the company seemed eager about the full project.
    Securing this project early on made Rob's role redundant, and we amicably parted ways. Half a year later Tim was offered a PhD, leaving Volkan and I (with minimal help from Berk and Ali).
    The preliminary project involved validating several not-quite-standardized Word documents, and I developed a VSTO plugin for Outlook to handle this task. It took longer than anticipated, mainly due to late-discovered requirements. Despite the iterative process, the client was ultimately very satisfied, and I focused on building trust with them during this phase.
    The full project aimed to execute business processes in response to incoming emails using multiple fine-tuned GPT-3 models in stages and incorporating as much context as possible into the prompts. Our first practical target was sorting emails in a shared mailbox and delegating tasks to different department members. Initial experiments suggested this process was likely feasible to automate.
    We were more intrigued by experiments demonstrating the opposite: certain business processes could not be replicated by GPT-3. This was particularly evident when deviations were necessary for common-sense reasons or when deviations would yield greater value. For example, a customer inquires if product delivery is possible before a specific date. The operator determines it's barely unattainable but recognizes the potential for high profits, which would prompt a human operator to deviate from standard procedures. We could not persuade GPT-3 to do this, and exploring such discrepancies in strategic reasoning seemed worthwhile.
    T...
    7 min
  • AF - Turning off lights with model editing by Sam Marks
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Turning off lights with model editing, published by Sam Marks on May 12, 2023 on The AI Alignment Forum.
    This post advertises an illustrative example of model editing that David Bau is fond of, and which I think should be better known. As a reminder, David Bau is a professor at Northeastern; the vibe of his work is "interpretability + interventions on model internals"; examples include the well-known ROME, MEMIT, and Othello papers.
    Consider the problem of getting a generative image model to produce an image of a bedroom containing unlit lamps (i.e. lamps which are turned off). Doesn't sound particularly interesting. Let's try the obvious prompts on DALL-E-2.
    Doesn't work great: my first three attempts only got one borderline hit, at the expense of turning the entire bedroom dark (which isn't really what I had in mind).
    As David Bau tells things, even after putting further effort into engineering an appropriate prompt, they weren't able to get what they wanted: a normal picture of a bedroom with lamps which are not turned on. (Apparently the captions for images containing unlit lamps don't often mention the lamps.)
    Despite not being able to get what they wanted with prompt engineering, they were able to get what they wanted via interpretability + model editing. Namely, they took a GAN which was trained to produce images of bedrooms, and then:
    identified a neuron which seemed to modulate the brightness of the lamps in the generated image
    intervened on the networks activations, setting the activation of this neuron so as to produce the desired level of brightness.
    The result is shown below.
    I like this example because it provides a template in a very toy setting for how model editing could be useful for alignment. The basic structure of this situation is:
    We have a model which is not behaving as desired, despite being able to do so in principle (our model doesn't output images of bedrooms with unlit lamps, despite being capable of doing so).
    Basic attempts at steering the model's behavior fail (prompt engineering isn't sufficient).
    But we are able to get the behavior that we want by performing a targeted model edit.
    This example also showcases some key weaknesses of this technique, which would need to be addressed for model editing to become a viable alignment strategy:
    Alignment tax. Looking closely at the image above, you'll notice that even though the direct light from the lamp is able to modified at will, certain second-order effects aren't affected (e.g. the light which is reflected off the wall). As David tells things, they also identified a whole suite of ~20 neurons in their GAN which modulated more subtle lighting effects.
    Not all behaviors can necessarily be targeted. The images on the lower row above contain two lamps, and these two lamps change their brightness together. The researchers were not able to find a neuron which would allow them to change the brightness of only one lamp in images that contained multiple lamps.
    No clear advantages over finetuning. The more obvious thing to do would be to finetune the model to output unlit lamps. As far as I know, no one tried to do that in this case, but I imagine it would work. I'll leave my most optimistic speculation about why model editing could have advantages over finetuning in certain situations in this footnote, but I don't currently find this speculation especially compelling.
    Overall, I'm not super bullish on the usefulness of model editing for alignment. But I do think it's intriguing, and it seems to have been useful in at least one case (though not necessarily one which is very analogous to x-risky cases of misalignment). Overall, I think that work like this is a risky bet, with the advantage that some of its failure modes might differ from the failure modes of other alignment techniques.
    That is, I f...
    5 min
  • EA - Why GiveWell funded the rollout of the malaria vaccine by GiveWell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why GiveWell funded the rollout of the malaria vaccine, published by GiveWell on May 12, 2023 on The Effective Altruism Forum.
    Author: Audrey Cooper, GiveWell Philanthropy Advisor
    Since our founding in 2007, GiveWell has directed over $600 million to programs that aim to prevent malaria, a mosquito-borne disease that causes severe illness and death. Malaria is preventable and curable, yet it killed over 600,000 people in 2021—mostly young children in Africa.[1]
    Following the World Health Organization’s approval of the RTS,S/AS01 malaria vaccine (RTS,S) in late 2021,[2] GiveWell directed $5 million to PATH to accelerate the rollout of the vaccine in certain areas of Ghana, Kenya, and Malawi. This grant aimed to enable these communities to gain access to the vaccine about a year earlier than they otherwise would, protecting hundreds of thousands of children from malaria.[3]
    Although we’re very excited about the potential of the RTS,S malaria vaccine to save lives, it isn’t a panacea. We still plan to support a range of malaria control interventions, including vaccines, nets, and antimalarial medicine.
    In this post, we will:
    Explain how we found the opportunity to fund the malaria vaccine
    Discuss why we funded this grant
    Share our plan for malaria funding moving forward
    Identifying a gap in vaccine access
    In October 2021, we shared our initial thoughts on the approval of the RTS,S malaria vaccine by the World Health Organization (WHO). At that point, we weren’t sure whether the vaccine would be cost-effective and were not aware of any opportunities for private donors to support the expansion of vaccine access.
    In the following months, our conversations with PATH, a large global health nonprofit that we’ve previously funded, revealed that there might be an opportunity to help deploy the vaccine more quickly in certain regions. PATH had been supporting the delivery of the vaccine in Ghana, Kenya, and Malawi as part of the WHO-led pilot—the Malaria Vaccine Implementation Program (MVIP)—since the pilot began in 2019.[4] In order to generate evidence about the effectiveness of the vaccine, randomly selected areas in each country received the vaccine during the early years of the pilot, while “comparison areas” would receive the vaccine at a later date, if the vaccine was recommended by the WHO.[5]
    Once the vaccine had received approval from the WHO, the WHO and PATH believed there was an opportunity to build on the momentum and groundwork of the pilot to roll out the vaccine to the comparison areas as soon as possible. However, the expectation at the time was that expanding use to the comparison areas would need to wait for the standard process through which low-income countries apply for support to access vaccines from Gavi, the Vaccine Alliance.[6] This process would have made it possible to introduce the vaccine at the end of 2023 at the earliest.[7]
    However, there was another path through which these vaccines could be provided more quickly. GlaxoSmithKline (GSK), the vaccine manufacturer, had committed to donate up to 10 million vaccine doses as part of its support for the MVIP.[8] This quantity of vaccine was set aside to allow completion of the pilot program, including vaccination in the comparison areas.[9] However, additional support was needed to be able to utilize these vaccines in advance of Gavi financing, including (for example) funding to cover the costs of safe injection supplies and vaccine shipping and handling, as well as the technical assistance required to support vaccine implementation.
    With funding from GiveWell, PATH believed it could provide the necessary technical assistance to the ministries of health in Ghana, Kenya, and Malawi to support them in using the donated vaccines from GSK and expand vaccine access to the comparison areas at the end of 202...
    9 min
  • LW - The way AGI wins could look very stupid by Christopher King
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The way AGI wins could look very stupid, published by Christopher King on May 12, 2023 on LessWrong.
    Computers don't have any sense of aesthetics or patterns that are standard the way people learn how to play chess. They play what they think is the objectively best move in any position, even if it looks absurd, and they can play any move no matter how ugly it is." - Murray Campbell quote about DeepBlue
    Vinge's principle states: "we usually think we can't predict exactly what a smarter-than-us agent will do, because if we could predict that, we would be that smart ourselves".
    A popular idea think this means that AGI would invent and use new technology such as nanorobotics to defeat us (this is the example Yudkowsky usually gives).
    However, this doesn't seem to jive with what happens in other domains where AI becomes superhuman. Usually what the AI does is understandable to humans. It's just that it looks, well, dumb.
    For example, in chess, computers use roughly the same piece evaluation that humans discovered in the 18th century, didn't discover any new openings, and generally seemed to play ugly moves. But they won anyways.
    If something like nanorobotics lets you take over the world, you'd expect a human group to be trying to create them to take over the world already because it seems to make sense. In reality, any plan that (for example) relies on DNA as a stepping stone will quickly run into regulatory problems.
    Instead, I imagine that the AGI's plan will elicit similar reactions as the following:
    You get a phone call advertising free Mayonnaise! You just need to follow a couple simple steps. The next day, you're confused and in some sort of Mayonnaise cult breaking into a military armory in Mexico.
    Is this plan something that humans can try? No, it seems pretty straight forward to attempt. So why haven't we tried it? Because it seems and likely is dumb. Why mayonnaise? Why a phone call? Why Mexico?
    But if AGI is similar to other superhuman AI, this is the type of thing we expect to see; a strategy that looks dumb but works. We have no way to predict which dumb strategy will be used, but given the large number of strategies that look dumb to humans, the AGI's strategy is likely to be one of them. And it has enough Yomi to predict which one will succeed.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - Towards Measures of Optimisation by Matt MacDermott
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Towards Measures of Optimisation, published by Matt MacDermott on May 12, 2023 on The AI Alignment Forum.
    We would like a mathematical theory which characterises the intuitive notion of ‘optimisation’.
    Before Shannon introduced his mathematical theory of communication, the concept of ‘information’ was vague and informal. Then Shannon devised several standardised measures of it, with useful properties and clear operational interpretations. It turned out the concept of information is universal, in that it can be quantified on a consistent scale (bits) across various contexts.
    Could something similar be true for ‘optimisation’?
    In this post we review a few proposed ways to measure optimisation power and play around with them a bit.
    Our general setup will be that we’re choosing actions from a set A to achieve an outcome in a set X. We have some beliefs about how our actions will affect outcomes in the form of a probability distribution pa∈ΔX for each a∈A. We also have some preferences, either in the form of a preference ordering over outcomes, or a utility function over outcomes. We will also assume there is some default distribution p∈ΔX, which we can either interpret as our beliefs about X if we don’t act, or if we take some default action[1].
    Yudkowsky
    Yudkowsky’s proposed definition just makes use of a preference ordering ⪰ over X. To measure the optimisation power of some outcome x, we count up all the outcomes which are at least as good as x, and divide that number by the total number of possible outcomes. It’s nice to take a negative log to turn this fraction into a number of bits: OP(x)=−log|{x′∈X∣x′⪰x}X|. If I achieve the second best outcome out of eight, that's −log28=2 bits of optimisation power.
    If the outcome space is infinite, then we can't count the number of outcomes at least as good as the one we got, so we need a measure to integrate over. If we make use of our default probability distribution here, the resulting quantity has a nice interpretation. ∫x′⪰xp(x′)dx′∫p(x′)dx′ is just ∫x′⪰xp(x′)dx′: the default probability of doing as well as we did. Since we're always assuming we've got a default distribution, we might as well define OP like this even in the finite-domain case. Again we’ll take a log to get OP(x)=−log∫x′⪰xp(x′)dx′. Now 2 bits of optimisation power means the default probability of doing this well was 14.
    So far we’ve just been thinking about the optimisation power of achieving a specific outcome. We can define the optimisation power of an action as the expected optimisation power under the distribution it induces over outcomes: OP(a)=∫pa(x)OP(x)dx. The above definitions just make use of a preference ordering. If we do have a utility function u:XR then we’d like our definitions to make use of that too. Intuitively, achieving the second best outcome out of three should constitute more optimisation power in a case where it’s almost as good as the first and much better than the third, compared to a case where it’s only slightly better than the third and much less good than the first[2].
    Analogously to how we previously asked ‘what fraction of the default probability mass is on outcomes at least as good as this one?’ we could try to ask ‘what fraction of the default expected utility comes from outcomes at least as good as this one?’.
    But making use of utility functions in the above definition is tricky. Recall that utility functions originate from the Von Neumann-Morgenstern theorem, which says that if an agent choosing between probabilistic mixtures of options satisfies some weak rationality criteria then it acts as if it maximises expected utility according to a utility function u:XR. The utility function produced by the VNM-theorem is only defined up to positive affine transformations, meaning that the utility function u′=au+b, for any a∈R>0 an...
    7 min
  • EA - US public opinion of AI policy and risk by Jamie Elsey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: US public opinion of AI policy and risk, published by Jamie Elsey on May 12, 2023 on The Effective Altruism Forum.
    Summary
    On April 21st 2023, Rethink Priorities conducted an online poll to assess US public perceptions of, and opinions about, AI risk. The poll was intended to conceptually replicate and extend a recent AI-related poll from YouGov, as well as drawing inspiration from some other recent AI polls from Monmouth University and Harris-MITRE.
    The poll covered opinions regarding:
    A pause on certain kinds of AI research
    Should AI be regulated (akin to the FDA)?
    Worry about negative effects of AI
    Extinction risk in 10 and 50 years
    Likelihood of achieving greater than human level intelligence
    Perceived most likely existential threats
    Expected harm vs. good from AI
    Our population estimates reflect the responses of 2444 US adults, poststratified to be representative of the US population. See the Methodology section of the Appendix for more information on sampling and estimation procedures.
    Key findings
    For each key finding below, more granular response categories are presented in the main text, along with demographic breakdowns of interest.
    Pause on AI Research. Support for a pause on AI research outstrips opposition. We estimate that 51% of the population would support, 25% would oppose, 20% remain neutral, and 4% don’t know (compared to 58-61% support and 19-23% opposition across different framings in YouGov’s polls). Hence, support is robust across different framings and surveys. The slightly lower level of support in our survey may be explained by our somewhat more neutral framing.
    Should AI be regulated (akin to the FDA)? Many more people think AI should be regulated than think it should not be. We estimate that 70% believe Yes, 21% believe No, and 9% don’t know.
    Worry about negative effects of AI. Worry in everyday life about the negative effects of AI appears to be quite low. We estimate 72% of US adults worry little or not at all about AI, 21% report a fair amount of worry, and less than 10% worry a lot or more.
    Extinction risk in 10 and 50 years. Expectation of extinction from AI is relatively low in the next 10 years but increases in the 50 year time horizon. We estimate 9% think AI-caused extinction to be moderately likely or more in the next 10 years, and 22% think this in the next 50 years.
    Likelihood of achieving greater than human level intelligence. Most people think AI will ultimately become more intelligent than people. We estimate 67% think this moderately likely or more, 40% highly likely or more, and only 15% think it is not at all likely.
    Perceived most likely existential threats. AI ranks low among other perceived existential threats to humanity. AI ranked below all 4 other specific existential threats we asked about, with an estimated 4% thinking it the most likely cause of human extinction. For reference, the most likely cause, nuclear war, is estimated to be selected by 42% of people. The other least likely cause - a pandemic - is expected to be picked by 8% of the population.
    Expected harm vs. good from AI. Despite perceived risks, people tend to anticipate more benefits than harms from AI. We estimate that 48% expect more good than harm, 31% more harm than good, 19% expecting an even balance, and 2% reporting no opinion.
    The estimates from this poll may inform policy making and advocacy efforts regarding AI risk mitigation. The findings suggest an attitude of caution from the public, with substantially greater support than opposition to measures that are intended to curb the evolution of certain types of AI, as well as for regulation of AI. However, concerns over AI do not yet appear to feature especially prominently in public perception of the existential risk landscape: people report worrying about it only a little, and rarely picked i...
    27 min
  • LW - Three Iterative Processes by LoganStrohl
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Three Iterative Processes, published by LoganStrohl on May 12, 2023 on LessWrong.
    Throughout all four phases of naturalism, there are three processes that are constantly repeating in the background: Story revision, POU loops, and a few kinds of oscillation that together I think of as "breathing".
    Story Revision
    Story revision begins at the very start of a study. If you started with “Catching the Spark”—the orientation procedure I suggested in "Getting Started"—then articulating a story was an explicit part of the process. Otherwise, you probably encountered at least one story about your topic along the way.
    I’ve presented “orientation” as something that happens only at the beginning, but that’s not really the case. Reorientation is always happening; as a mentor, I run through an abbreviated version of the orientation procedure at nearly every meeting, until I notice that a student has made a habit of asking and answering these questions on their own. In my own practice, I keep a running list of stories in my notes, a big-picture history of my studies.
    There are three parts of story revision:
    What's your story about what's going on?
    Draw out the assumptions in your story that you might be able to check through direct observation.
    Pick according to your curiosity or your sense of what would help you understand the whole space most, and try to investigate that.
    A real person's story statements often don’t make a lot of sense to onlookers, presumably because they're meant to reflect intuitions or felt senses more than rigid concepts; I often find myself reminding students, "Remember that this only needs to make sense to you." But for the sake of having some kind of illustration, here is a chronological list of story statements I recorded while studying a pattern in my learning-related actions that I called “failure, smallness, and tumbling”:
    Sept. 15: I think that if I focus on what I'm doing wrong, I'll learn faster from my mistakes.
    Sept. 16: When I pay attention to the outcome of a goal-directed behavior, I focus in on what I believe to have been under my control.
    Sept. 28: I care a lot about exerting control over myself.
    Oct. 3: I am very careful to control my words and actions so that I don't attract the attention of people who would hurt me if they knew I am different.
    Oct. 17: Part of me thinks that if I let go, everything will fall apart.
    Oct. 20: Part of me believes that caring in front of people about things you aren't already good at is too dangerous.
    Oct. 22: When I get small, it's because I feel that I've been surrounded by something dangerous, and contracting is as close as you can get to running away when you're surrounded.
    Nov. 22: When I'm “trying really hard”, I'm afraid that if I let go, things will fall apart.
    What you see in this list is how my responses to "What's my story about what's going on?" evolved over time. Each time I asked myself that question and got new response, I adjusted my study accordingly.
    For most people, it’s a good idea to deliberately revisit and perhaps revise your overall story on a regular basis (often around twice a month), and whenever you’re feeling floaty or like things have shifted. It’s important for navigation: When you stay focused on the details of your experience for too long, you may not notice that you’ve come to regard a topic in a completely new way. Knowing that your overall story has shifted can lead you to reconsider your strategy, rather than continuing to investigate experiences that no longer seem relevant.
    POU Loops
    I discussed Predict-Observe-Update loops in “Collection”, but I’ll say a few words about them here as well.
    These are the parts of a POU loop:
    Predict: Vividly simulate what you expect to experience—what phenomenology you expect to encounter—when your fulcrum experience occurs.
    Observe: When ...
    6 min