The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - EAGxNordics Unofficial Review Thread by Robert Praas
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EAGxNordics Unofficial Review Thread, published by Robert Praas on April 25, 2023 on The Effective Altruism Forum.
    I am usually really curious to get a taste of the overall atmosphere and insights gained from EAGs or EAGx events that I don’t attend. These gatherings, which host hundreds or even thousands of Effective Altruists, serve as valuable opportunities to exchange knowledge and potentially offer a snapshot of the most pressing EA themes and current projects. I attended EAGxNordics and, as a student, I will share my observations in a bullet-point format. Thank you Adash H-Moller for great comments and suggestions. Other attendees are very welcome to add their experiences / challenge these perspectives in the comments:
    Lessons learned:
    The majority of the participants seemed to have come from (obviously) Sweden, Norway, Finland, Estonia, Denmark and The Netherlands. Small countries with relatively tight EA communities.
    I was particularly impressed with the line-up of speakers from non-EA-labeled think tanks and institutes - I think it provides a strong benefit, especially to those EAs who are quite familiar with EA but would not find out about these adjacent initiatives. It also serves to reduce the extent to which we're in our own bubble.
    I talked to numerous participants of the Future Academy - who all learned about EA through that program. They shared great experiences in policy, entrepreneurship and education (from before they knew about EA) and I think they are a great addition to the community.
    Attendees can be more ambitious: Both in their conference experience and in their approach to EA. I spoke to too many students who had <5 1on1s planned, whereas these are regarded as one of the best ways to operate during a conference. Also, in terms of the career plans and EA projects I asked about, I would have loved to see bigger goals then the ones I heard.
    I attended talks by employees of (the) GFI, Charity Entrepreneurship and The Simon Institute. The things they had in common:
    They work on problems that are highly neglected (One speaker cited from a podcast: “No one is coming, it is up to us”)
    They do their homework thoroughly
    A key factor for their impact is their cooperation with local NGOs, governments and intergovernmental organizations.
    (Suggested by Adash) The talk by an employee of Nähtamatud Loomad about ‘invisible animals’ was great and provided useful insight into what corporate lobbying actually looks like on the ground - I think specific, object-level content is great for keeping us grounded.
    There could be more focus on analyzing EA as a community and considering what EA needs more of / needs to do differently, I asked a few people exactly those questions.
    David Nash mentioned a range of critiques:
    EA should maybe
    focus more on professionals to fill existing vacancies and close the mentorship gap
    be more centralized
    Consider more movement building on the field specific level
    leverage the value of it being a network more
    For the rest our fellow vikings and visitors tended to be less vocal in direct criticism
    A lot of people talked about AI Safety
    I felt there was a large group of students who were excited about contributing to this field
    Participants with other backgrounds mentioned this as well and multiple participants voiced the preference for (a) more balanced content/narrative around topics like global development, animal welfare, etc.
    (Suggested by Adash: N is small so take with a pinch of salt) I found the conversations I had with some early-career AI safety enthusiasts to show a lack of understanding of paths to x-risk and criticisms of key assumptions. I’m wondering if the early-stage AI field-building funnel might cause an echo chamber of unexamined AI panic that undermines general epistemics and cause-neutral principles.
    Thanks for list...
    4 min
  • EA - World Malaria Day: Reflecting on Past Victories and Envisioning a Malaria-Free Future by 2ndRichter
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: World Malaria Day: Reflecting on Past Victories and Envisioning a Malaria-Free Future, published by 2ndRichter on April 25, 2023 on The Effective Altruism Forum.
    World Malaria Day, inaugurated in 2017 by the United Nations World Health Organization, reminds us of malaria’s impact on humanity and the role we can take in preventing the disease. Malaria was only eradicated from areas like Europe as recently as the 1970s, and nearly half of the world’s population was still at risk of malaria in 2021. Over 600,000 people died of malaria in 2021 and 247 million people contracted the disease in 2021—and three-quarters of those deaths were children under five.
    Almost half of the world’s countries have eradicated malaria since 1945, and we have reason to hope that countries still affected can eradicate it as well. With significant scientific advancements, we know that effective malaria prevention can be impactful and relatively cheap. Typical interventions to prevent and treat malaria include insecticide-treated bednets, removing standing water from affected areas, and antiviral medications—and some of these interventions are relatively cheap. Only $5 USD can provide one malaria net and $7 can protect a child from malaria through malaria chemoprevention. Roughly $5000 USD will provide enough bednets or seasonal medicine doses to save someone's life. Recent advances in vaccines against malaria and in work exploring the use of gene drives provide further hope that we could eradicate malaria from countries that are still affected.
    On World Malaria Day, we encourage you to donate to Giving What We Can’s fundraiser partnering with the Against Malaria Foundation and the Malaria Consortium. Malaria is preventable and treatable; a lack of resources leaves people personally affected by the disease or affected by the loss of loved ones. Your giving can directly impact the lives of those affected by malaria: if we reached the $1 million USD fundraising goal, we could directly prevent roughly 200 deaths from malaria. Put simply, this is an area where we really can make a difference.
    Plasmodium falciparum prevalence from 2000 to 2019. The decreasing amount of red, orange, and yellow represents the decreasing prevalence of one of the deadliest strains of malaria due to prevention efforts.
    Data from, animation idea by Sam Deere.
    Where we've been
    Malaria has been a part of human history for thousands of years, from infections in ancient Rome to the infections of several U.S. presidents. Early treatment for malaria came in the form of quinine from the cinchona tree, first isolated by French chemists in 1820, and was commonly administered in the form of tonic water or the gin and tonic. French surgeon Alphonse Laveran discovered the plasmodium parasite as the cause of malaria in 1880, opening up further research that would identify antimalarials like chloroquine and DDT.
    Proportion of deaths from malaria to deaths from all causes in the eastern United States, 1870 US Census. From Our World In Data/Statistical Atlas from the 9th Census of the United States 1870 (published 1874).
    Fighting malaria was the impetus for developing public health infrastructure in a number of countries. The predecessor to the United States Centers for Disease Control was the Office of Malaria Control in War Areas, designed to limit the impact of malaria during World War II around US military bases in the Southern United States (hence its headquarters in Atlanta, Georgia rather than Washington DC).
    Roughly half of the world’s countries have eliminated malaria in their territories through public health efforts, including some in tropical regions where malaria is most likely to be prevalent. 79 countries eliminated malaria from 1945 to 2010, and several more since 2010. Countries must achieve at least three consecutive y...
    10 min
  • LW - My Assessment of the Chinese AI Safety Community by Lao Mein
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Assessment of the Chinese AI Safety Community, published by Lao Mein on April 25, 2023 on LessWrong.
    I've heard people be somewhat optimistic about this AI guideline from China. They think that this means Beijing is willing to participate in an AI disarmament treaty due to concerns over AI risk. Eliezer noted that China is where the US was a decade ago in regards to AI safety awareness, and expresses genuine hope that his ideas of an AI pause can take place with Chinese buy-in.
    I also note that no one expressing these views understands China well. This is a PR statement. It is a list of feel-good statements that Beijing publishes after any international event. No one in China is talking about it. They're talking about how much the Baidu LLM sucks in comparison to ChatGPT. I think most arguments about how this statement is meaningful are based fundamentally on ignorance - "I don't know how Beijing operates or thinks, so maybe they agree with my stance on AI risk!"
    Remember that these are regulatory guidelines. Even if they all become law and are strictly enforced, they are simply regulations on AI data usage and training. Not a signal that a willingness for an AI-reduction treaty is there. It is far more likely that Beijing sees near-term AI as a potential threat to stability that needs to be addressed with regulation. A domestic regulation framework for nuclear power is not a strong signal for a willingness to engage in nuclear arms reduction.
    Maybe it is true that AI risk in China is where it was in the US in 2004. But the US 2004 state was also similar to the US 1954 state, so the comparison might not mean that much. And we are not Americans. Weird ideas are penalized a lot more harshly here. Do you really think that a scientist is going to walk up to his friend from the Politburo and say "Hey, I know AI is a central priority of ours, but there are a few fringe scientists in the US asking for treaties limiting AI, right as they are doing their hardest to cripple our own AI development. Yes, I believe they are acting in good faith, they're even promising to not widen the current AI gap they have with us!" Well, China isn't in this race for parity or to be second best. China wants to win. But that's for another post.
    Remember that Chinese scientists are used to interfacing with our Western counterparts and know to say the right words like "diversity", "inclusion", and "no conflict of interest" that it takes to get our papers published. Just because someone at Beida makes a statement in one of their papers doesn't mean the intelligentsia is taking this seriously. I've looked through the EA/Rationalist/AI Safety forums in China, and they're mostly populated by expats or people physically outside of China. Most posts are in English, and they're just repeating/translating Western AI Safety concepts. A "moonshot idea" I saw brought up is getting Yudkowsky's Harry Potter fanfiction translated into Chinese (please never ever do this). The only significant AI safety group is Anyuan(), and they're only working on field-building. Also, there is only one group doing technical alignment work in China, the founder was paying for everything out of pocket and was unable to navigate Western non-profit funding.
    I've still not figured out why he wasn't getting funding from Chinese EA people (my theory is that both sides assume that if funding was needed, the other side would have already contacted them).
    You can't just hope an entire field into being in China. Chinese EAs have been doing field-building for the past 5+ years, and I see no field. If things keep on this trajectory, it will be the same in 5 more years. The main reason I could find is the lack of interfaces, people who can navigate both the Western EA sphere and the Chinese technical sphere. In many ways, the very conce...
    5 min
  • AF - Making Nanobots isn't a one-shot process, even for an artificial superintelligance by dankrad
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Making Nanobots isn't a one-shot process, even for an artificial superintelligance, published by dankrad on April 25, 2023 on The AI Alignment Forum.
    Summary: Yudkowsky argues that an unaligned AI will figure out a way to create self-replicating nanobots, and merely having internet access is enough to bring them into existence. Because of this, it can very quickly replace all human dependencies for its existence and expansion, and thus pursue an unaligned goal, e.g. making paperclips, which will most likely end up in the extinction of humanity.
    I however will write below why I think this description massively underestimates the difficulty in creating self-replicating nanobots (even assuming that they are physically possible), which requires focused research in the physical domain, and is not possible without involvement of top-tier human-run labs today.
    Why it matters? Some of the assumptions of pessimistic AI alignment researchers, especially by Yudkowsky, rest fundamentally on the fact that the AI will find quick ways to replace humans required for the AI to exist and expand.
    We have to get AI alignment right the first time we build a Super-AI, and there are no ways to make any corrections after we've built it
    As long as the AI does not have a way to replace humans outright, even if its ultimate goal may be non-aligned, it can pursue proximate goals that are aligned and safe for it to do. Alignment research can continue and can attempt to make the AI fully aligned or shut it down before it can create nanobots.
    The first time we build a Super-AI, we don't just have to make sure it's aligned, but we need it to perform a pivotal act like create nanobots to destroy all GPUs
    I argue below that this framing may be bad because it means performing one of the most dangerous steps first — creating nanobots — which may be best performed by an AI that is much more aligned than a first attempt
    What this post is not about: I make no argument about the feasibility of (non-biological) self-replicating nanobots. There may be fundamental reasons why they are impossible/difficult (even for superintelligent AIs)/will not outcompete biological life, an interesting question that is explored more by bhaut. I also don't claim that AI alignment doesn't matter. I think that it's extremely important, but I think it's unlikely that (1) a one-shot process will lead to it and also (2) that one-shot is necessary; I actually think that this kind of thinking increases risk.
    Finally, I don't claim that there aren't easier ways to kill all, or almost all, humans, for example pandemics or causing nuclear wars. However, most of these scenarios do not leave any good paths for an AI to expand because there would be no way to get more of its substrate (e.g. GPUs) or power supplies.
    Core argument
    Building something like a completely new type of nanobot is a massive research undertaking. Even under the assumption that an AI is much more intelligent and can learn and infer from much less data, it cannot do so from no data.
    Building a new type of nanobot (not based on biological life) requires not just the ability to design from existing capabilities, but actually doing completely new experiments on how the nanomachinery that is going to be used to do this interacts with itself and the external world. It isn't possible to completely cut out all experiments from the design process, because at least some of the experiments will be about how the physical world works. If you don't know anything about physics, you clearly can't design any kind of machine; I am pretty certain that right now we do not know enough about nanomachines to design a new kind of non-biological self-replicating nanobot that immediately works out of the box.
    To build it, you would need high quality labs to do very well specified experiments...
    10 min
  • EA - Student competition for drafting a treaty on moratorium of large-scale AI capabilities R&D by Nayanika
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Student competition for drafting a treaty on moratorium of large-scale AI capabilities R&D, published by Nayanika on April 24, 2023 on The Effective Altruism Forum. has announced a competition for the drafting of an international treaty on moratorium of large-scale AI capabilities research and development.
    The competition is open to all students of law, philosophy, and other relevant disciplines. The competition is organized by the Campaign for AI Safety, an Australian unincorporated association of people who are concerned about the risks of AI.
    Competition brief: The goal of the competition is to operationalize the suggestions of the article Pausing AI Developments Isn’t Enough. We Need to Shut it All Down, including the provisions on:
    Shutting down large GPU and TPU clusters (the large computer farms where the most powerful AIs are refined).
    Prohibition of training ML models (or combinations of models) with more than 500 million parameters.
    Prohibition of the use of quantum computers in any AI-related activities.
    A general moratorium of large-scale AI capabilities research and development.
    Passing of national laws criminalizing the development of any form of Artificial General Intelligence (AGI) or Artificial Superintelligence (ASI).
    Establishment of an international body to oversee the treaty. Effective mechanisms for enforcement of the treaty.
    The treaty must not expire until it is universally agreed that it is safe and ethical to resume large-scale AI capabilities research and development.
    Deadline for submissions: 15 June 2023.
    Prizes: The winner will receive a prize of AUD 4000. The runner-up will receive a prize of AUD 1000. The third place will receive a prize of AUD 500.
    How to participate:
    1)Read the competition brief above.
    2)Draft a treaty: The treaty should be in English and should be no longer than 10 pages. The treaty should be submitted in Word format.
    3)Submit your draft: Please e-mail your draft to [email protected]. Please include your name, university, and country in the e-mail.
    4)Wait for the results: The results will be announced on 1 July 2023. Judging criteria
    The judges will evaluate the drafts based on the following criteria:
    Clarity: The treaty should be clear and easy to understand.
    Legality: The treaty should be legally binding.
    Effectiveness: The treaty should be effective in achieving its goals.
    Comprehensiveness: The treaty should cover all the relevant issues.
    Judges’ discretion: The judges may use their discretion in evaluating the drafts. Judges’ decision is final. The judges’ decision is final and cannot be appealed. Prizes will be awarded only if submissions meet basic quality requirements for treaty drafts. By submitting a draft, you agree to publication of your draft on this website and waiving copyright to your draft.
    Panel of judges: We are currently assembling a panel of judges. If you are a public law professor, please e-mail us to express your interest in judging the competition.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Deep learning models might be secretly (almost) linear by beren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deep learning models might be secretly (almost) linear, published by beren on April 24, 2023 on LessWrong.
    Crossposted from my personal blog.
    Epistemic status: Pretty speculative, but there is a surprising amount of circumstantial evidence.
    I have been increasingly thinking about NN representations and slowly coming to the conclusion that they are (almost) completely secretly linear inside. This means that, theoretically, if we can understand their directions, we can very easily exert very powerful control on the internal representations, as well as compose and reason about them in a straightforward way. Finding linear directions for a given representation would allow us to arbitrarily amplify or remove it and interpolate along it as desired. We could also then directly 'mix' it with other representations as desired. Measuring these directions during inference would let us detect the degree of each feature that the network assigns to a given input. For instance, this might let us create internal 'lie detectors' (which there is some progress towards) which can tell if the model is telling the truth, or being deceptive.
    While nothing is super definitive (and clearly networks are not 100% linear), I think there is a large amount of fairly compelling circumstantial evidence for this position. Namely:
    Evidence for this:
    All the work from way back when about interpolating through VAE/GAN latent space. I.e. in the latent space of a VAE on CelebA there are natural 'directions' for recognizable semantic features like 'wearing sunglasses' and 'long hair' and linear interpolations along these directions produced highly recognizable images
    Rank 1 or low rank editing techniques such as ROME work so well (not perfectly but pretty well). These are effectively just emphasizing a linear direction in the weights.
    You can apparently add and mix LoRas and it works about how you would expect.
    You can merge totally different models. People working with Stable Diffusion community literally additively merge model weights with weighted sum and it works!
    Logit lens works.
    SVD directions.
    Linear probing definitely gives you a fair amount of signal.
    Linear mode connectivity and git rebasin.
    Colin Burns' unsupervised linear probing method works even for semantic features like 'truth'.
    You can merge together different models finetuned from the same initialization.
    You can do a moving average over model checkpoints and this improves performance and is better than any individual checkpoint!
    The fact that linear methods work pretty tolerably well in neuroscience.
    Various naive diff based editing and control techniques work at all.
    General linear transformations between models and modalities.
    You can often remove specific concepts from the model by erasing a linear subspace.
    Task vectors (and in general things like finetune diffs being composable linearly)
    People keep finding linear representations inside of neural networks when doing interpretability or just randomly.
    If this is true, then we should be able to achieve quite a high level of control and understanding of NNs solely by straightforward linear methods and interventions. This would mean that deep networks might end up being pretty understandable and controllable artefacts in the near future. Just at this moment, we just have not yet found the right levers yet (or rather lots of existing work does show this but hasn't really been normalized or applied at scale for alignment). Linear-ish network representations are a best case scenario for both interpretability and control.
    For a mechanistic, circuits-level understanding, there is still the problem of superposition of the linear representations. However, if the representations are indeed mostly linear than once superposition is solved there seem to be little other obstacles in front of a com...
    7 min
  • LW - Power laws in Speedrunning and Machine Learning by Jsevillamol
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Power laws in Speedrunning and Machine Learning, published by Jsevillamol on April 24, 2023 on LessWrong.
    Anticipating the future of AI necessarily requires anticipating results in performance. Ideally, we would like to understand the dynamics of improvements. This would help us preempt capabilities and understand how plausible sudden improvements are.
    However, this problem is notoriously difficult. Among other reasons, Machine Learning benchmarks use many different metrics for measuring performance. And the history of improvements in all domains is limited, spanning around a dozen improvements in the longest-running benchmarks.
    To circumvent these problems, we follow Sevilla (2021) and study video game speedrunning. Using data from speedrun.com, we investigate a previously noted regularity in world record progressions - an astounding fit to a power law pattern.
    Exploiting this regularity, we develop a random effects model for predicting the size of successive record improvements. We show that this model is significantly better than a baseline of predicting no improvement, and has a performance comparable to a model fit to the whole dataset ex-post. We combine this model with a simple model for predicting the timing of records to solve an outstanding problem from Sevilla (2021) - beating a baseline of predicting no improvement when forecasting new records out to some time horizon.
    After studying the model in a data-rich environment, we adapt it to the more difficult case of predicting benchmarks in Machine Learning. Due to the lack of longitudinal data, we fail to provide definitive evidence of a power-law decay in Machine Learning benchmark improvements. However, we still show that it improves over a constant prediction baseline.
    Assuming a power-law decay, we show that the model suggests two interesting patterns for machine learning:
    Machine Learning benchmarks aren’t close to saturation. They exhibit a pattern of improvements more aggressive than what we will see once the benchmarks are close to their irreducible loss.
    Sudden large improvements are infrequent but aren’t ruled out. According to the model, improvements over one order-of-magnitude in size above recent improvements happen once every fifty times.
    Overall, this investigation provides tentative evidence on two key questions for AI forecasting, and sets the ground for further study of record dynamics in Machine Learning and other domains.
    Read the paper here.
    We thank Tamay Besiroglu and the rest of Epoch for discussing the paper and for their support.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - A Hypothetical Takeover Scenario Twitter Poll by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Hypothetical Takeover Scenario Twitter Poll, published by Zvi on April 24, 2023 on LessWrong.
    I ran an experimental poll sequence on Twitter a few weeks back, on the subject of what would happen if a hostile ASI (Artificial Superintelligence) tried to take over the world and kill all humans, using only ordinary known techs.
    The hope was that by exploring this, it would become more clear where people’s true objections and cruxes were. Surely, at least, one could establish that once the AI was fully an agent, fully uncontrolled, had a fully hostile goal, was definitively smarter and more capable than us and could scale, that we were obviously super dead? Or, if not, one could at least figure out what wrong intuitions were blocking that realization, and figure out what to do about that.
    This post is an experiment as well. I wouldn’t say the results were a full success. I would say they were interesting enough that they’re worth writing up, if only as an illustration of how I am thinking about some of these questions and potential rhetorical strategies. If you’re not interested in that, it is safe to skip this one.
    Oh No
    Consider the following scenario:
    There come into existence an ASI (Artificial Super-Intelligence), defined here roughly as an AI system that is generically smarter than us, has all the cognitive capacities and skills any human has displayed on the internet, can operate orders of magnitude faster than us, and that can copy itself, and which is on the internet with the ability to interact and gather resources.
    That ASI decides to do something that would kill us – it wants a configuration of atoms in the universe that doesn’t involve any humans being alive. The AI does not love you, the AI does not hate you, you are composed of atoms it can use for something else and the AI’s goal is something else. Or maybe it does want everyone dead, who knows, that is also possible.
    The ASI kills us.
    Many people, for a variety of reasons many of which are good, do not expect step one to happen any time soon, or do not expect step 2 to happen any time soon conditional on step 1 happening.
    This post is not about how likely it is that steps 1 and 2 will happen. It is about:
    Q: If steps 1 and 2 do happen – oh no! – how likely is it under various scenarios that step 3 happens, that we all die?
    A remarkably common answer to this question is no, we will be fine.
    I don’t understand this. To me you have so obviously already lost once:
    The ASI has is loose on the internet.
    The ASI has access to enough resources that it can use them to get more.
    The ASI is much faster than you and is smarter than you.
    The ASI wants you to lose.
    It also seems obvious to me that such an AI would, if it desired to do so, in such a scenario, be able to kill all the people.
    Yet people demand to know exactly how it would do this, or they won’t believe it.
    Level the Playing Field a Bit
    By default, in such a scenario, you should expect to lose, lose fast and lose hard, due to the ASI that is smarter and more capable than you doing some combination of
    Making itself even smarter and more capable repeatedly until it is God-like AI (RSI: recursive self-improvement).
    The ASI doing things that you didn’t know how to physically do at all because it understands physics better than you (nanotech, synthetic biology, engineered viruses, new chip designs, or perhaps something you didn’t even conceive of or think was physically possible) that upend the whole game board.
    The ASI using super-advanced manipulation techniques like finding signals that directly hijack human brains or motor functions, or new highly effective undetectable brainwashing techniques, or anything like that.
    The general version of this, something you were not smart enough to figure out.
    Presumably, we can all agree that if one of these does indee...
    26 min
  • LW - The Toxoplasma of AGI Doom and Capabilities? by Robert AIZI
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Toxoplasma of AGI Doom and Capabilities?, published by Robert AIZI on April 24, 2023 on LessWrong.
    [Epistemic Status: I'm confident that the individual facts I lay out support the main claim, but I'm not fully confident its enough evidence to make a true or useful framework for understanding the world.]
    I'm going to give seven pieces of evidence to support this claim:
    AI Doomerism helps accelerate AI capabilities, and AI capabilities in turn proliferate the AI Doomerism meme.
    If these dynamics exist, they'd be not unlike the Toxoplasma of Rage. Here's my evidence:
    Sam Altman claims Eliezer "has IMO done more to accelerate AGI than anyone else":
    Technical talent who hear about AI doom might decide capabilities are technically sweet, or a race, or inevitable, and decide to work on it for those reasons (doomer -> capabilities transmission).
    Funders and executives who hear about AI doom might decide capabilities are a huge opportunity, or disruptive, or inevitable, and decide to fund it for those reasons (doomer -> capabilities transmission).
    Capabilities amplifies the memetic relevance of doomerism (capabilities -> doomer transmission).
    AI Doomerism says we should closely follow capabilities updates, discuss them, etc.
    Capabilities and doomerism gain and lose social status together - Eliezer Yudkowsky has been writing about doom for a long time, but got a Time article and TED talk only after significant capabilities advances.
    Memes generally benefit from conflict, and doomerism and capabilities can serve as adversaries for this purpose.
    I've been trying to talk about "AI doomerism" here as a separate meme than "AI safety", respectively something like "p(doom) is very large" and "we need to invest heavily into AI safety work", though these are obviously related and often cooccur. One could no doubt make a similar case for AI safety and capabilities supporting each other, but I think the evidence I listed above applies mostly to AI doom claims (if one uses Eliezer as synecdoche for AI doomerism, which I think is reasonable).
    I hope with this post I'm highlighting a something that is a combination of true and useful. Please keep in mind that the truth values of "AI doom is in a toxoplasma relationship with AI capabilities" and "AI doom is right" are independent.
    This post was inspired by one striking line Jan_Kulveit's helpful Talking publicly about AI risk:
    the AGI doom memeplex has, to some extent, a symbiotic relationship with the race toward AGI memeplex
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - AGI ruin mostly rests on strong claims about alignment and deployment, not about society by Rob Bensinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI ruin mostly rests on strong claims about alignment and deployment, not about society, published by Rob Bensinger on April 24, 2023 on LessWrong.
    Dustin Moskovitz writes on Twitter:
    My intuition is that MIRI's argument is almost more about sociology than computer science/security (though there is a relationship). People won't react until it is too late, they won't give up positive rewards to mitigate risk, they won't coordinate, the govt is feckless, etc.
    And that's a big part of why it seems overconfident to people, bc sociology is not predictable, or at least isn't believed to be.
    And Stefan Schubert writes:
    I think it's good @robbensinger wrote a list of reasons he expects AGI ruin. It's well-written.
    But it's notable and symptomatic that 9/10 reasons relate to the nature of AI systems and only 1/10 (discussed in less detail) to the societal response.
    Whatever one thinks the societal response will be, it seems like a key determinant of whether there'll be AGI ruin.
    Imo the debate on whether AGI will lead to ruin systematically underemphasises this factor, focusing on technical issues.
    It's useful to distinguish between warnings and all-things-considered predictions in this regard.
    When issuing warnings, it makes sense to focus on the technology itself. Warnings aim to elicit a societal response, not predict it.
    But when you actually try to predict what'll happen all-things-considered, you need to take the societal response into account in a big way
    As such I think Rob's list is better as a list of reasons we ought to take AGI risk seriously, than as a list of reasons it'll lead to ruin
    My reply is:
    It's true that in my "top ten reasons I expect AGI ruin" list, only one of the sections is about the social response to AGI risk, and it's a short section. But the section links to some more detailed discussions (and quotes from them in a long footnote):
    Four mindset disagreements behind existential risk disagreements in ML
    The inordinately slow spread of good AGI conversations in ML
    Inadequate Equilibria
    Also, discussing the adequacy of society's response before I've discussed AGI itself at length doesn't really work, I think, because I need to argue for what kind of response is warranted before I can start arguing that humanity is putting insufficient effort into the problem.
    If you think the alignment problem itself is easy, then I can cite all the evidence in the world regarding "very few people are working on alignment" and it won't matter.
    If you think a slowdown is unnecessary or counterproductive, then I can point out that governments haven't placed a ceiling on large training runs and you'll just go "So? Why should they?"
    Society's response can only be inadequate given some model of what's required for adequacy. That's a lot of why I factor out that discussion into other posts.
    More importantly, contra Dustin, I don't see myself as having strong priors or complicated models regarding the social situation.
    Eliezer Yudkowsky similarly says he doesn't have strong predictions about what governments or communities will do in this or that situation (beyond anti-predictions like "they probably won't do specific thing X that's wildly different from anything they've done before"):
    [Ngo][12:26]
    The other thing is that, for pedagogical purposes, I think it'd be useful for you to express some of your beliefs about how governments will respond to AI
    I think I have a rough guess about what those beliefs are, but even if I'm right, not everyone who reads this transcript will be
    [Yudkowsky][12:28]
    Why would I be expected to know that? I could talk about weak defaults and iterate through an unending list of possibilities.
    Thinking that Eliezer thinks he knows that to any degree of specificity feels like I'm being weakmanned!
    [Ngo][12:28]
    I'm not claiming you have any specifi...
    10 min