The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - How we could stumble into AI catastrophe by Holden Karnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How we could stumble into AI catastrophe, published by Holden Karnofsky on January 16, 2023 on The Effective Altruism Forum.
    This post will lay out a couple of stylized stories about how, if transformative AI is developed relatively soon, this could result in global catastrophe. (By “transformative AI,” I mean AI powerful and capable enough to bring about the sort of world-changing consequences I write about in my most important century series.)
    This piece is more about visualizing possibilities than about providing arguments. For the latter, I recommend the rest of this series.
    In the stories I’ll be telling, the world doesn't do much advance preparation or careful consideration of risks I’ve discussed previously, especially re: misaligned AI (AI forming dangerous goals of its own).
    People do try to “test” AI systems for safety, and they do need to achieve some level of “safety” to commercialize. When early problems arise, they react to these problems.
    But this isn’t enough, because of some unique challenges of measuring whether an AI system is “safe,” and because of the strong incentives to race forward with scaling up and deploying AI systems as fast as possible.
    So we end up with a world run by misaligned AI - or, even if we’re lucky enough to avoid that outcome, other catastrophes are possible.
    After laying these catastrophic possibilities, I’ll briefly note a few key ways we could do better, mostly as a reminder (these topics were covered in previous posts). Future pieces will get more specific about what we can be doing today to prepare.
    Backdrop
    This piece takes a lot of previous writing I’ve done as backdrop. Two key important assumptions (click to expand) are below; for more, see the rest of this series.
    In the most important century series, I argued that the 21st century could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.
    I focus on a hypothetical kind of AI that I call PASTA, or Process for Automating Scientific and Technological Advancement. PASTA would be AI that can essentially automate all of the human activities needed to speed up scientific and technological advancement.
    Using a variety of different forecasting approaches, I argue that PASTA seems more likely than not to be developed this century - and there’s a decent chance (more than 10%) that we’ll see it within 15 years or so.
    I argue that the consequences of this sort of AI could be enormous: an explosion in scientific and technological progress. This could get us more quickly than most imagine to a radically unfamiliar future.
    I’ve also argued that AI systems along these lines could defeat all of humanity combined, if (for whatever reason) they were aimed toward that goal.
    For more, see the most important century landing page. The series is available in many formats, including audio; I also provide a summary, and links to podcasts where I discuss it at a high level.-->
    It’s hard to talk about risks from transformative AI because of the many uncertainties about when and how such AI will be developed - and how much the (now-nascent) field of “AI safety research” will have grown by then, and how seriously people will take the risk, etc. etc. etc. So maybe it’s not surprising that estimates of the “misaligned AI” risk range from ~1% to ~99%.
    This piece takes an approach I call nearcasting: trying to answer key strategic questions about transformative AI, under the assumption that such AI arrives in a world that is otherwise relatively similar to today's.
    You can think of this approach like this: “Instead of asking where our ship will ultimately end up, let’s start by asking what destination it’s poi...
    48 min
  • EA - EA Organization Updates: January 2023 by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Organization Updates: January 2023, published by Lizka on January 16, 2023 on The Effective Altruism Forum.
    These monthly posts originated as the "Updates" section of the EA Newsletter. Organizations submit their own updates, which we edit for clarity.
    Job listings that these organizations highlighted (as well as a couple of other impactful jobs) are at the top of this post. Some of the jobs have pressing deadlines.
    You can see previous updates on the "EA Organization Updates (monthly series)" topic page, or in our repository of past newsletters. Notice that there’s also an “org update” tag, where you can find more news and updates that are not part of this consolidated series.
    The organizations are in alphabetical order, starting with L-Z, 0-A-K.
    Job listings
    Consider also exploring jobs listed on “Job listing (open).”
    GiveWell
    Senior Researcher (Remote / Oakland, CA, $181,400 - $199,800)
    Senior Research Associate (Remote / Oakland, CA, $127,000 - $139,900)
    Content Editor (Remote / Oakland, CA, $83,500 - $91,900)
    Global Priorities Institute
    Operations Coordinator (Maternity Cover) (Oxford, £29,614 - £35,326, apply by 24 January)
    IDinsight
    2023 Associate & Senior Associate Global Drive (Multiple locations)
    Technical Delivery Manager/Director (New Delhi, India or Nairobi, Kenya)
    Associate Product Manager (New Delhi, India or Nairobi, Kenya)
    Open Philanthropy
    Assorted jobs in Salesforce administration, operations, and recruiting (Remote; working hours must overlap with US hours for most roles. Salary range $84,303 - $127,021 across all jobs)
    Rethink Priorities
    Board Member (Remote, voluntary roles entail 3-10 hours/month while paid roles require 5-10 hours/week at a rate of $40.53/hour, apply by 20 January)
    Wild Animal Initiative
    Development Director (Remote, US preferred, open to UK applicants, $82,020 - $100,247, apply by 23 January)
    Organizational updates
    These are in alphabetical order, starting with L-Z, 0-A-K.
    Legal Priorities Project
    LPP’s Eric Martínez and Christoph Winter published a new working paper titled “Ordinary meaning of existential risk” investigating the ordinary meaning of legally relevant concepts in the existential risk literature. The paper aims to provide crucial insights for those tasked with drafting and interpreting existential risk laws, and for the coherence of ordinary meaning analysis more generally.
    José Villalobos and Christoph Winter participated in EAGxLatinAmerica. They hosted a Q&A on international law and existential risk.
    Matthijs Maas published a blog post titled “Existential risk mitigation: What I worry about when there are only bad options” as part of Draft Amnesty Day.
    LPP received a grant of $115,000 from the Survival and Flourishing Fund to support their general operations.
    One for the World
    One for the World mirrors the recommendations made by GiveWell for their own Nonprofit Partners portfolio. This year, GiveWell has updated its portfolio to contain a smaller list of nonprofits than before.
    In practice, this means that their Nonprofit Partners list has temporarily become much smaller, containing just four individual nonprofits. These nonprofits continue to offer gold-standard evidence that their method works and is incredibly cost-effective: Against Malaria Foundation, Malaria Consortium, New Incentives, and Helen Keller International.
    They are also adding a new option upon taking the 1% Pledge, which is GiveWell’s new All Grants Fund. This Fund will continue to make higher-risk grants, potentially including grants to nonprofits removed from GiveWell’s recommended nonprofits list. One for the World therefore thinks this gives donors the best chance to continue supporting a wider variety of granting opportunities.
    Open Philanthropy
    Open Philanthropy pre-announced its AI Worldviews Contest, which will launch in early 20...
    19 min
  • EA - Can GPT-3 produce new ideas? Partially automating Robin Hanson and others by NunoSempere
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Can GPT-3 produce new ideas? Partially automating Robin Hanson and others, published by NunoSempere on January 16, 2023 on The Effective Altruism Forum.
    Brief description of the experiment
    I asked a language model to replicate a few patterns of generating insight that humanity hasn't really exploited much yet, such as:
    Variations on "if you never miss a plane, you've been spending too much time at the airport".
    Variations on the Robin Hanson argument of "for common human behaviour X, its usual purported justification is Y, but it usually results in more Z than Y. If we cared about Y, we might do A instead".
    Variations on the genealogical argument: that the results of historical accidents are most likely not moral necessities or optimal systems.
    Motivation behind this experiment
    One of reasons to be afraid of artificial intelligence might be because, if you think in the abstract about how a system might behave as it becomes extremely intelligent, you might conclude that it might be able to completely outmanoeuvre us because of its superior ability to grasp the true structure of the world.
    This possibility is scary in the same sense that a modern chemist is scary to a historical alchemist. Our current chemist can completely outmanoeuvre previous alchemists by using their superior understanding of natural laws to produce better explosions, more subtle poisons, or more addictive and mind-blowing drugs.
    I do buy this fear in the limit for a being of God-like intelligence. But it's not clear to me whether it also applies to current systems or whether it will apply to their close descendants. In particular language models seem like they are powerful remixers and predictors but perhaps limited to drawing from the conceptual toolkit which humans already have. On the other hand, because they have access to so much information, they might be able to be prompted so as to reveal new relationships, connections, and insights.
    Some conceptual insights which have been historically important are:
    Explaining natural phenomena not in terms of Greek or Roman anthropomorphic gods, but with reference to naturalistic, physical explanations
    Understanding acceleration as distinct from motion
    Science as an experimental methodology
    The is/ought distinction
    Bayesian reasoning
    Ceasing to accept the divine right of kings as a justification for monarchical governance
    Randomized trials as a more robust way of generating generalizable knowledge
    The genealogical argument: understanding that systems (such as the details of the current prison system, our monetary system, the lack of color in men's clothes, or our attitudes towards gender and sex) are the result of historical accidents which could have gone differently. But often these systems are rationalized as being particularly adequate, or even morally necessary.
    But I don't think that language models are currently able to come up with original insights like the above from scratch (this would be very scary).
    Instead, I probe GPT-3's ability to come up with original variations of these three argumentative patterns:
    Variations on "if you never miss a plane, you've been spending too much time at the airport".
    Variations on the Robin Hanson argument of "for common human behaviour X, its usual purported justification is Y, but it usually results in more Z than Y. If we cared about Y, we might do A instead".
    Variations on the genealogical argument: that the results of historical accidents are most likely not moral necessities or optimal systems.
    The first pattern is known as an Umeshism. I associate the second pattern with Robin Hanson, who has had part of a fruitful career exploring some of its variations—though he is also known for other ideas, e.g., prediction markets and grabby aliens. I associate the third pattern with Nietzsche (who used it to criti...
    16 min
  • LW - We Need Holistic AI Macrostrategy by NickGabs
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We Need Holistic AI Macrostrategy, published by NickGabs on January 15, 2023 on LessWrong.
    Summary
    AI Macrostrategy is the study of high level questions having to do with prioritizing the use of resources on the current margin in order to achieve good AI outcomes. AI macrostrategy seems important if it is tractable. However, while few people are working on estimating particular parameters relevant to macrostrategy, even fewer are working on developing holistic macrostrategic models that combine estimates for different parameters to guide our actions. Moreover, while macrostrategy was less tractable in the past, recent developments (especially increased evidence for <10 year timelines) have made macrostrategy substantially more tractable. Thus, using the importance/tractability/neglectedness heuristics from EA, I conclude that on current margins macrostrategy should be a top priority.
    Acknowledgements:
    Thanks to Chinmay Deshpande, Carson Ezell, Nikola Jurkovic, and others for helping me to develop many of the ideas in this post, and to Thomas Larsen for both doing this and helping to directly edit the post.
    Epistemic status:
    Speculative, but I think the arguments are pretty straightforward and so I have >70% confidence in the main conclusion that more macrostrategy work should be done on current margins relative to other kinds of alignment work.
    What is AI Macrostrategy?
    AI Macrostrategy (henceforth just macrostrategy) is the study of high level questions having to do with prioritizing the use of resources to achieve good AGI outcomes on the current margin.
    Macrostrategic work can be divided broadly into two categories:
    Parameter estimates: attempts to forecast key variables such as timelines, takeoff speeds, and the difficulty of aligning AGI
    Holistic macrostrategy: attempts to combine these estimates and other pieces of data into a coherent, action-guiding model of AI alignment.
    For examples of macrostrategic questions, Holden mentions several central macrostrategic questions in this post.
    Importance of Macrostrategy
    I think that attempting to answer macrostrategic questions is extremely important for four primary reasons.
    Importance of Prioritization in Heavy Tailed Domains
    It is widely accepted among Effective Altruists that the distribution of impactfulness among different cause areas is heavy tailed. However, while I expect that the distribution of impactfulness among different AI interventions is not as heavy tailed as the distribution of impactfulness among cause areas in general, I do expect it to be at least somewhat heavy tailed, with the best interventions being >2 orders of magnitude more effective in expectation than the median intervention. Thus, it is critical to identify the best interventions rather than settling for interventions that seem vaguely pointed in the direction of solving alignment/making AGI go well. However, identifying these interventions requires some kind of macrostrategic model. Thus, applying the basic heuristic that prioritization is important in heavy tailed domains already suggests that macrostrategy is quite important.
    Achieving The Best Long Term Outcomes Requires Macrostrategy
    In addition to the distribution of the impactfulness of different AI interventions, the distribution of value across possible long run futures is also likely to be heavy tailed. This is because if what happens in the long run future will be controlled by powerful optimizers such as superintelligent AI, then due to the fact that tails come apart, most of the expected value relative to a particular utility function lies in futures where the powerful optimizers controlling the future in question are optimizing that specific utility function (or something extremely close to it). As a result, if you have consequentialist values, you should be focused on tryi...
    13 min
  • EA - Don’t Balk at Animal-friendly Results by Bob Fischer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don’t Balk at Animal-friendly Results, published by Bob Fischer on January 16, 2023 on The Effective Altruism Forum.
    Key Takeaways
    The Moral Weight Project assumes hedonism. It also assumes that in the absence of good direct measures of the intensities of valenced experiences, the best way to assess differences in the potential intensities of animals’ valenced experiences is to look at differences in other capacities that might serve as proxies for differences in hedonic potential.
    Suppose that these assumptions lead to the conclusion that chickens and humans can realize roughly the same amount of welfare at any given time. Call this “the Equality Result.” The key question: Would the Equality Result alone be a good reason to think that one or both of these assumptions is mistaken?
    We don’t think so. To explain why not, we consider three bases for skepticism about the Equality Result. Then, we consider whether the Equality Result should be surprising given hedonism.
    Three Bases for Skepticism
    First, someone might balk at the implications of the Equality Result given certain independent theses. For instance, given utilitarianism, the Equality Result probably implies that there should be a massive shift in neartermist resources toward animals, and someone might find this unbelievable. But the Equality Result isn’t to blame: utilitarianism is.
    Second, someone might be inclined to accept some theory of welfare that does not support the Equality Result. Fair enough, but as we’ve argued elsewhere, that only gets you so far.
    Third, someone might balk at the Equality Result even given hedonism. The basic problem with this is that the anti-Equality-Result intuition is uncalibrated, uncalibrated intuitions are vulnerable to various biases, and there are some highly relevant biases in the present context.
    If Hedonism, Then the Equality Result Shouldn’t Be Surprising
    We quickly consider three popular theories valenced states and argue that there are plausible assumptions on which each one leads to the Equality Result.
    This isn’t an argument for the Equality Result. It is, however, a check against knee-jerk skepticism.
    Introduction
    This is the seventh post in the Moral Weight Project Sequence. The aim of the sequence is to provide an overview of the research that Rethink Priorities conducted between May 2021 and October 2022 on interspecific cause prioritization—i.e., making resource allocation decisions across species.
    As EAs, we want to compare the cost-effectiveness of all interventions, including ones that benefit (sentient) animals with the cost-effectiveness of interventions that benefit humans. To do that, we need to estimate the value of each kind of animal relative to humans. If we understand each individual’s value in terms of the welfare they generate, whether positive or negative, then this means we need to estimate the amount of welfare that each kind of animal realizes relative to the amount of welfare that humans realize.
    How should we react if our method for generating these estimates produces a surprising result? Suppose, for instance, that we make various assumptions and generate a method for estimating how much welfare animals can realize relative to how much welfare humans can realize (which is the first step toward estimating how much welfare animals actually realize). And suppose that, when applied, our method suggests that chickens and humans can realize roughly the same amount of welfare at any given time. Call this “the Equality Result.” Would getting the Equality Result itself be a reason to think we made a mistake?
    Let’s make this concrete. The Moral Weight Project assumes hedonism—i.e., that all and only positively valenced experiences are good for you and all and only negatively valenced experiences are bad for you. Moreover, it assumes that, in the ab...
    26 min
  • EA - 80,000 Hours career review: Information security in high-impact areas by 80000 Hours
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 80,000 Hours career review: Information security in high-impact areas, published by 80000 Hours on January 16, 2023 on The Effective Altruism Forum.
    This is a cross-post of a career review from the 80,000 Hours website written by Jarrah Bloomfield. See the original here.
    Introduction
    As the 2016 US presidential campaign was entering a fractious round of primaries, Hillary Clinton’s campaign chair, John Podesta, opened a disturbing email.[1] The March 19 message warned that his Gmail password had been compromised and that he urgently needed to change it.
    The email was a lie. It wasn’t trying to help him protect his account — it was a phishing attack trying to gain illicit access.
    Podesta was suspicious, but the campaign’s IT team erroneously wrote the email was “legitimate” and told him to change his password. The IT team provided a safe link for Podesta to use, but it seems he or one of his staffers instead clicked the link in the forged email. That link was used by Russian intelligence hackers known as “Fancy Bear,” and they used their access to leak private campaign emails for public consumption in the final weeks of the 2016 race, embarrassing the Clinton team.
    While there are plausibly many critical factors in any close election, it’s possible that the controversy around the leaked emails played a non-trivial role in Clinton’s subsequent loss to Donald Trump. This would mean the failure of the campaign’s security team to prevent the hack — which might have come down to a mere typo[2] — was extraordinarily consequential.
    These events vividly illustrate how careers in infosecurity at key organisations have the potential for outsized impact. Ideally, security professionals can develop robust practices that reduce the likelihood that a single slip-up will result in a significant breach. But this key component for the continued and unimpaired functioning of important organisations is often neglected.
    And the need for such protection stretches far beyond hackers trying to cause chaos in an election season. Information security is vital to safeguard all kinds of critical organisations such as those storing extremely sensitive data about biological threats, nuclear weapons, or advanced artificial intelligence, that might be targeted by criminal hackers or aggressive nation states. Such attacks, if successful, could contribute to dangerous competitive dynamics (such as arms races) or directly lead to catastrophe.
    Some infosecurity roles involve managing and coordinating organisational policy, working on technical aspects of security, or a combination of both. We believe many such roles have thus far been underrated among those interested in effective altruism and reducing global catastrophic risks, and we’d be excited to see more altruistically motivated candidates move into this field.
    In a nutshell
    Organisations with influence, financial power, and advanced technology are targeted by actors seeking to steal or abuse these assets. A career in information security is a promising avenue to support high-impact organisations by protecting against these attacks, which have the potential to disrupt an organisation’s mission or even increase existential risk.
    Jeffrey Ladish contributed to this career review. We also thank Wim van der Schoot for his helpful comments.
    Why might information security be a high-impact career?
    Information security protects against events that hamper an organisation’s ability to fulfil its mission, such as attackers gaining access to confidential information. Information security specialists play a vital role in supporting the mission of organisations, similar to roles in operations.
    So if you want an impactful career, expertise in information security could enable you to make a significant positive difference in the world by helping important organis...
    20 min
  • EA - Consider paying for literature or book reviews using bounties and dominant assurance contracts by Arjun Panickssery
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consider paying for literature or book reviews using bounties and dominant assurance contracts, published by Arjun Panickssery on January 15, 2023 on The Effective Altruism Forum.
    Cross-posted to LessWrong here.
    In September 2021, a LessWrong pilot program paid $500 for high-quality book reviews related to "science, history, and rationality." They got 36 submissions (they don't say how many were rejected for low quality, but at least 9 were accepted) and a bunch of them were popular on the forum.
    There's a bounty tag on LW (and on the EA Forum) but it isn't used much. A culture of posting bounties—either individually or in groups of people interested in the same information—has benefits for patrons, writers, and the community generally:
    For patrons—If there's a question you want investigated or a book you want reviewed, you can save your valuable time by tossing the question into the LW void and waiting for a piece that's worth accepting. Others probably want the same thing and can contribute to a pool. Ambitious patrons can also influence the direction of the community by sponsoring posts on topics they think people should think about more. You don't have to worry about filtering for visible credentials and writers don't have to worry about having any.
    For writers—Bounties motivate people to write both through a direct monetary incentive, but also because a lot of people dissuade themselves from writing on the Internet to avoid looking vain or self-important. Bounties cover for this awkwardness by providing non-status-related reasons to post.
    For the community—This whole exchange provides a positive externality to the lurkers who can read more posts for free.
    The simplest way this could work is for people to post individual bounties of e.g. $500 for posts drawing conclusions that would have taken them just too long to justify at the hourly value of their time. These bounties can guide writers who may be looking for things to read and write about anyway.
    An obstacle to bounty markets is that writers incur the risk of being outshone by a better post written around the same time. They could also be snubbed by picky benefactors. If most bounties are posted by a small group of people who post many individual bounties, then reputation effects can manage this. Group bounties could be difficult to coordinate since people aren't motivated to post "I'd contribute $100 to a post analyzing the top 10 writing advice books for insights" if the bounty is unlikely ever to be fulfilled. And they're not motivated to join existing pools when they could free-ride instead.
    One solution is to use the opposite direction: Kickstarter-style writing proposals. In monthly threads, users post advertisements for book reviews/literature reviews/investigations that they'd be willing to produce at some price and specifications. This puts the reputational demands on the writers and the risk on the sponsors who might pay for a poor product.
    In theory, writers could kickstart posts using dominant assurance contracts. An example (this is a real offer): If you send $20 to arjun.panickssery at Gmail via PayPal by noon New York time on January 21st, I'll send you back $25 if fewer than 10 people sent me money. If 10 or more people send me money, I'll post a review of Steven Pinker's The Sense of Style: The Thinking Person's Guide to Writing in the 21st Century by the end of the month. I'm not sure whether I'm just giving away free money right now.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • AF - Understanding the diffusion of large language models: summary by Ben Cottier
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding the diffusion of large language models: summary, published by Ben Cottier on January 16, 2023 on The AI Alignment Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - Consider paying for literature or book reviews using bounties and dominant assurance contracts by Arjun Panickssery
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consider paying for literature or book reviews using bounties and dominant assurance contracts, published by Arjun Panickssery on January 15, 2023 on LessWrong.
    Cross-posted to the EA Forum here.
    In September 2021, a LessWrong pilot program paid $500 for high-quality book reviews related to "science, history, and rationality." They got 36 submissions (they don't say how many were rejected for low quality, but at least 9 were accepted) and a bunch of them were popular on the forum.
    There's a bounty tag on LW (and on the EA Forum) but it isn't used much. A culture of posting bounties—either individually or in groups of people interested in the same information—has benefits for patrons, writers, and the community generally:
    For patrons—If there's a question you want investigated or a book you want reviewed, you can save your valuable time by tossing the question into the LW void and waiting for a piece that's worth accepting. Others probably want the same thing and can contribute to a pool. Ambitious patrons can also influence the direction of the community by sponsoring posts on topics they think people should think about more. You don't have to worry about filtering for visible credentials and writers don't have to worry about having any.
    For writers—Bounties motivate people to write both through a direct monetary incentive, but also because a lot of people dissuade themselves from writing on the Internet to avoid looking vain or self-important. Bounties cover for this awkwardness by providing non-status-related reasons to post.
    For the community—This whole exchange provides a positive externality to the lurkers who can read more posts for free.
    The simplest way this could work is for people to post individual bounties of e.g. $500 for posts drawing conclusions that would have taken them just too long to justify at the hourly value of their time. These bounties can guide writers who may be looking for things to read and write about anyway.
    An obstacle to bounty markets is that writers incur the risk of being outshone by a better post written around the same time. They could also be snubbed by picky benefactors. If most bounties are posted by a small group of people who post many individual bounties, then reputation effects can manage this. Group bounties could be difficult to coordinate since people aren't motivated to post "I'd contribute $100 to a post analyzing the top 10 writing advice books for insights" if the bounty is unlikely ever to be fulfilled. And they're not motivated to join existing pools when they could free-ride instead.
    One solution is to use the opposite direction: Kickstarter-style writing proposals. In monthly threads, users post advertisements for book reviews/literature reviews/investigations that they'd be willing to produce at some price and specifications. This puts the reputational demands on the writers and the risk on the sponsors who might pay for a poor product.
    In theory, writers could kickstart posts using dominant assurance contracts. An example (this is a real offer): If you send $20 to arjun.panickssery at Gmail via PayPal by noon New York time on January 21st, I'll send you back $25 if fewer than 10 people sent me money. If 10 or more people send me money, I'll post a review of Steven Pinker's The Sense of Style: The Thinking Person's Guide to Writing in the 21st Century by the end of the month. I'm not sure whether I'm just giving away free money right now.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • AF - Speculation on Path-Dependance in Large Language Models. by NickyP
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Speculation on Path-Dependance in Large Language Models., published by NickyP on January 15, 2023 on The AI Alignment Forum.
    Epistemic Status: Highly Speculative. I spent less than a day thinking about this in particular, and though I have spent a few months studying large language models, I have never trained a language model. I am likely wrong about many things. I have not seen research on this, so it may be useful for for someone to do a real deep dive.
    Thanks to Anthony from the Center on Long Term Risk for sparking the discussion earlier today for this post. Also thanks to conversations with Evan Hubinger ~1 year ago that got me thinking about the topic previously.
    Summary
    My vague suspicions at the moment are somewhat along the lines of:
    Training an initial model: moderate to low path-dependance
    Running a model: high "prompt-dependance"
    Reinforcement Learning a Model: moderate to high path-dependance.
    Definitions of "low" and "high" seem somewhat arbitrary, but I guess what I mean is how different behaviours of the model can be. I expect some aspects to be quite path dependant, and others not so much. This is trying to quantify how many aspects might have path-dependance based on vibe.
    Introduction
    Path dependence is thinking about the "butterfly effect" for machine learning models. For highly path-dependant models, small changes in how a model is trained can lead to big differences in how it performs. If a model is highly path-dependant, then if we want to understand how our model will behave and make sure it's doing what we want, we need to pay attention to the nitty-gritty details of the training process, like the order in which it's learning things, or the random weights initialisation. And, if we want to influence the final outcome, we have to intervene early on in the training process.
    I think having an understanding of path-dependance is likely useful, but have not really seen any empirical results on the topic. I think that in general, it is likely to depend on different training methods a lot, and in this post I will give some vague impressions I have on the path dependance of Large Language Models (LLMs).
    In this case, I will also include "prompt-dependance" as another form of "path-dependance" when it comes to the actual outputs of the models, though this is not technically correct since it does not depend on the actual training of the model.
    Initial Training of a Model
    My Understanding: Low to Moderate Path-Dependance
    So with Large Language Models at the moment, the main way they are trained it that you should have a very large dataset, randomise the order, and use each text exactly once. In practice, many datasets have a lot of duplicate data of things that are particularly common (possible example: transcripts of a well-know speech) though people try to avoid this. While this may seem there should be a large degree of path dependance, my general impression is that, at least in most current models, this does not happen that often. In general, LLMs can tend to struggle with niche facts, so I would perhaps expect that in some cases it learns a niche fact that it does not learn in another case, but the LLMs seems to be at least directionally accurate. (An example I have seen, is that it might say "X did Mathematics in Cambridge" instead of "X did Physics in Oxford", but compared to possibility space, it is not that far.)
    I suspect that having a completely different dataset would impact the model outputs significantly, but from my understanding of path dependance, this does not particularly fall under the umbrella of path dependance, since it is modelling a completely different distribution. Though even in this case, I would suspect that in text from categories in the overlapping distribution, that the models would have similar looking outputs (though ...
    11 min