The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Forum + LW relationship: What is the effect? by Rockwell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Forum + LW relationship: What is the effect?, published by Rockwell on January 23, 2023 on The Effective Altruism Forum.
    At the risk of starting a messy discussion, I'm curious how folks feel the heavy linking between the EA Forum and LessWrong affects both content on the Forum itself and the EA community more generally.I know many people identify as part of both spaces (and I myself peruse LW), but I'm wondering if the connection has larger cultural effects, from normalized writing styles to a perhaps disproportionate rationalist representation. Thoughts?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - (Incorrectly) Overemphasizing Effective Careers by Davidmanheim
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: (Incorrectly) Overemphasizing Effective Careers, published by Davidmanheim on January 23, 2023 on The Effective Altruism Forum.
    Effective altruism can be compatible with most moral viewpoints, and there is nothing fundamental about effective altruism that requires it to be a near-exclusive focus. There is, however, an analysis of effective altruism that seems to implicitly disagree, sneaking in a near-complete focus on effectiveness via implicit assumptions in analysis. I think that this type of analysis, which is (incorrectly) assumed by many people I have spoken to in the community, is not a necessary conclusion, and in fact runs counter to the assumptions for fiscal effective altruism. Effective careers are great, but making them the only "real" way to be an Effective Altruist should be strongly rejected.
    The mistaken analysis goes as follows; if we are balancing priorities, and take a consequentialist view, we should prioritize our decisions on the basis of overall impact. However, effective altruism has shown that different interventions differ in their impact by orders of magnitude. Therefore, if we give any non-trivial weight to improving the world, then it is such a large impact, it will overbalance other considerations.
    This can be illustrated with a notional career choice model. In this model, someone has several different goals. Perhaps they wish to have a family, and think that impacts on their family is almost half of the total reason to pick a career, while their personal happiness is another almost-half. Finally, in line with the financial obligation to give 10% of their income, they “tithe” their career choice, assigning 10% of the weight to their positive impact on the broader world. Now, they must choose between an “effective career,” or working a typical office job.
    FactorFamilyPersonal HappinessBeneficenceOverallWeight45%45%10%100%PriorityRatingImpactRatingImpactRatingImpactRatingImpactEffective Career3/1032/1029/1010003.15102.5Office Job9/1099/1091/1018.28.2
    As the table illustrates, there are two ways to weigh the options; either rating how preferred the option is as a preference, or assessing its impact. The office job effectively only has impact via donations, while the effective career addresses a global need. The first method leads to choosing the office job, the second to choosing the effective career. In this way, the non-trivial weight put on impact becomes overwhelming. (I will note that this analysis often fails to account for another issue that Effective Altruism used to focus on more strongly, that of replaceability. But even assuming a neglected area, where replaceability is negligible, the totalizing critique obtains.)
    The equivalent fiscal analysis certainly fails; dedicating 10% of your money to effective causes does not imply that if the cause is very effective, it requires you to give more than 10% of your money. This is not to say that the second analysis is confused - but it does require accepting that under a sufficiently utilitarian viewpoint, where your decisions weight your benefit against others, even greatly prioritizing your own happiness creates a nearly-totalizing obligation to others. And that's not what Effective Altruism generally suggests.
    And to be clear, my claim is not particularly novel. To quote a recent EA Forum post from 80,000 hours: “It feels important that working to improve the world doesn’t prevent me from achieving any of the other things that are really significant to me in life — for example, having a good relationship with my husband and having close, long-term friendships.”
    It seems important, however, to clarify that in many scenarios it simply is not the case that an effective career requires anything like the degree of sacrifice that the example above implies. While charities and altruistic end...
    5 min
  • EA - My highly personal skepticism braindump on existential risk from artificial intelligence. by NunoSempere
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My highly personal skepticism braindump on existential risk from artificial intelligence., published by NunoSempere on January 23, 2023 on The Effective Altruism Forum.
    Summary
    This document seeks to outline why I feel uneasy about high existential risk estimates from AGI (e.g., 80% doom by 2070). When I try to verbalize this, I view considerations like
    selection effects at the level of which arguments are discovered and distributed
    community epistemic problems, and
    increased uncertainty due to chains of reasoning with imperfect concepts
    as real and important.
    I still think that existential risk from AGI is important. But I don’t view it as certain or close to certain, and I think that something is going wrong when people see it as all but assured.
    Discussion of weaknesses
    I think that this document was important for me personally to write up. However, I also think that it has some significant weaknesses:
    There is some danger in verbalization leading to rationalization.
    It alternates controversial points with points that are dead obvious.
    It is to a large extent a reaction to my imperfectly digested understanding of a worldview pushed around the ESPR/CFAR/MIRI/LessWrong cluster from 2016-2019, which nobody might hold now.
    In response to these weaknesses:
    I want to keep in mind that do want to give weight to my gut feeling, and that I might want to updating on a feeling of uneasiness rather than on its accompanying reasonings or rationalizations.
    Readers might want to keep in mind that parts of this post may look like a bravery debate. But on the other hand, I've seen that the points which people consider obvious and uncontroversial vary from person to person, so I don’t get the impression that there is that much I can do on my end for the effort that I’m willing to spend.
    Readers might want to keep in mind that actual AI safety people and AI safety proponents may hold more nuanced views, and that to a large extent I am arguing against a “Nuño of the past” view.
    Despite these flaws, I think that this text was personally important for me to write up, and it might also have some utility to readers.
    Uneasiness about chains of reasoning with imperfect concepts
    Uneasiness about conjunctiveness
    It’s not clear to me how conjunctive AI doom is. Proponents will argue that it is very disjunctive, that there are lot of ways that things could go wrong. I’m not so sure.
    In particular, when you see that a parsimonious decomposition (like Carlsmith’s) tends to generate lower estimates, you can conclude:
    That the method is producing a biased result, and trying to account for that
    That the topic under discussion is, in itself, conjunctive: that there are several steps that need to be satisfied. For example, “AI causing a big catastrophe” and “AI causing human exinction given that it has caused a large catastrophe” seem like they are two distinct steps that would need to be modelled separately,
    I feel uneasy about only doing 1.) and not doing 2.) I think that the principled answer might be to split some probability into each case. Overall, though, I’d tend to think that AI risk is more conjunctive than it is disjunctive
    I also feel uneasy about the social pressure in my particular social bubble. I think that the social pressure is for me to just accept Nate Soares’ argument here that Carlsmith’s method is biased, rather than to probabilistically incorporate it into my calculations. As in “oh, yes, people know that conjunctive chains of reasoning have been debunked, Nate Soares addressed that in a blogpost saying that they are biased”.
    I don’t trust the concepts
    My understanding is that MIRI and others’ work started in the 2000s. As such, their understanding of the shape that an AI would take doesn’t particularly resemble current deep learning approaches.
    In particular, I think that man...
    23 min
  • EA - Announcing Introductions for Collaborative Truth Seeking Tools by brook
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing Introductions for Collaborative Truth Seeking Tools, published by brook on January 23, 2023 on The Effective Altruism Forum.
    EA involves working together remotely a lot. Tools exist to make this experience better, but they may be underused.
    In particular, we're trying to do truth seeking. It's easier to do that when it's easy to explain your view, understand others', and quickly see agreements, disagreements and cruxes. In addition, taking some time to lay out your model clearly is probably good for your own thinking and understanding.
    This sequence will introduce tools for doing that, alongside tutorials & guidance. Each day's post will also have a challenge you can use the tool to solve, to get a feel for the tool and see if it's for you.
    We’ll also be running a short session each day in the EA GatherTown; if you want to try the tools as part of a group, make sure to come along!
    Why do this?
    EA involves a lot of collaborating, often remotely, and in particular communicating complex models or concepts.
    There are a lot of software tools available to help with this, but they don't seem to be used as often as they could be, or as broadly. Often, when tutorials exist, they're poorly written and/or not targeted at the way EAs would likely be using this tool.
    This project has 3 key goals:
    Increase the use of collaborative truth-seeking tools (so EAs can find truth better together)
    Improve how EAs are using these tools (by introducing them to the key features quickly)
    Save EAs time (searching for tutorials or looking through poorly-written documentation)
    What is it?
    I've collated tutorials (where they exist) and written or recorded them (where they don't) which I think are useful for EAs. There'll be a forum post each day for the next 7 week days, each consisting of video and text tutorials for either one complex tool or a handful of smaller tools.
    Sometimes posts will consist primarily of links elsewhere, if good tutorials already exist. To help people find tools which might be useful for a given project, there's also a wiki page with short summaries of tools.
    Here are the posts you can expect to see over the next [??]:
    Guesstimate: Why and How to Use it
    Visualisation of Probability Mass
    Squiggle: Why and How to Use it
    Loom: Why and How to Use it
    Excalidraw: Why and How to Use it
    Forecasting tools and Prediction Markets: Why and How
    Polis: Why and How to Use it
    What kinds of tasks are we talking about?
    To make this a little more concrete than 'collaborative truth seeking', here are some examples of tasks EAs might want to do which could benefit from the use of one of these tools:
    Get feedback on your model of the impact of a grant to provide malaria bed nets (guesstimate)
    Summarise a forum post visually (excalidraw)
    Summarise a forum post in video form (loom)
    View an aggregate prediction for when we might expect AGI before deciding which area of AI to work in, and how (metaculus)
    Compare theories of what kinds of interventions might be useful to reduce biorisk (excalidraw, guesstimate)
    Calculate the expected value of trialling meditating for 2 weeks (guesstimate)
    Estimate the success of an ongoing project providing access to birth control for monitoring and evaluation (squiggle)
    Record a 3 minute video summarising your progress instead of having an hour-long meeting with your supervisor (loom)
    Learn to quantitatively forecast with a short feedback loop (pastcasting)
    So, if you've tried to do tasks of this nature, keep an eye out! Tomorrow: Guesstimate, a tool for quantifying intuitions.
    As a final request, we’d also really appreciate any feedback you have about the tools or the posts, or if you have any suggestions for tools you think deserve to have posts made about them!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please ...
    4 min
  • EA - My thoughts on parenting and having an impactful career by 80000 Hours
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My thoughts on parenting and having an impactful career, published by 80000 Hours on January 23, 2023 on The Effective Altruism Forum.
    When my husband and I decided to have children, we didn’t put much thought into the broader social impact of the decision. We got together at secondary school and had been discussing the fact we were going to have kids since we were 18, long before we found effective altruism.
    We made the actual decision to have a child much later, but how it would affect our careers or abilities to help others still wasn’t a large factor in the decision. As with most people though, the decision has, in fact, had significant effects on our careers.
    Raising my son, Leo — now three years old — is one of the great joys of my life, and I’m so happy that my husband and I decided to have him. But having kids can be challenging for anyone, and there may be unique challenges for people who aim to have a positive impact with their careers.
    I’m currently the director of the one-on-one programme at 80,000 Hours and a fund manager for the Effective Altruism Infrastructure Fund. So I wanted to share my experience with parenting and working for organisations whose mission I care about deeply. Here are my aims:
    Give readers an example of a working parent who also thinks a lot about 80,000 Hours’ advice.
    Discuss some of the ways having kids is likely to affect the impact you have in your career, for people who want to consider that when deciding whether to have kids.
    Discuss challenges people might face in their careers related to having kids and how they might handle them.
    Help people feel less alone if they’re finding some of the standard parenting advice alienating — particularly any mothers who feel the literature tends to underestimate how much they care about their career.
    Write out some of the lessons I’ve learned and things I would have liked to have known beforehand (I still find some of this hard to keep in mind!).
    Start a conversation with the hope that other like-minded parents will share their lessons and suggestions.
    Highlight some of the ways the effective altruism community supports parents.
    Note different people find very different advice useful, and people’s situations vary greatly by how many children they have, whether they have a partner and what that person’s situation is like, what family help they have nearby, their socioeconomic condition, and so on. I’ve been very fortunate to live in a wealthy country like the UK with a lot of social support, and I’ve been paid well enough to always meet my needs. My experiences will be most relevant to people who are similarly situated.
    And some of what follows will be speculative, because I consider counterfactuals and possibilities that are inevitably uncertain. Also, my son is only three, so I have fairly limited experience. I’d love for others to contribute to this conversation and offer additional perspectives.
    Deciding whether to have children
    It feels important that working to improve the world doesn’t prevent me from achieving any of the other things that are really significant to me in life — for example, having a good relationship with my husband and having close, long-term friendships.
    Becoming a parent was another personal priority in my life. For that reason, I didn’t think much about how having a child would affect the impact I had over my life. While I think it’s important to consider how we can best have a positive impact on the world, I don’t think it’s required or practical to think we might have to give up some of the things that are most important to us in the name of impact.
    I did think about it some when considering whether to have more children. The potential negative effects on my ability to have an impact with my career counted against having any more kids, but my husband also was ...
    22 min
  • EA - Questions I ask myself when making decisions (or communicating) by Catherine Low
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Questions I ask myself when making decisions (or communicating), published by Catherine Low on January 23, 2023 on The Effective Altruism Forum.
    I work on CEA’s Community Health and Special Projects Team. But these thoughts are my own (apart from question 6 which I inherited from Nicole Ross).
    I have an internal list of questions that I often go through when I’m making a decision (including when that decision is whether or what to communicate to others).
    I often bring these questions up when I’m trying to help others in the EA community make decisions, and some folks seem to find them helpful. I also sometimes see people making (what I believe to be) sub-optimal decisions and one or other of these questions may have helped things go better.
    Maybe you’ll find them helpful too, and maybe you have your own questions you ask yourself (please share them in the comments!).
    I’m afraid they don’t tell you how to make a decision (sorry - I haven’t worked that bit out!), and they aren’t exhaustive. They are just questions that seem to help when I am making decisions - either by nudging me to generate new options, or to become more aware of consequences.
    1. “Is this really where the tradeoff bites?”
    Some decisions are framed as “A” or “B”. More decisions are framed as tradeoffs between amounts of A and amounts of B. Considering tradeoffs is very useful! But I often notice people assuming they’re making a decision under difficult tradeoffs between values A and B, when I see A and B as being values on two quite different axes. There are often alternatives where we can get more A without sacrificing B or vice versa, or we can get more A AND more B (i.e. there are Pareto improvements we can make).
    The A and B I’ve been thinking most about recently are (caricatured) “clearly stating what you believe” and “being sensitive to others”. I think Richard Chappell’s “Don’t be an Edgelord” explains this particular A and B well. But I’ve noticed the general pattern a lot.
    2. “What does my shoulder X think?” or. (higher cost) “what does X think?”
    You’re not always going to please everyone. That’s the way things are. But plenty of people seem to be surprised by the negative reaction they get from their decisions or words. This problem seems tractable.
    It would be nice if people could ask lots of people their opinion before making a decision, but that is costly (and choosing to ask is a decision in itself). So I’ve been trying to develop a range of people/groups that metaphorically pop up on my shoulder to help me think:
    Some individuals whose thinking I respect - some who I usually agree with, some who I regularly disagree with.
    Individuals or groups who would be affected by my decisions
    I’m sure my shoulder people OFTEN don’t match the real person’s thoughts. But I still find them helpful to create new perspectives.
    Some tips for developing or using shoulder people:
    If the stakes are high - don’t rely on your shoulder person. Talk to the real people, the real stakeholders, people who have a different background or a different way of thinking to you. This especially goes if you’ve noticed you are frequently surprised by people’s reactions to your actions or communications, or if you don’t know the stakeholders well.
    You don’t have to know the person/group to make a shoulder person - just read a bunch of what they’ve written. Some examples: I’ve often found insight from Julia Wise, so through reading her blog I created a shoulder Julia long before I started working with her. I’ve only met Habryka once or twice, but I often get new perspectives from reading their contributions on the Forum. This seems helpful, so I’d like to improve my shoulder Habryka.
    Before publishing something, read your writing several times with several different shoulder people in mind.
    Practice! One way to develop should...
    6 min
  • EA - There should be a public adversarial collaboration on AI x-risk by pradyuprasad
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There should be a public adversarial collaboration on AI x-risk, published by pradyuprasad on January 23, 2023 on The Effective Altruism Forum.
    I think that adversarial collaborations are a good way of understanding competing perspectives on an idea, especially if it is polarising or especially controversial.
    The term was first introduced by Daniel Kahneman. The basic idea is that two people with competing perspectives on an issue work together towards a joint belief. Two people working in good faith would be able to devise various experiments and discussions that clarify the idea and work towards a joint belief. (Kahneman uses the word "truth", but I think the word "belief" is more justified in this context).
    AI x-risk is a good place to have a public adversarial collaboration
    First the issue is especially polarising. The beliefs of people working on AI risk are that AI presents one of the greatest challenges to humanity's survival. On the other hand, AI research organisations by revealed preference (they're going full speed ahead on building AI capabilities) and stated preference (see this survey too) think the risk is much lower.
    In my opinion having an adversarial collaboration between a top AI safety person (who works on x-risk from AI) and someone who did not think that the x risks were substantial would have clear benefits.
    It would make the lines of disagreement clearer. To me, an outsider in the space it's not very clear where exactly people disagree and to what extent. This would clear that up and possibly provide a baseline for future debate to be based on.
    It would also legitimise x-risk concerns quite a bit if this was to be co-written by someone respected in the field.
    Finally, it would make both sides of the debate evaluate the other side clearly and see their own blindspots better. This would improve the overall epistemic quality of the AI x-risk debate.
    How could this go wrong?
    The main failure mode is that the parties writing it aren't doing it in good faith. If they're trying to write it out with the purpose of proving the other side wrong, it will fail terribly.
    The second failure mode is that the arguments for either sides are based too much on thought experiments and it is hard to find a resolution because there isn't much empirical grounding for either side. In Kahenman's example, even with actual experiments they could infer from, both parties couldn't agree with it for 8 years. That's entirely possible with this as well.
    Other key considerations
    Finding the right people from both sides of the debate might be more difficult than I assume. I think there are people who can do it (eg. Richard Ngo and Jacob Buckman have said that they have done it in private) and Boaz Bark and Ben Edelman have also published a thoughtful critique (although not an adversarial collaboration), but it maybe that they're too busy or aren't interested enough in doing it
    A similar version has been done before and this might risk duplicating it. I don't think this is the case because the debate was hard to follow and not explicitly written with the indent of finding a joint belief.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - What a compute-centric framework says about AI takeoff speeds - draft report by Tom Davidson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What a compute-centric framework says about AI takeoff speeds - draft report, published by Tom Davidson on January 23, 2023 on The Effective Altruism Forum.
    I’ve written a draft report on AI takeoff speeds, the question of how quickly AI capabilities might improve as we approach and surpass human-level AI. Will human-level AI be a bolt from the blue, or will we have AI that is nearly as capable many years earlier?
    Most of the analysis is from the perspective of a compute-centric framework, inspired by that used in the Bio Anchors report, in which AI capabilities increase continuously with more training compute and work to develop better AI algorithms.
    This post doesn’t summarise the report. Instead I want to explain some of the high-level takeaways from the research which I think apply even if you don’t buy the compute-centric framework.
    The framework
    h/t Dan Kokotajlo for writing most of this section
    This report accompanies and explains (h/t Epoch for building this!), a user-friendly quantitative model of AGI timelines and takeoff, which you can go play around with right now. (By AGI I mean “AI that can readily[1] perform 100% of cognitive tasks” as well as a human professional; AGI could be many AI systems working together, or one unified system.)
    Takeoff simulation with Tom’s best-guess value for each parameter.
    The framework was inspired by and builds upon the previous “Bio Anchors” report. The “core” of the Bio Anchors report was a three-factor model for forecasting AGI timelines:
    Dan’s visual representation of Bio Anchors report
    Compute to train AGI using 2020 algorithms. The first and most subjective factor is a probability distribution over training requirements (measured in FLOP) given today’s ideas. It allows for some probability to be placed in the “no amount would be enough” bucket.
    The probability distribution is shown by the coloured blocks on the y-axis in the above figure.
    Algorithmic progress. The second factor is the rate at which new ideas come along, lowering AGI training requirements. Bio Anchors models this as a steady exponential decline.
    It’s shown by the falling yellow lines.
    Bigger training runs. The third factor is the rate at which FLOP used on training runs increases, as a result of better hardware and more $ spending. Bio Anchors assumes that hardware improves at a steady exponential rate.
    The FLOP used on the biggest training run is shown by the rising purple lines.
    Once there’s been enough algorithmic progress, and training runs are big enough, we can train AGI. (How much is enough? That depends on the first factor!)
    This draft report builds a more detailed model inspired by the above. It contains many minor changes and two major ones.
    The first major change is that algorithmic and hardware progress are no longer assumed to have steady exponential growth. Instead, I use standard semi-endogenous growth models from the economics literature to forecast how the two factors will grow in response to hardware and software R&D spending, and forecast that spending will grow over time. The upshot is that spending accelerates as AGI draws near, driving faster algorithmic (“software”) and hardware progress.
    The key dynamics represented in the model. “Software” refers to the quality of algorithms for training AI.
    The second major change is that I model the effects of AI systems automating economic tasks – and, crucially, tasks in hardware and software R&D – prior to AGI. I do this via the “effective FLOP gap:” the gap between AGI training requirements and training requirements for AI that can readily perform 20% of cognitive tasks (weighted by economic-value-in-2022). My best guess, defended in the report, is that you need 10,000X more effective compute to train AGI. To estimate the training requirements for AI that can readily perform x% of cognit...
    30 min
  • AF - What a compute-centric framework says about AI takeoff speeds - draft report by Tom Davidson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What a compute-centric framework says about AI takeoff speeds - draft report, published by Tom Davidson on January 23, 2023 on The AI Alignment Forum.
    As part of my work for Open Philanthropy I’ve written a draft report on AI takeoff speeds, the question of how quickly AI capabilities might improve as we approach and surpass human-level AI. Will human-level AI be a bolt from the blue, or will we have AI that is nearly as capable many years earlier?
    Most of the analysis is from the perspective of a compute-centric framework, inspired by that used in the Bio Anchors report, in which AI capabilities increase continuously with more training compute and work to develop better AI algorithms.
    This post doesn’t summarise the report. Instead I want to explain some of the high-level takeaways from the research which I think apply even if you don’t buy the compute-centric framework.
    The framework
    h/t Dan Kokotajlo for writing most of this section
    This report accompanies and explains (h/t Epoch for building this!), a user-friendly quantitative model of AGI timelines and takeoff, which you can go play around with right now. (By AGI I mean “AI that can readily[1] perform 100% of cognitive tasks” as well as a human professional; AGI could be many AI systems working together, or one unified system.)
    Takeoff simulation with Tom’s best-guess value for each parameter.
    The framework was inspired by and builds upon the previous “Bio Anchors” report. The “core” of the Bio Anchors report was a three-factor model for forecasting AGI timelines:
    Dan’s visual representation of Bio Anchors report
    Compute to train AGI using 2020 algorithms. The first and most subjective factor is a probability distribution over training requirements (measured in FLOP) given today’s ideas. It allows for some probability to be placed in the “no amount would be enough” bucket.
    The probability distribution is shown by the coloured blocks on the y-axis in the above figure.
    Algorithmic progress. The second factor is the rate at which new ideas come along, lowering AGI training requirements. Bio Anchors models this as a steady exponential decline.
    It’s shown by the falling yellow lines.
    Bigger training runs. The third factor is the rate at which FLOP used on training runs increases, as a result of better hardware and more $ spending. Bio Anchors assumes that hardware improves at a steady exponential rate.
    The FLOP used on the biggest training run is shown by the rising purple lines.
    Once there’s been enough algorithmic progress, and training runs are big enough, we can train AGI. (How much is enough? That depends on the first factor!)
    This draft report builds a more detailed model inspired by the above. It contains many minor changes and two major ones.
    The first major change is that algorithmic and hardware progress are no longer assumed to have steady exponential growth. Instead, I use standard semi-endogenous growth models from the economics literature to forecast how the two factors will grow in response to hardware and software R&D spending, and forecast that spending will grow over time. The upshot is that spending accelerates as AGI draws near, driving faster algorithmic (“software”) and hardware progress.
    The key dynamics represented in the model. “Software” refers to the quality of algorithms for training AI.
    The second major change is that I model the effects of AI systems automating economic tasks – and, crucially, tasks in hardware and software R&D – prior to AGI. I do this via the “effective FLOP gap:” the gap between AGI training requirements and training requirements for AI that can readily perform 20% of cognitive tasks (weighted by economic-value-in-2022). My best guess, defended in the report, is that you need 10,000X more effective compute to train AGI. To estimate the training requirements for AI th...
    30 min
  • AF - What a compute-centric framework says about AI takeoff speeds - draft report by Tom Davidson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What a compute-centric framework says about AI takeoff speeds - draft report, published by Tom Davidson on January 23, 2023 on The AI Alignment Forum.
    As part of my work for Open Philanthropy I’ve written a draft report on AI takeoff speeds, the question of how quickly AI capabilities might improve as we approach and surpass human-level AI. Will human-level AI be a bolt from the blue, or will we have AI that is nearly as capable many years earlier?
    Most of the analysis is from the perspective of a compute-centric framework, inspired by that used in the Bio Anchors report, in which AI capabilities increase continuously with more training compute and work to develop better AI algorithms.
    This post doesn’t summarise the report. Instead I want to explain some of the high-level takeaways from the research which I think apply even if you don’t buy the compute-centric framework.
    The framework
    h/t Dan Kokotajlo for writing most of this section
    This report accompanies and explains (h/t Epoch for building this!), a user-friendly quantitative model of AGI timelines and takeoff, which you can go play around with right now. (By AGI I mean “AI that can readily[1] perform 100% of cognitive tasks” as well as a human professional; AGI could be many AI systems working together, or one unified system.)
    Takeoff simulation with Tom’s best-guess value for each parameter.
    The framework was inspired by and builds upon the previous “Bio Anchors” report. The “core” of the Bio Anchors report was a three-factor model for forecasting AGI timelines:
    Dan’s visual representation of Bio Anchors report
    Compute to train AGI using 2020 algorithms. The first and most subjective factor is a probability distribution over training requirements (measured in FLOP) given today’s ideas. It allows for some probability to be placed in the “no amount would be enough” bucket.
    The probability distribution is shown by the coloured blocks on the y-axis in the above figure.
    Algorithmic progress. The second factor is the rate at which new ideas come along, lowering AGI training requirements. Bio Anchors models this as a steady exponential decline.
    It’s shown by the falling yellow lines.
    Bigger training runs. The third factor is the rate at which FLOP used on training runs increases, as a result of better hardware and more $ spending. Bio Anchors assumes that hardware improves at a steady exponential rate.
    The FLOP used on the biggest training run is shown by the rising purple lines.
    Once there’s been enough algorithmic progress, and training runs are big enough, we can train AGI. (How much is enough? That depends on the first factor!)
    This draft report builds a more detailed model inspired by the above. It contains many minor changes and two major ones.
    The first major change is that algorithmic and hardware progress are no longer assumed to have steady exponential growth. Instead, I use standard semi-endogenous growth models from the economics literature to forecast how the two factors will grow in response to hardware and software R&D spending, and forecast that spending will grow over time. The upshot is that spending accelerates as AGI draws near, driving faster algorithmic (“software”) and hardware progress.
    The key dynamics represented in the model. “Software” refers to the quality of algorithms for training AI.
    The second major change is that I model the effects of AI systems automating economic tasks – and, crucially, tasks in hardware and software R&D – prior to AGI. I do this via the “effective FLOP gap:” the gap between AGI training requirements and training requirements for AI that can readily perform 20% of cognitive tasks (weighted by economic-value-in-2022). My best guess, defended in the report, is that you need 10,000X more effective compute to train AGI. To estimate the training requirements for AI th...
    30 min