The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Applications Open: CEA Career Planning Program Pilot by Cian M
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Applications Open: CEA Career Planning Program Pilot, published by Cian M on August 20, 2023 on The Effective Altruism Forum.
    The CEA Virtual Programs team is piloting a 5-week Career Planning Program! Apply by August 27th.
    Key info
    Dates: September 11th - October 15th, 2023
    Application deadline: August 27th
    Target Audience: Anyone who is considering pursuing a career in an EA cause area and who has an existing understanding of EA ideas.
    Content: Participants will complete an adapted version of 80,000 Hours' career planning template, and have weekly discussions with a facilitator and a group of peers interested in pursuing similar career paths.
    We think we can help you to plan your career by providing structure, accountability and feedback during your career planning process. Through a combination of reading, exercises and discussions, you'll explore what a high-impact career looks like for you, and start making progress on next steps. Find out more about the program here.
    We highly recommend this program if you:
    Are interested in finding a career that is high-impact and a good fit for you.
    Have a good understanding of core ideas in effective altruism and/or have completed an Introductory EA Program or In-Depth EA Program.
    Can commit at least 1.5 hours a week to exercises and readings, and 1.5 hours to discussion sessions.
    Note: Though the materials for this program are based on the 80,000 Hours 8-week career planning course and 80,000 Hours was consulted when creating this program, they are not the organisers. However, we would still encourage people to apply to 80,000 Hours 1:1 advising - the accountability and structure of this program makes a great compliment to 1:1 advising!
    Ready to apply? Find out more about the program here and register here by Sunday, August 27th.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • LW - "Dirty concepts" in AI alignment discourses, and some guesses for how to deal with them by Nora Ammann
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Dirty concepts" in AI alignment discourses, and some guesses for how to deal with them, published by Nora Ammann on August 20, 2023 on LessWrong.
    Meta: This is a short summary & discussion post of a talk on the same topic by Javier Gomez-Lavin, which he gave as part of the PIBBSS speaker series. The speaker series features researchers from both AI Alignment and adjacent fields studying intelligent behavior in some shape or form. The goal is to create a space where we can explore the connections between the work of these scholars and questions in AI Alignment.
    This post doesn't provide a comprehensive summary of the ideas discussed in the talk, but instead focuses on exploring some possible connections to AI Alignment. For a longer version of Gomez-Levin's ideas, you can check out a talk here.
    "Dirty concepts" in the Cognitive Sciences
    Gomez-Lavin argues that cognitive scientists engage in a form of "philosophical laundering," wherein they associate, often implicitly, philosophically loaded concepts (such as volition, agency, etc.) into their concept of "working memory."
    He refers to such philosophically laundered concepts as "dirty concepts" insofar as they conceal potentially problematic assumptions being made. For instance, if we implicitly assume that working memory requires, for example, volition, we have now stretched our conception of working memory to include all of cognition. But, if we do this, then the concept of working memory loses much of its explanatory power as one mechanism among others underlying cognition as a whole.
    Often, he claims, cognitive science papers will employ such dirty concepts in the abstract and introduction but will identify a much more specific phenomena being measured in the methods and results section.
    What to do about it? Gomez-Lavin's suggestion in the case of CogSci
    The pessimistic response (and some have suggested this) would be to quit using any of these dirty concept (e.g. agency) all together. However, it appears that this would amount to throwing the baby out with the bathwater.
    To help remedy the problem of dirty concepts in working memory literature, Gomez-Lavin proposes creating an ontology of the various operational definitions of working memory employed in cognitive science by mining a wide range of research articles. The idea is that, instead of insisting that working memory be operationally defined in a single way, we ought to embrace the multiplicity of meanings associated with the term by keeping track of them more explicitly.
    He refers to this general approach as "productive pessimism." It is pessimistic insofar as it starts from the assumption that dirty concepts are being problematically employed, but it is productive insofar as it attempts to work with this trend rather than fight against it.
    While it is tricky to reason with those fuzzy concepts, once we are rigorous about proposing working definitions / operationalization of these terms as we use them, we can avoid some of the main pitfalls and improve our definitions over time.
    Relevance to AI alignment?
    It seems fairly straightforward that AI alignment discourse, too, suffers from dirty concepts.
    If this is the case (and we think it is), a similar problem diagnosis (e.g. how dirty concepts can hamper research/intellectual progress) and treatment (e.g. ontology mapping) may apply.
    A central example here is the notion of "agency". Alignment researchers often speak of AI systems as agents. Yet, there are often multiple, entangled meanings intended when doing so. High-level descriptions of AI x-risk often exploit this ambiguity in order to speak about the problem in general, but ultimately imprecise terms. This is analogous to how cognitive scientists will often describe working memory in general terms in the abstract and operationalize the term only in the m...
    6 min
  • EA - Longtermism Fund: August 2023 Grants Report by Michael Townsend
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Longtermism Fund: August 2023 Grants Report, published by Michael Townsend on August 20, 2023 on The Effective Altruism Forum.
    Introduction
    In this grants report, the Longtermism Fund team is pleased to announce that the following grants have been recommended by Longview and are in the process of being disbursed:
    Two grants promoting beneficial AI:
    Supporting AI interpretability work conducted by Martin Wattenberg and Fernanda Viegas at Harvard University ($110,000 USD)
    Funding AI governance work conducted by the evaluations project at the Alignment Research Center ($220,000 USD)
    Two biosecurity and pandemic prevention grants:
    Supporting a specific project at NTI | Bio working to disincentivize biological weapons programmes ($100,000 USD)
    Partially funding the salary of a Director of Research and Administration for the Center for Communicable Disease Dynamics ($80,000 USD)
    One grant improving nuclear security:
    Funding a project by the Carnegie Endowment for International Peace to better understand and advocate for policies that avoid escalation pathways to nuclear war ($52,000 USD).
    This report will provide information on what the grants will fund, and why they were made. It was written by Giving What We Can, which is responsible for the Fund's communications. Longview Philanthropy is responsible for the Fund's research and grantmaking.
    We would also like to acknowledge and apologise for the report being released two months later than we would have liked, in part due to delays in the process of disbursing these grants. In future, we will aim to take potential delays into account so that we can better keep to our target of releasing a report once every six months.
    Scope of the Fund
    These grants were decided by the general grantmaking process outlined in our previous grants report and the Fund's launch announcement.
    As a quick summary, the Fund supports work that:
    Reduces existential and catastrophic risks, such as those coming from misaligned artificial intelligence, pandemics, and nuclear war.
    Promotes, improves, and implements key longtermist ideas.
    In addition, the Fund focuses on organisations with a compelling and transparent case in favour of their cost-effectiveness, and/or that will benefit from being funded by a large number of donors. Longview Philanthropy decides the grants and allocations based on its past and ongoing work to evaluate organisations in this space.
    Grantees
    AI interpretability work at Harvard University - $110,000
    This grant is to support the work of Martin Wattenberg and Fernanda Viegas to develop their AI interpretability work at Harvard University. The grant aims to fund research that enhances our understanding of how modern AI systems function - better understanding how these systems work is among the more straightforward ways we can ensure these systems are safe. Profs. Wattenberg and Viegas have a strong track record (with both having excellent references from other experts) and their future plans are likely to advance the interpretability field.
    Longview: "We recommended a grant of $110,000 to support Martin Wattenberg and Fernanda Viegas' interpretability work on the basis of excellent reviews of their prior work. These funds will go primarily towards setting up a compute cluster and hiring graduate students or possibly postdoctoral fellows."
    Learn more about this grant.
    ARC Evals - $220,000
    The evaluations project at the Alignment Research Center ("ARC Evals") works on "assessing whether cutting-edge AI systems could pose catastrophic risks to civilization." ARC Evals is contributing to the following AI governance approach:
    Before a new large-scale system is released, assess whether it is capable of potentially catastrophic activities.
    If so, require strong guarantees that the system will not carry out such activities.
    ARC Evals wor...
    10 min
  • AF - "Dirty concepts" in AI alignment discourses, and some guesses for how to deal with them by Nora Ammann
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Dirty concepts" in AI alignment discourses, and some guesses for how to deal with them, published by Nora Ammann on August 20, 2023 on The AI Alignment Forum.
    Meta: This is a short summary & discussion post of a talk on the same topic by Javier Gomez-Lavin, which he gave as part of the PIBBSS speaker series. The speaker series features researchers from both AI Alignment and adjacent fields studying intelligent behavior in some shape or form. The goal is to create a space where we can explore the connections between the work of these scholars and questions in AI Alignment.
    This post doesn't provide a comprehensive summary of the ideas discussed in the talk, but instead focuses on exploring some possible connections to AI Alignment. For a longer version of Gomez-Levin's ideas, you can check out a talk here.
    "Dirty concepts" in the Cognitive Sciences
    Gomez-Lavin argues that cognitive scientists engage in a form of "philosophical laundering," wherein they associate, often implicitly, philosophically loaded concepts (such as volition, agency, etc.) into their concept of "working memory."
    He refers to such philosophically laundered concepts as "dirty concepts" insofar as they conceal potentially problematic assumptions being made. For instance, if we implicitly assume that working memory requires, for example, volition, we have now stretched our conception of working memory to include all of cognition. But, if we do this, then the concept of working memory loses much of its explanatory power as one mechanism among others underlying cognition as a whole.
    Often, he claims, cognitive science papers will employ such dirty concepts in the abstract and introduction but will identify a much more specific phenomena being measured in the methods and results section.
    What to do about it? Gomez-Lavin's suggestion in the case of CogSci
    The pessimistic response (and some have suggested this) would be to quit using any of these dirty concept (e.g. agency) all together. However, it appears that this would amount to throwing the baby out with the bathwater.
    To help remedy the problem of dirty concepts in working memory literature, Gomez-Lavin proposes creating an ontology of the various operational definitions of working memory employed in cognitive science by mining a wide range of research articles. The idea is that, instead of insisting that working memory be operationally defined in a single way, we ought to embrace the multiplicity of meanings associated with the term by keeping track of them more explicitly.
    He refers to this general approach as "productive pessimism." It is pessimistic insofar as it starts from the assumption that dirty concepts are being problematically employed, but it is productive insofar as it attempts to work with this trend rather than fight against it.
    While it is tricky to reason with those fuzzy concepts, once we are rigorous about proposing working definitions / operationalization of these terms as we use them, we can avoid some of the main pitfalls and improve our definitions over time.
    Relevance to AI alignment?
    It seems fairly straightforward that AI alignment discourse, too, suffers from dirty concepts.
    If this is the case (and we think it is), a similar problem diagnosis (e.g. how dirty concepts can hamper research/intellectual progress) and treatment (e.g. ontology mapping) may apply.
    A central example here is the notion of "agency". Alignment researchers often speak of AI systems as agents. Yet, there are often multiple, entangled meanings intended when doing so. High-level descriptions of AI x-risk often exploit this ambiguity in order to speak about the problem in general, but ultimately imprecise terms. This is analogous to how cognitive scientists will often describe working memory in general terms in the abstract and operationalize the term ...
    6 min
  • LW - AI Forecasting: Two Years In by jsteinhardt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Forecasting: Two Years In, published by jsteinhardt on August 20, 2023 on LessWrong.
    Two years ago, I commissioned forecasts for state-of-the-art performance on several popular ML benchmarks. Forecasters were asked to predict state-of-the-art performance on June 30th of 2022, 2023, 2024, and 2025. While there were four benchmarks total, the two most notable were MATH (a dataset of free-response math contest problems) and MMLU (a dataset of multiple-choice exams from the high school to post-graduate level).
    One year ago, I evaluated the first set of forecasts. Forecasters did poorly and underestimated progress, with the true performance lying in the far right tail of their predicted distributions. Anecdotally, experts I talked to (including myself) also underestimated progress. As a result of this, I decided to join the fray and registered my own forecasts for MATH and MMLU last July.
    June 30, 2023 has now passed, so we can resolve the forecasts and evaluate my own performance as well as that of other forecasters, including both AI experts and generalist "superforecasters". I'll evaluate the original forecasters that I commissioned through Hypermind, the crowd forecasting platform Metaculus, and participants in the XPT forecasting competition organized by Karger et al. (2023), which was stratified into AI experts and superforecasters.
    Overall, here is how I would summarize the results:
    Metaculus and I did the best and were both well-calibrated, with the Metaculus crowd forecast doing slightly better than me.
    The AI experts from Karger et al. did the next best. They had similar medians to me but were (probably) overconfident in the tails.
    The superforecasters from Karger et al. did the next best. They (probably) systematically underpredicted progress.
    The forecasters from Hypermind did the worst. They underpredicted progress significantly on MMLU.
    Interestingly, this is a reverse of my impressions from last year, where even though forecasters underpredicted progress, I thought of experts as underpredicting progress even more. In this case, it seems the experts did pretty well and better than generalist forecasters.
    What accounts for the difference? Some may be selection effects (experts who try to register forecasts are more likely to be correct). But I'd guess some is also effort: the expert "forecasts" I had in mind last year were from informal hallway conversations, while this year they were formal quantitative predictions with some (small) monetary incentive to be correct. In general, I think we should trust expert predictions more in this setting (relative to their informal statements), and I'm now somewhat more optimistic that experts can give accurate forecasts given a bit of training and the correct incentives.
    In the rest of the post, I'll first dive into everyone's forecasts and evaluate each in turn. Then, I'll consider my own forecast in detail, evaluating not just the final answer but the reasoning I used (which was preregistered and can be found here).
    My forecasts, and others
    As a reminder, forecasts are specified as probability distributions over some (hopefully unambiguously) resolvable future outcome. In this case the outcome was the highest credibly claimed benchmark accuracy by any ML system on the MATH and MMLU benchmarks as of June 30, 2023.
    My forecasts from July 17, 2022 are displayed below as probability density functions, as well as cumulative distribution functions and the actual result:
    MATHMMLUResult: 69.6% (Lightman et al., 2023)Result: 86.4% (GPT-4)
    Orange is my own forecast, while green is the crowd forecast of Metaculus on the same date. For MATH, the true result was at my 41st percentile, while for MMLU it was at my 66th percentile. I slightly overestimated progress on MATH and underestimated MMLU, but both were within my range of e...
    20 min
  • LW - Is Chinese total factor productivity lower today than it was in 1956? by Ege Erdil
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is Chinese total factor productivity lower today than it was in 1956?, published by Ege Erdil on August 19, 2023 on LessWrong.
    tl;dr: Multifactor productivity data from a famous economic dataset, used often to proxy for technological progress and innovation, might be significantly biased by poor estimates of the social returns to education, the capital share of income, and real GDP. Such estimates should be treated with caution.
    Introduction
    Total factor productivity is an economic concept that is used to quantify how efficiently a country can make use of its economic resources. It's a rather nebulous concept in general because it's not directly measurable but instead corresponds to latent variables in growth models that account for "unexplained variation" in output.
    If we stick to the abstract realm of growth models, there is often a clear definition: for instance, we might model a country's real GDP Y by a function such as
    Y=AL1-αKα
    where L and K denote the country's total labor force and capital stock respectively, 0<α<1 is a parameter, and A is total factor productivity, hereafter abbreviated as TFP. If we have two countries with the same capital stock and labor force but one of them has a higher economic output, we say that country has a higher TFP.
    Of course, the same principle works in general, not just with this specific functional form and for labor and capital inputs. If we have some economically relevant inputs f1,f2,.,fn (sometimes called factors of production), we can imagine a model where economic output in a country is given by
    Y=AH(f1,f2,.,fn)
    for some function H. In this case, the TFP A is just a factor that's present to "close the model": changes in output not accounted for by changes in the inputs fi are automatically accounted for by changes in A. For a specific H, we can see A=Y/H as a measure of economic productivity which controls for "obvious factors" such as changes in the labor or capital stocks. Making the right choice of H is very important, of course: we don't want to use a random H, but one that is actually informed by the data. I'll talk more about how we do this later.
    For instance, if a country has increasing TFP over time, that suggests the country's output is rising because of things that are out of our model. If our model only had labor force and capital stock, perhaps the quality of the workers in the country has gone up: they have more human capital, even though they have the same number of people because each person is on average more productive. It could be that the country's institutions have improved and this enables them to make more efficient use of their existing resources. Anything that is out of the model goes into TFP, which is why it's sometimes called a "trash can statistic".
    Importantly, TFP is a concept that only makes sense relative to a model, in particular relative to a collection of factors of production that we choose to consider in our model. There is no such thing as a country's "absolute TFP" because TFP is a latent variable and its value depends on the model specification. Still, TFP is often used as a proxy for technological progress, because we intuitively think that better technology should involve something other than raw accumulations of labor, capital, and human talent. This approach is used by Bloom et al. (2020), the famous "are ideas getting harder to find?" paper, for the United States specifically. Guzey et al. (2021) criticizes them on this point, among others.
    The puzzle of declining total factor productivity
    Now, we come to the question in the title of the post. Check out the plot below:
    According to this plot, China had a higher TFP in 1956 (when it was poorer per capita than most African countries) than in 2019! What's going on here?
    Saying that China regressed technologically doesn't make s...
    15 min
  • EA - Launching Foresight Institute's AI Grant for Underexplored Approaches to AI Safety - Apply for Funding! by elteerkers
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Launching Foresight Institute's AI Grant for Underexplored Approaches to AI Safety - Apply for Funding!, published by elteerkers on August 19, 2023 on The Effective Altruism Forum.
    Summary
    We are excited to launch our new grant programme that will fund areas that we consider underexplored when it comes to AI safety. In light of the potential for shorter AGI timelines, we will re-grant $1-1.2 million per year to support much needed development in one of the following areas:
    Neurotechnology, Whole Brain Emulation, and Lo-fi uploading for AI safety
    Cryptography and Security approaches for Infosec and AI security
    Safe Multipolar AI scenarios and Multi-Agent games
    Apply for funding- to be reviewed on a rolling basis.
    See below, or visit our website to learn more!
    Areas that We're Excited to Fund
    Neurotechnology, Whole Brain Emulation, and Lo-fi uploading for AI safety
    We are interested in exploring the potential of neurotechnology, particularly Whole Brain Emulation (WBE) and cost-effective lo-fi approaches to uploading, that could be significantly sped up, leading to a re-ordering of technology arrival that might reduce the risk of unaligned AGI by the presence of aligned software intelligence.
    We are particularly excited by the following:
    WBE as a potential technology that may generate software intelligence that is human-aligned simply by being based directly on human brains
    Lo-fi approaches to uploading (e.g. extensive lifetime video of a laboratory mouse to train a model of a mouse without referring to biological brain data)
    Neuroscience and neurotech approaches to AI Safety (e.g. BCI development for AI safety)
    Other concrete approaches in this area
    General scoping/mapping opportunities in this area, especially from a differential technology development perspective, as well as understanding the reasons why this area may not be a suitable focus
    Cryptography and Security approaches for Infosec and AI security
    To explore the potential benefits of Cryptography and Security technologies in securing AI systems. This includes:
    Computer security to help with AI Infosecurity or approaches for scaling up security techniques to potentially apply to more advanced AI systems
    Cryptographic and auxiliary techniques for building coordination/governance architectures across different AI(-building) entities
    Privacy-preserving verification/evaluation techniques
    Other concrete approaches in this area
    General scoping/mapping opportunities in this area, especially from a differential technology development perspective, or exploring why this area is not a good focus area
    Safe Multipolar AI scenarios and Multi-Agent games
    Exploring the potential of safe Multipolar AI scenarios, such as:
    Multi-agent game simulations or game theory
    Scenarios avoiding collusion and deception, and pareto-preferred and positive-sum dynamics
    Approaches for tackling principal agent problems in multipolar systems
    Other concrete approaches in this area
    General scoping/mapping opportunities in this area, especially from a differential technology development perspective, or exploring why this area is not a good focus area
    Interested in Applying?
    We look forward to receiving your submissions. Applications will be reviewed on a rolling basis-apply here.
    For the initial application, you'll be required to submit:
    A background of yourself and your work
    A short summary and budget for your project highlighting which area you are applying to, and outlining what you would like to investigate and why
    At least two references
    We will aim to get back to applicants within 8 weeks of receiving their application.
    If you are interested then please find more information about the grant here.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    4 min
  • EA - New probabilistic simulation tool by ProbabilityEnjoyer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New probabilistic simulation tool, published by ProbabilityEnjoyer on August 19, 2023 on The Effective Altruism Forum.
    Dagger (usedagger.com) is a new tool for calculations with uncertainty. It uses Monte Carlo simulation.
    There are two ways to specify your simulation model:
    Import an existing spreadsheet
    use Probly, a Python dialect designed for probabilistic simulation
    ℹ️ Each of the two links above has 4 interactive examples. You might want to start there.
    Spreadsheet
    Example
    In this 15-second video, we take a complex existing spreadsheet (100+ rows) from GiveWell and turn it into a Monte Carlo simulation
    The sheet already gives us "optimistic/pessimistic" values, so it's as simple as adding one column to specify the distribution as (e.g.) uniform
    (Longer version of this video)
    Features
    Dependency graph
    Intuitive and mathematically rigorous sensitivity analysis
    Our sensitivity analysis uses Sobol' global sensitivity indices. The approach and the intuition behind it are explained in more detail here.
    ℹ️ You need to enable the sensitivity analysis under "Advanced options"
    Summary table
    This table exposes the structure of your model by showing the dependency graph as a tree. Similar to Workflowy, you can zoom to any variable, or expand/collapse.
    Probly
    Probly feels very like Python, except that any number can also be a probability distribution:
    Example
    Here's a fuller example of the syntax and resulting output. It's part of a GiveWell CEA of iron and folic acid supplementation.
    Distribution support
    Probly supports 9 probability distributions. Each can be constructed in multiple ways.
    For example, you can construct a normal distribution in 5 ways:
    This clickable table shows you everything that's supported, and includes example code:
    ℹ️ Shortcut: probly.dev redirects to usedagger.com/probly
    Limitations
    There are at the moment numerous limitations. A small selection of them:
    Probly:
    Doesn't support the Sobol' sensitivity analysis
    Doesn't show the dependency graph
    Spreadsheet
    There is no UI in Dagger to edit the model. All changes must go via the spreadsheet.
    The spreadsheet must specify probability distributions in a specific format.
    All models are public
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • EA - Taking prioritisation within 'EA' seriously by CEvans
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Taking prioritisation within 'EA' seriously, published by CEvans on August 19, 2023 on The Effective Altruism Forum.
    NB: This post is arguably slightly info hazardous (for lack of a better term) for anyone in the community who might feel particular anxiety about questioning their own career decisions. Perhaps consider reading this first: You have more than one goal, and that's fine - EA Forum (effectivealtruism.org). This piece is about the importance of having non-impact focussed goals, and that it is extremely ok to have them. This post is intended to suggest what people should do in so far as they want to have more impact, rather than being a suggestion of what everyone in the EA community should do.
    Your decisions matter. The precise career path you pick really matters for your impact. I think many people in the EA community would say they believe this if asked, but haven't really internalised what this means for them. I think many people would have great returns for impact to thinking more carefully about prioritisation for their career, even within the "EA careers space. Here are some slight caricatures of statements I hear regularly:
    "I want an 'impactful' job"
    "I am working on a very important problem, so within that I will do what I think is interesting"
    "I was already interested in something mentioned on 80,000 Hours, so I will work on that"
    "People seem to think this area is important, so I suppose I should work on that"
    "I am not a researcher, so I shouldn't work on that problem"
    I think these are all major mistakes for those who say them, in so far as impact is their primary career goal. My goals for this post are to make the importance of prioritisation feel more salient to members of the community (which is the first half of my post), and to help making progress feel and be more attainable (the second half from "What does thinking seriously about prioritisation look like".
    Key Claims
    For any given person, their best future 'EA career paths' are at least an order of magnitude more impactful than their median 'EA career path'.
    For over 50% of self identifying effective altruists, in their current situation:
    Thinking more carefully about prioritisation will increase their expected impact by several times.
    There will be good returns to thinking more about the details of prioritising career options for yourself, not just uncritically deferring to others or doing very high-level "cause prioritisation".
    They overvalue personal fit and prior experience when determining what to work on.
    I think the conclusions of my argument should excite you.
    Helping people is amazing.. This community has enabled many of us to help orders of magnitude more people than we otherwise would have. I am claiming that you might be able to make a similar improvement again with careful thought, and that as a community we might be able to achieve a lot more.
    Defining prioritisation in terms of your career
    Prioritisation: Determining which particular actions are most likely to result in your career having as much impact as possible. In practice, this looks like a combination of career planning and cause/intervention prioritisation. So, "your prioritisation" means something like "your current best guess of what precisely you should be doing with your career for impact".
    Career path: The hyper-specific route which you take through your career, factoring decisions in such as which specific issues/interventions to work on, which organisations to work at, and which specific roles to take. I do not mean to be as broad as "AI alignment researcher" or "EA community builder".
    'EA' person/career path: By this I mean a person or choice which is motivated by impact, not necessarily in so-called 'EA' organisations or explicitly identifying as part of the community.
    For any given person, the best "EA"...
    37 min
  • AF - We can do better than DoWhatIMean by Luke H Miles
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We can do better than DoWhatIMean, published by Luke H Miles on August 19, 2023 on The AI Alignment Forum.
    Lately, "alignment" means "follow admin rules and do what users mean". Admins can put in rules like "don't give instructions for bombs, hacking, or bioweapons" and "don't take sides in politics". As AI gets more powerful, we can use it to write better rules and test for loopholes. And "do what users mean" implies some low-impactfulness: it won't factory reset your computer when you ask it to fix the font size.
    In other words, the plan has two steps:
    Make a machine that can do literally anything
    Configure it to do good things
    I don't like this plan very much because it puts power in the hands of the few and has big security liabilities.
    Temptation
    Imagine you're on the team of three people that writes the constitution for the training run for the gigaAI1. Or you're the one actually typing it in. Or you do infra and write all the code that strings pass through on the way to and from the training cluster.
    Might you want to take advantage of the opportunity? I can think of a few subtle things I might add.
    If I think about the people in my life I know well, I can only name one that I would nominate for the constitution writing position.
    One solution I've heard for this is to use the powerful DWIM machine to construct a second unalterable machine, then delete the first machine. You might call the time between creating the first and second machine the temptation mountain (in a plot where X is time and Y is the maximum power of any individual person). Then you want to minimize the volume of this mountain. It is difficult to shrink the mountain very much though.
    Security risks
    If you're 90% through your DWIM AI development and it seems very promising, then you must be very careful with your code and data. If someone else gets your secrets, then they can make it do what they mean, and they might mean for it to do something quite different. You should be careful with your team members too since threats and blackmail and bribes are all quite cheap and effective. (Hard to be careful with them though since they put "on constitution AI team" on their linkedins, their phone numbers are practically public information, and they live in a group house with employees from your competitor.)
    If you're doing open source AI development and start having big successes then you might start looking at the github profiles of your stargazers and wonder what they mean for it to do.
    If altLab is also getting close to AGI then you have to figure out whether your stop-clause means you help them or their stop-clause means they help you.
    Doing better
    Could we come up with a plan so rock solid that we'd be comfortable giving the code, the data, or the team to anyone? Like um how you could give Gandhi to Hitler without stressing.
    For lack of a better term[1], let's call such a system intrinsically aligned AI. The key innovation required might be a means to strongly couple capabilities and alignment. This is in contrast to the current regime where we build Do Anything Now then tell it to do good.
    Such an innovation would unblock coordination and ease security burdens, and not require the board of directors & the engineers to be saints. It would also give the open source community something to work on that doesn't give AI safety folks gray hairs.
    To be clear, I don't have a good idea for how to achieve this. I'm not certain it is even possible.
    However, it's not obviously impossible to me. And I would be much more at ease if all the AI labs were building intrinsically aligned AI instead of something that can be stolen and directed to arbitrary goals.[2]
    Let's at least give it a shot.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min