The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Review of The Good It Promises, the Harm It Does by Richard Y Chappell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Review of The Good It Promises, the Harm It Does, published by Richard Y Chappell on May 2, 2023 on The Effective Altruism Forum.
    [TL;DR: I didn't find much of value in the book. The quality of argumentation is worse than on most blogs I read. Maybe others will have better luck discerning any hidden gems in the mix?]
    The Good It Promises, the Harm It Does: Critical Essays on Effective Altruism (eds. Adams, Crary, & Gruen) puts me in mind of Bastiat’s Candlestick Makers' Petition. For any proposed change—be it the invention of electricity, or even the sun rising—there will be some in a position to complain. This is a book of such complaints. There is much recounting of various “harms” caused by EA (primarily to social justice activists who are no longer as competitive for grant funding). But nowhere in the volume is there any serious attempt to compare these costs against the gains to others—especially the populations supposedly served by charitable work, as opposed to the workers themselves—to determine which is greater. (One gets the impression that cost-benefit analysis is too capitalistic for these authors to even consider.) The word “trade-off” does not appear in this volume.
    A second respect in which the book’s title may be misleading is that it is exclusively about the animal welfare wing of EA. (There is a ‘Coda’ that mentions longtermism, but merely to sneer at it. There was no substantive engagement with the ideas.)
    I personally didn’t find much of value in the volume, but I’ll start out by flagging what good I can. I’ll then briefly explain why I wasn’t much impressed with the rest—mainly by way of sharing representative quotes, so readers can judge it for themselves.
    The Good
    The more empirically-oriented chapters raise interesting challenges about animal advocacy strategy. We learn that EA funders have focused on two main strategies to reform or eventually upend animal agriculture: (i) corporate cage-free (and similar) campaigns, and (ii) investment in meat alternatives. Neither involves the sort of “grassroots” activism that the contributors to this volume prefer. So some of the authors discuss potential shortcomings of the above two strategies, and potential benefits of alternatives like (iii) vegan outreach in Black communities, and (iv) animal sanctuaries.
    I expect EAs will welcome discussion of the effectiveness of different strategies. That’s what the movement is all about, after all. By far the most constructive article in the volume (chapter 4, ‘Animal Advocacy’s Stockholm Syndrome’) noted that “cage- free campaigns. can be particularly tragic in a global context” where factory-farms are not yet ubiquitous:
    The conscientious urban, middle-class Indian consumer cannot see that there is a minor difference between the cage-free egg and the standard factory-farmed egg, and a massive gulf separating both of these from the traditionally produced egg [where birds freely roam their whole lives] for a simple reason: the animal protection groups the consumer is relying upon are pointing to the (factory- farmed) cage-free egg instead of alternatives to industrial farming. (p. 45)
    Such evidence of localized ineffectiveness (or counterproductivity) is certainly important to identify & take into account!
    There’s a larger issue (not really addressed in this volume) of when it makes sense for a funder to go “all in” on their best bets vs. when they’d do better to “diversify” their philanthropic portfolio. This is something EAs have discussed a bit before (often by making purely theoretical arguments that the “best bet” maximizes expected value), but I’d be excited to see more work on this problem using different methodologies, including taking into account the risk of “model error” or systemic bias in our initial EV estimates. (Maybe such work is already out ther...
    18 min
  • EA - Apply Now: First-Ever EAGxNYC This August by Arthur Malone
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Apply Now: First-Ever EAGxNYC This August, published by Arthur Malone on May 2, 2023 on The Effective Altruism Forum.
    TL;DR: Applications are now open for EAGxNYC 2023, taking place in Manhattan this August 18-20!
    We’re thrilled to announce that this summer, EAGx comes to New York City for the first time!
    Application:
    Reviewed on a rolling basis, apply here before the deadline of July 31, 2023. Applying early means you'll have more time to prep and help us plan for your needs!
    When:
    August 18-20, 2023
    Where:
    Convene, 225 Liberty Street, New York, NY, in Lower Manhattan near the World Trade Center complex
    Who:
    EAGxNYC is intended for both individuals new to the movement and those already professionally engaged with EA, and will cover a diverse range of high-impact cause areas. We believe the conference will be of particular value to those currently exploring new ways they can have an impact, such as students, young professionals, and mid-career professionals looking to shift into EA-aligned work. We also invite established organizations looking to share their work and grow their pool of potential collaborators or hirees. Due to venue and funding capacity, the conference will be capped at 500 attendees.
    Geographic scope: As a locally-organized supplement to the Centre for Effective Altruism-organized EAG conferences, EAGxNYC aims to primarily serve, and foster connections between, those living in the NYC area. While we are also excited to welcome individuals from around the globe, due to limited capacity we will prioritize applicants who have a connection to our New York metropolitan area or are seriously considering relocating here, followed by applicants from throughout the East Coast. However, if you are uncertain about your eligibility, don't hesitate to apply!
    Travel Grants: Limited travel grants of up to $500 are available to individuals from outside of NYC who would not be able to attend EAGxNYC without financial assistance. Applications for financial assistance have no bearing on admissions to the conference.
    Programming:
    EAGxNYC will take place from Friday, August 18th through Sunday, August 20th with registration opening in the early afternoon Friday, followed by dinner and opening talks that evening. Content will be scheduled and the venue will be open for networking until 10PM Friday, 8AM-10PM Saturday, and 8AM-7PM Sunday. Along with dinner on Friday, the venue will be providing breakfast, lunch, and snacks and drinks on Saturday and Sunday. Dinner will not be served on the premises Saturday or Sunday, but the EAGxNYC team will help coordinate group dinners nearby and encourage all attendees to make use of the venue throughout the evening.
    We aim to program content covering all effective altruism cause areas with a special emphasis on the intersection between EA and New York City. If you are interested in presenting at the conference, please reach out to the organizing team.
    Satellite Programming: If you’re already in the New York City area and want to get involved leading up to or following the conference, check out the local EA NYC group for public events, cause-related and professional subgroup events, opportunities for online engagement, and more!
    More info: Detailed information on the agenda, speakers, and content will be available closer to the conference via Swapcard and updates to this website page. Periodically checking in on our website will help you stay up to date in the meantime, and if you have any questions or concerns, drop email us at [email protected].
    We can't wait to see you in NYC this Summer!
    The organizing team :)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Legal Priorities Project – Annual Report 2022 by Legal Priorities Project
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Legal Priorities Project – Annual Report 2022, published by Legal Priorities Project on May 2, 2023 on The Effective Altruism Forum.
    Summary
    Note: A private version of this report with additional confidential updates has been shared with a few major donors and close collaborators. If you fall into this category and would like to access the extended version, please get in touch with us at [email protected].
    This report presents a review of the Legal Priorities Project’s work in 2022, including key successes, statistics, bottlenecks, and issues. We also offer a short overview of our priorities for 2023 and briefly describe our methodology for updating our priorities. You can learn more about how to support our work at the end of the report.
    A PDF version of this report is available here.
    In 2022:
    Our research output per FTE was high to very high: With only 3.6 FTE researchers, we had 7 peer-reviewed papers (2 journal articles and 5 book chapters) accepted for publication, 3 papers currently under review, and one book under contract with Oxford University Press. We also added 7 papers to our Working Paper Series (for a total of 18), published a new chapter of our research agenda, and published 6 shorter pieces in online forums. We also spent significant time on reports aimed at informing our prioritization on artificial intelligence and biosecurity in particular (which we plan to publish in Q2 of 2023) and ran a writing competition on “Improving Cost-Benefit Analysis to Account for Existential and Catastrophic Risks” with a judging panel composed of eminent figures in law. Based on our experience, our research output was much higher than typical legal academic research groups of similar size.
    Beyond academic research, we analyzed ongoing policy efforts, and our research received positive feedback from policymakers. Relevant feedback and discussions provided valuable insight into what research would support decision-making and favorable outcomes, which we believe improved our prioritization.
    We experimented with running several events targeting different audiences and received hundreds of applications from students and academics at top institutions worldwide. Some participants have already reported significant updates to their plans as a result. Feedback on our events was overwhelmingly positive, and we gained valuable information about the different types of programs and their effectiveness, which will inform future events.
    Team morale remained high, including during stressful developments, and our operations ran smoothly.
    In 2023:
    Our research will increasingly focus on reducing specific types of existential risk based on concrete risk scenarios, shifting more focus toward AI risk. While this shift started in 2022, AI risk will become more central to our research in 2023. We will publish an update to our research agenda and theory of change accordingly.
    We will continue to publish research of various types. However, we will significantly increase our focus on non-academic publications, such as policy/technical reports and blog posts, in order to make our work more accessible to policymakers and a wider audience. As part of this strategy, we will also launch a blog featuring shorter pieces by LPP staff and invited researchers.
    We would like to run at least one, but ideally two, flagship field-building programs: The Legal Priorities Summer Institute and the Summer Research Fellowship.
    We will seek to raise at least $1.1m to maintain our current level of operations for another year. More optimistically, we aim to increase our team size by 1–3 additional FTE, ideally hiring a senior researcher with a background in US law to work on risks from advanced artificial intelligence.
    Introduction
    The Legal Priorities Project is an independent, global research and field-bui...
    59 min
  • LW - Systems that cannot be unsafe cannot be safe by Davidmanheim
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Systems that cannot be unsafe cannot be safe, published by Davidmanheim on May 2, 2023 on LessWrong.
    Epistemic Status: Trying to clarify a confusion people outside of the AI safety community seem to have about what safety means for AI systems.
    In engineering and design, there is a process that includes, among other stages, specification, creation, verification and validation, and deployment. Verification and validation are where most people focus when thinking about safety - can we make sure the system performs correctly? I think this is a conceptual error that I want to address.
    "Verification and validation (also abbreviated as V&V) are independent procedures that are used together for checking that a product, service, or system meets requirements and specifications and that it fulfills its intended purpose." - Wikipedia
    Both of these terms are used slightly differently across fields, but in general, verification is the process of making sure that the system fulfills the design requirements and/or other standards. This pre-supposes that the system has some defined requirements or a standard, at least an implicit one, and that it could fail to meet that bar. That is, the specification of the system includes what it must and must not do, and if the system does not do what it should, or does something that it should not, it fails.
    Machine learning systems, especially language models, aren't well understood. The potential applications are varied and uncertain, entire classes of new and surprising failure modes are still being found, and we have nothing like a specification of what the system should or should not do, must or must not do, and where it can and cannot be used.
    To take a very concrete example, metal rods have safety characteristics, and they might be rated for use up to some weight limit, under some specific load for some amount of time, in certain temperature ranges, for some amount of time. These can all be tested. If the bar does not stay within a predefined range of characteristics at a given temperature, with a given load, it fails. It can also be found to be acceptable in one temperature range, but not another, or similar. At the end of verification and validation, the bar is deemed to have passed or failed for a given application, based on what the requirements for that larger system are.
    At its best, red-teaming and safety audits of ML systems check lots of known failure modes, and determine whether they are susceptible. There is no pre-defined standard or set of characteristics that are checked, no real ability to consider application specific requirements, and no ability to specify where the system should not or must not be used.
    Until we have some safety standard for machine learning models, they aren't "partly safe" or "assumed safe," or "good enough for consumer use." If we lack a standard for safety, ideally one where there is consensus that it is sufficient for a specific application, then exploration or verification of the safety of a machine learning model is meaningless. If a model is released to the public without a clear indication about what the system can safely be used for, with verification that it passed a relevant standard, and clear instruction that it cannot be used elsewhere, it is an unsafe model. Anyone who claims otherwise seems fundamentally confused about what safety means for such systems.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Intermediate goals for reducing risks from nuclear weapons: A shallow review (part 1/4) by MichaelA
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Intermediate goals for reducing risks from nuclear weapons: A shallow review (part 1/4), published by MichaelA on May 1, 2023 on The Effective Altruism Forum.
    This is a blog post, not a research report, meaning it was produced relatively quickly and is not to Rethink Priorities' typical standards of substantiveness and careful checking for accuracy.
    Summary
    What is this post?
    This post is the first part of what was intended to be a shallow review of potential “intermediate goals” one could pursue in order to reduce nuclear risk (focusing especially on the contribution of nuclear weapons to existential risk). The full review would’ve broken intermediate goals down into:
    goals aimed at reducing the odds of nuclear conflict or other non-test nuclear detonations
    goals aimed at changing how nuclear conflict plays out if it does occur (in a way that reduces its harms)
    goals aimed at improving resilience to or ability to recover from the harms of nuclear conflict
    goals that are cross-cutting, focused on field-building, or otherwise have indirect effects
    This first part of the shallow review focuses just on the first of those categories: goals aimed at reducing the odds of nuclear conflict or other non-test nuclear detonations. We tentatively think that, on the margin, this is the least promising of those four categories of goals, but that there are still some promising interventions in this category.
    Within this category, we review multiple potential goals. For most of those goals, we briefly discuss:
    What we mean by the goal
    Why progress on this goal might reduce or increase nuclear risk
    Examples of specific interventions or organizations that could advance the goal
    Our very tentative bottom-line beliefs about:
    How much progress on the goal would reduce or increase nuclear risk
    What resources are most needed to make progress on the goal
    How easy making progress on the goal would be
    What key effects making progress on the goal might have on things other than nuclear risk
    Note that, due to time constraints, this post is much less comprehensive and thoroughly researched and reviewed than we’d like.
    The intermediate goals we considered, and our tentative bottom line beliefs on them
    This post and table breaks down high-level goals into increasingly granular goals, and shares our current best guesses on the relatively granular goals. Many goals could be pursued for multiple reasons and could hence appear in multiple places in this table, but we generally just showed each goal in the first relevant place anyway. This means in some cases a lot of the benefits of a given goal may be for higher-level goals we haven’t shown it as nested under.
    We unfortunately did even less research on the goals listed from From 1.1.2.6 onwards than the earlier ones, and the bottom-line views for that later set are mostly Will’s especially tentative personal views.
    Potential intermediate goalWhat effect would progress on this goal have on nuclear risk?How easy would it be to make progress on this goal?What resources are most needed for progress on this goal?Key effects this goal might have on things other than nuclear risk?1.1.1.1Reduce the odds of armed conflict in generalModerate reduction in riskHardUnsure.1.1.1.2Reduce the odds of (initially non-nuclear) armed conflict, with a focus on those involving at least one nuclear-armed stateMajor reduction in riskHard Similar to “1.1.1.1: Reduce the odds of armed conflict in general”1.1.1.3Reduce proliferationModerate reduction in riskHardMore non-nuclear conflict?
    Or maybe less?1.1.1.4Promote complete nuclear disarmamentMajor reduction in riskAlmost impossible to fully achieve this unless the world changes radically (e.g., a world government or transformative artificial intelligence is created, or a great power war occurs) 1.1.2.1Promote no first use (N...
    28 min
  • EA - AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now by Greg Colbourn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now, published by Greg Colbourn on May 2, 2023 on The Effective Altruism Forum.
    Content note: discussion of a near-term, potentially hopeless life-and-death situation that affects everyone.
    Tldr: AGI is basically here. Alignment is nowhere near ready. We may only have a matter of months to get a lid on this (strictly enforced global limits to compute and data) in order to stand a strong chance of survival. This post is unapologetically alarmist because the situation is highly alarming. Please help. Fill out this form to get involved. Here is a list of practical steps you can take.
    We are in a new era of acute risk from AGI
    Artificial General Intelligence (AGI) is now in its ascendency. GPT-4 is already ~human-level at language and showing sparks of AGI. Large multimodal models – text-, image-, audio-, video-, VR/games-, robotics-manipulation by a single AI – will arrive very soon (from Google DeepMind) and will be ~human-level at many things: physical as well as mental tasks; blue collar jobs in addition to white collar jobs. It’s looking highly likely that the current paradigm of AI architecture (Foundation models), basically just scales all the way to AGI. These things are “General Cognition Engines”.
    All that is stopping them being even more powerful is spending on compute. Google & Microsoft are worth $1-2T each, and $10B can buy ~100x the compute used for GPT-4. Think about this: it means we are already well into hardware overhang territory.
    Here is a warning written two months ago by people working at applied AI Alignment lab Conjecture: “we are now in the end-game for AGI, and we (humans) are losing”. Things are now worse. It’s looking like GPT-4 will be used to meaningfully speed up AI research, finding more efficient architectures and therefore reducing the cost of training more sophisticated models.
    And then there is the reckless fervour of plugin development to make proto-AGI systems more capable and agent-like to contend with. In very short succession from GPT-4, OpenAI announced the ChatGPT plugin store, and there has been great enthusiasm for AutoGPT. Adding Planners to LLMs (known as LLM+P) seems like a good recipe for turning them into agents. One way of looking at this is that the planners and plugins act as the System 2 to the underlying System 1 of the general cognitive engine (LLM). And here we have agentic AGI. There may not be any secret sauce left.
    Given the scaling of capabilities observed so far for the progression of GPT-2 to GPT-3 to GPT3.5 to GPT-4, the next generation of AI could well end up superhuman. I think most people here are aware of the dangers: we have no idea how to reliably control superhuman AI or make it value-aligned (enough to prevent catastrophic outcomes from its existence). The expected outcome from the advent of AGI is doom. This is in large part because AI Alignment research has been completely outpaced by AI capabilities research and is now years behind where it needs to be.
    To allow Alignment time to catch up, we need a global moratorium on AGI, now.
    A short argument for uncontrollable superintelligent AI happening soon (without urgent regulation of big AI):
    This is a recipe for humans extincting themselves that appears to be playing out along the mainline of future timelines:
    Either of
    GPT-4 + curious (but ultimately reckless) academics -> more efficient AI -> next generation foundation model AI (which I’ll call NextAI for short); Or
    Google DeepMind just builds NextAI (they are probably training it already)
    NextAI + planners + AutoGPT + plugins + further algorithmic advancements + gung ho humans (e/acc etc) = NextAI2 in short order. Weeks even. Access to compute for training is not a bottleneck because that cyborg syste...
    32 min
  • LW - Some Thoughts on Virtue Ethics for AIs by peligrietzer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some Thoughts on Virtue Ethics for AIs, published by peligrietzer on May 2, 2023 on LessWrong.
    This post argues for the desirability and plausibility of AI agents whose values have a structure I call ‘praxis-based.’ The idea draws on various aspects of virtue ethics, and basically amounts to an RL-flavored take on that philosophical tradition.
    Praxis-based values as I define them are, informally, reflective decision-influences matching the description ‘promote x x-ingly’: ‘promote peace peacefully,’ ‘promote corrigibility corrigibly,’ ‘promote science scientifically.’
    I will later propose a quasi-formal definition of this values-type, but the general idea is that certain values are an ouroboros of means and end. Such values frequently come up in human “meaning of life” activities (e.g. math, art, craft, friendship, athletics, romance, technology), as well as in complex forms of human morality (e.g. peace, democracy, compassion, respect, honesty). While this is already indirect reason to suspect that a human-aligned AI should have ‘praxis-based’ values, there is also a central direct reason: traits such as corrigibility, transparency, and niceness can only function properly in the form of ‘praxis-based’ values.
    It’s widely accepted that if early strategically aware AIs possess values like corrigibility, transparency, and perhaps niceness, further alignment efforts are much more likely to succeed. But values like corrigibility or transparency or niceness don’t easily fit into an intuitively consequentialist form like ‘maximize lifetime corrigible behavior’ or ‘maximize lifetime transparency.’ In fact, an AI valuing its own corrigibility or transparency or niceness in an intuitively consequentialist way can lead to extreme power-seeking whereby the AI violently remakes the world to (at a minimum) protect itself from the risk that humans will modify said value.
    On the other hand, constraints or taboos or purely negative values (a.k.a. ‘deontological restrictions’) are widely believed to be weak, in the sense that an advanced AI will come to work around them or uproot them: ‘never lie’ or ‘never kill’ or ‘never refuse a direct order from the president’ are poor substitutes for active transparency, niceness, and corrigibility.
    The idea of ‘praxis-based’ values is meant to capture the normal, sensible way we want an agent to value corrigibility or transparency or niceness, which intuitively-consequentialist values and deontology both fail to capture. We want an agent that (e.g.) actively tries to be transparent, and to cultivate its own future transparency and its own future valuing of transparency, but that will not (for instance) engage in deception and plotting when it expects a high future-transparency payoff.
    Having lightly motivated the idea that ‘praxis-based’ values are desirable from an alignment point of view, the rest of this post will survey key premises of the hypothesis that ‘praxis-based’ values are a viable alignment goal. I’m going to assume an agent with some form of online reinforcement learning going on, and draw on ‘shards’ talk pretty freely.
    I informally described a ‘praxis-based’ value as having the structure 'promote x x-ingly.' Here is a rough formulation of what I mean, put in terms of a utility-theoretic description of a shard that implements an alignment-enabling value x:
    Actions (or more generally 'computations') get an x-ness rating. We define the x shard's expected utility conditional on a candidate action a as the sum of two utility functions: a bounded utility function on the x-ness of a and a more tightly bounded utility function on the expected aggregate x-ness of the agent's future actions conditional on a. (So the shard will choose an action with mildly suboptimal x-ness if it gives a big boost to expected aggregate future x-ness, but r...
    10 min
  • EA - Exploring Metaculus’s AI Track Record by Peter Scoblic
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exploring Metaculus’s AI Track Record, published by Peter Scoblic on May 1, 2023 on The Effective Altruism Forum.
    By Peter Mühlbacher, Research Scientist at Metaculus, and Peter Scoblic, Director of Nuclear Risk at Metaculus
    Metaculus is a forecasting platform where an active community of thousands of forecasters regularly make probabilistic predictions on topics of interest ranging from scientific progress to geopolitics. Forecasts are aggregated into a time-weighted median, the “Community Prediction”, as well as the more sophisticated “Metaculus Prediction”, which weights forecasts based on past performance and extremises in order to compensate for systematic human cognitive biases. Although we feature questions on a wide range of topics, Metaculus focuses on issues of artificial intelligence, biosecurity, climate change and nuclear risk.
    In this post, we report the results of a recent analysis we conducted exploring the performance of all AI-related forecasts on the Metaculus platform, including an investigation of the factors that enhance or degrade accuracy.
    Most significantly, in this analysis we found that both the Community and Metaculus Predictions robustly outperform naïve baselines. The recent claim that performance on binary questions is “near chance” requires sampling on only a small subset of the forecasting questions we have posed or on the questionable proposition that a Brier score of 0.207 is akin to a coin flip. What’s more, forecasters performed better on continuous questions, as measured by the continuous ranked probability score (CRPS). In sum, both the Community Prediction and the Metaculus Prediction—on both binary and continuous questions—provide a clear and useful insight into the future of artificial intelligence, despite not being “perfect”.
    Summary Findings
    We reviewed Metaculus’s resolved binary questions (“What is the probability that X will happen?”) and resolved continuous questions (“What will be the value of X?”) that were related to the future of artificial intelligence. For the purpose of this analysis, we defined AI-related questions as those which belonged to one or more of the following categories: “Computer Science: AI and Machine Learning”; “Computing: Artificial Intelligence”; “Computing: AI”; and “Series: Forecasting AI Progress.” This gave us: 64 resolved binary questions (with 10,497 forecasts by 2,052 users) and 88 resolved continuous questions (with 13,683 predictions by 1,114 users). Our review of these forecasts found:
    Both the community and Metaculus predictions robustly outperform naïve baselines.
    Analysis showing that the community prediction’s Brier score on binary questions is 0.237 relies on sampling only a small subset of our AI-related questions.
    Our analysis of all binary AI-related questions finds that the score is actually 0.207 (a point a recent analysis agrees with), which is significantly better than “chance”.
    Forecasters performed better on continuous questions than binary ones.
    Top-Line Results
    This chart details the performance of both the Community and Metaculus predictions on binary and continuous questions. Please note that, for all scores, lower is better and that Brier scores, which range from 0 to 1 (where 0 represents oracular omniscience and 1 represents complete anticipatory failure) are roughly comparable to continuous ranked probability scores (CRPS) given the way we conducted our analysis. (For more on scoring methodology, see below.)
    Brier (binary questions)CRPS (continuous questions)Community Prediction0.2070.096Metaculus Prediction0.1820.103baseline prediction0.250.172
    Results for Binary Questions
    We can use Brier scores to measure the quality of a forecast on binary questions. Given that a Brier score is the mean squared error of a forecast, the following things are true:
    If you alread...
    13 min
  • AF - Systems that cannot be unsafe cannot be safe by David Manheim
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Systems that cannot be unsafe cannot be safe, published by David Manheim on May 2, 2023 on The AI Alignment Forum.
    In engineering and design, there is a process that includes, among other stages, specification, creation, verification and validation, and deployment. Verification and validation are where most people focus when thinking about safety - can we make sure the system performs correctly? I think this is a conceptual error that I want to address.
    "Verification and validation (also abbreviated as V&V) are independent procedures that are used together for checking that a product, service, or system meets requirements and specifications and that it fulfills its intended purpose." - Wikipedia
    Both of these terms are used slightly differently across fields, but in general, verification is the process of making sure that the system fulfills the design requirements and/or other standards. This pre-supposes that the system has some defined requirements or a standard, and that it could fail to meet that bar. That is, the specification of the system includes what it must and must not do, and if the system does not do what it should, or does something that it should not, it fails.
    Machine learning systems, especially language models, aren't well understood. The potential applications are varied and uncertain, entire classes of new and surprising failure modes are still being found, and we have nothing like a specification of what the system should or should not do, must or must not do, and where it can and cannot be used.
    To take a very concrete example, metal rods have safety characteristics, and they might be rated for use up to some weight limit, under some specific load for some amount of time, in certain temperature ranges, for some amount of time. These can all be tested. If the bar does not stay within a predefined range of characteristics at a given temperature, with a given load, it fails. It can also be found to be acceptable in one temperature range, but not another, or similar. At the end of verification and validation, the bar is deemed to have passed or failed for a given application, based on what the requirements for that larger system are.
    At its best, red-teaming and safety audits of ML systems check lots of known failure modes, and determine whether they are susceptible. There is no pre-defined standard or set of characteristics that are checked, no real ability to consider application specific requirements, and no ability to specify where the system should not or must not be used.
    Until we have some safety standard for machine learning models, they aren't "partly safe" or "assumed safe," or "good enough for consumer use." If we lack a standard for safety, one where there is consensus that it is sufficient for a specific application, exploration or verification of the safety of a machine learning model is meaningless. If a model is released to the public without a clear indication about what the system can safely be used for, with verification that it passed a relevant standard, and clear instruction that it cannot be used elsewhere, it is an unsafe model. Anyone who claims otherwise seems fundamentally confused about what safety means for such systems.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • AF - AGI safety career advice by Richard Ngo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI safety career advice, published by Richard Ngo on May 2, 2023 on The AI Alignment Forum.
    People often ask me for career advice related to AGI safety. This post summarizes the advice I most commonly give. I’ve split it into three sections: general mindset, alignment research and governance work. See also this post I wrote two years ago, containing a bunch of fairly general career advice.
    General mindset
    In order to have a big impact on the world you need to find a big lever. This document assumes that you think, as I do, that AGI safety is the biggest such lever. There are many ways to pull on that lever, though—from research and engineering to operations and field-building to politics and communications. I encourage you to choose between these based primarily on your personal fit—a combination of what you're really good at and what you really enjoy. In my opinion the difference between being a great versus a mediocre fit swamps other differences in the impactfulness of most pairs of AGI-safety-related jobs.
    How should you find your personal fit? To start, you should focus on finding work where you can get fast feedback loops. That will typically involve getting hands-on or doing some kind of concrete project (rather than just reading and learning) and seeing how quickly you can make progress. Eventually, once you've had a bunch of experience, you might notice a feeling of confusion or frustration: why is everyone else missing the point, or doing so badly at this? For some people that involves investigating a specific topic (for me, the question “what’s the best argument that AGI will be misaligned?“); for others it's about applying skills like conscientiousness (e.g. "why can't others just go through all the obvious steps?") Being excellent seldom feels like you’re excellent, because your own abilities set your baseline for what feels normal. (Though note that a few top researchers commented on a draft to say that they disagreed with this point.)
    What if you have that experience for something you don't enjoy doing? I expect that this is fairly rare, because being good at something is often very enjoyable. But in those cases, I'd suggest trying it until you observe that even a string of successes doesn't make you excited about what you're doing; and at that point, probably trying to pivot (although this is pretty dependent on the specific details).
    Lastly: AGI safety is a young and small field; there’s a lot to be done, and still very few people to do it. I encourage you to have agency when it comes to making things happen: most of the time the answer to “why isn’t this seemingly-good thing happening?” or “why aren’t we 10x better at this particular thing?” is “because nobody’s gotten around to it yet”. And the most important qualifications for being able to solve a problem are typically the ability to notice it and the willingness to try. One anecdote to help drive this point home: a friend of mine has had four jobs at four top alignment research organizations; none of those jobs existed before she reached out to the relevant groups to suggest that they should hire someone with her skillset. And this is just what’s possible within existing organizations—if you’re launching your own project, there are far more opportunities to do totally novel things. (The main exception is when it comes to outreach and political advocacy.
    Alignment is an unusual field because the base of fans and supporters is much larger than the number of researchers, and so we should be careful to avoid alignment discourse being dominated by advocates who have little familiarity with the technical details, and come across as overconfident. See the discussion here for more on this.)
    Alignment research
    I’ll start with some high-level recommendations, then give a brief overview of how I see the f...
    20 min