The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Cost-effectiveness of professional field-building programs for AI safety research by Center for AI Safety
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cost-effectiveness of professional field-building programs for AI safety research, published by Center for AI Safety on July 10, 2023 on The Effective Altruism Forum.
    Summary
    This post explores the cost-effectiveness of AI safety field-building programs aimed at ML professionals, specifically the Trojan Detection Challenge (a prize), NeurIPS ML Safety Social, and NeurIPS ML Safety Workshop.
    We estimate the benefit of these programs in ‘Quality-Adjusted Research Years’, using cost-effectiveness models built for the Center for AI Safety (introduction post here, full code here).
    We intend for these models to support - not determine - strategic decisions. We do not believe, for instance, that programs which a model rates as lower cost-effectiveness are necessarily not worthwhile as part of a portfolio of programs.
    The models’ tentative results, summarized below, suggest that field-building programs for professionals compare favorably to ‘baseline’ programs - directly funding a talented research scientist or PhD student working on trojans research for 1 year or 5 years respectively. Further, the cost-effectiveness of these programs can be significantly improved with straightforward modifications - such as focusing a hypothetical prize on a more ‘relevant’ research avenue, or running a hypothetical workshop with a much smaller budget.
    ProgramCost (USD)Benefit (counterfactual expected QARYs)Cost-effectiveness (QARYs per $1M)Trojan Detection Challenge65,00026390NeurIPS ML Safety Social520015029,000NeurIPS ML Safety Workshop110,0003603300Hypothetical: Power Aversion Prize50,0004909900Hypothetical: Cheaper Workshop35,0002507000Baseline: Scientist Trojans500,000 84170Baseline: PhD Trojans250,0008.735
    For readers who are after high-level takeaways, including which factors are driving these results, skip ahead to the cost-effectiveness in context section. For those keen on understanding the model and results in more detail, read on as we:
    Give important disclaimers. (Read more.)
    Direct you to background information about this project. (Read more.)
    Walk through the model. (Read more.)
    Contrast these programs with one another, and with funding researchers directly. (Read more.)
    Consider the scalability and robustness properties of the model. (Read more.)
    Disclaimer
    This analysis is a starting point for discussion, not a final verdict. The most critical reasons for this are that:
    These models are reductionist. Even if we have avoided other pitfalls associated with cost-effectiveness analyses, the models might ignore factors that turn out to be crucial in practice, including (but not limited to) interactions between programs, threshold effects, and diffuse effects.
    The models’ assumptions are first-pass guesses, not truths set in stone. Most assumptions are imputed second-hand following a short moment of thought, before being adjusted ad-hoc for internal consistency and differences of beliefs between Center for AI Safety (CAIS) staff and external practitioners. In some cases, parameters have been redefined since initial practitioner input.
    Instead, the analyses in this post represent an initial effort in explicitly laying out assumptions, in order to take a more systematic approach towards AI safety field-building.
    Background
    For an introduction to our approach to modeling - including motivations for using models, the benefits and limitations of our key metric, guidance for adopting or adapting the models for your own work, comparisons between programs for students and professionals, and more - refer to the introduction post.
    2. The models’ default parameters are based on practitioner surveys and the expertise of CAIS staff. Detailed information on the values and definitions of these parameters, and comments on parameters with delicate definitions or contestable views, can be found i...
    38 min
  • EA - Cost-effectiveness of student programs for AI safety research by Center for AI Safety
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cost-effectiveness of student programs for AI safety research, published by Center for AI Safety on July 10, 2023 on The Effective Altruism Forum.
    Summary
    This post explores the cost-effectiveness of field-building programs for students, specifically the Atlas Fellowship (a rationality program, with some AI safety programming), MLSS (an ML safety course for undergraduates), a top-tier university student group, and undergraduate research stipends.
    We estimate the benefit of these programs in ‘Quality-Adjusted Research Years’, using cost-effectiveness models built for the Center for AI Safety (introduction post here, full code here). Since our framework focuses on benefits for technical AI safety research exclusively, we will not account for other benefits of programs with broader objectives, such as the Atlas Fellowship.
    We intend for these models to support - not determine - strategic decisions. We do not believe, for instance, that programs which a model rates as lower cost-effectiveness are necessarily not worthwhile as part of a portfolio of programs.
    The models’ tentative results, summarized below, suggest that student groups and undergraduate research stipends are considerably more cost-effective than Atlas and MLSS. (With many important caveats and uncertainties, discussed in the post.) Additionally, student groups and undergraduate research stipends compare favorably to ‘baseline’ programs - directly funding a talented research scientist or PhD student working on trojans research for 1 year or 5 years respectively.
    ProgramCost (USD) Benefit (counterfactual expected QARYs)Cost-effectiveness (QARYs per $1M)Atlas9,000,000 434.7MLSS330,0006.419Student Group350,00050140Undergraduate Stipends50,00017340Baseline: Scientist Trojans500,00084170Baseline: PhD Trojans250,0008.735
    For readers who are after high-level takeaways, including which factors are driving these results, skip ahead to the cost-effectiveness in context section. For those keen on understanding the model and results in more detail, read on as we:
    Give important disclaimers. (Read more.)
    Direct you to background information about this project. (Read more.)
    Walk through the model. (Read more.)
    Contrast these programs with one another, and with funding researchers directly. (Read more.)
    Test the robustness of the model. (Read more.)
    Disclaimers
    This analysis is a starting point for discussion, not a final verdict. The most critical reasons for this are that:
    These models are reductionist. Even if we have avoided other pitfalls associated with cost-effectiveness analyses, the models might ignore factors that turn out to be crucial in practice, including (but not limited to) interactions between programs, threshold effects, and diffuse effects.
    The models’ assumptions are first-pass guesses, not truths set in stone. Most assumptions are imputed second-hand following a short moment of thought, before being adjusted ad-hoc for internal consistency and differences of beliefs between Center for AI Safety (CAIS) staff and external practitioners. In some cases, parameters have been redefined since initial practitioner input.
    This caveat is particularly important for the Atlas Fellowship, where we have not discussed parameter values with key organizers.
    Instead, the analyses in this post represent an initial effort in explicitly laying out assumptions, in order to take a more systematic approach towards AI safety field-building.
    Background
    For an introduction to our approach to modeling - including motivations for using models, the benefits and limitations of our key metric, guidance for adopting or adapting the models for your own work, comparisons between programs for students and professionals, and more - refer to the introduction post.
    The models’ default parameters are based on practitioner surveys and the ex...
    34 min
  • LW - I think Michael Bailey's dismissal of my autogynephilia questions for Scott Alexander and Aella makes very little sense by tailcalled
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I think Michael Bailey's dismissal of my autogynephilia questions for Scott Alexander and Aella makes very little sense, published by tailcalled on July 10, 2023 on LessWrong.
    There's been a lot of autogynephilia talk lately. One subject that sometimes comes up, but hasn't been discussed much on LessWrong, is how common autogynephilia is in cis women.
    Two datasets that are sometimes used for this question are Scott Alexander's and Aella's. Part of how they happened to be made is that I reached out to Scott and Aella, suggesting experimental questions for assessing autogynephilia, in ways that might function in cis women too and not just cis men:
    Picture a very beautiful woman. How sexually arousing would you find it to imagine being her?
    Do you find the thought of masturbating alone as a woman to be erotic?
    This is somewhat different from usual autogynephilia measures, which contain questions such as:
    Did you ever feel sexually aroused when putting on females' underwear or clothing?
    Have you ever become sexually aroused while picturing your nude female breasts?
    The reason I didn't include these latter questions is because it seems likely to me that they will be interpreted differently for males and females (e.g. if males do not have female anatomy, then they cannot be aroused by it literally, so instead they get aroused by imagining some other female anatomy that they don't actually have), and because these usual questions seem very bad if taken literally ("ever" and "while" rather than "how frequently" and "by" seem like there could be a lot of ways to get affirmative answers while not actually being autogynephilic - though the low rate of endorsement among women suggests to me that they are not taking it literally?).
    Anyway, Michael Bailey (activist researcher for autogynephilia ideology) responds in Aporia Magazine, saying "It's important not to confound y'know being seen as a woman or having a female body from having sex with a partner while having a female body or the prospect of going out on a date while wearing sexy clothes".
    I don't think this makes any sense. "Do you find the thought of masturbating alone as a woman to be erotic?" explicitly states that one doesn't have a partner around. "Picture a very beautiful woman. How sexually arousing would you find it to imagine being her?" doesn't suggest a partner any more than "Did you ever feel sexually aroused when putting on females' underwear or clothing?" or "Have you ever become sexually aroused while picturing your nude female breasts?" does. This seems like a completely unfounded critique.
    I think people who dismiss these questions should admit that they are starting with The Bottom Line that cis women are not autogynephilic, and reasoning backwards about what sorts of measures they do or do not want to endorse, rather than pretending that they are driven by evidence and desire for good measurement.
    Since nobody really knows how to assess this.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - “Reframing Superintelligence” + LLMs + 4 years by Eric Drexler
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: “Reframing Superintelligence” + LLMs + 4 years, published by Eric Drexler on July 10, 2023 on LessWrong.
    Background
    In January 2019, FHI published Reframing Superintelligence, a book-length technical report on prospects for advanced AI. OpenAI published the first paper on GPT-2 a month later. Advances since then have been strange and rapid, and I’d like to revisit the report in light of what we have learned. In brief, I think that the abstract conceptual model of AI development and organization proposed in Reframing fits today’s reality quite well, even though LLM-based technologies have diverged far from anything I’d anticipated.
    Below, you'll find an abstract of the abstract of the report, followed by a series of section-level mini-summaries with update comments. I’ve omitted sections that are either outside the intended focus of this article or are too broad and forward-looking to summarize.
    A significant impetus behind “Reframing Superintelligence” was to challenge a prevailing notion of advanced AI (equating superintelligent-level AI with a superintelligent agent), which has, in my view, been assigned disproportionate weight and skewed the balance of alignment research. The report offers an alternative framework that includes both risks and opportunities that are overlooked by agent-centric perspectives.
    Note that this reframing is additive rather than subtractive: My intention is not to disregard agent-focused concerns - their importance is assumed, not debated. Indeed, the AI services model anticipates a world in which dangerous superintelligent agents could emerge with relative ease, and perhaps unavoidably. My aim is to broaden the working ontology of the community to include systems in which superintelligent-level capabilities can take a more accessible, transparent, and manageable form, open agencies rather than unitary agents. This framework highlights different risks and expands the the solution-space for familiar problems.
    Finally, when I refer “LLMs”, please read this as encompassing multimodal models (GPT-4!) with considerations that carry over to a wider range of foundation models.
    Abstract of the Abstract
    “Reframing Superintelligence” reviews the concept of superintelligent AI systems as utility-driven agents and suggests expanding our ontology of superintelligence to include compositions of AI systems that can best be understood through their structures, relationships, development processes, and the services they can provide - services that can include AI research and development itself. This perspective gives rise to the “Comprehensive AI Services” (CAIS) model, which proposes general intelligence as a property of flexible systems of services in which task-focused agents are among the components. The CAIS model envisions AI services expanding toward asymptotically comprehensive superintelligent-level performance, including the service of providing new services in line with human objectives and informed by strong models of human (dis)approval.
    This reframing has broad implications for AI prospects, including AI safety and strategy, practical applications of advanced AI systems, and the fundamental relationship between goals and intelligence. In this context, the emergence of strongly self-modifying agents with superintelligent-level capabilities remains a concern, yet the desirability and potential instrumental value of such agents is greatly diminished.
    Section mini-summaries + updates
    1. R&D automation provides the most direct path to an intelligence explosion
    Self-transforming AI agents have no natural role in recursive improvement. A more direct path would instead involve AI-enabled AI development in which new capabilities are implemented without any system being self-modifying.
    Today’s most striking applications of AI to AI development are app...
    28 min
  • LW - Consider Joining the UK Foundation Model Taskforce by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consider Joining the UK Foundation Model Taskforce, published by Zvi on July 10, 2023 on LessWrong.
    A few months ago, Ian Hogarth wrote the Financial Times Op-Ed headlined “We must slow down the race to God-like AI.”
    A few weeks ago, he was appointed head of the UK Foundation Model Taskforce, and given 100 million pounds to dedicate to AI safety, to universal acclaim. Soon there will also be a UK Global AI Summit.
    He wrote an op-ed in The Times asking everyone for their help, with accompanying Twitter thread. Based on a combination of sources, I am confident that this effort has strong backing for the time being although that is always fragile, and that it is aimed squarely at the real target of extinction risk from AI, with a strong understanding of what it would mean to have an impact on that.
    Once again: The real work begins now.
    The UK Taskforce will need many things in order to succeed. It will face opposition within and outside the government, and internationally. There is a narrow window until the AI summit to hit the ground running and establish capability and credibility.
    The taskforce represents a startup government mindset that makes me optimistic, and that seems like the best hope for making government get things done again, including on other vital causes that are not AI, and not only in the UK. We likely only get one shot at this. If the taskforce fails, there will probably not be another such effort.
    Right now, the main bottleneck is that the taskforce is talent constrained. There is an urgent need to scale up rapidly with people who can hit the ground running and allow the taskforce to orient.
    If you are in position to help, then with the possible exception of creating your own organization at scale, I believe this is the highest leverage opportunity currently available.
    To reach out and see if you can help, you can fill out this Google Form here.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Incentives from a causal perspective by Tom Everitt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Incentives from a causal perspective, published by Tom Everitt on July 10, 2023 on The AI Alignment Forum.
    Post 4 of Towards Causal Foundations of Safe AGI, preceded by Post 1: Introduction, Post 2: Causality, and Post 3: Agency.
    By Tom Everitt, James Fox, Ryan Carey, Matt MacDermott, Sebastian Benthall, and Jon Richens, representing the Causal Incentives Working Group. Thanks also to Toby Shevlane and Aliya Ahmad.
    “Show me the incentive, and I’ll show you the outcome” - Charlie Munger
    Predicting behaviour is an important question when designing and deploying agentic AI systems. Incentives capture some key forces that shape agent behaviour, which don’t require us to fully understand the internal workings of a system.
    This post shows how a causal model of an agent and its environment can reveal what the agent wants to know and what it wants to control, as well as how it will respond to commands and influence its environment. A complementary result shows that some incentives can only be inferred from a causal model, so a causal model of the agent’s environment is strictly necessary for a full incentive analysis.
    Value of information
    What information would an agent like to learn? Consider, for example, Mr Jones deciding whether to water his lawn, based on the weather report, and whether the newspaper arrived in the morning. Knowing the weather means that he can water more when it will be sunny than when it will be raining, which saves water and improves the greenness of the grass. The weather forecast therefore has information value for the sprinkler decision, and so does the weather itself, but the newspaper arrival does not.
    We can quantify how useful observing the weather is for Mr Jones, by comparing his expected utility in a world in which he does observe the weather, to a world in which he doesn’t. (This measure only makes sense if we can assume that Mr Jones adapts appropriately to the different worlds, i.e. he needs to be agentic in this sense.)
    The causal structure of the environment reveals which variables provide useful information. In particular, the d-separation criterion captures whether information can flow between variables in a causal graph when a subset of variables are observed. In single-decision graphs, value of information is possible when there is an information-carrying path from a variable to the agent’s utility node, when conditioning on the decision node and its parents (i.e. the “observed” nodes).
    For example, in the above graph, there is an information-carrying path from forecast to weather to grass greenness, when conditioning on the sprinkler, forecast and newspaper. This means that the forecast can (and likely will) provide useful information about optimal watering. In contrast, there is no such path from the newspaper arrival. In that case, we call the information link from the newspaper to the sprinkler nonrequisite.
    Understanding what information an agent wants to obtain is useful for several reasons. First, in e.g. fairness settings, the question of why a decision was made is often as important as what the decision was. Did gender determine a hiring decision? Value of information can help us understand what information the system is trying to glean from its available observations (though a formal understanding of proxies remains an important open question).
    More philosophically, some researchers consider an agent’s cognitive boundary as the events that the agent cares to measure and influence. Events that lack value of information must fall outside the measuring part of this boundary.
    Response Incentives
    Related to the value of information are response incentives: what changes in the environment would a decision chosen by an optimal policy respond to? Changes are operationalised as post-policy interventions, i.e. as interventions that...
    11 min
  • EA - Basefund Has Entered Its Trial Phase by bob
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Basefund Has Entered Its Trial Phase, published by bob on July 10, 2023 on The Effective Altruism Forum.
    We are excited to announce that Basefund has entered its trial phase. Starting today, individuals who have previously donated to effective charities and are currently facing financial trouble can apply for hardship assistance at basefund.org.
    During this trial phase, if your application is accepted, you will receive the lowest amount among the following three options:
    The payout suggested by our hardship examiners
    50% of your donations to cost-effective charities made in 2022 and 2023
    1,000 USD or the equivalent amount in another currency
    Beware that during the trial phase, Basefund may halt operations or change its rules without warning.
    If you're aware of anyone who has previously donated to effective charities and is currently facing financial hardship, please let them know about our fund. If you're not quite sure whether you qualify as experiencing hardship, we recommend you to submit an application.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • LW - Consciousness as a conflationary alliance term by Andrew Critch
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consciousness as a conflationary alliance term, published by Andrew Critch on July 10, 2023 on LessWrong.
    Tl;dr: In this post, I argue that the concept of 'consciousness' is more conflated than people realize, in that there's a lot of divergence in what people mean by "consciousness", and people are unaware of the degree of divergence. This confusion allows the formation of broad alliances around the value of consciousness, even when people don't agree on what it actually means. I call alliances built around conflated terms "conflationary alliances".
    Executive Summary
    Part 1: Mostly during my PhD, I somewhat-methodically interviewed a couple dozen people to figure out what they meant by consciousness, and found that (a) there seems to be a surprising amount of diversity in what people mean by the "consciousness", and (b) they are often surprised to find out that other people mean different things when they say "consciousness". This has implications for AI safety advocacy because AI will sometimes be feared and/or protected on the grounds that it is "conscious", and it's good to be able to navigate these debates wisely.
    (Other heavily conflated terms in AI discourse might include "fairness", "justice", "alignment", and "safety", although I don't want to debate any of those cases here. This post is going to focus on consciousness, and general ideas about the structure of alliances built around confused concepts in general.)
    Part 2: When X is a conflated term like "consciousness", large alliances can form around claims like "X is important" or "X should be protected". Here, the size of the alliance is a function of how many concepts get conflated with X. Thus, the alliance grows because of the confusion of meanings, not in spite of it. I call this a conflationary alliance. Persistent conflationary alliances resist disambiguation of their core conflations, because doing so would break up the alliance into factions who value the more precisely defined terms. The resistance to deconflation can be deliberate, or merely a social habit or inertia.
    Part 1: What people mean by "consciousness".
    "Consciousness" is an interesting word, because many people have already started to notice that it's a confused term, yet there is still widespread agreement that conscious beings have moral value. You'll even find some people taking on strange positions like "I'm not conscious" or "I don't know if I'm conscious" or "lookup tables are conscious", as if rebelling against the implicit alliance forming around the "consciousness" concept. What's going on here?
    To investigate, over about 10 years between 2008 and 2018 I informally interviewed dozens of people who I noticed were interested in talking about consciousness, for 1-3 hours each. I did not publish these results, and never intended to, because I was mainly just investigating for my own interest. In retrospect, it would have been better, for me and for anyone reading this post, if I'd made a proper anthropological study of it. I'm sorry that didn't happen. In any case, here is what I have to share:
    "Methodology"
    Extremely informal; feel free to skip or just come back to this part if you want to see my conclusions first.
    Whom did I interview? Mostly academics I met in grad school, in cognitive science, AI, ML, and mathematics. In an ad hoc manner at academic or other intellectually-themed gatherings, whenever people talked about consciousness, I gravitated toward the conversation and tried to get someone to spend a long conversation with me to unpack what they meant.
    How did I interview them? What I asked each person was to take some time to look inside their own minds - sometimes starting out by paying attention to just their bodies, if introspection was hard for them - and try to describe to me in more detail the thing they were calli...
    17 min
  • AF - “Reframing Superintelligence” + LLMs + 4 years by Eric Drexler
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: “Reframing Superintelligence” + LLMs + 4 years, published by Eric Drexler on July 10, 2023 on The AI Alignment Forum.
    Background
    In January 2019, FHI published Reframing Superintelligence, a book-length technical report on prospects for advanced AI. OpenAI published the first paper on GPT-2 a month later. Advances since then have been strange and rapid, and I’d like to revisit the report in light of what we have learned. In brief, I think that the abstract conceptual model of AI development and organization proposed in Reframing fits today’s reality quite well, even though LLM-based technologies have diverged far from anything I’d anticipated.
    Below, you'll find an abstract of the abstract of the report, followed by a series of section-level mini-summaries with update comments. I’ve omitted sections that are either outside the intended focus of this article or are too broad and forward-looking to summarize.
    A significant impetus behind “Reframing Superintelligence” was to challenge a prevailing notion of advanced AI (equating superintelligent-level AI with a superintelligent agent), which has, in my view, been assigned disproportionate weight and skewed the balance of alignment research. The report offers an alternative framework that includes both risks and opportunities that are overlooked by agent-centric perspectives.
    Note that this reframing is additive rather than subtractive: My intention is not to disregard agent-focused concerns - their importance is assumed, not debated. Indeed, the AI services model anticipates a world in which dangerous superintelligent agents could emerge with relative ease, and perhaps unavoidably. My aim is to broaden the working ontology of the community to include systems in which superintelligent-level capabilities can take a more accessible, transparent, and manageable form, open agencies rather than unitary agents. This framework highlights different risks and expands the the solution-space for familiar problems.
    Finally, when I refer “LLMs”, please read this as encompassing multimodal models (GPT-4!) with considerations that carry over to a wider range of foundation models.
    Abstract of the Abstract
    “Reframing Superintelligence” reviews the concept of superintelligent AI systems as utility-driven agents and suggests expanding our ontology of superintelligence to include compositions of AI systems that can best be understood through their structures, relationships, development processes, and the services they can provide - services that can include AI research and development itself. This perspective gives rise to the “Comprehensive AI Services” (CAIS) model, which proposes general intelligence as a property of flexible systems of services in which task-focused agents are among the components. The CAIS model envisions AI services expanding toward asymptotically comprehensive superintelligent-level performance, including the service of providing new services in line with human objectives and informed by strong models of human (dis)approval.
    This reframing has broad implications for AI prospects, including AI safety and strategy, practical applications of advanced AI systems, and the fundamental relationship between goals and intelligence. In this context, the emergence of strongly self-modifying agents with superintelligent-level capabilities remains a concern, yet the desirability and potential instrumental value of such agents is greatly diminished.
    Section mini-summaries + updates
    1. R&D automation provides the most direct path to an intelligence explosion
    Self-transforming AI agents have no natural role in recursive improvement. A more direct path would instead involve AI-enabled AI development in which new capabilities are implemented without any system being self-modifying.
    Today’s most striking applications of AI to AI develo...
    28 min
  • AF - Open-minded updatelessness by Nicolas Macé
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open-minded updatelessness, published by Nicolas Macé on July 10, 2023 on The AI Alignment Forum.
    Summary
    Bounded agents might be unaware of possibilities relevant to their decision-making. That is, they may not just be uncertain, but fail to conceive of some relevant hypotheses entirely. What's more, commitment races might pressure early AGIs into adopting an updateless policy from a position of limited awareness. What happens then when a committed AGI becomes aware of a possibility that’d have changed which commitment it’d have wanted to make in the first place? Motivated by this question, we develop "open-minded" extensions of updatelessness, where agents revise their priors upon experiencing awareness growth and reevaluate their commitment to a plan relative to the revised prior.
    Introduction
    Bounded agents may be unaware of propositions relevant to the decision problem they face. That is, they don’t merely have uncertainty, but also fail to conceive of the full set of possibilities relevant to their decision-making. (For example, when playing a board game, one might be unaware of some crucial rule. Moreover, one’s awareness might grow, e.g. when one discovers such a rule midgame.)Awareness growth raises questions for commitment. What if one commits, and then discovers an important consideration, whose conception would have changed the plan one would have wanted to commit to? The earlier one commits, the less time one has to think about the relevant considerations, and the more likely this problem is to arise.
    We are interested in preventing AGI systems from making catastrophic strategic commitments. One reason that not-fully-aware AGI systems could make bad commitments is that important hypotheses are missing for their priors. For example, they might fail to conceive of certain attitudes towards fairness that bargaining counterparts might possess. One might think that AGI agents would quickly become aware of all relevant hypotheses, and make commitments only then. But commitment race dynamics might pressure early AGIs into making commitments before thinking carefully, and in particular, in a position of limited awareness. From The Commitment Races problem (emphasis ours):
    If two consequentialists are playing a game of Chicken, the first one to throw out their steering wheel wins. [.] More generally, consequentialist agents are motivated to make commitments as soon as possible, since that way they can influence the behavior of other consequentialist agents who may be learning about them. Of course, they will balance these motivations against the countervailing motive to learn more and think more before doing drastic things. The problem is that the first motivation will push them to make commitments much sooner than would otherwise be optimal. So they might not be as smart as us when they make their commitments, at least not in all the relevant ways.
    In a commitment race, agents who are known to be updateless have a strategic advantage over their updateful counterparts. Therefore, commitment races might introduce pressures for agents to become updateless as soon as possible, and one might worry that early-stage AGIs hastily adopt a version of updatelessness that mishandles awareness growth.
    So we think it’s important to map out different ways one can go about being updateless when one’s awareness can grow. The aims of this post are to (1) argue for the relevance of unawareness to updatelessness and (2) explore several approaches to extending updatelessness to cases of awareness growth. Specifically, we introduce closed- and open-minded versions of updatelessness:
    An agent is closed-mindedly updateless if they ignore awareness growth and continue being committed to their current plan. This approach is dynamically consistent, but clearly undesirable if one’s prior l...
    26 min