The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Cognitive Emulation: A Naive AI Safety Proposal by Connor Leahy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cognitive Emulation: A Naive AI Safety Proposal, published by Connor Leahy on February 25, 2023 on LessWrong.
    This is part of the work done at Conjecture.
    This post has been reviewed before publication as per our infohazard policy. We thank our external reviewers for their comments and feedback.
    This post serves as a signpost for Conjecture’s new primary safety proposal and research direction, which we call Cognitive Emulation (or “CoEm”). The goal of the CoEm agenda is to build predictably boundable systems, not directly aligned AGIs. We believe the former to be a far simpler and useful step towards a full alignment solution.
    Unfortunately, given that most other actors are racing for as powerful and general AIs as possible, we won’t share much in terms of technical details for now. In the meantime, we still want to share some of our intuitions about this approach.
    We take no credit for inventing any of these ideas, and see our contributions largely in taking existing ideas seriously and putting them together into a larger whole.
    In Brief
    The core intuition is that instead of building powerful, Magical end-to-end systems (as the current general paradigm in AI is doing), we instead focus our attention on trying to build emulations of human-like things. We want to build systems that are “good at chess for the same reasons humans are good at chess.”
    CoEms are a restriction on the design space of AIs to emulations of human-like stuff. No crazy superhuman blackbox Magic, not even multimodal RL GPT5. We consider the current paradigm of developing AIs that are as general and as powerful as possible, as quickly as possible, to be intrinsically dangerous, and we focus on designing bounded AIs as a safer alternative to it.
    Logical, Not Physical Emulation
    We are not interested in direct physical emulation of human brains or simulations of neurons, but of “logical” emulation of thought processes. We don’t care about whether underlying functions are implemented in the same way as they are in the system we are trying to emulate, just that the abstraction over their function holds, and is not leaky.
    Minimize Magic
    In the current paradigm, we generally achieve new capabilities through an increase in Magic. We throw more compute at black boxes that develop internal algorithms we have no insight into. Instead of continually increasing the amount of Magic present in our systems, we want to actively decrease this amount, to more cleanly implement and understand how new capabilities are achieved. Some amount of Magic will realistically be needed to implement many useful functions, but we want to minimize the amount of times we have to use such uninterpretable methods, and clearly keep track of where we are using them, and why.
    CoEms are much “cleaner” than Ems, which are still ultimately big black boxes of weird computation, while in the CoEm paradigm, we keep careful track of where the Magic is and try to keep its presence to a minimum.
    Predict, Track and Bound Capabilities
    In the current dominant machine learning paradigm, there are absolutely no guarantees nor understanding of what is being created. Power laws don’t tell us anything about what capabilities will emerge or what other properties our systems will actually have.
    One of the core hopes of shifting to a CoEm paradigm is that far more deeply understanding what we are building should allow us to predictively bound our system’s capabilities to a human-like regime. This eliminates the problem of being unable to know when an ostensibly harmless system passes from an understandable, harmless capabilities regime into an unprecedented, dangerous regime.
    Exploit the Human Regime
    We want systems that are as safe as humans, for the same reasons that humans have (or don’t have) those safety properties. Any scheme that involves building s...
    10 min
  • AF - Cognitive Emulation: A Naive AI Safety Proposal by Connor Leahy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cognitive Emulation: A Naive AI Safety Proposal, published by Connor Leahy on February 25, 2023 on The AI Alignment Forum.
    This is part of the work done at Conjecture.
    This post has been reviewed before publication as per our infohazard policy. We thank our external reviewers for their comments and feedback.
    This post serves as a signpost for Conjecture’s new primary safety proposal and research direction, which we call Cognitive Emulation (or “CoEm”). The goal of the CoEm agenda is to build predictably boundable systems, not directly aligned AGIs. We believe the former to be a far simpler and useful step towards a full alignment solution.
    Unfortunately, given that most other actors are racing for as powerful and general AIs as possible, we won’t share much in terms of technical details for now. In the meantime, we still want to share some of our intuitions about this approach.
    We take no credit for inventing any of these ideas, and see our contributions largely in taking existing ideas seriously and putting them together into a larger whole.
    In Brief
    The core intuition is that instead of building powerful, Magical end-to-end systems (as the current general paradigm in AI is doing), we instead focus our attention on trying to build emulations of human-like things. We want to build systems that are “good at chess for the same reasons humans are good at chess.”
    CoEms are a restriction on the design space of AIs to emulations of human-like stuff. No crazy superhuman blackbox Magic, not even multimodal RL GPT5. We consider the current paradigm of developing AIs that are as general and as powerful as possible, as quickly as possible, to be intrinsically dangerous, and we focus on designing bounded AIs as a safer alternative to it.
    Logical, Not Physical Emulation
    We are not interested in direct physical emulation of human brains or simulations of neurons, but of “logical” emulation of thought processes. We don’t care about whether underlying functions are implemented in the same way as they are in the system we are trying to emulate, just that the abstraction over their function holds, and is not leaky.
    Minimize Magic
    In the current paradigm, we generally achieve new capabilities through an increase in Magic. We throw more compute at black boxes that develop internal algorithms we have no insight into. Instead of continually increasing the amount of Magic present in our systems, we want to actively decrease this amount, to more cleanly implement and understand how new capabilities are achieved. Some amount of Magic will realistically be needed to implement many useful functions, but we want to minimize the amount of times we have to use such uninterpretable methods, and clearly keep track of where we are using them, and why.
    CoEms are much “cleaner” than Ems, which are still ultimately big black boxes of weird computation, while in the CoEm paradigm, we keep careful track of where the Magic is and try to keep its presence to a minimum.
    Predict, Track and Bound Capabilities
    In the current dominant machine learning paradigm, there are absolutely no guarantees nor understanding of what is being created. Power laws don’t tell us anything about what capabilities will emerge or what other properties our systems will actually have.
    One of the core hopes of shifting to a CoEm paradigm is that far more deeply understanding what we are building should allow us to predictively bound our system’s capabilities to a human-like regime. This eliminates the problem of being unable to know when an ostensibly harmless system passes from an understandable, harmless capabilities regime into an unprecedented, dangerous regime.
    Exploit the Human Regime
    We want systems that are as safe as humans, for the same reasons that humans have (or don’t have) those safety properties. Any scheme that involv...
    10 min
  • EA - Worldview Investigations Team: An Overview by Rethink Priorities
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Worldview Investigations Team: An Overview, published by Rethink Priorities on February 25, 2023 on The Effective Altruism Forum.
    Introduction
    Rethink Priorities’ Worldview Investigations Team (WIT) exists to improve resource allocation within the effective altruism movement, focusing on tractable, high-impact questions that bear on philanthropic priorities. WIT builds on Rethink Priorities’ strengths as a multi-cause, stakeholder-driven, interdisciplinary research organization: it takes action-relevant philosophical, methodological, and strategic problems and turns them into manageable, modelable problems. Rethink Priorities is currently hiring multiple roles to build out the team:
    Worldview Investigations Philosophy Researcher
    Worldview Investigations Quantitative Researcher
    Worldview Investigations Programmer
    These positions offer a significant opportunity for thoughtful and curious individuals to shift the priorities, research areas, and philanthropic spending strategies of major organizations through interdisciplinary work. WIT tackles problems like:
    How should we convert between the units employed in various cost-effectiveness analyses (welfare to DALYs-averted; DALYs-averted to basis points of existential risk averted, etc.)?
    What are the implications of moral uncertainty for work on different cause areas?
    What difference would various levels of risk- and ambiguity-aversion have on cause prioritization? Can those levels of risk- and/or ambiguity-aversion be justified?
    The work involves getting up to speed with the literature in different fields, contacting experts, writing up reasoning in a manner that makes sense to experts and non-experts alike, and engaging with quantitative models.
    The rest of this post sketches WIT’s history, strategy, and theory of change.
    WIT’s History
    Worldview investigation has been part of Rethink Priorities from the beginning, as some of Rethink Priorities’ earliest work was on invertebrate sentience. Invertebrate animals are far more numerous than vertebrate animals, but the vast majority of animal-focused philanthropic resources go to vertebrates rather than invertebrates. If invertebrates aren’t sentient, then this is as it should be, given that sentience is necessary for moral status. However, if invertebrates are sentient, then it would be very surprising if the current resource allocation were optimal. So, this project involved sorting through the conceptual issues associated with assessing sentience, identifying observable proxies for sentience, and scouring the academic literature for evidence with respect to each proxy. In essence, this project developed a simple, transparent tool for making progress on fundamental questions about the distribution of consciousness.
    If the members of a species have a sufficient number of relevant traits, then they probably deserve more philanthropic attention than they’ve received previously.
    Rethink Priorities’ work on invertebrate sentience led directly to its next worldview investigation project, as even if animals are equally sentient, they may not have equal capacity for welfare. For all we know, some animals may be able to realize much more welfare than others. Jason Schukraft took up this question in his five-post series about moral weight, again trying to sort out the conceptual issues and make empirical progress by finding relevant proxies for morally relevant differences. His influential work laid the foundation for the Moral Weight Project, which, again, created a simple, transparent tool for assessing differences in capacity for welfare. Moreover, it developed a way to implement those differences in cost-effectiveness analyses.
    In addition to its work on animals, Rethink Priorities has done research on standard metrics for evaluating health interventions and estimating the burden...
    8 min
  • EA - How major governments can help with the most important century by Holden Karnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How major governments can help with the most important century, published by Holden Karnofsky on February 24, 2023 on The Effective Altruism Forum.
    I’ve been writing about tangible things we can do today to help the most important century go well. Previously, I wrote about helpful messages to spread; how to help via full-time work; and how major AI companies can help.
    What about major governments1 - what can they be doing today to help?
    I think governments could play crucial roles in the future. For example, see my discussion of standards and monitoring.
    However, I’m honestly nervous about most possible ways that governments could get involved in AI development and regulation today.
    I think we still know very little about what key future situations will look like, which is why my discussion of AI companies (previous piece) emphasizes doing things that have limited downsides and are useful in a wide variety of possible futures.
    I think governments are “stickier” than companies - I think they have a much harder time getting rid of processes, rules, etc. that no longer make sense. So in many ways I’d rather see them keep their options open for the future by not committing to specific regulations, processes, projects, etc. now.
    I worry that governments, at least as they stand today, are far too oriented toward the competition frame (“we have to develop powerful AI systems before other countries do”) and not receptive enough to the caution frame (“We should worry that AI systems could be dangerous to everyone at once, and consider cooperating internationally to reduce risk”). (This concern also applies to companies, but see footnote.2)
    In a previous piece, I talked about two contrasting frames for how to make the best of the most important century:
    The caution frame. This frame emphasizes that a furious race to develop powerful AI could end up making everyone worse off. This could be via: (a) AI forming dangerous goals of its own and defeating humanity entirely; (b) humans racing to gain power and resources and “lock in” their values.
    Ideally, everyone with the potential to build something powerful enough AI would be able to pour energy into building something safe (not misaligned), and carefully planning out (and negotiating with others on) how to roll it out, without a rush or a race. With this in mind, perhaps we should be doing things like:
    Working to improve trust and cooperation between major world powers. Perhaps via AI-centric versions of Pugwash (an international conference aimed at reducing the risk of military conflict), perhaps by pushing back against hawkish foreign relations moves.
    Discouraging governments and investors from shoveling money into AI research, encouraging AI labs to thoroughly consider the implications of their research before publishing it or scaling it up, working toward standards and monitoring, etc. Slowing things down in this manner could buy more time to do research on avoiding misaligned AI, more time to build trust and cooperation mechanisms, and more time to generally gain strategic clarity
    The “competition” frame. This frame focuses less on how the transition to a radically different future happens, and more on who's making the key decisions as it happens.
    If something like PASTA is developed primarily (or first) in country X, then the government of country X could be making a lot of crucial decisions about whether and how to regulate a potential explosion of new technologies.
    In addition, the people and organizations leading the way on AI and other technology advancement at that time could be especially influential in such decisions.
    This means it could matter enormously "who leads the way on transformative AI" - which country or countries, which people or organizations.
    Some people feel that we can make confident statements today a...
    8 min
  • LW - Relationship Orientations by DaystarEld
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Relationship Orientations, published by DaystarEld on February 24, 2023 on LessWrong.
    Jacob Falkovich has written a good post on Entertaining vs Building orientations, which made me realize I should probably cross-post my own articles on these sorts of things.(Note: these articles refer largely to normative modern western culture. When I say “most people” or “most relationships,” I’m speaking descriptively, not prescriptively. There are absolutely exceptions to all of it, and if you’re in one of those, or in a subculture in which that exception is the norm, I don’t want to give the impression that there’s anything wrong with that)
    One of the things I’ve noticed after nearly a decade of therapy is that the word “preference” seems insufficiently strong for a lot of things people want that nevertheless don’t rise to the level of being called a need. For most people, not getting their preferred ice cream flavor won’t ruin an otherwise good day, but for some, coming home to find dishes in the sink and laundry on the floor can make the world feel like it’s falling apart.
    This becomes most clear in relationship counseling, where two or more people are trying to live together and accommodate each other’s desires while having their own respected. On some level we know “I prefer a clean home” is not the same as “I prefer vanilla ice cream,” but people don’t often consider how this difference in intensity-of-preferences can impact relationships when they’re unaligned.
    On the other hand, there are some “preferences” we generally understand to be inflexible and important. Asking a heterosexual person to enjoy intercourse with someone of the same sex, or asking a pansexual person to only enjoy porn involving heterosexual pairings, would be considered not just rude but basically impossible. In extreme situations someone might try to enjoy something they don’t, or have a physical reaction while being mentally uncomfortable, and this would generally be understood to be tragic.
    That brings us to a commonly used word that is generally understood to mean more than simple preference: “orientation.”
    I’ve found that a lot of difficulties people have in relationships come from treating things more like preferences than orientations. To be clear, even this is a spectrum. There are clusters on the far ends which can easily be labeled one or the other, but any sort of comprehensive universal list is impossible.
    What we can do is notice the sorts of things that are more useful to treat as orientations. Here’s the list of things I believe most people in relationships explicitly and consciously treat this way:
    Attraction (sexual orientation included as implicit)
    Children (how many, and usually a rough idea of of when they’ll be had)
    Career (roughly how much money each person is expected to make/how many hours worked)
    Religion (decreasingly, but many would still end a marriage if their partner came out as atheist or converted to a different faith)
    Politics (increasingly, particularly among younger folk; “swipe left if you voted for X.”)
    To some degree this feels like a good summary of the sorts of “impersonal” things it makes sense to be explicit and upfront about with your partners as deal-breakers.
    But when we dig deeper into the day-to-day lives of those in relationships to observe the sorts of things that cause ongoing conflict, we see more. Here’s an incomplete list of what I believe people implicitly and often unconsciously treat this way:
    Pets (how many and what kind)
    Living location (assuming you will live together)
    Extended family (how involved will they be)
    Diet (Increasingly common for vegetarians and vegans)
    Cleanliness (both hygiene and home)
    Relationship type (Monogamy vs some form of open or poly. Some make this explicit, but for most people a monogamy is the unquestioned assumption...
    5 min
  • LW - Are there rationality techniques similar to staring at the wall for 4 hours? by trevor
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are there rationality techniques similar to staring at the wall for 4 hours?, published by trevor on February 24, 2023 on LessWrong.
    I'm wondering if one exists, and what is the name of that technique or that family of techniques. It seems like something that CFAR has researched extensively and layered a lot of things on top of.
    4 hours is necessary for anything substantially longer than 1-2 hours, since 6 hours is too long under most circumstances. Obviously, whiteboards and notepads are allowed, but screens and books absolutely must be kept in a different room. I'm not sure how sporadic google searching and book-searching and person-consulting factors into this, because those queries will suck you in and interrupt the state.
    If people are using sleep deprivation, showers, lying in bed, or long drives to think, it's probably primarily the absence of interruption (from books and screens and people) that triggers valuable thoughts and thinking, not the tasks themselves. (although freeway driving might potentially do more good than harm by consistently keeping certain parts of the brain stimulated).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Updates from the Mental Health Funder's Circle by wtroy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Updates from the Mental Health Funder's Circle, published by wtroy on February 24, 2023 on The Effective Altruism Forum.
    The Mental Health Funder’s Circle held its first grant round in the Fall/Winter of 2022. To those of you who applied for this round, we appreciate your patience. As this was our first round of funding, everything took longer than expected. We will iterate on the structure of the funding circle over time, and intend to develop a process that adds value for members and grantees alike.
    A total of $254,000 was distributed to the following three organizations:
    A matching grant of $44,000 to Vida Plena for their work on cost-effective community mental health in Ecuador.
    Two grants totaling $100,000 to Happier Lives Institute for their continued work on subjective wellbeing and cause prioritization research.
    Two grants totaling $110,000 to Rethink Wellbeing to support mental health initiatives for the EA community.
    Our next round of funding is now open, with initial 1-pagers due April 1st. After applications have been reviewed, we will contact promising grantees and make final funding decisions by June 1st. Applications can be found on our website. For more information on the MHFC, see this forum post. Unfortunately, we lack the ability to respond to every application.
    We are excited to find and support impactful organizations working on cost-effective and catalytic mental health interventions. We encourage you to apply, even if you think your project might be outside of our scope!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - How popular is ChatGPT? Part 1: more popular than Taylor Swift by Harlan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How popular is ChatGPT? Part 1: more popular than Taylor Swift, published by Harlan on February 24, 2023 on LessWrong.
    Harlan Stewart, 23 February 2023
    Introduction
    Public attention toward AI seems much higher after the release of ChatGPT at the end of November. But how much higher is it? To better understand this, I looked at search data from Google Trends about ChatGPT, OpenAI, AI, and AI Alignment. Unfortunately, Google Trends only shares relative search volumes instead of the number of searches made for a term or topic. I compared these relative search volumes to other non-AI topics, such as Taylor Swift, to make them more useful. This is similar to adding a familiar “for scale” object in a product photo.
    How to read these graphs
    In the first graph, the data is about searches for the terms in quotation marks, which are exact search terms. In the others, the data is about search “topics,” which are collections of various search terms related to a topic, as defined by Google Trends.
    The vertical axes of these graphs are relative search volume, defined as the percentage of the peak search volume in that graph.
    Data
    ChatGPT is mainstream
    For the time that ChatGPT has been publicly available since November 30 2022, US searches for it outnumbered US searches for Taylor Swift or Drake. However, there were only around a third as many searches for ChatGPT as searches for Wordle, and Wordle itself had only around a third of the search volume that it did in Spring 2022.
    Americans suddenly know about OpenAI
    For the time that OpenAI has existed, since December 10 2015, Americans usually searched for it less than for Blockbuster Video, a retailer that closed in 2014. In the months since ChatGPT was announced, American searches for OpenAI have increased by around 15x to a volume similar to that for Samsung.
    Interest in AI evolved from dinosaurs to birds
    For most of the last decade, there has been a similar number of global searches about AI as about dinosaurs. In the time since DALL-E 2’s beta was announced less than a year ago, global searches about AI have roughly tripled, rising to a volume of global searches similar to that about birds.
    Alignment interest is at an all-time high but still pretty low
    Over the last 10 years, global searches about AI alignment have risen from “digital scent technology” level to “colonization of the moon” level and possibly beyond. Searches about AI alignment seem to have roughly quadrupled in the last two years. Eyeballing this graph, it’s unclear to me whether the announcements of DALL-E 2 or ChatGPT had any significant effect on search volume.
    Discussion
    ChatGPT is receiving mainstream attention. Although I have not done any statistical analysis of these trends, it appears to me that the popularity of ChatGPT is also driving interest in both OpenAI as a company and AI in general. Interest in alignment is also on the rise but still about as obscure an interest as colonization of the moon.
    It’s unclear whether interest in AI will continue to grow, plateau, or drop back to previous levels. This will likely depend on what near-term future progress in AI will look like. If you expect that AI-related news as interesting as ChatGPT will be rare, you might expect interest to decline as the hype fizzles out. If you expect that the pace of interesting AI advancements will continue at its current fast rate, you might expect interest in AI to continue to grow, perhaps becoming even more popular than birds.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Christiano (ARC) and GA (Conjecture) Discuss AI Alignment Cruxes - Summary by Andrea Miotti
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Christiano (ARC) and GA (Conjecture) Discuss AI Alignment Cruxes - Summary, published by Andrea Miotti on February 24, 2023 on LessWrong.
    The following is a summary of a discussion between Paul Christiano (ARC) and Gabriel Alfour, hereafter GA (Conjecture), which took place on December 11, 2022 on Slack. It was held as part of a series of discussions between Conjecture and people from other organizations in the AGI and alignment field. See our retrospective on the Discussions for more information about the project and the format.
    You can read the full transcript of this discussion here (note that it has been lightly edited for readability).
    Introduction
    GA is pessimistic about alignment being solved because he thinks there is (1) an AGI race to the bottom, (2) alignment is hard in ways that we are bad at dealing with, and (3) we don't have a lot of time to get better, given the pace of the race.
    Christiano clarifies: does GA expect a race to the bottom because investment in alignment will be low, people won’t be willing to slow development/deployment if needed, or something else? He predicts alignment investment will be 5-50% of total investment, depending on how severe risk appears. If the risks look significant-but-kind-of-subtle, he expects getting 3-6 months of delay based on concern. In his median doomy case, he expects 1-2 years of delay.
    GA expects lower investment (1-5%). More crucially, though, GA expects it to be hard to turn funding and time into effective research given alignment’s difficulty.
    Alignment Difficulty, Feedback Loops, & Phase Shifts
    GA’s main argument for alignment difficulty is that getting feedback on our research progress is difficult, because
    Core concepts and desiderata in alignment are complex and abstract.
    We are bad at factoring complex, abstract concepts into smaller more tractable systems without having a lot of quantitative feedback.
    We are bad at building feedback loops when working on abstract concepts
    We are bad at coming to agreement on abstract concepts.
    All this will make it difficult to predict when phase shifts – eg qualitative changes to how systems are representing information, which might break our interpretability methods – will occur.
    Such phase shifts seem likely to occur when we shift from in vitro to in vivo, which makes it particularly likely that the alignment techniques we build in vitro won’t be robust to them.
    Despite theorists arguing connecting AI systems to e.g. the internet is dangerous for this reason, labs will do it, because the path from current systems to future danger is complex and we may not see legibly catastrophic failures until it is too late. So, even getting better at predicting may not help.
    Christiano disagrees building feedback loops is hard in alignment. We can almost certainly study reward hacking in vitro in advance, together with clear measurements of whether we are succeeding at mitigating the problem in a way that should be expected to generalize to AI coup. Conditioned on deceptive alignment being a problem that emerges, there’s a >50% chance that we can study it in the same sense.
    Furthermore, Christiano argues most plausible approaches to AI alignment have much richer feedback loops than the general version of either of these problems. For example, if you have an approach that requires building a kind of understanding of the internals of your model then you can test whether you can build that kind of understanding in not-yet-catastrophic models. If you have an approach that requires your model being unable to distinguish adversarial examples from deployment cases, you can test whether your models can make that distinction. You can generally seek methods that don’t have particular reasons to break at the same time that things become catastrophic.
    GA is skeptical that alignment techni...
    14 min
  • LW - Retrospective on the 2022 Conjecture AI Discussions by Andrea Miotti
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Retrospective on the 2022 Conjecture AI Discussions, published by Andrea Miotti on February 24, 2023 on LessWrong.
    At the end of 2022, following the success of the 2021 MIRI Conversations, Conjecture started a project to host discussions about AGI and alignment with key people in the field. The goal was simple: surface positions and disagreements, identify cruxes, and make these debates public whenever possible for collective benefit.
    Given that people and organizations will have to coordinate to best navigate AI's increasing effects, this is the first, minimum-viable coordination step needed to start from. Coordination is impossible without at least common knowledge of various relevant actors' positions and models.
    People sharing their beliefs, discussing them and making as much as possible of that public is strongly positive for a series of reasons.
    First, beliefs expressed in public discussions count as micro-commitments or micro-predictions, and help keep the field honest and truth-seeking. When things are only discussed privately, humans tend to weasel around and take inconsistent positions over time, be it intentionally or involuntarily.
    Second, commenters help debates progress faster by pointing out mistakes.
    Third, public debates compound. Knowledge shared publicly leads to the next generation of arguments being more refined, and progress in public discourse.
    We circulated a document about the project to various groups in the field, and invited people from OpenAI, DeepMind, Anthropic, Open Philanthropy, FTX Future Fund, ARC, and MIRI, as well as some independent researchers to participate in the discussions. We prioritized speaking to people at AGI labs, given that they are focused on building AGI capabilities.
    The format of discussions was as follows:
    A brief initial exchange with the participants to decide on the topics of discussion. By default, the discussion topic was “How hard is Alignment?”, since we've found we disagree with most people about this, and the reasons for it touch on many core cruxes about AI.
    We held the discussion synchronously for ~120 minutes, in writing, each on a dedicated, private Slack channel.
    We involved a moderator when possible. The moderator's role was to help participants identify and address their cruxes, move the conversation forward, and summarize points of contention.
    We planned to publish cleaned up versions of the transcripts and summaries to Astral Codex Ten, LessWrong, and the EA Forum. Participants were given the opportunity to clarify positions and redact information they considered infohazards or PR risks, as well as veto publishing altogether. We included this clause specifically to address the concerns expressed by people at AI labs, who expected heavy scrutiny by leadership and communications teams on what they can state publicly.
    People from ARC, DeepMind, and OpenAI, as well as one independent researcher agreed to participate. The two discussions with Paul Christiano and John Wentworth will be published shortly. One discussion with a person working at DeepMind is pending approval before publication. After a discussion with an OpenAI researcher took place, OpenAI strongly recommended against publishing, so we will not publish it.
    Most people we were in touch with were very interested in participating. However, after checking with their own organizations, many returned saying their organizations would not approve them sharing their positions publicly.
    This was in spite of the extensive provisions we made to reduce downsides for them: making it possible to edit the transcript, veto publishing, strict comment moderation, and so on. We think organizations discouraging their employees from speaking openly about their views on AI risk is harmful, and we want to encourage more openness.
    We are pausing the project for...
    5 min