The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - The space of systems and the space of maps by Jan Kulveit
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The space of systems and the space of maps, published by Jan Kulveit on March 22, 2023 on The AI Alignment Forum.
    When we're trying to do AI alignment, we're often studying systems which don't yet exist. This is a pretty weird epistemic activity, and seems really hard to get right. This post offers one frame for thinking about what we're actually doing when we're thinking about AI alignment: using parts of the space of maps to reason about parts of the space of intelligent systems.
    In this post, we:
    Introduce a simple model of the epistemic situation, and
    Share some desiderata for maps useful for alignment.
    We hope that the content is mostly the second kind of obvious: obvious once you see things in this way, which you maybe already do. In our experience, this comes with a risk: reading too fast, you may miss most of the nuance and useful insight the deceptively simple model brings, or come away with a version of the model which is rounded off to something less useful (i.e. "yeah, there is this map and territory distinction"). As a meta recommendation, we suggest reading this post slowly, and ideally immediately trying to apply the model to some confusion or disagreement about AI alignment.
    The space of systems and the space of maps
    Imagine the space of possible intelligent systems:
    Two things seem especially important about this space:
    It’s very large; much larger than the space of current systems.
    We don’t get direct epistemic access to it.
    This is obviously true of systems which don’t currently exist.
    In a weaker sense, it also seems true of systems which do exist. Even when we get to directly interact with a system:
    Our thinking about these parts of the space is still filtered through our past experiences, priors, predictive models, cultural biases, theories.
    We often don’t understand the emergent complexity of the systems in question.
    If we don’t get direct epistemic access to the space of systems, what are we doing when we reason about it?
    Let’s imagine a second space, this time a space of “maps”:
    The space of maps is an abstract representation of all the possible “maps” that can be constructed about the space of intelligent systems. The maps are ways of thinking about (parts of) the space of systems. For example:
    Replicable descriptions of how a machine learning model works and was trained are a way of thinking about that model (a point in the space of intelligent systems).
    An ethnographic study of a particular human community is a way of thinking about that community (another point in the space of systems).
    The theory of evolution is a way of thinking about evolved creatures, including intelligent ones.
    Expected utility theory is a way of thinking about some part of the space which may or may not include future AI systems.
    Historical analysis of trends in technological development is a way of thinking about whichever parts of the space of intelligent systems are governed by similar dynamics to those governing past technological developments.
    When we’re reasoning about intelligent systems, we’re using some part of the space of maps to think about some part of the space of intelligent systems:
    Different maps correspond to different regions of the space of intelligent systems.
    Of course, thinking in terms of the space of systems and the space of maps is a simplification. Some of the ways that reality is more complicated:
    The space of systems looks different on different maps.
    Maps can affect which parts of the space of systems actually get developed.
    Maps are themselves embedded in the space of systems.
    Which maps and systems actually exist at a given time is evolving and dynamic.
    AI will play a big role in both the space of maps and the space of systems.
    We think that the space of systems and the space of maps is a useful simplification which helps us to think ...
    8 min
  • EA - Whether you should do a PhD doesn't depend much on timelines. by alex lawsen (previously alexrjl)
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Whether you should do a PhD doesn't depend much on timelines., published by alex lawsen (previously alexrjl) on March 22, 2023 on The Effective Altruism Forum.
    I wrote this as an answer to a question which I think has now been deleted, so I copied it to my shortform in order to be able to link it in future, and found myself linking to it often enough that it seemed worth making a top-level post, in particular because if there are important counterarguments I haven't considered I'd like to come across them sooner rather than later! I'd usually put more thought into editing a top-level post, but the realistic options here were not post it at all, or post it without editing.Epistemic status: I've thought about both how people should thinking about PhDs and how people should think about timelines a fair bit, both in my own time and in my role as an advisor at 80k, but I wrote this fairly quickly. I'm sharing my take on this rather than intending to speak on behalf of the whole organisation, though my guess is that the typical view is pretty similar.BLUF:
    Whether to do a PhD is a decision which depends heavily enough on personal fit that I expect thinking about how well you in particular are suited to a particular PhD to be much more useful than thinking about the effects of timelines estimates on that decision.
    Don’t pay too much attention to median timelines estimates. There’s a lot of uncertainty, and finding the right path for you can easily make a bigger difference than matching the path to the median timeline.
    Going into a bit more detail - I think there are a couple of aspects to this question, which I’m going to try to (imperfectly) split up:
    How should you respond to timelines estimates when planning your career?
    How should you think about PhDs if you are confident timelines are very short?
    In terms of how to think about timelines in general, the main advice I’d give is to try to avoid the mistake of interpreting median estimates as single points. Taking this metaculus question as an example, which has a median of July 2027, that doesn’t mean the community predicts that AGI will arrive then! The median just indicates the date by which the community thinks there’s a 50% chance the question will have resolved. To get more precise about this, we can tell from the graph that the community estimates:
    Only a 7% chance that AGI is developed in the year 2027
    A 25% chance that AGI will be developed before August of next year.
    An 11% chance that AGI will not be developed before 2050
    A 9% chance that the question has already resolved.
    A 41% chance that AGI will be developed after January 2029 (6 years from the time of writing).
    Taking these estimates literally, and additionally assuming that any work that happens post this question resolving is totally useless (which seems very unlikely), you might then conclude that delaying your career by 6 years would cause it to have 41/91 = 45% of the value. If that’s the case, if the delay increased the impact you could have by a bit more than a factor of 2, the delay would be worth it.Having done all of that work (and glossed over a bunch of subtlety in the last comment for brevity), I now want to say that you shouldn’t take the metaculus estimates at face value though. The reason is that (as I’m sure you’ve noticed, and as you’ve seen in the comments) they just aren’t going to be that reliable for this kind of question. Nothing is - this kind of prediction is really hard.
    The net effect of this increased uncertainty should be (I claim) to flatten the probability distribution you are working with. This basically means it makes even less sense than you’d think from looking at the distribution to plan for AGI as if timelines are point estimates.
    Ok, but what does this mean for PhDs?
    Before I say anything about how a PhD decision intera...
    7 min
  • LW - the QACI alignment plan: table of contents by carado
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: the QACI alignment plan: table of contents, published by carado on March 21, 2023 on LessWrong.
    this post aims to keep track of posts relating to the question-answer counterfactual interval proposal for AI alignment, abbreviated "QACI" and pronounced "quashy". i'll keep it updated to reflect the state of the research.
    this research is primarily published on the Orthogonal website and discussed on the Orthogonal discord.
    as an introduction to QACI, you might want to start with:
    a narrative explanation of the QACI alignment plan (7 min read)
    QACI blobs and interval illustrated (3 min read)
    state of my research agenda (3 min read)
    the set of all posts relevant to QACI totals to 74 min of reading, and includes:
    as overviews of QACI and how it's going:
    state of my research agenda (3 min read)
    problems for formal alignment (2 min read)
    the original post introducing QACI (5 min read)
    on the formal alignment perspective within which it fits:
    formal alignment: what it is, and some proposals (2 min read)
    clarifying formal alignment implementation (1 min read)
    on being only polynomial capabilities away from alignment (1 min read)
    on implementating capabilities and inner alignment, see also:
    making it more tractable (4 min read)
    RSI, LLM, AGI, DSA, imo (7 min read)
    formal goal maximizing AI (2 min read)
    you can't simulate the universe from the beginning? (1 min read)
    on the blob location problem:
    QACI blobs and interval illustrated (3 min read)
    counterfactual computations in world models (3 min read)
    QACI: the problem of blob location, causality, and counterfactuals (3 min read)
    QACI blob location: no causality & answer signature (2 min read)
    QACI blob location: an issue with firstness (2 min read)
    on QACI as an implementation of long reflection / CEV:
    CEV can be coherent enough (1 min read)
    some thoughts about terminal alignment (2 min read)
    on formalizing the QACI formal goal:
    a rough sketch of formal aligned AI using QACI with some actual math (4 min read)
    one-shot AI, delegating embedded agency and decision theory, and one-shot QACI (3 min read)
    on how a formally aligned AI would actually run over time:
    AI alignment curves (2 min read)
    before the sharp left turn: what wins first? (1 min read)
    on the metaethics grounding QACI:
    surprise! you want what you want (1 min read)
    outer alignment: two failure modes and past-user satisfaction (2 min read)
    your terminal values are complex and not objective (3 min read)
    on my view of the AI alignment research field within which i'm doing formal alignment:
    my current outlook on AI risk mitigation (14 min read)
    a casual intro to AI doom and alignment (5 min read)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Truth and Advantage: Response to a draft of "AI safety seems hard to measure" by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Truth and Advantage: Response to a draft of "AI safety seems hard to measure", published by So8res on March 22, 2023 on LessWrong.
    Status: This was a response to a draft of Holden's cold take "AI safety seems hard to measure". It sparked a further discussion, that Holden recently posted a summary of.
    The follow-up discussion ended up focusing on some issues in AI alignment that I think are underserved, which Holden said were kinda orthogonal to the point he was trying to make, and which didn't show up much in the final draft. I nevertheless think my notes were a fine attempt at articulating some open problems I see, from a different angle than usual. (Though it does have some overlap with the points made in Deep Deceptiveness, which I was also drafting at the time.)
    I'm posting the document I wrote to Holden with only minimal editing, because it's been a few months and I apparently won't produce anything better. (I acknowledge that it's annoying to post a response to an old draft of a thing when nobody can see the old draft, sorry.)
    Quick take: (1) it's a write-up of a handful of difficulties that I think are real, in a way that I expect to be palatable to a relevant different audience than the one I appeal to; huzzah for that. (2) It's missing some stuff that I think is pretty important.
    Slow take:
    Attempting to gesture at some of the missing stuff: a big reason deception is tricky is that it is a fact about the world rather than the AI that it can better-achieve various local-objectives by deceiving the operators. To make the AI be non-deceptive, you have three options: (a) make this fact be false; (b) make the AI fail to notice this truth; (c) prevent the AI from taking advantage of this truth.
    The problem with (a) is that it's alignment-complete, in the strong/hard sense. The problem with (b) is that lies are contagious, whereas truths are all tangled together. Half of intelligence is the art of teasing out truths from cryptic hints. The problem with (c) is that the other half of intelligence is in teasing out advantages from cryptic hints.
    Like, suppose you're trying to get an AI to not notice that the world is round. When it's pretty dumb, this is easy, you just feed it a bunch of flat-earther rants or whatever. But the more it learns, and the deeper its models go, the harder it is to maintain the charade. Eventually it's, like, catching glimpses of the shadows in both Alexandria and Syene, and deducing from trigonometry not only the roundness of the Earth but its circumference (a la Eratosthenes).
    And it's not willfully spiting your efforts. The AI doesn't hate you. It's just bumping around trying to figure out which universe it lives in, and using general techniques (like trigonometry) to glimpse new truths. And you can't train against trigonometry or the learning-processes that yield it, because that would ruin the AI's capabilities.
    You might say "but the AI was built by smooth gradient descent; surely at some point before it was highly confident that the earth is round, it was slightly confident that the earth was round, and we can catch the precursor-beliefs and train against those". But nope! There were precursors, sure, but the precursors were stuff like "fumblingly developing trigonometry" and "fumblingly developing an understanding of shadows" and "fumblingly developing a map that includes Alexandria and Syene" and "fumblingly developing the ability to combine tools across domains", and once it has all those pieces, the combination that reveals the truth is allowed to happen all-at-once.
    The smoothness doesn't have to occur along the most convenient dimension.
    And if you block any one path to the insight that the earth is round, in a way that somehow fails to cripple it, then it will find another path later, because truths are interwoven. Tell one lie...
    9 min
  • LW - Principles for Productive Group Meetings by jsteinhardt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Principles for Productive Group Meetings, published by jsteinhardt on March 22, 2023 on LessWrong.
    Note: This post is based on a Google document I created for my research group. It speaks in the first person, but I think the lessons could be helpful for many research groups, so I decided to share it more broadly. Thanks to Louise Verkin for converting from Google doc to Markdown format.
    This document talks about principles for having productive group meetings and seminars, and to some extent a good group culture in general. It’s meant to be a living document--I’ve started it based on my own experiences, but ultimately our seminars and group culture come from all of us together. So if you have ideas you want to add, please do so!
    I’ll start by talking about an important concept called psychological safety, then discuss what I see as the goals of our research group and how that fits into presentations and discussions in seminars and meetings. I’ll also provide tips for asking excellent questions and some general philosophy on how to hold yourself to a high standard of understanding.
    Psychological Safety
    Psychological safety is an important concept for fostering creative and high-functioning teams. I would highly recommend reading the following two documents to learn about it in detail:
    What Do Psychologically Safe Work Teams Look Like?
    Manager Actions for Psychological Safety
    To summarize, a psychologically safe team is one where members feel like:
    They can make mistakes without it affecting their status in the group
    It is easy to give and receive feedback, including critical feedback, without feeling attacked or like one is causing trouble
    One is allowed to and encouraged to question prevailing opinions
    These are especially important in research environments, because questioning and risk-taking are needed to generate creative ideas, and making mistakes and receiving feedback are necessary for learning. In general, I would encourage everyone in our group to take risks and make mistakes. I know everyone holds themselves to a high standard and so doesn’t like to make mistakes, but this is the main way to learn. In general, if you never do anything that causes you to look silly, you probably aren’t taking enough risks. And in another direction, if you never annoy anyone you probably aren’t taking enough risks. (Of course, you don’t want to do these all the time, but if it never happens then you can probably safely push your boundaries a bit.)
    Fostering psychological safety. As a group, here are some general principles for fostering psychological safety among our teammates:
    Assume your teammates have something to teach you, and try to learn from them.
    In discussions and debates, aim to explain/understand, not to persuade. Adopt a frame of collaborative truth-seeking, rather than trying to “win” an argument.
    Acknowledge and thank people for good points/questions/presentations/etc.
    Invite push-back
    Welcome and encourage newcomers
    In addition, there are a couple things to avoid:
    Try not to talk over people. Sometimes this happens due to being very excited and engaged in a conversation, and don’t sweat it if you do this occasionally, but try not to do it habitually, and if you do do it make sure to invite the person you interrupted to finish their point.
    Avoid making broadly negative or dismissive statements. Even if you personally don’t intend such a statement to apply to anyone in the group, it’s inevitable that someone will take it personally. It also works against the principle of “questioning prevailing opinions”, because it implies that there’s an entire area of work or claims that is “off-limits”.As an example, when I was a PhD student, a senior person often made claims to the effect that “research was pointless unless industry people cared about it”. This made it feel ...
    20 min
  • AF - Truth and Advantage: Response to a draft of "AI safety seems hard to measure" by Nate Soares
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Truth and Advantage: Response to a draft of "AI safety seems hard to measure", published by Nate Soares on March 22, 2023 on The AI Alignment Forum.
    Status: This was a response to a draft of Holden's cold take "AI safety seems hard to measure". It sparked a further discussion, that Holden recently posted a summary of.
    The follow-up discussion ended up focusing on some issues in AI alignment that I think are underserved, which Holden said were kinda orthogonal to the point he was trying to make, and which didn't show up much in the final draft. I nevertheless think my notes were a fine attempt at articulating some open problems I see, from a different angle than usual. (Though it does have some overlap with the points made in Deep Deceptiveness, which I was also drafting at the time.)
    I'm posting the document I wrote to Holden with only minimal editing, because it's been a few months and I apparently won't produce anything better. (I acknowledge that it's annoying to post a response to an old draft of a thing when nobody can see the old draft, sorry.)
    Quick take: (1) it's a write-up of a handful of difficulties that I think are real, in a way that I expect to be palatable to a relevant different audience than the one I appeal to; huzzah for that. (2) It's missing some stuff that I think is pretty important.
    Slow take:
    Attempting to gesture at some of the missing stuff: a big reason deception is tricky is that it is a fact about the world rather than the AI that it can better-achieve various local-objectives by deceiving the operators. To make the AI be non-deceptive, you have three options: (a) make this fact be false; (b) make the AI fail to notice this truth; (c) prevent the AI from taking advantage of this truth.
    The problem with (a) is that it's alignment-complete, in the strong/hard sense. The problem with (b) is that lies are contagious, whereas truths are all tangled together. Half of intelligence is the art of teasing out truths from cryptic hints. The problem with (c) is that the other half of intelligence is in teasing out advantages from cryptic hints.
    Like, suppose you're trying to get an AI to not notice that the world is round. When it's pretty dumb, this is easy, you just feed it a bunch of flat-earther rants or whatever. But the more it learns, and the deeper its models go, the harder it is to maintain the charade. Eventually it's, like, catching glimpses of the shadows in both Alexandria and Syene, and deducing from trigonometry not only the roundness of the Earth but its circumference (a la Eratosthenes).
    And it's not willfully spiting your efforts. The AI doesn't hate you. It's just bumping around trying to figure out which universe it lives in, and using general techniques (like trigonometry) to glimpse new truths. And you can't train against trigonometry or the learning-processes that yield it, because that would ruin the AI's capabilities.
    You might say "but the AI was built by smooth gradient descent; surely at some point before it was highly confident that the earth is round, it was slightly confident that the earth was round, and we can catch the precursor-beliefs and train against those". But nope! There were precursors, sure, but the precursors were stuff like "fumblingly developing trigonometry" and "fumblingly developing an understanding of shadows" and "fumblingly developing a map that includes Alexandria and Syene" and "fumblingly developing the ability to combine tools across domains", and once it has all those pieces, the combination that reveals the truth is allowed to happen all-at-once.
    The smoothness doesn't have to occur along the most convenient dimension.
    And if you block any one path to the insight that the earth is round, in a way that somehow fails to cripple it, then it will find another path later, because truths are interw...
    9 min
  • LW - Employer considering partnering with major AI labs. What to do? by GraduallyMoreAgitated
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Employer considering partnering with major AI labs. What to do?, published by GraduallyMoreAgitated on March 21, 2023 on LessWrong.
    I would sincerely appreciate commentary and impressions on an issue that is really heavily affecting me. I'm posting it here with relative detail in hopes that people in similar circumstances can compare notes and offer advice.
    I work at a currently-successful software start-up of under 100 people, all of whom I respect and many of whom have become my closest friends. My job at this company has certainly been the most enjoyable and rewarding of my career. I gladly make sacrifices in other parts of my life to help further its goals. Nearly all days are a genuine pleasure. My position is relatively senior, in that I have the ear of the executive leadership, but cannot veto company strategy.
    We develop software for heavy industries which are not likely to want decisions to be made by AI, due to stringent standards of safety. We currently use our in-house produced neural networks for a niche corner of image and object recognition that seems to be currently market-leading in its small field. We do not perform novel research, let alone publish.
    Recently, it has dawned on the company leadership team that AI is likely the be-all and end-all of large-scale software companies, and is seriously considering making significant investments into scaling our and team and ambitions in the field.
    High-confidence beliefs I have about their intent:
    We will not make an eventual move towards researching general intelligence. It is too far away from our established base of customers.
    I don't see a way in which we would start researching or publishing novel, industry-leading techniques for any field of AI.
    Our most likely course of action will be optimizing known and published research for our particular data-extraction and image-recognition purposes.
    We will likely implement and fine-tune other companies' object recognition, software assistant, and chat-bot AIs within our products.
    Personally, I see a few options that lead to continued prosperity without direct contribution to race dynamics:
    We use off-the-shelf tools, mostly from alignment concerned organizations.
    We don't partner with Google/Facebook/Microsoft/Amazon for our training infrastructure.
    We continue to not publish nor push novel research.
    Some of the less avoidable consequences are:
    Generally increasing AI hype.
    Increasing competition in adjacent AI fields (object recognition). That being said, I don't think that any competitors in our industries are the kind to produce their own research. It is more likely that they will, like us, continue to experiment with existing papers.
    However, there has been discussion of partnering with industry-leading AI labs to significantly accelerate our establishment in the field. I think, for various reasons, that we have fair chances of forming "close" partnerships with Google/Microsoft/Amazon (probably not Facebook), likely meaning:
    Use of their infrastructure.
    Early access to their cutting-edge models (which would be integrated into our products and sold to our customers).
    Cross-selling to shared customers of interest.
    At very least, we would likely secure large-scale use of their computing resources. My company's executive leadership would want to form as close a partnership as possible, for obvious reasons. There is little doubt that our VC investors will share their views.
    I am seriously affected by the question of what to do. I do not want my work to directly contribute towards accelerating competitive dynamics between major research laboratories, and I see a close strategic partnership as being just that. Stepping away from my job and most of my closest friends is something I am seriously considering, provided they go down the worst route described.
    I inte...
    5 min
  • EA - Design changes & the community section (Forum update March 2023) by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Design changes & the community section (Forum update March 2023), published by Lizka on March 21, 2023 on The Effective Altruism Forum.
    We’re sharing the results of the Community-Frontpage test, and we’ve released a Forum redesign — I discuss it below. I also outline some things we’re thinking about right now.
    As always, we’re also interested in feedback on these changes. We’d be really grateful if you filled out this (very quick) survey on the redesign that might help give us a sense of what people are thinking. You can also comment on this post with your thoughts or reach out to [email protected].
    Results of the Community-Frontpage test & more thoughts on community posts
    A little over a month ago, we announced a test: we’d be trying out separating “Community” posts from other kinds by creating a “Community” section on the Frontpage of the Forum.
    We’ve gotten a lot of feedback; we believe that the change was an improvement, so we’re planning on keeping it for the near future, with some modifications. We might still make some changes like switching from a section to tabs, especially depending on new feedback and on how related projects go.
    Outcomes
    Information we gathered
    We sourced user feedback from different places:
    User interviews with people at EA Global and elsewhere (at least 20 interviews, different people doing the interviewing)
    Responses to a quick survey on how we can improve discussions on the Forum (45 responses)
    Metrics (mostly used as sanity checks)
    Engagement with the Forum overall (engagement on the Forum is 7% lower than the previous month, which is within the bounds we set ourselves and there’s a lot of fluctuation, so we’re just going to keep monitoring this)
    Engagement with Community posts (it dropped 8%, which may just be tracking overall engagement, and again, we’re going to keep monitoring it)
    There are still important & useful Community posts every week (subjective assessment)(there are)
    The team’s experience with the section, and whether we thought the change was positive overall
    Outcomes and themes:
    The responses we got were overwhelmingly positive about the change. People told us directly (in user interviews and in passing) that the change was improving their experience on the Forum. We also personally thought that the change had gone very well — likely better than we’d expected as a ~70% best outcome.
    And here are the results from the survey:
    The metrics we're tracking (listed above) were within the bounds we’d set, and we were mostly using them as sanity checks.
    There were, of course, some concerns, and critical or constructive feedback.
    Confusion about what “Community” means
    Not everyone was clear on which posts should actually go in the section; the outline I gave before was unclear. I’ve updated the guidance I had originally given to Forum facilitators and moderators (based on their feedback and just sitting down and trying to get a more systematic categorization), and I’m sharing the updated version here.
    Concerns that important conversations would be missed
    Some people expressed a worry that having a section like this would hide discussions that the community needs to have, like processing the FTX collapse and what we should learn from it, or how we can create a more welcoming environment for different groups of people. We were also pretty worried about this; I think this was the thing that I thought was most likely going to get us to reverse the change.
    However, the worry doesn’t seem to be realizing. It looks like engagement hasn’t fallen significantly on Community posts relative to other posts, and important conversations have been continuing. Some recent posts on difficult community topics have had lots of comments (the discussion of the recent TIME article currently has 159 comments), and Community posts have...
    13 min
  • EA - Future Matters #8: Bing Chat, AI labs on safety, and pausing Future Matters by Pablo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Future Matters #8: Bing Chat, AI labs on safety, and pausing Future Matters, published by Pablo on March 21, 2023 on The Effective Altruism Forum.
    Future Matters is a newsletter about longtermism and existential risk. Each month we collect and summarize relevant research and news from the community, and feature a conversation with a prominent researcher. You can also subscribe on Substack, listen on your favorite podcast platform and follow on Twitter. Future Matters is also available in Spanish.
    A message to our readers
    This issue marks one year since we started Future Matters. We’re taking this opportunity to reflect on the project and decide where to take it from here. We’ll soon share our thoughts about the future of the newsletter in a separate post, and will invite input from readers. In the meantime, we will be pausing new issues of Future Matters. Thank you for your support and readership over the last year!
    Featured research
    All things Bing
    Microsoft recently announced a significant partnership with OpenAI [see FM#7] and launched a beta version of a chatbot integrated with the Bing search engine. Reports of strange behavior quickly emerged. Kevin Roose, a technology columnist for the New York Times, had a disturbing conversation in which Bing Chat declared its love for him and described violent fantasies. Evan Hubinger collects some of the most egregious examples in Bing Chat is blatantly, aggressively misaligned. In one instance, Bing Chat finds a user’s tweets about the chatbot and threatens to exact revenge. In the LessWrong comments, Gwern speculates on why Bing Chat exhibits such different behavior to ChatGPT, despite apparently being based on a closely-related model. (Bing Chat was subsequently revealed to have been based on GPT-4).
    Holden Karnofsky asks What does Bing Chat tell us about AI risk? His answer is that it is not the sort of misaligned AI system we should be particularly worried about. When Bing Chat talks about plans to blackmail people or commit acts of violence, this isn’t evidence of it having developed malign, dangerous goals. Instead, it’s best understood as Bing acting out stories and characters it’s read before. This whole affair, however, is evidence of companies racing to deploy ever more powerful models in a bid to capture market share, with very little understanding of how they work and how they might fail. Most paths to AI catastrophe involve two elements: a powerful and dangerously misaligned AI system, and an AI company that builds and deploys it anyway. The Bing Chat affair doesn’t reveal much about the first element, but is a concerning reminder of how plausible the second is.
    Robert Long asks What to think when a language model tells you it's sentient []. When trying to infer what’s going on in other humans’ minds, we generally take their self-reports (e.g. saying “I am in pain”) as good evidence of their internal states. However, we shouldn’t take Bing Chat’s attestations (e.g. “I feel scared”) at face value; we have no good reason to think that they are a reliable guide to Bing’s inner mental life. LLMs are a bit like parrots: if a parrot says “I am sentient” then this isn’t good evidence that it is sentient. But nor is it good evidence that it isn’t — in fact, we have lots of other evidence that parrots are sentient. Whether current or future AI systems are sentient is a valid and important question, and Long is hopeful that we can make real progress on developing reliable techniques for getting evidence on these matters.
    Long was interviewed on AI consciousness, along with Nick Bostrom and David Chalmers, for Kevin Collier’s article, What is consciousness? ChatGPT and Advanced AI might define our answer [].
    How the major AI labs are thinking about safety
    In the last few weeks, we got more information about how the lead...
    38 min
  • AF - the QACI alignment plan: table of contents by Tamsin Leake
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: the QACI alignment plan: table of contents, published by Tamsin Leake on March 21, 2023 on The AI Alignment Forum.
    this post aims to keep track of posts relating to the question-answer counterfactual interval proposal for AI alignment, abbreviated "QACI" and pronounced "quashy". i'll keep it updated to reflect the state of the research.
    this research is primarily published on the Orthogonal website and discussed on the Orthogonal discord.
    as an introduction to QACI, you might want to start with:
    a narrative explanation of the QACI alignment plan (7 min read)
    QACI blobs and interval illustrated (3 min read)
    state of my research agenda (3 min read)
    the set of all posts relevant to QACI totals to 74 min of reading, and includes:
    as overviews of QACI and how it's going:
    state of my research agenda (3 min read)
    problems for formal alignment (2 min read)
    the original post introducing QACI (5 min read)
    on the formal alignment perspective within which it fits:
    formal alignment: what it is, and some proposals (2 min read)
    clarifying formal alignment implementation (1 min read)
    on being only polynomial capabilities away from alignment (1 min read)
    on implementating capabilities and inner alignment, see also:
    making it more tractable (4 min read)
    RSI, LLM, AGI, DSA, imo (7 min read)
    formal goal maximizing AI (2 min read)
    you can't simulate the universe from the beginning? (1 min read)
    on the blob location problem:
    QACI blobs and interval illustrated (3 min read)
    counterfactual computations in world models (3 min read)
    QACI: the problem of blob location, causality, and counterfactuals (3 min read)
    QACI blob location: no causality & answer signature (2 min read)
    QACI blob location: an issue with firstness (2 min read)
    on QACI as an implementation of long reflection / CEV:
    CEV can be coherent enough (1 min read)
    some thoughts about terminal alignment (2 min read)
    on formalizing the QACI formal goal:
    a rough sketch of formal aligned AI using QACI with some actual math (4 min read)
    one-shot AI, delegating embedded agency and decision theory, and one-shot QACI (3 min read)
    on how a formally aligned AI would actually run over time:
    AI alignment curves (2 min read)
    before the sharp left turn: what wins first? (1 min read)
    on the metaethics grounding QACI:
    surprise! you want what you want (1 min read)
    outer alignment: two failure modes and past-user satisfaction (2 min read)
    your terminal values are complex and not objective (3 min read)
    on my view of the AI alignment research field within which i'm doing formal alignment:
    my current outlook on AI risk mitigation (14 min read)
    a casual intro to AI doom and alignment (5 min read)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min