The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Retrospective on recent activity of Riesgos Catastróficos Globales by Jaime Sevilla
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Retrospective on recent activity of Riesgos Catastróficos Globales, published by Jaime Sevilla on May 1, 2023 on The Effective Altruism Forum.
    The new team of Riesgos Catastróficos Globales started their job two months ago.
    During this time, they have been working on two reports on what we have identified as top priorities for the management of Global Catastrophic Risks from Spanish-Speaking countries: food security during Abrupt Sunlight-Reduction Scenarios (e.g. nuclear winter) and AI regulation.
    In this article, I will cover their output in more depth and future plans, with some reflections on how the project is going.
    The short version is that I am reasonably pleased, and the directive board has decided to continue the project for two more months. The team's productivity has exceeded my expectations, though I see opportunities for improvement in our quality assurance, formation and outreach. We remain short of funding; if you want to support our work you can donate through our donation portal.
    Intellectual output
    In the last two months, the team has been working on two major reports and several minor outputs.
    1) Report on food security in Argentina during abrupt sun-reducing scenarios (ASRS), in collaboration with ALLFED. In this report, we explain the important role Argentina could have during ASRS to mitigate global famine. We sketch several policies that would be useful inclusions in an emergency plan, such as resilient food deployment, together with suggestions on which public organisms could implement them.
    2) Report on AI regulation for the EU AI Act Spanish sandbox (forthcoming). We are interviewing and eliciting opinions from several experts, to compile an overview of AI risk for Spanish policymakers and proposals to make the most out of the upcoming EU AI sandbox.
    3) An article about AI regulation in Spain. In this short article, we explain the relevance of Spain for AI regulation in the context of the EU AI Act. We propose four policies that could be tested in the upcoming sandbox. It serves as a preview of the report I mentioned above.
    4) An article about the new GCR mitigation law in USA, reporting on its meaning and proposing similar initiatives for Spanish-Speaking countries.
    5) Two statements about Our Common Agenda Policy Briefs, in collaboration with the Simon Institute.
    Overall, I think we have done a good job of contextualizing the research done in the international GCR community. However, I feel we rely a lot on the involvement of the direction board for quality assurance, and our limited time means that some mistakes and misconceptions will likely have made it to publication.
    Having said that, I am pleased with the results. The team has been amazingly productive, publishing a 60-page report in two months and several minor publications alongside it.
    In the future, we will be involving more experts for a more thorough review process. This also means that we will be erring towards producing shorter reports, which can be more thoroughly checked and are better for engaging policy-makers.
    Formation
    Early in the project, we identified the education of our staff as a key challenge to overcome. Our staff has work experience and credentials, but their exposure to the GCR literature was limited.
    We undertook several activities to address this lack of formation:
    Knowledge transfer talks with Spanish-speaking experts from our directive board and advisory network (Juan García from ALLFED, Jaime Sevilla from Epoch, Clarissa Rios Rojas from CSER).
    A GCR reading group with curated reading recommendations.
    An online course taught by Sandra Malagón from Carreras con Impacto.
    A dedicated course on the basics of Machine Learning.
    I am satisfied with the results, and I see a clear progression in the team. In hindsight, I think we erred on the side of too much form...
    8 min
  • EA - [Linkpost] ‘The Godfather of A.I.’ Leaves Google and Warns of Danger Ahead by Darius1
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] ‘The Godfather of A.I.’ Leaves Google and Warns of Danger Ahead, published by Darius1 on May 1, 2023 on The Effective Altruism Forum.
    Geoffrey Hinton—a pioneer in artificial neural networks—just left Google, as reported by the New York Times: ‘The Godfather of A.I.’ Leaves Google and Warns of Danger Ahead (archive version).
    Some highlights from the article [emphasis added]:
    Dr. Hinton said he has quit his job at Google, where he has worked for more than decade and became one of the most respected voices in the field, so he can freely speak out about the risks of A.I. A part of him, he said, now regrets his life’s work.
    “I console myself with the normal excuse: If I hadn’t done it, somebody else would have,” Dr. Hinton said
    “It is hard to see how you can prevent the bad actors from using it for bad things,” Dr. Hinton said.
    Dr. Hinton, often called “the Godfather of A.I.,” did not sign either of those letters and said he did not want to publicly criticize Google or other companies until he had quit his job.
    In 1972, as a graduate student at the University of Edinburgh, Dr. Hinton embraced an idea called a neural network...At the time, few researchers believed in the idea. But it became his life’s work.
    In 2012, Dr. Hinton and two of his students in Toronto, Ilya Sutskever and Alex Krishevsky, built a neural network that could analyze thousands of photos and teach itself to identify common objects, such as flowers, dogs and cars.
    Google spent $44 million to acquire a company started by Dr. Hinton and his two students. And their system led to the creation of increasingly powerful technologies, including new chatbots like ChatGPT and Google Bard. Mr. Sutskever went on to become chief scientist at OpenAI. In 2018, Dr. Hinton and two other longtime collaborators received the Turing Award, often called “the Nobel Prize of computing,” for their work on neural networks.
    last year, as Google and OpenAI built systems using much larger amounts of data, his view changed. He still believed the systems were inferior to the human brain in some ways but he thought they were eclipsing human intelligence in others. “Maybe what is going on in these systems,” he said, “is actually a lot better than what is going on in the brain.”
    As companies improve their A.I. systems, he believes, they become increasingly dangerous. “Look at how it was five years ago and how it is now,” he said of A.I. technology. “Take the difference and propagate it forwards. That’s scary.”
    Until last year, he said, Google acted as a “proper steward” for the technology, careful not to release something that might cause harm. But now that Microsoft has augmented its Bing search engine with a chatbot — challenging Google’s core business — Google is racing to deploy the same kind of technology. The tech giants are locked in a competition that might be impossible to stop, Dr. Hinton said.
    Down the road, he is worried that future versions of the technology pose a threat to humanity because they often learn unexpected behavior from the vast amounts of data they analyze. This becomes an issue, he said, as individuals and companies allow A.I. systems not only to generate their own computer code but actually run that code on their own.
    “The idea that this stuff could actually get smarter than people — a few people believed that,” he said. “But most people thought it was way off. And I thought it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.”
    Dr. Hinton believes that the race between Google and Microsoft and others will escalate into a global race that will not stop without some sort of global regulation.
    But that may be impossible, he said. Unlike with nuclear weapons, he said, there is no way of knowing whether companies or countries are working on the technol...
    5 min
  • LW - The Apprentice Thread 2 by hath
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Apprentice Thread 2, published by hath on May 1, 2023 on LessWrong.
    A while back, @Zvi set up a Schelling point for mentorship. I, personally, had some pretty good experiences from that thread, and wanted to continue the tradition, now that the old thread's been inactive for some time.
    To see some of my thoughts on mentorship, check out this dialogue between Elizabeth and I.
    I'll be copying Zvi's formatting for comments:
    Replies to this post should take the form of any of the following:
    [MENTOR]: A non-binding indication of potential interest in mentorship. Mention that you might, at some point, be interested in taking on an apprentice. This commits you to nothing. Make sure to indicate what you’d be teaching them and what project would likely be involved, and open with [MENTOR]. You are free to include contact info, or not include it and monitor replies.
    Replies to this comment to indicate potential interest in being the apprentice, marked [APPRENTICE], which should include a method of further contact.
    [APPRENTICE]: A non-binding indication of potential interest in being an apprentice. Mention that you might, at some point, be interested in being an apprentice. This commits you to nothing. Make sure to indicate what you’re interested in being an apprentice in and learning, and an indication of what’s motivating you.
    Replies to this comment to indicate potential interest in being the mentor, marked with [MENTOR], which should include a method of further contact.
    [NORMAL] You’re free to comment as per normal, but start with [NORMAL] in the top-level for clarity.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - What 2025 looks like by Ruby
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What 2025 looks like, published by Ruby on May 1, 2023 on LessWrong.
    I wrote almost all of this in mid-March before the FLI Open Letter and Eliezer's TIME piece. Weirdly, after just six weeks I'd likely write something different. This isn't as finished/polished as I'd like, but better to ship it as is than languish incomplete forever.
    Not quite two years ago, Daniel Kokotaljo wrote a highly acclaimed post about What 2026 looks like that aimed to tell a single detail future history ("trajectory") about how world events play out in coming years.
    As I'm trying to orient myself to what is about to happen, I figured it'd be useful to make my own attempt at this kind of thing. Daniel was bolder than me and tried to imagine 2026 from 2021; I simply don't think I can imagine anything five years out and writing out the rest of 2023, 2024, and 2025 has given me plenty to think about further.
    Daniel's vignette places a lot of attention on the size (parameters, compute) and capabilities of models. Daniel and others when imagining the future also want to describe changes in world economy (of which GDP may or may not be a good measure). Those elements feel less interesting to me to thank about directly than other effects.
    Major Variables
    Over the years, it feels like the following are key to track.
    Object-level capabilities of the models. Elaboration unnecessary
    Adoption and application. It's become salient to me recently that not only is the raw "inherent" power level of models relevant to the world, but also how much they're being harnessed. Widespread use and application of AI will determine things like attention, hype, societal effects, competition, government involvement, etc.
    Hype and attention. Importance, neglectedness, tractability. Many of us were used to thinking about how to achieve AI alignment in a world where not that many people were focused on Alignment or even AGI at all. As we gradually (or not that gradually) get into a world where everyone is thinking about AI, it's a pretty different gameboard.
    For example, I joined the LessWrong dev team in 2018/2019 and in those days we were still trying to revive LessWrong and restore it's activity levels. Now we're preparing to handle the droves of new users who've found their way to the site due to newfound interest in AI. It significantly alters the strategy and projects we're working on, and I expect similar effects for most people in the Alignment ecosystem.
    Economic effects. How much is AI actually creating value? Is it making people more productive at their jobs? Is it replacing jobs? Etc.
    Social Effects. There's already a question of how much did Russian chatbots influence the last election. With LLMs capable of producing human-quality text, I do think we should worry about how the online discourse goes from this point on.
    Politicization. AI hasn't been deeply politicized yet. I would be surprised if it stays that way, and tribal affiliation affecting people's views and actions will be notable.
    Government attention and action. Governments already are paying attention to AI, and I think that will only increase over time, and eventually they'll take actions in response.
    Effects on the Alignment and EA community. The increased attention on AI (and making AI good) means that many more people will find their way to our communities, and recruiting will be much easier (it already is compared to a few years ago, as far as I can tell). The Great Influx as I've been calling it, is going to change things and strain things.
    Alignment progress. As other events unfold and milestones are reached, I think it's worth thinking about how much progress has realistically been made on various technical Alignment questions.
    This is not an exhaustive list on things worth predicting or tracking, but it's some of the major ones according to...
    27 min
  • LW - Geoff Hinton Quits Google by Adam Shai
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Geoff Hinton Quits Google, published by Adam Shai on May 1, 2023 on LessWrong.
    The NYTimes reports that Geoff Hinton has quit his role at Google:
    On Monday, however, he officially joined a growing chorus of critics who say those companies are racing toward danger with their aggressive campaign to create products based on generative artificial intelligence, the technology that powers popular chatbots like ChatGPT.
    Dr. Hinton said he has quit his job at Google, where he has worked for more than a decade and became one of the most respected voices in the field, so he can freely speak out about the risks of A.I. A part of him, he said, now regrets his life’s work.
    “I console myself with the normal excuse: If I hadn’t done it, somebody else would have,” Dr. Hinton said during a lengthy interview last week in the dining room of his home in Toronto, a short walk from where he and his students made their breakthrough.
    Some clarification from Hinton followed:
    It was already apparent that Hinton considered AI potentially dangerous, but this seems significant.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - The Rocket Alignment Problem, Part 2 by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Rocket Alignment Problem, Part 2, published by Zvi on May 1, 2023 on LessWrong.
    Previously (Eliezer Yudkowsky): The Rocket Alignment Problem.
    Recently we had a failure to launch, and a failure to communicate around that failure to launch. This post explores that failure to communicate, and the attempted message.
    Some Basic Facts about the Failed Launch
    Elon Musk’s SpaceX launched a rocket. Unfortunately, the rocket blew up, and failed to reach orbit. SpaceX will need to try again, once the launch pad is repaired.
    There was various property damage, but from what I have seen no one was hurt.
    I’ve heard people say the whole launch was a sshow and the grounding was ‘well earned.’ How the things that went wrong were absurd, SpaceX is the worst, and so on.
    The government response? SpaceX Starship Grounded Indefinitely By FAA.
    An FAA spokesperson told FLYING that mishap investigations, which are standard in cases such as this, “might conclude in a matter of weeks,” but more complex investigations “might take several months.”
    Perhaps this will be a standard investigation, and several months later everything will be fine. Perhaps it won’t be, and SpaceX will never fly again because those in power dislike Elon Musk and want to seize this opportunity.
    There are also many who would be happy that humans won’t get to go into space, if in exchange we get to make Elon Musk suffer, perhaps including those with power. Other signs point to the relationships with regulators remaining strong, yet in the wake of the explosion the future of Starship is for now out of SpaceX’s hands.
    A Failure to Communicate
    In light of these developments, before we knew the magnitude or duration of the grounding, Eliezer wrote the following, which very much failed in its communication.
    If the first prototype of your most powerful rocket ever doesn’t make it perfectly to orbit and land safely after, you may be a great rocket company CEO but you’re not qualified to run an AGI company.
    (Neither is any other human. Shut it down.)
    Eliezer has been using the rocket metaphor for AI alignment for a while, see The Rocket Alignment Problem.
    I knew instantly both what the true and important point was here, and also the way in which most people would misunderstand.
    The idea is that in order to solve AGI alignment, you need to get it right on the first try. If you create an AGI and fail at its alignment, you do not get to scrap the experiment, learn from what happened. You do not get to try, try again until you succeed, the way we do for things like rocket launches.
    That is because you created an unaligned AGI. Which kills you.
    Eliezer’s point here was to say that the equivalent difficulty level and problem configuration to aligning an AGI successfully would be if Musk stuck the landing on Starship on the first try. His first attempt to launch a rocket would need to end up safely back on the launching pad.
    The problem is that the rocket blowing up need not even get one person killed, let alone kill everyone. The rocket blowing up caused a bunch of property damage. Why Play in Hard Mode (or Impossible Mode) when you only need to Play in Easy Mode?
    Here were two smart people pointing out exactly this issue.
    Jeffrey Ladish: I like the rocket analogy but in this case I don’t think it holds since Elon’s plans didn’t depend on getting it right the first try. With rockets, unlike AGI, it’s okay to fail first try because you can learn (I agree that Elon isn’t qualified to run an AGI company)
    Eliezer: Okay if J Ladish didn’t get it, this was probably too hard to follow reliably.
    The analogy is valid because Elon would’ve preferred to stick the landing first try, and wasn’t setting up deliberately to fail where Starship failed. If he had the power to build a non-omnicidal superintelligence on his first try, he could’ve also ...
    15 min
  • EA - Overview: Reflection Projects on Community Reform by Joris P
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Overview: Reflection Projects on Community Reform, published by Joris P on May 1, 2023 on The Effective Altruism Forum.
    TL;DR: We’re sharing this short collection of active projects that we’re aware of, that focus on community reform and reflections on recent events.
    We don’t know much about the projects and are probably missing some projects (please let us know in the comments!), but we’ve found it hard to keep up with what is and is not happening, and noticed that we wished that a list like this existed.
    We’re posting this in our personal capacity. Any mistakes are our own. We did ask colleagues at CEA for input and feedback; thank you to Lizka & Chana (CEA), and Robert (not CEA) in particular!
    A lot has happened in the EA community over the past half year. In light of all the developments, people have discussed all kinds of changes EA could make.
    As community members, we’ve been finding it quite hard to follow what work is being done to reflect on the various ways in which we could reform. That's why we tried to put together a (incomplete!) list of (mostly ‘official’) projects that are being done to reflect on all sorts of questions that came up over the past six months.
    Some Caveats
    The topics listed below are not in any way a complete overview of the things that could or should be discussed - we don’t think we’re covering all the things community members are thinking about, and aren’t trying to!
    A topic isn’t “covered” or “taken” if there’s already a project that’s focused on it. Many of these projects are trying to cover a lot of ground, and the people working on them would probably appreciate other groups trying to do something on the topic, too.
    We chose to include mostly reflection projects that are being done in an official capacity, or by more than one person. That means we’re not including a lot of interesting Forum posts or news articles that have been written about possible problems and reforms! See the section 'Assorted written criticisms and reflections' below for some examples.
    This list is almost certainly incomplete. For instance, some projects we've heard about aren't public, and we think it's likely that there are other projects we haven't heard about. We encourage people to share what they’re working on in the comments!
    We’re not really sharing our views on how excited we are about the projects.
    The projects we’re aware of
    The EA survey (run by Rethink Priorities in collaboration with CEA) was updated and re-sent to people to ask about FTX community response — you can see the results here
    Community health concerns
    Investigations into the CEA Community Health Team’s processes and past actions:
    There’s an external investigation into how the CEA Community Health Team responded to concerns about Owen Cotton-Barratt (source)
    An internal review of the CEA Community Health Team’s processes is also ongoing (same source as above)
    Members of the Community Health team shared they hope that both investigations will be concluded sometime in the next month. They noted that the team does not have control over the timeline of the external investigation
    The CEA Community Health Team is (separately) conducting “a project to get a better understanding of the experiences of women and gender minorities in the EA community” (source)
    Governance in EA institutions
    Julia Wise and Ozzie Gooen are setting up a taskforce that will recommend potential governance reforms for various EA organizations. They are currently looking to get in touch with people with relevant expertise - see here
    Note that they write: “This project doesn’t aim to be a retrospective on what happened with FTX, and won’t address all problems in EA, but we hope to make progress on some of them.”
    EA leadership & accountability
    An investigation by the law firm Mintz, commissioned by EVF UK and EVF US, “to...
    6 min
  • EA - The costs of caution by Kelsey Piper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The costs of caution, published by Kelsey Piper on May 1, 2023 on The Effective Altruism Forum.
    Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author's permission. The author may not see or respond to comments on this post.
    If you thought we might be able to cure cancer in 2200, then I think you ought to expect there’s a good chance we can do it within years of the advent of AI systems that can do the research work humans can do.
    Josh Cason on Twitter raised an objection to recent calls for a moratorium on AI development:
    April 2, 2023
    Or raise your hand if you or someone you love has a terminal illness, believes Ai has a chance at accelerating medical work exponentially, and doesn't have til Christmas, to wait on your make believe moratorium. Have a heart man ❤️
    I’ve said that I think we should ideally move a lot slower on developing powerful AI systems. I still believe that. But I think Josh’s objection is important and deserves a full airing.
    Approximately 150,000 people die worldwide every day. Nearly all of those deaths are, in some sense, preventable, with sufficiently advanced medical technology. Every year, five million families bury a child dead before their fifth birthday. Hundreds of millions of people live in extreme poverty. Billions more have far too little money to achieve their dreams and grow into their full potential. Tens of billions of animals are tortured on factory farms.
    Scientific research and economic progress could make an enormous difference to all these problems. Medical research could cure diseases. Economic progress could make food, shelter, medicine, entertainment and luxury goods accessible to people who can't afford it today. Progress in meat alternatives could allow us to shut down factory farms.
    There are tens of thousands of scientists, engineers, and policymakers working on fixing these kinds of problems — working on developing vaccines and antivirals, understanding and arresting aging, treating cancer, building cheaper and cleaner energy sources, developing better crops and homes and forms of transportation. But there are only so many people working on each problem. In each field, there are dozens of useful, interesting subproblems that no one is working on, because there aren’t enough people to do the work.
    If we could train AI systems powerful enough to automate everything these scientists and engineers do, they could help.
    As Tom discussed in a previous post, once we develop AI that does AI research as well as a human expert, it might not be long before we have AI that is way beyond human experts in all domains. That is, AI which is way better than the best humans at all aspects of medical research: thinking of new ideas, designing experiments to test those ideas, building new technologies, and navigating bureaucracies.
    This means that rather than tens of thousands of top biomedical researchers, we could have hundreds of millions of significantly superhuman biomedical researchers.[1]
    That’s more than a thousand times as much effort going into tackling humanity’s biggest killers. If you thought we might be able to cure cancer in 2200, then I think you ought to expect there’s a good chance we can do it within years of the advent of AI systems that can do the research work humans can do.[2]
    All this may be a massive underestimate. This envisions a world that’s pretty much like ours except that extraordinary talent is no longer scarce. But that feels, in some senses, like thinking about the advent of electricity purely in terms of ‘torchlight will no longer be scarce’. Electricity did make it very cheap to light our homes at night. But it also enabled vacuum cleaners, washing machines, cars, smartphones, airplanes, video recording, Twitter — entirely new things, not just cheaper access to thi...
    6 min
  • LW - Shah (DeepMind) and Leahy (Conjecture) Discuss Alignment Cruxes - Summary & Transcript by Olivia Jimenez
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shah (DeepMind) and Leahy (Conjecture) Discuss Alignment Cruxes - Summary & Transcript, published by Olivia Jimenez on May 1, 2023 on LessWrong.
    Preface
    In December 2022, Rohin Shah (DeepMind) and Connor Leahy (Conjecture) discussed why Leahy is pessimistic about AI risk, and Shah is less so. Below is a summary and transcript.
    Summary
    Leahy expects discontinuities - capabilities rapidly increasing and behavior diverging far from what we aim towards - to be a core alignment difficulty. While this concept is similar to the sharp left turn (SLT), Leahy notes he doesn’t like the term SLT. SLT seems to imply deception or a malicious treacherous turn, which Leahy doesn’t see as an interesting, relevant, or necessary part.
    Shah suggests they start by discussing Leahy’s SLT views.
    Leahy explains he expects there to be some properties that are robustly useful for achieving many goals. As systems become powerful and applied to an increasing variety of tasks, these properties will be favored, and systems will generalize well to increasingly novel tasks.
    However, the goals we intend for our AIs to learn might not generalize. Perhaps it will seem that we’ve gotten them relatively aligned them to the goal of helping humans, but once they become smart enough to access the action space of "hook all humans up to super heroin", we’ll realize they learned some superficial distortion of our intentions that allows for bizarre and undesirable actions like that. And an AI might access this intelligence level rapidly, such that it’s not visibly misaligned ahead of time. In this sense, as systems get smarter, they might become harder to understand and manage.
    Shah is mostly on board: capabilities will improve, convergent instrumental goals exist, generalization will happen, goals can fail to generalize, etc. (However, he doesn’t think goal misgeneralization will definitely happen.)
    Shah asks why Leahy expects AI capabilities to increase rapidly. What makes the sharp left turn sharp?
    Leahy answers if we can’t rule a sharp left turn, and given a sharp left turn we expect it would be catastrophic, that’s sufficient cause for concern.
    The reason Leahy expects capabilities to increase sharply, though, relates to how humanity’s capabilities developed. He sees humans as having undergone sharp left turns in the past 1000 and 150 years. Our intelligence increased rapidly, such that we went from agriculture to the space age incredibly quickly. These shifts seem to have happened for three subsequent reasons. First, because of increases in brain size. Second, because of the development of culture, language and tools. And third, because of improvements in epistemology: science, rationality and emotional regulation. All three of these mechanisms will also exist for AIs, and AIs will be less constrained than humans in several respects, such as no biological limits on their brain size. So, as we scale AI systems, give them access to tools, and guide them towards discovering better ways of solving problems, Leahy expects they’ll become rapidly smarter just like we did.
    [Editor’s note expanding the parallel: When humans became rapidly smarter, they became increasingly unaligned with evolution’s goal of inclusive genetic fitness. Humans never cared about inclusive genetic fitness. Evolution just got us to care about proxies like sex. Early on, that wasn’t a big deal; sex usually led to offspring. Once we were smart enough to invent birth control, though, it was a big deal. Nowadays, human behavior is significantly unaligned with evolution’s intention of inclusive genetic fitness. If AIs become rapidly smarter, and they’ve only learned shallow proxies of our goals, a direction that we currently don’t know how to avoid, we might expect their behavior will stray very far from what we value too.]
    Shah notes Leahy’...
    49 min
  • AF - Shah (DeepMind) and Leahy (Conjecture) Discuss Alignment Cruxes by Olivia Jimenez
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shah (DeepMind) and Leahy (Conjecture) Discuss Alignment Cruxes, published by Olivia Jimenez on May 1, 2023 on The AI Alignment Forum.
    Preface
    In December 2022, Rohin Shah (DeepMind) and Connor Leahy (Conjecture) discussed why Leahy is pessimistic about AI risk, and Shah is less so. Below is a summary and transcript.
    Summary
    Leahy expects discontinuities - capabilities rapidly increasing and behavior diverging far from what we aim towards - to be a core alignment difficulty. While this concept is similar to the sharp left turn (SLT), Leahy notes he doesn’t like the term SLT. SLT seems to imply deception or a malicious treacherous turn, which Leahy doesn’t see as an interesting, relevant, or necessary part.
    Shah suggests they start by discussing Leahy’s SLT views.
    Leahy explains he expects there to be some properties that are robustly useful for achieving many goals. As systems become powerful and applied to an increasing variety of tasks, these properties will be favored, and systems will generalize well to increasingly novel tasks.
    However, the goals we intend for our AIs to learn might not generalize. Perhaps it will seem that we’ve gotten them relatively aligned them to the goal of helping humans, but once they become smart enough to access the action space of "hook all humans up to super heroin", we’ll realize they learned some superficial distortion of our intentions that allows for bizarre and undesirable actions like that. And an AI might access this intelligence level rapidly, such that it’s not visibly misaligned ahead of time. In this sense, as systems get smarter, they might become harder to understand and manage.
    Shah is mostly on board: capabilities will improve, convergent instrumental goals exist, generalization will happen, goals can fail to generalize, etc. (However, he doesn’t think goal misgeneralization will definitely happen.)
    Shah asks why Leahy expects AI capabilities to increase rapidly. What makes the sharp left turn sharp?
    Leahy answers if we can’t rule a sharp left turn, and given a sharp left turn we expect it would be catastrophic, that’s sufficient cause for concern.
    The reason Leahy expects capabilities to increase sharply, though, relates to how humanity’s capabilities developed. He sees humans as having undergone sharp left turns in the past 1000 and 150 years. Our intelligence increased rapidly, such that we went from agriculture to the space age incredibly quickly. These shifts seem to have happened for three subsequent reasons. First, because of increases in brain size. Second, because of the development of culture, language and tools. And third, because of improvements in epistemology: science, rationality and emotional regulation. All three of these mechanisms will also exist for AIs, and AIs will be less constrained than humans in several respects, such as no biological limits on their brain size. So, as we scale AI systems, give them access to tools, and guide them towards discovering better ways of solving problems, Leahy expects they’ll become rapidly smarter just like we did.
    [Editor’s note expanding the parallel: When humans became rapidly smarter, they became increasingly unaligned with evolution’s goal of inclusive genetic fitness. Humans never cared about inclusive genetic fitness. Evolution just got us to care about proxies like sex. Early on, that wasn’t a big deal; sex usually led to offspring. Once we were smart enough to invent birth control, though, it was a big deal. Nowadays, human behavior is significantly unaligned with evolution’s intention of inclusive genetic fitness. If AIs become rapidly smarter, and they’ve only learned shallow proxies of our goals, a direction that we currently don’t know how to avoid, we might expect their behavior will stray very far from what we value too.]
    Shah notes Leahy’s points t...
    48 min