The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Product Endorsement: Apollo Neuro by Elizabeth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Product Endorsement: Apollo Neuro, published by Elizabeth on May 8, 2023 on LessWrong.
    Short version: This $310 vibrating bracelet dramatically improved my sleep and moderately improved my emotional regulation. The return policy is pretty liberal so if this seems at all appealing I recommend trying it, or one of the cheaper alternatives I haven’t investigated. Between now and Mothers’ Day they are $300.
    [note: the link I use here is an affiliate link that gives you a $40 discount and me a $40 Amazon gift card, but I think it won’t apply for either of us if you buy during the sale]
    Backstory
    I have a pretty liberal threshold for trying shit Facebook advertises to me. Most of it is crap, but every once in a while there is something amazing that justifies all the work and return fees. Previous purchases include resistance band clips that measure force applied, sleep-safe headphones that claimed to measure your HRV, and an infrared heat massage. But when I first looked at the Apollo Neuro’s website, it was too dumb even for me. The explanation for how it worked was a mix of absent and stupid, and the rush to provide scientific evidence was somehow worse than nothing. I only tried it because a friend raved about it. He had also thought it sounded deeply stupid and only bought it because his friend raved about it, and I can only assume she also thought it was stupid until someone raved about it to her, in a great circle of life. Now it is your turn to be told it sounds stupid but it works.
    [For some people. My sample size is only three people.]
    Benefits to me
    My sleep improved a lot. I don’t have precise metrics for this because fitbit is stupid, but 1-3 times a week I wake up feeling drugged (positive valence) because my muscles are so relaxed. This never happened before the neuro unless I took actual drugs. I also estimate my number of remembered wake-ups has been ~halved.
    The Neuro has about 8 hours of battery. When I started using it I always woke up with the battery drained, meaning I’d activated it repeatedly. I now wake up with it at 70-90% charge, partially because I wake up less and thus am not turning it back on, and partially because I use less intense vibration.
    [TBF I regularly take supplements for sleep, so what I really mean here is “unless I took more drugs than baseline”]
    My emotional regulation improved as well. There were a number of stressful things I handled better than my baseline. Some of this is subjective, but there are a few things with obvious before-pictures. Most notably:I have pretty bad medical anxiety, due most notably to dental malpratice leaving me with painful nerve damage, but also some other stuff. Last fall I had a doctor’s appointment on Friday, followed by friend’s child’s medical emergency on Saturday that, due to their newness to the country, I needed to be in charge for. I did it, but I was wrecked for at least a day afterwards, possibly more, and my partner had to put in a lot of emotional energy helping me recover. ~4 weeks into using the Apollo Neuro I had a dentist appointment in the afternoon, followed 8 hours later by a friend’s medical emergency requiring my attention and eventually a 3AM field trip to the ER for which I was the only available driver.
    I did have a little freak out once I was home, but I recovered to normal faster than I did from the incident in the fall despite that having been a less intense day with more help.
    I also found myself more decisively liking and disliking things, and noticing when things shifted. At parties the transition from “I’m enjoying this” to “I’m done” is clearer, without a lot of “maybe I should hold out and it will get better”.
    Mechanics
    The website is stupid, so let me tell you how it really works. The Neuro is shaped like a watch with a large rectangular face. You can wear it around...
    8 min
  • LW - LeCun’s “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem by Steven Byrnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LeCun’s “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem, published by Steven Byrnes on May 8, 2023 on LessWrong.
    Summary
    This post is about the paper A Path Towards Autonomous Machine Intelligence (APTAMI) by Yann LeCun. It’s a high-level sketch of an AI architecture inspired by the brain.
    APTAMI is mostly concerned with arguing that this architecture is a path towards more-capable AI. However, it is also claimed (both in the paper itself and in associated public communication) that this architecture is a path towards AI that is “controllable and steerable”, kind, empathetic, and so on.
    I argue that APTAMI is in fact, at least possibly, a path towards that latter destination, but only if we can solve a hard and currently-unsolved technical problem.
    This problem centers around the Intrinsic Cost module, which performs a role loosely analogous to “innate drives” in humans—e.g. pain being bad, sweet food being good, a curiosity drive, and so on.
    APTAMI does not spell out explicitly (e.g. with pseudocode) how to create the Intrinsic Cost module. It offers some brief, vague ideas of what might go into the Intrinsic Cost module, but does not provide any detailed technical argument that an AI with such an Intrinsic Cost would be controllable / steerable, kind, empathetic, etc.
    I will argue that, quite to the contrary, if we follow the vague ideas in the paper for building the Intrinsic Cost module, then there are good reasons to expect the resulting AI to be not only unmotivated by human welfare, but in fact motivated to escape human control, seek power, self-reproduce, etc., including by deceit and manipulation.
    Indeed, it is an open technical problem to write down any Intrinsic Cost function (along with training environment and other design choices) for which there is a strong reason to believe that the resulting AI would be controllable and/or motivated by human welfare, while also being sufficiently competent to do the hard intellectual tasks that we’re hoping for (e.g. human-level scientific R&D).
    I close by encouraging LeCun himself, his colleagues, and anyone else to try to solve this open problem. It’s technically interesting, very important, and we have all the information we need to start making progress now. I've been working on that problem myself for years, and I think I’m making more than zero progress, and if anyone reaches out to me I’d be happy to discuss the current state of the field in full detail.
    .And then there’s an epilogue, which steps away from the technical discussion of the Intrinsic Cost module, and instead touches on bigger-picture questions of research strategy & prioritization. I will argue that the question of AI motivations merits much more than the cursory treatment that it got in APTAMI—even given the fact that APTAMI was a high-level early-stage R&D vision paper in which every other aspect of the AI is given an equally cursory treatment.
    (Note: Anyone who has read my Intro to Brain-Like AGI Safety series will notice that much of this post is awfully redundant with it—basically an abbreviated subset with various terminology changes to match the APTAMI nomenclature. And that’s no coincidence! As mentioned, the APTAMI architecture was explicitly inspired by the brain.)
    1. Background: the paper’s descriptions of the “Intrinsic Cost module”
    For the reader’s convenience, I’ll copy everything specific that APTAMI says about the Intrinsic Cost module. (Emphasis in original.)
    PAGES 7-8: The Intrinsic Cost module is hard-wired (immutable, non trainable) and computes a single scalar, the intrinsic energy that measures the instantaneous “discomfort” of the agent – think pain (high intrinsic energy), pleasure (low or negative intrinsic energy), hunger, etc. The input to the module is the current state of t...
    26 min
  • AF - LeCun’s “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem by Steve Byrnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LeCun’s “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem, published by Steve Byrnes on May 8, 2023 on The AI Alignment Forum.
    Summary
    This post is about the paper A Path Towards Autonomous Machine Intelligence (APTAMI) by Yann LeCun. It’s a high-level sketch of an AI architecture inspired by the brain.
    APTAMI is mostly concerned with arguing that this architecture is a path towards more-capable AI. However, it is also claimed (both in the paper itself and in associated public communication) that this architecture is a path towards AI that is “controllable and steerable”, kind, empathetic, and so on.
    I argue that APTAMI is in fact, at least possibly, a path towards that latter destination, but only if we can solve a hard and currently-unsolved technical problem.
    This problem centers around the Intrinsic Cost module, which performs a role loosely analogous to “innate drives” in humans—e.g. pain being bad, sweet food being good, a curiosity drive, and so on.
    APTAMI does not spell out explicitly (e.g. with pseudocode) how to create the Intrinsic Cost module. It offers some brief, vague ideas of what might go into the Intrinsic Cost module, but does not provide any detailed technical argument that an AI with such an Intrinsic Cost would be controllable / steerable, kind, empathetic, etc.
    I will argue that, quite to the contrary, if we follow the vague ideas in the paper for building the Intrinsic Cost module, then there are good reasons to expect the resulting AI to be not only unmotivated by human welfare, but in fact motivated to escape human control, seek power, self-reproduce, etc., including by deceit and manipulation.
    Indeed, it is an open technical problem to write down any Intrinsic Cost function (along with training environment and other design choices) for which there is a strong reason to believe that the resulting AI would be controllable and/or motivated by human welfare, while also being sufficiently competent to do the hard intellectual tasks that we’re hoping for (e.g. human-level scientific R&D).
    I close by encouraging LeCun himself, his colleagues, and anyone else to try to solve this open problem. It’s technically interesting, very important, and we have all the information we need to start making progress now. I've been working on that problem myself for years, and I think I’m making more than zero progress, and if anyone reaches out to me I’d be happy to discuss the current state of the field in full detail.
    .And then there’s an epilogue, which steps away from the technical discussion of the Intrinsic Cost module, and instead touches on bigger-picture questions of research strategy & prioritization. I will argue that the question of AI motivations merits much more than the cursory treatment that it got in APTAMI—even given the fact that APTAMI was a high-level early-stage R&D vision paper in which every other aspect of the AI is given an equally cursory treatment.
    (Note: Anyone who has read my Intro to Brain-Like AGI Safety series will notice that much of this post is awfully redundant with it—basically an abbreviated subset with various terminology changes to match the APTAMI nomenclature. And that’s no coincidence! As mentioned, the APTAMI architecture was explicitly inspired by the brain.)
    1. Background: the paper’s descriptions of the “Intrinsic Cost module”
    For the reader’s convenience, I’ll copy everything specific that APTAMI says about the Intrinsic Cost module. (Emphasis in original.)
    PAGES 7-8: The Intrinsic Cost module is hard-wired (immutable, non trainable) and computes a single scalar, the intrinsic energy that measures the instantaneous “discomfort” of the agent – think pain (high intrinsic energy), pleasure (low or negative intrinsic energy), hunger, etc. The input to the module is the curren...
    26 min
  • EA - How quickly AI could transform the world (Tom Davidson on The 80,000 Hours Podcast) by 80000 Hours
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How quickly AI could transform the world (Tom Davidson on The 80,000 Hours Podcast), published by 80000 Hours on May 8, 2023 on The Effective Altruism Forum.
    Over at The 80,000 Hours Podcast we just published an interview that is likely to be of particular interest to people who identify as involved in the effective altruism community: Tom Davidson on how quickly AI could transform the world.
    You can click through for the audio, a full transcript and related links. Below is the episode summary and some key excerpts.
    Episode Summary
    By the time that the AIs can do 20% of cognitive tasks in the broader economy, maybe they can already do 40% or 50% of tasks specifically in AI R&D. So they could have already really started accelerating the pace of progress by the time we get to that 20% economic impact threshold.
    At that point you could easily imagine that really it’s just one year, you give them a 10x bigger brain. That’s like going from chimps to humans — and then doing that jump again. That could easily be enough to go from [AIs being able to do] 20% [of cognitive tasks] to 100%, just intuitively. I think that’s kind of the default, really.
    Tom Davidson
    It’s easy to dismiss alarming AI-related predictions when you don’t know where the numbers came from.
    For example: what if we told you that within 15 years, it’s likely that we’ll see a 1,000x improvement in AI capabilities in a single year? And what if we then told you that those improvements would lead to explosive economic growth unlike anything humanity has seen before?
    You might think, “Congratulations, you said a big number — but this kind of stuff seems crazy, so I’m going to keep scrolling through Twitter.”
    But this 1,000x yearly improvement is a prediction based on real economic models created by today’s guest Tom Davidson, Senior Research Analyst at Open Philanthropy. By the end of the episode, you’ll either be able to point out specific flaws in his step-by-step reasoning, or have to at least consider the idea that the world is about to get — at a minimum — incredibly weird.
    As a teaser, consider the following:
    Developing artificial general intelligence (AGI) — AI that can do 100% of cognitive tasks at least as well as the best humans can — could very easily lead us to an unrecognisable world.
    You might think having to train AI systems individually to do every conceivable cognitive task — one for diagnosing diseases, one for doing your taxes, one for teaching your kids, etc. — sounds implausible, or at least like it’ll take decades.
    But Tom thinks we might not need to train AI to do every single job — we might just need to train it to do one: AI research.
    And building AI capable of doing research and development might be a much easier task — especially given that the researchers training the AI are AI researchers themselves.
    And once an AI system is as good at accelerating future AI progress as the best humans are today — and we can run billions of copies of it round the clock — it’s hard to make the case that we won’t achieve AGI very quickly.
    To give you some perspective: 17 years ago we saw the launch of Twitter, the release of Al Gore’s An Inconvenient Truth, and your first chance to play the Nintendo Wii.
    Tom thinks that if we have AI that significantly accelerates AI R&D, then it’s hard to imagine not having AGI 17 years from now.
    Wild.
    Host Luisa Rodriguez gets Tom to walk us through his careful reports on the topic, and how he came up with these numbers, across a terrifying but fascinating three hours.
    Luisa and Tom also discuss:
    How we might go from GPT-4 to AI disaster
    Tom’s journey from finding AI risk to be kind of scary to really scary
    Whether international cooperation or an anti-AI social movement can slow AI progress down
    Why it might take just a few years to go from pretty good AI to superhum...
    24 min
  • LW - What does it take to ban a thing? by qbolec
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What does it take to ban a thing?, published by qbolec on May 8, 2023 on LessWrong.
    Epistemic status: I am not an expert. I just took several things which people banned (child labor, chemical weapons, ozone-depleting substances) and for each just searched for the first article which seriously engages with the question "how did we succeed to ban it?", read it once, and summarized how I understand it. If someone has more examples, or better explanations, I'd be glad to learn.
    I think that there's something to learn from examples of bad things that we have banned in the past despite some people benefiting from those bad things. A rosy-eyed, but wrong, image of how that happened is "well, people just realized the thing was bad so, they've banned it". Turns out it is not at all how it has happened.
    Ban of chemical weapons
    TL;DR: They seem to not be very effective (face masks), may backfire in case of wind ("blowback")
    Source:/
    Quotes I've found interesting:
    One answer is that while gas attacks are terrifying, the weapon has proved to be militarily ineffective. After Ypres, the allies provided masks to their front-line troops, who stood in their trenches killing onrushing Germans as clouds of gas enveloped their legs. That was true even as both sides climbed the escalatory ladder, introducing increasingly lethal chemicals (phosgene and mustard gas), that were then matched by increasingly effective countermeasures. The weapon also proved difficult to control. In several well-documented instances, gases deployed by front-line troops blew back onto their own trenches — giving a literalist tinge to the term “blowback,” now used to describe the unintended consequences of an intelligence operation.
    The world’s militaries are loath to ban weapons that kill effectively, while acceding to bans of weapons that they don’t need.
    At the end of World War I, a precise tabulation of casualties showed that some 91,000 soldiers on all sides were killed in gas attacks — less than 10 percent of the total deaths for the entire war. Machine guns and artillery shells, it turns out, were far more effective systems for delivering death.
    Among the ban supporters was a Norwegian foreign ministry official who issued an impassioned plea for the adoption of a treaty banning the weapon. In the midst of his talk (which I attended), a British colonel leaned across the table at which I was sitting, a wry smile on his face. “You know why the Norwegians favor a ban?” he asked. I shook my head: no. “Because they don’t have any,” he said.
    Note, that cluster bombs and mines are still not banned, despite similar "moral" problems with them:
    Additionally, key senior military officers believed agreeing to the ban would set a dangerous precedent — that the military could be pressured into banning weapons by what they described as left-leaning humanitarian organizations.
    The world’s militaries don’t want to ban weapons that are efficient killers. So while it is true that the land mine and cluster munitions bans have gained widespread international support (162 countries have signed the land-mine ban, 108 countries have signed onto the Convention on Cluster Munitions), the countries most likely to use both (the U.S., China, Russia and India) remain nonsignatories.
    Ban of child labor
    TL;DR: In Great Depression children were considered stealing jobs from adults
    Source:
    Quotes I've found interesting:
    By the 1870s, unions condemned child labor on the basis that overly young workers competed for jobs, making it harder for adults to obtain higher pay and better conditions – not due to concerns about the well-being of kids.
    Despite Southern opposition, reformers argued that state-level regulations were rife with loopholes and difficult to enforce. In 23 states, for instance, there was no official way to determine children’s ...
    10 min
  • EA - EA Anywhere Slack: consolidating professional and affinity groups by Sasha Berezhnoi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Anywhere Slack: consolidating professional and affinity groups, published by Sasha Berezhnoi on May 8, 2023 on The Effective Altruism Forum.
    Summary
    In an effort to improve communication infrastructure, the EA Anywhere Slack workspace has recently undergone major changes to accommodate several professional and affiliation Slack workspaces like EA Entrepreneurs, EA Global Discussion, EA Creatives & Communicators, and more. The project was initiated by Pineapple Operations because the old structure was inefficient and overwhelming - others have made similar observations. We believe making it easier to track discussions and reducing the number of workspaces will increase activity, avoid information loss and prevent duplicated communications.
    If you have a Slack workspace you’d like to merge with EA Anywhere’s, reach out to us!
    Why consolidate?
    With too many workspaces, the infrastructure of the EA movement becomes increasingly overwhelming and confusing, and it’s difficult to keep up with new workspaces. Having a central Slack gives people access to a broader range of communities at once.
    People don’t have the time or energy to check multiple Slacks, which results in low activity. Some discussions just don’t reach the critical mass, and valuable connections are not happening.
    Most of the workspaces were on a free plan that hid messages and files older than 90 days. Consolidation around a few paid workspaces prevents groups from losing historical information.
    There is overlapping membership between Slacks (we estimate between 10-50%), so consolidation makes it easier to track communications.
    These reasons were true for the dozens of Slack workspaces we identified with low activity and limited facilitation.
    Why EA Anywhere?
    EA Anywhere is an online discussion space for the global EA community and a touchpoint for people without local groups nearby. It plays an important role in supporting other virtual ecosystems in EA: we host EAGxVirtual conferences, provide support and share knowledge with other online groups.
    EA Anywhere Slack is on a paid Pro plan and has active facilitation from a full-time community organizer, which makes it a good choice for this project. We can provide support for groups joining the space, including events promotion, Zoom accounts, Slack integrations, and knowledge-sharing calls.
    There is a demand for informal conversation spaces and networking that the EA Forum doesn’t currently provide. We are inspired by Slack-based communities that thrive and create value for thousands of members without becoming too overwhelming.
    Progress to date
    We reached out to workspace owners with our proposal and received positive feedback. In most cases, we merged the users and message history.
    List of Slack workspaces we have already merged or consolidated (thanks to the admins of these groups!):
    EA Global Discussions
    EA Entrepreneurs
    EA Creatives & Communicators
    EA Generalists
    EA Housing
    EA Tech Network
    Public Interest Tech
    EA Project Management
    EA Supply Chain Logistics
    EA Math and Physics
    Product in EA
    Effective Environmentalism
    We already see the benefits of increased coordination:
    Members are engaging with other groups and projects that have been locked into small workspaces.
    Members are more willing to ask questions and ask for advice. Activity in the #all-questions-welcome channel increased four-fold compared to the previous three-month average, with both new and former members engaging in discussions.
    Organizers have an easier way to promote opportunities and events.
    As of April 30th, a month after the merger, the initial hype subsided but the activity is still twice as high:
    90 members who posted weekly (44 before)
    360 weekly active members (210 before the merger)
    We will continue using Slack Analytics to track the activity and send a follow-up user surve...
    5 min
  • EA - The Legend of Dr. Oguntola Sapara by jai
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Legend of Dr. Oguntola Sapara, published by jai on May 8, 2023 on The Effective Altruism Forum.
    This is a true story.
    At the dawn of the long war's final century, the tide was turning. The Abomination was in retreat, driven back by warriors wielding the Lance of Jenner. But, as in all wars, some battles could not be fought so directly. Battles waged in cunning and lies. Battles where the enemy lurked in shadow. In that darkness an unholy alliance was forged between the Pox Abomination and a conspiracy of traitors to the Yoruba people.
    They named it "Sopona", gave it a face, and proclaimed themselves the Abomination’s priests. They claimed that they alone could intercede with the Abomination on behalf of innocent victims. To cross them, they said, was to incur Sopona's wrath. They would unleash the Abomination upon any who dared oppose them - and sometimes they would simply inflict Sopona's torture indiscriminately. Amid death and devastation their victims would beg the Priests for help, further cementing their grip on power. And all the while they kept Jenner's Lance at bay, for their power rested on fear of the Abomination, and should it be slain their power, too, would come to an end.
    They operated in secrecy: the better to obscure their lies, and the better to hide from those who would challenge it. Through blackmail and terror, they maintained their iron grip for generations. None dared utter "Sopona" lest they invoke its wrath - and so even the true name was hidden.
    Every measure by every authority failed to contain the Abomination. They could never understand why, for they were blind to the enemy's allies. The Deathly Priests and their twisted methods were beyond the grasp of governments, warriors, and weapons. Here the global campaign could not reach. Here, harbored by its murderous allies, the Abomination reigned, and the Yoruba people resigned themselves to an abominable god against which there seemed no hope.
    It is inadvisable to try to hide from humans. They are curious, relentless, ruthless creatures, fearless when determined and cunning as well. And none were more human than Dr. Oguntola Sapara.
    Oguntola was a proud child of the Yoruba people. His father, born in chains, together with his mother, raised a family of prodigies: not only Oguntula, but his brother Alexander and his sister Clementina. But Clementina's story was all too short, for when she was to bring life into the world, she was instead taken by death.
    There are no records of how Oguntola felt that day; All we know is that this was the moment that Oguntula dedicated his life to defending the innocent from the inhuman evils of the world, to master the protective arts and wield them against any who would dare threaten his people.
    For years he studied, and toiled, and healed, growing ever stronger in the art through talent and sheer force of will. Ten years on his quest took him across the seas to study in a far away land, and here yet more obstacles greeted him. For among the practitioners of the art were counted a great many fools. They would forfeit the privilege of working alongside one of humanity's best for the most vapid and meaningless of reasons, and worse still actively stymied his efforts in all things lest their foolishness be revealed for the lie it was.
    But Oguntola persisted, surmounting every obstacle lesser humans would set before him. In time he not only prevailed, but he proved himself among the greatest of the practitioners. He was recognized as a master of the art, and elected to the Royal Institute of healers.
    (In the midst of everything, he even assisted the legendary truth-seeker Ida Wells in her crusades against evil and ignorance - but that is another story.)
    His training complete and his mastery assured, Dr. Oguntola Sapara returned to Lagos to confront his true e...
    10 min
  • EA - The Rethink Priorities Existential Security Team's Strategy for 2023 by Ben Snodin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Rethink Priorities Existential Security Team's Strategy for 2023, published by Ben Snodin on May 8, 2023 on The Effective Altruism Forum.
    The Rethink Priorities Existential Security Team's Strategy for 2023
    Summary
    This post contains a moderately rough, high-level description of the Rethink Priorities Existential Security team’s (XST’s) strategy for the period April-October 2023.
    XST is a team of researchers focused on improving the world according to a longtermist outlook through research and other projects, and is part of Rethink Priorities (RP).
    Note that until very recently we were called the General Longtermism team (GLT). We have now renamed ourselves the Existential Security team (XST), which is slightly more descriptive and more closely reflects our focus on reducing existential risk.
    XST’s three focus areas for 2023 will be:
    Longtermist entrepreneurship (65%): Making highly impactful longtermist projects happen by finding and developing ideas for highly promising longtermist projects, identifying potential founders, and supporting them as they get these projects started. Our main activities will be:
    Identifying and detailing the most promising ideas for longtermist projects, with a goal of having ~5 detailed project ideas by the end of June, that we can bring to a potential meeting of talented entrepreneurs in July/August, organized by Mike McCormick.
    A relatively brief founder-first-style founder search (looking for highly promising founders and finding projects that they are an especially good fit for).
    Exploring founder-in-residence MVPs (hiring potential founders and giving them space to develop their own ideas for promising projects).
    Supporting founders once they’re identified.
    Strategic clarity research (25%): Research that helps shed light on high-level strategic questions relevant for the EA community and for people working on reducing existential risk. This year, we plan to focus on high-level EA movement-building strategy questions (such as “What kind of EA movement do we want?” or “What’s the optimal portfolio among priority cause areas we should aim at building?”), and possibly on high-level questions that seem important for assessing whether and how to help launch entrepreneurial projects. Most of our work on this will happen in the second half of the year.
    Flexible time for high-impact opportunities (10%): Time allocated for i) team members working on projects that they are very keen on and ii) highly impactful and time-sensitive projects that arise due to changes in external circumstances.
    Concrete outputs we’ll aim for:
    5 project idea memos by the end of June that are of a standard equal to or better than the 2023 Q1 megaproject speedruns that we posted on the EA Forum in February.
    1 strategic clarity research output by the end of October.
    1 new promising project launched by the end of October.
    11 publicly shared research or project idea outputs by the end of the year.
    From mid-May onwards, we’re planning to have 4 FTE executing this strategy: me (Ben), Marie, Jam, and Renan. Linch is pursuing a separate research agenda related to longtermist strategic clarity.
    The high-level timeline is:
    [completed] March: The team winds down current projects and begins work on executing the team strategy from the start of April.
    [in progress] April-July: The team focuses on the entrepreneurship program, and works on founder-first activities, founder support, and project research. The project research is focused on generating a new prioritization model and shallow project ranking by the end of April, and 5 project ideas memos by the end of June for a potential meeting of promising entrepreneurs in July/August.
    August-October: Jam and Ben continue working on the entrepreneurship program, while Marie and Renan switch to strategic clarity research.
    Start o...
    26 min
  • AF - A technical note on bilinear layers for interpretability by Lee Sharkey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A technical note on bilinear layers for interpretability, published by Lee Sharkey on May 8, 2023 on The AI Alignment Forum.
    Summary
    In this short theoretical note (now on Arxiv) I examine bilinear layers, which are MLP layers that take the form
    MLPBilinear(x)=(W1x)⊙(W2x).
    When used in language models, they perform better than standard MLPs with elementwise activation functions (but appear very slightly below state of the art).
    Despite their competitiveness, they are mathematically much easier to analyze: Although they are nonlinear functions of their input, bilinear layers can be expressed using only linear operations and third order tensors.
    Because they can be linearized, we can extend 'A Mathematical Framework for Transformer Circuits' (Elhage et al. 2022) beyond attention-only transformers to transformers with both attention and MLP layers.
    In a similar way to how the analysis of Elhage et al. (2022) helped to reveal QK- and OV-circuits, induction heads, and virtual attention heads, the analyzability of bilinear layers may lend them to deeper safety insights by allowing us to talk more formally about circuits in large language models.
    Additionally, and more speculatively, bilinear layers might offer an alternative path for mechanistic interpretability through understanding the mechanisms of feature construction instead of having to enumerate and understand a (potentially exponentially) large number of features in large models.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - An artificially structured argument for expecting AGI ruin by Rob Bensinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An artificially structured argument for expecting AGI ruin, published by Rob Bensinger on May 7, 2023 on LessWrong.
    Philosopher David Chalmers asked:
    [I]s there a canonical source for "the argument for AGI ruin" somewhere, preferably laid out as an explicit argument with premises and a conclusion?
    Unsurprisingly, the actual reason people expect AGI ruin isn't a crisp deductive argument; it's a probabilistic update based on many lines of evidence. The specific observations and heuristics that carried the most weight for someone will vary for each individual, and can be hard to accurately draw out.
    That said, Eliezer Yudkowsky's So Far: Unfriendly AI Edition might be a good place to start if we want a pseudo-deductive argument just for the sake of organizing discussion. People can then say which premises they want to drill down on.
    In The Basic Reasons I Expect AGI Ruin, I wrote:
    When I say "general intelligence", I'm usually thinking about "whatever it is that lets human brains do astrophysics, category theory, etc. even though our brains evolved under literally zero selection pressure to solve astrophysics or category theory problems".
    It's possible that we should already be thinking of GPT-4 as "AGI" on some definitions, so to be clear about the threshold of generality I have in mind, I'll specifically talk about "STEM-level AGI", though I expect such systems to be good at non-STEM tasks too.
    STEM-level AGI is AGI that has "the basic mental machinery required to do par-human reasoning about all the hard sciences", though a specific STEM-level AGI could (e.g.) lack physics ability for the same reasons many smart humans can't solve physics problems, such as "lack of familiarity with the field".
    A simple way of stating the argument in terms of STEM-level AGI is:
    Substantial Difficulty of Averting Instrumental Pressures: As a strong default, absent alignment breakthroughs, STEM-level AGIs that understand their situation and don't value human survival as an end will want to kill all humans if they can.
    Substantial Difficulty of Value Loading: As a strong default, absent alignment breakthroughs, STEM-level AGI systems won't value human survival as an end.
    High Early Capabilities. As a strong default, absent alignment breakthroughs or global coordination breakthroughs, early STEM-level AGIs will be scaled to capability levels that allow them to understand their situation, and allow them to kill all humans if they want.
    Conditional Ruin. If it's very likely that there will be no alignment breakthroughs or global coordination breakthroughs before we invent STEM-level AGI, then given 1+2+3, it's very likely that early STEM-level AGI will kill all humans.
    Inadequacy. It's very likely that there will be no alignment breakthroughs or global coordination breakthroughs before we invent STEM-level AGI.
    Therefore it's very likely that early STEM-level AGI will kill all humans. (From 1–5)
    I'll say that the "invention of STEM-level AGI" is the first moment when an AI developer (correctly) recognizes that it can build a working STEM-level AGI system within a year. I usually operationalize "early STEM-level AGI" as "STEM-level AGI that is built within five years of the invention of STEM-level AGI".
    I think humanity is very likely to destroy itself within five years of the invention of STEM-level AGI. And plausibly far sooner — e.g., within three months or a year of the technology's invention. A lot of the technical and political difficulty of the situation stems from this high level of time pressure: if we had decades to work with STEM-level AGI before catastrophe, rather than months or years, we would have far more time to act, learn, try and fail at various approaches, build political will, craft and implement policy, etc.
    This argument focuses on "human survival", but from my perspec...
    51 min