The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - LessWrong Community Weekend 2023 – Applications Open by Henry Prowbell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LessWrong Community Weekend 2023 – Applications Open, published by Henry Prowbell on May 1, 2023 on LessWrong.
    Previous attendees can skip to the application form and note that we’re processing applications in 3 batches this year. If you need to know quickly whether you’re being offered a place, apply by May 15th to be included in batch 1.
    September 22-25th is the 10th annual LessWrong Community Weekend (LWCW). This is Europe's largest rationalist social gathering which brings together 150 aspiring rationalists from across Europe and beyond for four days of socialising, fun and intellectual exploration.
    The majority of the content will be unconference style and participant driven. On Friday afternoon we put up six wall-sized daily planners and by Saturday morning the attendees fill them up with 100+ workshops, talks and activities of their own devising.
    Previous years’ schedules have included.
    Double Cruxing
    Hamming Circles
    Gendlin Focusing
    Applied Rationality workshops by CFAR instructors and instructors-in-training
    Circling
    Authentic Relating games
    Improv theatre
    Introduction to stand up comedy
    Writing rationalist fiction
    Dance workshops
    Acapella singing
    Icebreaker games
    Lightning talks
    Celebrating failure groups
    Giant outdoor chess Penultima
    Dungeons & Dragons
    Kung Fu basics
    Board games
    Breathwork workshops
    Ecstatic dancing
    Radical Honesty workshops
    Playfighting for adults
    Polyamory and relationships workshops
    Sex Q&A roundtable
    Quantified self workshops
    Moral philosophy debates
    AI safety Q&A
    How to handle fear of AI Doom
    Value drift in EA
    The neurobiology of psychedelics
    The science of longevity
    Morning runs and yoga
    Meditation in the rooftop winter garden
    Night time swimming
    Ted Chiang and Greg Egan bedtime story readings
    If things like ecstatic dancing, radical honesty and polyamory workshops sound too intense for you, rest assured everything is optional. (I’m British and very awkward so a lot of this stuff terrifies me.)
    The event takes place in the natural environs of Lake Wannsee on the outskirts of Berlin. So you can spend some time recharging in between making new friends by hiking in the forests, sunbathing or swimming in the lake.
    For those who want to extend their stay in Berlin, there will very likely be meetups arranged in the days before and after the event too.
    LWCW is LGBTQIA+ friendly, people are welcome to bring their children (potentially there will be a professional childminder provided throughout) and this year we’re putting extra effort into creating an event where people of all ages, genders, backgrounds and adjacent interests (EA, circling, philosophy, meditation.) feel at home.
    This event has a special place in my heart and I truly think there’s nothing else quite like it. It’s where I’ve made friends who have more in common with me than I knew was possible and it’s where I’ve been introduced to ideas that have altered the course of my life – which is something I never truly got from the online version of LessWrong.
    Essential Information
    When: Friday 22nd September - Monday 25th September 2023
    Where: jh-wannsee.de (Berlin)
    Prices: Nobody makes any money from this event and the organiser team are unpaid. All your money goes into paying for the venue, food, equipment and other expenses.
    Regular ticket: €200
    Supporter ticket: €250/300/400
    If you want to attend but the ticket or travel cost is the only thing holding you back send us a message briefly explaining your situation. We have a small fund set aside for people who require financial aid.
    Apply here: http://tiny.cc/LWCW2023signup
    Contact: If you have any questions post them in the comments section below or email lwcw.europe[at]gmail.com
    Schedule
    Friday lunch: Meet in central Berlin at lunchtime for covid tests and vegan food followed by a short bus journey to JH Wannsee (all included in ...
    11 min
  • EA - First clean water, now clean air by finm
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: First clean water, now clean air, published by finm on April 30, 2023 on The Effective Altruism Forum.
    The excellent report from Rethink Priorities was my main source for this. Many of the substantial points I make are taken from it, though errors are my own. It’s worth reading! The authors are Gavriel Kleinwaks, Alastair Fraser-Urquhart, Jam Kraprayoon, and Josh Morrison.
    Clean water
    In the mid 19th century, London had a sewage problem. It relied on a patchwork of a few hundred sewers, of brick and wood, and hundreds of thousands of cesspits. The Thames — Londoners’ main source of drinking water — was near-opaque with waste. Here is Michael Faraday in an 1855 letter to The Times:
    Near the bridges the feculence rolled up in clouds so dense that they were visible at the surface even in water of this kind [.] The smell was very bad, and common to the whole of the water. It was the same as that which now comes up from the gully holes in the streets. The whole river was for the time a real sewer [.] If we neglect this subject, we cannot expect to do so with impunity; nor ought we to be surprised if, ere many years are over, a season give us sad proof of the folly of our carelessness.
    That “sad proof” arrived more than once. London saw around three outbreaks of cholera, killing upwards of 50,000 people in each outbreak.
    But early efforts to address the public health crisis were guided by the wrong theory about how diseases spread. On the prevailing view, epidemics were caused by ‘miasma’ (bad air) — a kind of poisonous mist from decomposing matter. Parliament commissioned a report on the ‘Sanitary Condition of the Labouring Population’, which showed a clear link between poverty and disease, and recommended a bunch of excellent and historically significant reforms. But one recommendation backfired because of this scientific misunderstanding: according to the miasma theory, it made sense to remove human waste through wastewater — but that water flowed into the Thames and contaminated it further.
    But in one of these outbreaks, the physician John Snow has spotted how incidence of cholera clustered around a single water pump in Soho, suggesting that unclean water was the major source of the outbreak. A few years later, the experiments of Louis Pasteur helped foster the germ theory of disease, sharpening the understanding of how and why to treat drinking water for public health. These were well-timed discoveries
    Because soon things got even worse. Heat exacerbated the smell; and the summer of 1858 was unusually hot. 1858 was the year of London’s ‘Great Stink’, and the Thames “a Stygian pool, reeking with ineffable and intolerable horrors” in Prime Minister Disraeli’s words. The problem had become totally unignorable.
    Parliament turned to Joseph Bazalgette, chief engineer of London’s Metropolitan Board of Works. Spurred by the Great Stink, he was given licence to oversee the construction of an ambitious plan to rebuild London’s sewage system, to his own design. 1,800km of street sewers would feed into 132km of main interconnecting sewers. A network of pumping stations was built, to lift sewage from streets below the high water mark. 18 years later, the result was the kind of modern sewage system we mostly take for granted: a system to collect wastewater and dump it far from where it could contaminate food and drinking water; in this case a dozen miles eastwards to the Thames estuary. "The great sewer that runs beneath Londoners”, wrote Bazalgette’s obituarist, “has added some 20 years to their chance of life”.
    Remarkably, most of the system remains in use. London’s sewage system has obviously been expanded, and wastewater treatment is much better. Bazalgette’s plan was built to last, and succeeded.
    As London built ways of expelling wastewater, it also built ways of channelling c...
    33 min
  • EA - Bridging EA's Gender Gap: Input From 60 People by Alexandra Bos
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bridging EA's Gender Gap: Input From 60 People, published by Alexandra Bos on April 30, 2023 on The Effective Altruism Forum.
    TLDR
    We hosted a session at EAGxRotterdam during which 60 participants discussed potential reasons why there are fewer women in EA and how this could be improved. The main categories of solutions people came up with were (1) adjusting outreach strategies, (2) putting women in more visible positions, (3) making EA’s atmosphere more female-friendly, (4) pursuing strategies to empower women in EA, and (5) adjusting certain attributes of EA thought. The goal of this post is to facilitate a solution-oriented discussion within the EA community so we can make tangible progress on its currently skewed gender ratio and underlying problems.
    Some notes before we start:
    Whether gender diversity is something to strive for is beyond this discussion. We will simply assume that it is and go from there. You could for example check out these posts (1, 2, 3) for a discussion on (gender) diversity if you want to read about this or discuss it.
    To keep the scope of this post and the session we hosted manageable, we focused on women specifically. However, we do not claim gender is binary and acknowledge that to increase diversity there are many more groups to focus on than just women (such as POC or other minorities).
    The views we describe in this post don’t necessarily correspond with our (Veerle Bakker's & Alexandra Bos') own but rather we are describing others’ input.
    Please view this post as crowdsourcing hypotheses from community members as a starting point for further discussion rather than as presenting The Hard Truth. You can also view posts such as these (1, 2, 3) for additional views on EA’s gender gap.
    EA & Its Gender Gap
    It is no secret that more men than women are involved with the EA community currently. In the last EA survey (2020), only 26.9% of respondents identified as female. This is similar to the 2019 survey.
    Graph source: EA Survey 2020.
    The goal of this post is to get a solution-oriented discussion started within the wider EA community to take steps towards tangible change. We aim to do this by sharing the insights from a discussion session at EAGxRotterdam in November 2022 titled "Discussion: how to engage more women with EA". In this post, we will be going through the different problems the EAGx’ers suspected may underlie the gender gap. Each problem will be paired with the potential solutions they proposed.
    Methodology
    This post summarises and categorises the insights from group discussions from a workshop at EAGxRotterdam. Around 60 people attended this session, approximately 40 of whom were women. In groups of 5, participants brainstormed on both 1) what may be the reasons for the relatively low number of women in EA (focusing on the causes, 15 mins), and 2) in what ways the EA community could attract more women to balance this out (focusing on solutions, 15 mins). The discussions were based on these prompts. We asked the groups to take notes on paper during their discussions so that this could be turned into this forum post. We informed them of this in advance. If you want to take a deep dive and look at the source materials, you are welcome to take a look at the participants’ written discussion notes.
    Limitations
    This project has some considerable limitations. First of all, the groups’ ideas are based on short brainstorming sessions, so they are not substantiated with research or confirmed in other ways. It is also worth mentioning that not all attendees had a lot of experience with the EA community - some only knew about EA for a couple of weeks or months. Furthermore, a considerable amount of information may have gotten lost in translation because it was transferred through hasty discussion notes. Additionally, the information was the...
    12 min
  • EA - More global warming might be good to mitigate the food shocks caused by abrupt sunlight reduction scenarios by Vasco Grilo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More global warming might be good to mitigate the food shocks caused by abrupt sunlight reduction scenarios, published by Vasco Grilo on April 29, 2023 on The Effective Altruism Forum.
    Disclaimer: this is not a project from Alliance to Feed the Earth in Disasters (ALLFED).
    Summary
    Global warming increases the risk from climate change. This “has the potential to result in—and to some extent is already resulting in—increased natural disasters, increased water and food insecurity, and widespread species extinction and habitat loss”.
    However, I think global warming also decreases the risk from food shocks caused by abrupt sunlight reduction scenarios (ASRSs), which can be a nuclear winter, volcanic winter, or impact winter. In essence, because low temperature is a major driver for the decrease in crop yields that can lead to widespread starvation (see Xia 2022, and this post from Luisa Rodriguez).
    Factoring in both of the above, my best guess is that additional emissions of greenhouse gases (GHGs) are beneficial up to an optimal median global warming in 2100 relative to 1880 of 3.3 ºC, after which the increase in the risk from climate change outweighs the reduction in that from ASRSs. This suggests delaying decarbonisation is good at the margin if one trusts (on top of my assumptions!):
    Metaculus’ community median prediction of 2.41 ºC.
    Climate Action Tracker’s projections of 2.6 to 2.9 ºC for current policies and action.
    Nevertheless, I am not confident the above conclusion is resilient. My sensitivity analysis indicates the optimal median global warming can range from 0.1 to 4.3 ºC. So the takeaway for me is that we do not really know whether additional GHG emissions are good/bad.
    In any case, it looks like the effect of global warming on the risk from ASRSs is a crucial consideration, and therefore it must be investigated, especially because it is very neglected. Another potentially crucial consideration is that an energy system which relies more on renewables, and less on fossil fuels is less resilient to ASRSs.
    Robustly good actions would be:
    Improving civilisation resilience.
    Prioritising the risk from nuclear war over that from climate change (at the margin).
    Keeping options open by:
    Not massively decreasing/increasing GHG emissions.
    Researching cost-effective ways to decrease/increase GHG emissions.
    Learning more about the risks posed by ASRSs and climate change.
    Introduction
    In the sense that matters most for effective altruism, climate change refers to large-scale shifts in weather patterns that result from emissions of greenhouse gases such as carbon dioxide and methane largely from fossil fuel consumption. Climate change has the potential to result in—and to some extent is already resulting in—increased natural disasters, increased water and food insecurity, and widespread species extinction and habitat loss.
    In What We Owe to the Future (WWOF), William MacAskill argues “decarbonisation [decreasing GHG emissions] is a proof of concept for longtermism”, describing it as a “win-win-win-win-win”. In addition to (supposedly) improving the longterm future:
    “Moving to clean energy has enormous benefits in terms of present-day human health. Burning fossil fuels pollutes the air with small particles that cause lung cancer, heart disease, and respiratory infections”.
    “By making energy cheaper [in the long run], clean energy innovation improves living standards in poorer countries”.
    “By helping keep fossil fuels in the ground, it guards against the risk of unrecovered collapse”.
    “By furthering technological progress, it reduces the risk of longterm stagnation”.
    I agree decarbonisation will eventually be beneficial, but I am not sure decreasing GHG emissions is good at the margin now. As I said in my hot takes on counterproductive altruism:
    Mitigating global warming dec...
    26 min
  • LW - A small update to the Sparse Coding interim research report by Lee Sharkey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A small update to the Sparse Coding interim research report, published by Lee Sharkey on April 30, 2023 on LessWrong.
    This is a linkpost to a set of slides containing an update to a project that was the subject of a previous post ([Interim research report] Taking features out of superposition with sparse autoencoders).
    The update is very small and scrappy. We haven't had much time to devote to this project since posting the Interim Research Report.
    TL;DR for the slides:
    We trained a minuscule language model (LM) (residual size = 16; 6 layers) and then trained sparse autoencoders on MLP activations (dimension = 64) from the third layer of that model.
    We found that, when we compared the 'ground truth feature recovery' plots, the plots for the toy data and LM data were much more similar than in the Interim Research Report.
    Very, very tentatively, we found the layer had somewhere between 512-1024 features. By labelling a subset of these features, we estimate there are roughly 600 easily labellable (monosemantic) features. For instance, we found a feature that activates for a period immediately after 'Mr', 'Mrs', or 'Dr'.
    We suspect that the reason the toy data and LM data plots had previously looked different was due to severely undertrained sparse autoencoders.
    We're hopeful that with more time to devote to this project we can confirm the results and apply the method to larger LMs. If it works, it would give us the ability to tell mechanistic stories about what goes on inside large LMs in terms of monosemantic features.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Hell is Game Theory Folk Theorems by jessicata
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hell is Game Theory Folk Theorems, published by jessicata on May 1, 2023 on LessWrong.
    [content warning: simulated very hot places; extremely bad Nash equilibria]
    (based on a Twitter thread)
    Rowan: "If we succeed in making aligned AGI, we should punish those who committed cosmic crimes that decreased the chance of an positive singularity sufficiently."
    Neal: "Punishment seems like a bad idea. It's pessimizing another agent's utility function. You could get a pretty bad equilibrium if you're saying agents should be intentionally harming each others' interests, even in restricted cases."
    Rowan: "In iterated games, it's correct to defect when others defect against you; that's tit-for-tat."
    Neal: "Tit-for-tat doesn't pessimize, though, it simply withholds altruism sometimes. In a given round, all else being equal, defection is individually rational."
    Rowan: "Tit-for-tat works even when defection is costly, though."
    Neal: "Oh my, I'm not sure if you want to go there. It can get real bad. This is where I pull out the game theory folk theorems."
    Rowan: "What are those?"
    Neal: "They're theorems about Nash equilibria in iterated games. Suppose players play normal-form game G repeatedly, and are infinitely patient, so they don't care about their positive or negative utilities being moved around in time. Then, a given payoff profile (that is, an assignment of utilities to players) could possibly be the mean utility for each player in the iterated game, if it satisfies two conditions: feasibility, and individual rationality."
    Rowan: "What do those mean?"
    Neal: "A payoff profile is feasible if it can be produced by some mixture of payoff profiles of the original game G. This is a very logical requirement. The payoff profile could only be the average of the repeated game if it was some mixture of possible outcomes of the original game. If some player always receives between 0 and 1 utility, for example, they can't have an average utility of 2 across the repeated game."
    Rowan: "Sure, that's logical."
    Neal: "The individual rationality condition, on the other hand, states that each player must get at least as much utility in the profile as they could guarantee getting by min-maxing (that is, picking their strategy assuming other players make things as bad as possible for them, even at their own expense), and at least one player must get strictly more utility."
    Rowan: "How does this apply to an iterated game where defection is costly? Doesn't this prove my point?"
    Neal: "Well, if defection is costly, it's not clear why you'd worry about anyone defecting in the first place."
    Rowan: "Perhaps agents can cooperate or defect, and can also punish the other agent, which is costly to themselves, but even worse for the other agent. Maybe this can help agents incentivize cooperation more effectively."
    Neal: "Not really. In an ordinary prisoner's dilemma, the (C, C) utility profile already dominates both agents' min-max utility, which is the (D, D) payoff. So, game theory folk theorems make mutual cooperation a possible Nash equilibrium."
    Rowan: "Hmm. It seems like introducing a punishment option makes everyone's min-max utility worse, which makes more bad equilibria possible, without making more good equilibria possible."
    Neal: "Yes, you're beginning to see my point that punishment is useless. But, things can get even worse and more absurd."
    Rowan: "How so?"
    Neal: "Let me show you my latest game theory simulation, which uses state-of-the-art generative AI and reinforcement learning. Don't worry, none of the AIs involved are conscious, at least according to expert consensus."
    Neal turns on a TV and types some commands into his laptop. The TV shows 100 prisoners in cages, some of whom are screaming in pain. A mirage effect appears across the landscape, as if the area is very hot.
    Rowan: "Wow, tha...
    9 min
  • LW - How can one rationally have very high or very low probabilities of extinction in a pre-paradigmatic field? by shminux
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How can one rationally have very high or very low probabilities of extinction in a pre-paradigmatic field?, published by shminux on April 30, 2023 on LessWrong.
    It is generally accepted in the local AI alignment circles that the whole field is pre-paradigmatic, in the Kuhnian sense (phase 1, as summarized here, if longer reading is not your thing). And yet, plenty of people are quite confident in their predictions of either doom or fizzle. A somewhat caricature way of representing their logic is, I think, "there are so many disjunctive ways to die, only one chance to get it right, and we don't have a step-by-step how-to, so we are hooped" vs "this is just one of many disruptive inventions whose real impact can only be understood way down the road, and all of them so far have resulted in net benefit, AI is just another example" (I have low confidence in the accuracy of the latter description, feel free to correct.) I can see the logic in both of those, what I do not see is how one can rationally have very high or very low confidence, given how much inherent uncertainty there is in our understanding of what is going on.
    My default is something more cautious, akin to Scott Alexander's/
    where one has to recognize their own reasoning limitations in the absence of hard empirical data, not "The Lens That Sees Its Flaws", but more like "The Lens That Knows It Has Flaws" without necessarily being able to identify them.
    So, how can one be very very very sure of something that has neither empirical confirmation, nor sound science behind it? Or am I misrepresenting the whole argument?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Discussion about AI Safety funding (FB transcript) by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Discussion about AI Safety funding (FB transcript), published by Akash on April 30, 2023 on The Effective Altruism Forum.
    Kat Woods recently wrote a Facebook post about Nonlinear's new funding program.
    This led to a discussion (in the comments section) about funding norms, the current funding bar, concerns about lowering the bar, and concerns about the current (relatively centralized) funding situation.
    I'm posting a few of the comments below. I'm hoping this might promote more discussion about the funding landscape. Such discussion could be especially valuable right now, given that:
    Many people are starting to get interested in AI safety (including people who are not from the EA/rationalist communities)
    AGI timeline estimates have generally shortened
    Investment in overall AI development is increasing quickly
    There may be opportunities to spend large amounts of money in the upcoming year (e.g., scalable career transition grant programs, regranting programs, 2024 US elections, AI governance/policy infrastructure, public campaigns for AI safety).
    Many ideas with high potential upside also have noteworthy downside risks (phrased less vaguely, I think that among governance/policy/comms projects that have high potential upside, >50% also have non-trivial downside risks).
    We might see pretty big changes in the funding landscape over the next 6-24 months
    New funders appear to be getting interested in AI safety
    Governments are getting interested in AI safety
    Major tech companies may decide to invest more resources into AI safety
    Selected comments from FB thread
    Note: I've made some editorial decisions to keep this post relatively short. Bolding is added by me. See the full thread here. Also, as usual, statements from individuals don't necessarily reflect the views of their employers.
    Kat Woods (Nonlinear)
    I often talk to dejected people who say they tried to get EA funding and were rejected
    And what I want to do is to give them a rousing speech about how being rejected by one funder doesn't mean that their idea is bad or that their personal qualities are bad.
    The evaluation process is noisy. Even the best funders make mistakes. They might just have a different world model or value system than you. They might have been hangry while reading your application.
    That to succeed, you'll have to ask a ton of people, and get a ton of rejections, but that's OK, because you only need a handful of yeses.
    (Kat then describes the new funding program from Nonlinear. TLDR: People submit an application that can then be reviewed by a network of 50+ funders.)
    Claire Zabel (Program Officer at Open Philanthropy)
    Claire's comment:
    (Claire quoting Kat:) The evaluation process is noisy. Even the best funders make mistakes. They might just have a different world model or value system than you. They might have been hangry while reading your application.
    (Claire's response): That's true. It's also possible the project they are applying for is harmful, but if they apply to enough funders, eventually someone will fund the harmful project (unilateralist's curse). In my experience as a grantmaker, a substantial fraction (though certainly very far from all) rejected applications in the longtermist space seem harmful in expectation, not just "not cost-effective enough"
    Selected portions of Kat's response to Claire:
    1. We’re probably going to be setting up channels where funders can discuss applicants. This way if there are concerns about net negativity, other funders considering it can see that. This might even lead to less unilateralist curse because if lots of funders think that the idea is net negative, others will be able to see that, instead of the status quo, where it’s hard to know what other funders think of an application.
    2. All these donors were giving anyways, with all the possibilities of the u...
    11 min
  • AF - A small update to the Sparse Coding interim research report by Lee Sharkey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A small update to the Sparse Coding interim research report, published by Lee Sharkey on April 30, 2023 on The AI Alignment Forum.
    This is a linkpost to a set of slides containing an update to a project that was the subject of a previous post ([Interim research report] Taking features out of superposition with sparse autoencoders).
    The update is very small and scrappy. We haven't had much time to devote to this project since posting the Interim Research Report.
    TL;DR for the slides:
    We trained a minuscule language model (LM) (residual size = 16; 6 layers) and then trained sparse autoencoders on MLP activations (dimension = 64) from the third layer of that model.
    We found that, when we compared the 'ground truth feature recovery' plots, the plots for the toy data and LM data were much more similar than in the Interim Research Report.
    Very, very tentatively, we found the layer had somewhere between 512-1024 features. By labelling a subset of these features, we estimate there are roughly 600 easily labellable (monosemantic) features. For instance, we found a feature that activates for a period immediately after 'Mr', 'Mrs', or 'Dr'.
    We suspect that the reason the toy data and LM data plots had previously looked different was due to severely undertrained sparse autoencoders.
    We're hopeful that with more time to devote to this project we can confirm the results and apply the method to larger LMs. If it works, it would give us the ability to tell mechanistic stories about what goes on inside large LMs in terms of monosemantic features.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Connectomics seems great from an AI x-risk perspective by Steven Byrnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Connectomics seems great from an AI x-risk perspective, published by Steven Byrnes on April 30, 2023 on LessWrong.
    Context
    Numerous people are in a position to accelerate certain areas within science or technology, whether by directing funds and resources, or by working in the area directly. But which areas are best to accelerate?
    One possible consideration (among others) is the question: “Is accelerating this technology going to increase the chance that our future transition to superhuman artificial general intelligence (AGI) goes well? Or decrease it? Or make no difference?” My goal here is to try to answer that question for connectomics (the science & technology of mapping how neurons connect to each other in a brain).
    This blog post is an attempt to contribute to Differential Technology Development (DTD) (part of the broader field of Differential Intellectual Progress). Successful DTD involves trying to predict complicated and deeply uncertain future trajectories and scenarios. I think the best we can hope for is to do better than chance. But I’m optimistic that we can at least exceed that low bar.
    My qualifications: I’m probably as qualified as anyone to discuss AI x-risk and how it relates to neuroscience. As for connectomics, I’m not too familiar with the techniques, but I’m quite familiar with how the results are used—in the past few years I have scrutinized probably hundreds of journal articles describing neural tracer measurements. (Think of neural tracer measurements as the traditional, “artisanal”, small-scale version of connectomics.) I find such articles extremely useful; I would happily trade away 20 fMRI papers for one neural tracer paper. This post is very much “my opinions” as opposed to consensus, and I’m happy for further discussion.
    TL;DR
    Improved connectomics technology seems like it would be very helpful for the project of reverse-engineering circuitry in the hypothalamus and brainstem that implement the “innate drives” upstream of human motivations and morality. And that’s a good thing! We may wind up in a situation where future researchers face the problem of designing “innate drives” for an AI; knowing how they work in humans would be helpful for various reasons.
    Improved connectomics technology seems like it would NOT be very helpful for the project of reverse-engineering the learning algorithms implemented by various parts of the brain, particularly the neocortex. And that’s a good thing too! I think that this reverse-engineering effort would lead directly to knowledge of how to build superhuman AGI, whereas I would like us to collectively make much more progress on AGI safety & alignment first, and to learn exactly how to build AGI second.
    Improved connectomics technology might open up a path to achieving Whole Brain Emulation (WBE) earlier than non-WBE AGI. And that’s a good thing too! Generally, a WBE-first future seems difficult to pull off, because (I claim) as soon as we understand the brain well enough for WBE, then we already understand the brain well enough to make non-WBE AGI, and someone will probably do that first. But if we could pull it off, it would potentially be very useful for a safe transition to AGI. I have previously been very skeptical that WBE is a possibility at all, but when I imagine a scenario where radically improved human connectomics technology is available in the near future, then it does actually seem like a possibility to have WBE come before non-WBE AGI, at least by a year or two, given enough effort and luck.
    1. Background considerations
    1.1 The race between reverse-engineering the cortex versus reverse-engineering the hypothalamus & brainstem
    My theory is that parts of the brain (esp. cortex, thalamus, striatum, and cerebellum) are running large-scale learning algorithms, while other parts of the brain (...
    18 min