The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Introducing Stanford’s new Humane & Sustainable Food Lab by MMathur
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing Stanford’s new Humane & Sustainable Food Lab, published by MMathur on April 30, 2023 on The Effective Altruism Forum.
    We are excited to announce the new Humane & Sustainable Food Lab at Stanford University’s School of Medicine (California, USA). Our mission is to end factory farming through cutting-edge scientific research that we are uniquely positioned to conduct. I am the principal investigator of the lab, an Assistant Professor at the Stanford School of Medicine with dual appointments in the Quantitative Sciences Unit and Department of Pediatrics. Because arguments for reducing factory farming as a cause area have been detailed elsewhere, here I focus on describing:
    Our approach
    Our research and publications to date
    Our upcoming research priorities
    Why we are funding-constrained
    1. Our approach
    1.1. Breadth, then depth
    Empirical research on how to reduce factory farming is still nascent, with many low-hanging fruit and unexplored possibilities. As such, it is critical to explore broadly to see what general directions are most promising and in what real-world contexts (e.g., educational interventions that appeal to animal welfare [1, 2, 3], choice-architecture “nudges” that subtly shift food-service environments, etc.). We are conducting studies on a range of individual- and society-level interventions (see below), ultimately aiming to find and refine the most tractable, cost-effective, and scalable interventions. As we home in on candidate interventions, we expect our research to become more deeply focused on a smaller number of interventions.
    1.2. Collaborating with food service to conduct and disseminate research in real-world contexts
    We have a unique collaboration with the Director and Executive Chefs at the Stanford dining halls, allowing us to conduct controlled trials in real-world settings to assess interventions to reduce consumption of meat and animal products. Some of our interventions have been as simple and scalable as reducing the size of spoons used to serve these foods. Also, Stanford Residential & Dining Enterprises is a founding member of the Menus of Change University Research Collaborative (MCURC), a nationwide research consortium of 74 colleges and universities that conduct groundbreaking, collaborative studies on healthy and sustainable food choices in food service. MCURC provides evidence-based recommendations for promoting healthier and more sustainable food choices in food service operations, providing a natural route to dissemination.
    Our established research model involves conducting initial pilot studies at Stanford's dining halls to assess interventions' real-world feasibility and obtain preliminary effect-size estimates, then conducting large-scale, multisite studies by partnering with collaborating members of MCURC. We also have ongoing collaborations with restaurants and plant-based food startups in which we are studying whether adding modern plant-based analogs (e.g., Impossible Burgers or JUST Egg) to a menu reduces sales of animal-based foods.
    1.3. Building a new academic field
    The large majority of empirical research on reducing factory farming has been conducted by nonprofits. In contrast, academics have engaged comparatively little with this cause area (but with notable, commendable exceptions). Academics have a chick’n-and-JUST Egg problem: without a robust academic field for farmed animal welfare, academics remain largely unaware of this cause area and lack the necessary mentorship and career incentives to pursue it; conversely, without individual labs pursuing this research, a robust academic field cannot emerge. Our lab is designed as a prototype, demonstrating that it is feasible – and indeed rather joyful! – for a lab to focus on an EA-aligned, neglected cause area, while also succeeding robustly by the stri...
    11 min
  • AF - Connectomics seems great from an AI x-risk perspective by Steve Byrnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Connectomics seems great from an AI x-risk perspective, published by Steve Byrnes on April 30, 2023 on The AI Alignment Forum.
    Context
    Numerous people are in a position to accelerate certain areas within science or technology, whether by directing funds and resources, or by working in the area directly. But which areas are best to accelerate?
    One possible consideration (among others) is the question: “Is accelerating this technology going to increase the chance that our future transition to superhuman artificial general intelligence (AGI) goes well? Or decrease it? Or make no difference?” My goal here is to try to answer that question for connectomics (the science & technology of mapping how neurons connect to each other in a brain).
    This blog post is an attempt to contribute to Differential Technology Development (DTD) (part of the broader field of Differential Intellectual Progress). Successful DTD involves trying to predict complicated and deeply uncertain future trajectories and scenarios. I think the best we can hope for is to do better than chance. But I’m optimistic that we can at least exceed that low bar.
    My qualifications: I’m probably as qualified as anyone to discuss AI x-risk and how it relates to neuroscience. As for connectomics, I’m not too familiar with the techniques, but I’m quite familiar with how the results are used—in the past few years I have scrutinized probably hundreds of journal articles describing neural tracer measurements. (Think of neural tracer measurements as the traditional, “artisanal”, small-scale version of connectomics.) I find such articles extremely useful; I would happily trade away 20 fMRI papers for one neural tracer paper. This post is very much “my opinions” as opposed to consensus, and I’m happy for further discussion.
    TL;DR
    Improved connectomics technology seems like it would be very helpful for the project of reverse-engineering circuitry in the hypothalamus and brainstem that implement the “innate drives” upstream of human motivations and morality. And that’s a good thing! We may wind up in a situation where future researchers face the problem of designing “innate drives” for an AI; knowing how they work in humans would be helpful for various reasons.
    Improved connectomics technology seems like it would NOT be very helpful for the project of reverse-engineering the learning algorithms implemented by various parts of the brain, particularly the neocortex. And that’s a good thing too! I think that this reverse-engineering effort would lead directly to knowledge of how to build superhuman AGI, whereas I would like us to collectively make much more progress on AGI safety & alignment first, and to learn exactly how to build AGI second.
    Improved connectomics technology might open up a path to achieving Whole Brain Emulation (WBE) earlier than non-WBE AGI. And that’s a good thing too! Generally, a WBE-first future seems difficult to pull off, because (I claim) as soon as we understand the brain well enough for WBE, then we already understand the brain well enough to make non-WBE AGI, and someone will probably do that first. But if we could pull it off, it would potentially be very useful for a safe transition to AGI. I have previously been very skeptical that WBE is a possibility at all, but when I imagine a scenario where radically improved human connectomics technology is available in the near future, then it does actually seem like a possibility to have WBE come before non-WBE AGI, at least by a year or two, given enough effort and luck.
    1. Background considerations
    1.1 The race between reverse-engineering the cortex versus reverse-engineering the hypothalamus & brainstem
    My theory is that parts of the brain (esp. cortex, thalamus, striatum, and cerebellum) are running large-scale learning algorithms, while other parts of...
    18 min
  • AF - The voyage of novelty by Tsvi Benson-Tilsen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The voyage of novelty, published by Tsvi Benson-Tilsen on April 30, 2023 on The AI Alignment Forum.
    [Metadata: crossposted from. First completed January 17, 2023. This essay is more like research notes than exposition, so context may be missing, the use of terms may change across essays, and the text might be revised later; only the versions at tsvibt.blogspot.com are definitely up to date.]
    Novelty is understanding that is new to a mind, that doesn't readily correspond or translate to something already in the mind. We want AGI in order to understand stuff that we haven't yet understood. So we want a system that takes a voyage of novelty: a creative search progressively incorporating ideas and ways of thinking that we haven't seen before. A voyage of novelty is fraught: we don't understand the relationship between novelty and control within a mind.
    What takes the voyage, from where to where? The voyage is taken by some AI system, from not understanding what we want it to understand, to understanding all that. A schematic:
    Novelty
    Novelty is structure that is new to a mind. A mind acquires novelty via creativity.
    Endosystemic novelty is novelty that fits in with the preexisting mind in ways analogous to how preexisting elements of the mind fit in with the mind; e.g., a new sub-skill such as "opening a car door", or a new word. Diasystemic novelty is novelty (new structure) that involves through-going changes to the mind; e.g. a new heuristic like "investigate boundaries" applied automatically across many contexts. At the extreme end of diasystemic novelty are cognitive realms distinct from the cognitive realm currently occupied by the mind.
    Structure hand-holdable by humanity
    Where does the voyage of novelty start? It starts wherever the mind is. But where can it safely start?
    A lot of structure has the property that it is easy enough for humans to render non-dangerous a mind's exploration (discovery, creation) and exercise (application, expression) of that structure. Humanity can "hold a mind's hand" through its voyage of novelty through such structure.
    Some overlapping reasons that structure can be hand-holdable, with examples:
    The structure just doesn't contribute much to a mind having large effects on the world. For example, a mind might know about some obscure species of grass that no human knows about; a human can't understand the mind's thoughts in absolutely full detail, at least without zerself learning about the species of grass. But it just doesn't matter that much, assuming the grass isn't a source of extremely potent toxin or something.
    The structure is easy to intervene on to limit the extent of its influence over the rest of the mind. For example, one can limit the information flow between graspable elements. (However, this sort of hand-holding is very fraught, for example because of conceptual Doppelgängers: the rest of the mind can just reconstitute the supposedly boxed structure in a Doppelgänger outside of the box.)
    The structure is easy for humans to understand fully, including all its implications for and participation in thinking and action, so that it doesn't contribute inscrutably to the mind's activity. See gemini modeling. For example, the mental model [coffee cup] is hand-holdable, as humans have it. This is a central example of hand-holdability: the mind (hypothetically, at least) in question does the same things, thinks the same way, with respect to coffee cups, as do humans; what's in common between the mind's activity in different contexts with respect to coffee cups, is also in common with humans.
    (It's not clear whether and how this is possible in practice, for one thing due to essential provisionality: the way a mind thinks about something can always change and expand, so that [the structure that currently plays the analogous role in ...
    10 min
  • LW - Research agenda: Supervising AIs improving AIs by Quintin Pope
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Research agenda: Supervising AIs improving AIs, published by Quintin Pope on April 29, 2023 on LessWrong.
    [This post summarizes some of the work done by Owen Dudney, Roman Engeler and myself (Quintin Pope) as part of the SERI MATS shard theory stream.]
    TL;DR
    Future prosaic AIs will likely shape their own development or that of successor AIs. We're trying to make sure they don't go insane.
    Summary
    There are two main ways AIs can get better: by improving their training algorithms or by improving their training data.
    We consider both scenarios, and tentatively believe that data-based improvement is riskier than architecture based improvement. Current models mostly derive their behavior from their training data, and not training algorithms (meaning their architectures, hyperparameters, loss functions, optimizers or the like). So far, most improvements to AI training algorithms seem 'value neutral'. Also note that most of human value drift currently derives from cultural shifts changing the 'training data' available in the environment, not biological evolution over the brain's base learning algorithms.
    We imagine a future where AIs self-augment by continuously seeking out more and better training data, and either creating successor AIs or training themselves on that data. Often, these data will come from the AIs running experiments in the real world (doing science), deliberately seeking data that would cover a specific gap in its current capabilities, analogous to how human scientists seek data from domains where our current understanding is limited. With AI, this could involve AgentGPT-like systems that spin up many instances of themselves to run experiments in parallel, potentially leading to quick improvements if we are in an agency overhang.
    We want to find methods of ensuring such 'automated science' processes remain safe and controllable, even after many rounds of self-directed data collection and training. In particular, we consider problems such as:
    Preventing self-training from amplifying undesirable behaviors
    Preventing semantic drift in concept representations during self-training
    Ensuring cross-modality actions (such as a generated image for a text-to-image model, or robot movement for a text-and-image-to-actuator-motion model) remain grounded in their natural language descriptions after self-training in a non-lingual modality
    Preventing value drift during multiple, iterated steps of self-retraining
    Currently, we're focusing on scalable methods of tracking behavioral drift in language models, as well as benchmarks for evaluating a language model's capacity for stable self-modification via self-training.
    Introduction
    So far, most improvements in AI capabilities fall into two categories:
    Algorithmic: better architectures, better loss functions, better optimizers, etc.
    Data-driven: more training data, higher quality data, domain-specific data, etc.
    We expect the future to resemble the past, and so we expect that future capabilities improvements will come from these two sources. However, it also seems likely that AIs will increasingly be the ones responsible for such capabilities advances. In fact, current work has already started using language models as part of the data curation process, or to generate future training data directly. Moreover, with GPT-3's widespread adoption, it is probable that GPT-4's training data contains content generated by its predecessor. This phenomenon extends to fine-tuning processes like RLHF, where earlier versions' output influences the cognition of subsequent iterations.
    Researchers are likely to use the most capable models available to them in whatever AI-driven improvement process they devise. Thus, such a process is iterative, with the first AI shaping the training of the second AI, shaping the training of the third AI, and so ...
    38 min
  • EA - If you’d like to do something about sexual misconduct and don’t know what to do. by Habiba
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: If you’d like to do something about sexual misconduct and don’t know what to do., published by Habiba on April 30, 2023 on The Effective Altruism Forum.
    Photo by Andrew Moca
    Following on from posts from women in the community discussing recent events (like this and this) I wanted to provide some specific suggestions for people who want to do something about sexual misconduct and harassment in EA but don’t know where to start. If you find the word useful, you could consider this post a guide to being an “ally”
    I was inspired to write this after some of my male friends asked what they could do to help and/or were pretty surprised by how well the things they did went down. But it’s hopefully useful advice for everyone, myself included.
    I’ve written this most with sexual harassment in mind. However, some of the sections are applicable in thinking about sexual assault, violence or even rape.
    Caveats:
    The post is not attempting to persuade anyone who isn’t already convinced that it’s worth taking action - I’ll leave that to other posts.
    It’s just my take on what might be helpful to do
    Navigating this stuff is tricky - what makes most sense for you to do is going to be very context dependent
    My thoughts are fairly focused on the culture and norms I’m most familiar with (I’m a woman of colour, from the UK)
    Summary
    Remember we’re all within the system - One useful frame can be thinking about what actions are the “paths of least resistance” in social situations that lead to harm, and trying to not take those ourselves as a way of trying to change them
    Act with compassion - In general act as if people in the discussion have a decent chance of having been personally affected. And be extra compassionate as a result.
    Learn about the issue - You can still be really helpful even if you don’t feel like a world expert on the topic. That said, do carry on learning more, including beyond the EA community, especially before suggesting improvements. This can be either by yourself (there’s some suggestions at the end) or with others (e.g. start a discussion group)
    Listen to community members empathetically - Be an empathetic listener to people in discussions about sexual misconduct. Both reactively when they raise things with you and proactively by reaching out to people who might feel affected. Consider pausing discussing your opinions and solutions with someone who is upset at least until you’ve done the listening bit first.
    Support people who tell you about their experience - Listen to and carry on being supportive to someone who shares a story with you about an experience they had. Respect their preferences on confidentiality and autonomy over what to do next. And also look after yourself.
    Reflect on your own behaviour - Spend a little time reflecting on your own past behaviour and if there is anything you want to do about past actions or change for the future. Check in with someone else if so. This is tough but courageous to do.
    Interject when you see harmful behaviour - Whether it's happening online or in person interject as an active bystander. Challenge the behaviour and/or look out for the person affected
    Take action about people who have harmed others - Take responsibility for doing something about people you interact with who have harmed others, but handle this with care. Consult with friends / experts to work out what to do. Feel free to maintain your own distance even if behaviour isn’t bad enough to face professional repercussions
    Participate in the discussion - When there is community wide discussion about issues like sexual misconduct participate by signal boosting those affected and contributing your own takes
    Contribute to or start community wide initiatives - Contribute to existing projects in the EA space, take actions within your local EA group or workplace and conside...
    34 min
  • LW - One bit of observation can unlock many of optimization - but at what cost? by dr s
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: One bit of observation can unlock many of optimization - but at what cost?, published by dr s on April 29, 2023 on LessWrong.
    This question by johnswentworth nerd sniped me, so I ended up thinking a lot about the relationship between information and control over the world in the simplified scenario of a single tape of bits. The question asked how many bits of optimization one could unlock with a single bit of observation; the answer ended up being "arbitrarily many", as proven by a simple example: suppose you have a rule that says that if the first bits of your action Amatch those of your observation O, then the rest of A gets copied to your target value Y; then there's no limit to how many bits you can copy, and the only information you need to leverage is the knowledge of the secret "key". We know this works because this is exactly how locks work in real life too.
    If I possess someone's bank account password, or the combination to a safe, or the US nuclear codes, then I'm able to produce disproportionate effects to the tiny size of that knowledge, just because I can also rely on the state of the world and its laws being set up in such a way that I can use that knowledge as a pivot to trigger much bigger effects.
    The thing I wanted to focus on then was those conditions: even given that I know the "password", what else do I need to know about the world at large, and what limits are there on my power to optimize the final state? I decided to focus on the following model:
    a world string W of N bits, prepared in some initial state W0, with some regions known and some randomized;
    a discrete map f that determines the evolution of this world, such that W1=f(W0), W2=f(f(W0)) and so on. The map is reversible, such that there exists an inverse Wi−1=f−1(Wi);
    The focus on the map being reversible is because in the real world the laws of physics are time symmetric too and microscopically should not destroy information. Irreversible computing allows the deletion of information, which reduces the entropy of the system. In a computer embedded in a larger world this can be compensated by creating entropy somewhere else, but if our string has to represent the entire world, then it should preserve information. The world string has several identifiable regions:
    an action region A, within which we can set up bits arbitrarily in the initial state;
    an observation region O, which we can't affect but whose contents we know exactly in the initial state;
    a target region Y, to which we aim at writing certain bits so as to maximize the mutual information I(Y;G) with some goal string G;
    two fuel regions F0 and F1, at the ends of the string, filled respectively with all 0s and all 1s. We'll see the use for them in a moment.
    The map f can be defined as a series of instructions. Since we're doing reversible computing, we can use only one universal logic gate, like the Toffoli gate (CCNOT) or the Fredkin gate (CSWAP). These take three bits as arguments, so once we've decided our gate of choice an entire program can be composed simply of triples of addresses of bits, wherein each address will require L=⌈log2(N)⌉ bits, meaning a program's size is 3L bits/instruction. Consider the simplest possible version of a "lock-like" program, which compares b1 and b2, and if they're identical, it swaps b3 and b4 (note that we can't simply copy b3: that would erase information and not be reversible). We will also need two "fuel" bits f1 and f2 prepared respectively in the 0 and 1 states. The program can be written with just three Fredkin gates:
    CSWAP b_2 f_1 f_2
    CSWAP f_2 b_3 b_4
    After this, the two "fuel" bits are used up and can't be relied on any more for future calculations. From the viewpoint of the entire string, of course, the entropy is constant and the process is entirely reversible; but if you only looke...
    9 min
  • LW - Biological DOOM: a brief overview of biological computation by Metacelsus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Biological DOOM: a brief overview of biological computation, published by Metacelsus on April 29, 2023 on LessWrong.
    (no, not that kind of biological doom)
    DOOM is a classic first-person shooter game released in 1993 by id Software. Because it’s from 1993, it doesn’t require much computing power compared to modern games. Additionally, the code (written in C) is easy to compile to run on a variety of processors.
    Over the years, hackers have made DOOM run on things such as an ATM, a touchbar of a MacBook, a Porsche 911, and even a TI-84 calculator powered by potato batteries.
    But what about cells?
    Requirements for DOOM
    The inputs to DOOM are based on button presses, traditionally on a keyboard. 9 keys in total are required (assuming “switch weapon” is implemented as one key that cycles through weapons).
    For computation, the original 1993 release required:
    4 MB of RAM and 12 MB of hard-drive storage
    Intel 386 (bare minimum) or 486 processor.
    There is some flexibility regarding the processor, but slower processors will have worse frame-rates. The Intel 386 had 275,000 transistors in its most basic configuration.
    DOOM also requires a graphical output. The smallest resolution I’ve seen is 128x32 pixels, and that was cutting it a bit close. We’ll assume we need 4096 black-and-white pixels for the display.
    Finally, DOOM has audio. For the purposes of this thought experiment, we can ignore this output. Although the soundtrack is great, it’s not strictly required to play the game.
    Approaches to biological computation
    So, how could we potentially run DOOM? Biological systems can perform computations in several ways:
    Nucleic acid hybridization
    These logic gates are based on strand displacement between complementary DNA sequences. A recent paper demonstrated a set of DNA-based logic gates that could add two 6-bit binary numbers.
    Pros and cons:
    Memory capacity is good (encoded in DNA or RNA)
    Switching speed is OK (rate constants vary by design but are typically around 106M−1s−1)
    Visual output could be provided by fluorophore/quencher conjugated oligonucleotides, but . . .
    Coupling to a macroscopic output display would be far too slow, because it would have to rely on diffusion (taking a few minutes to cover a millimeter-scale distance). So, the game would have to be played using a microscope.
    It’s hard to “reset” gates after using them, this requires coupling to some energy source
    It’s also hard to integrate DNA-based logic gates into other biological systems, since not many organisms use short pieces of ssDNA. RNA might be used instead.
    Transcription and translation
    These logic gates use the same tools that cells use to regulate gene expression. For example, the classic lac operon in bacteria implements:
    Biologists have exploited similar systems to build logic gates, as well as systems involving the regulation of translation (the production of proteins using mRNAs as templates). A recent paper used Cas9 binding to a sgRNA-like sequence inserted in an mRNA to control its translation. To form a NAND gate, they split Cas9 into two fragments; if both were present, the output protein was not produced.
    Pros and cons:
    Memory capacity is acceptable (encoded in DNA or RNA)
    There will be challenges with implementing the number of logic gates required while avoiding cross-talk
    The dealbreaker: far too slow to run DOOM. RNA and protein half-lives are on the order of minutes to hours.
    Protein phosphorylation (kinases/phosphatases)
    Many cell signaling pathways use protein phosphorylation as a signal. This is much faster than transcription and translation, since no new RNAs or proteins need to be produced. A paper in 2021 built a toggle switch in yeast out of several kinases and phosphatases.
    Pros and cons:
    Response speed is adequate, similar to nucleic acid hybridization gates (i.e., largely l...
    10 min
  • LW - Which technologies are stuck on initial adoption? by Vanessa Kosoy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Which technologies are stuck on initial adoption?, published by Vanessa Kosoy on April 29, 2023 on LessWrong.
    Video calls have been with us for a while. Except, they were rarely used. IME, people sometimes had Skype calls with relatives abroad and that's about it. And then, COVID happened. Suddenly, Zoom skyrocketed, with Google Meet not far behind. The reason is obvious.
    Now, the time of lockdowns and restrictions on gatherings is over, the incentives to do video calls are (AFAICT) more or less the same as pre-COVID, and yet video calls persist. They became a completely routine way of doing business meetings, academic seminars and occasional social events. Why? AFAICT it's just the initial adoption barrier: once everyone did lots of video calls, and realized they are actually pretty convenient, they just kept using them.
    So, here's a fun question: What other things are like video calls in the pre-COVID era? That is, the technology exists (more or less: maybe the UX needs some trivial improvements), the use-cases exist, only nobody uses it just because they're unaware or because it's not a "normal" thing everyone does. Given something to create initial adoption (like COVID did for video calls), everyone would start using it and never go back.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Research agenda: Supervising AIs improving AIs by Quintin Pope
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Research agenda: Supervising AIs improving AIs, published by Quintin Pope on April 29, 2023 on The AI Alignment Forum.
    [This post summarizes some of the work done by Owen Dudney, Roman Engeler and myself (Quintin Pope) as part of the SERI MATS shard theory stream.]
    TL;DR
    Future prosaic AIs will likely shape their own development or that of successor AIs. We're trying to make sure they don't go insane.
    Summary
    There are two main ways AIs can get better: by improving their training algorithms or by improving their training data.
    We consider both scenarios, and tentatively believe that data-based improvement is riskier than architecture based improvement. Current models mostly derive their behavior from their training data, and not training algorithms (meaning their architectures, hyperparameters, loss functions, optimizers or the like). So far, most improvements to AI training algorithms seem 'value neutral'. Also note that most of human value drift currently derives from cultural shifts changing the 'training data' available in the environment, not biological evolution over the brain's base learning algorithms.
    We imagine a future where AIs self-augment by continuously seeking out more and better training data, and either creating successor AIs or training themselves on that data. Often, these data will come from the AIs running experiments in the real world (doing science), deliberately seeking data that would cover a specific gap in its current capabilities, analogous to how human scientists seek data from domains where our current understanding is limited. With AI, this could involve AgentGPT-like systems that spin up many instances of themselves to run experiments in parallel, potentially leading to quick improvements if we are in an agency overhang.
    We want to find methods of ensuring such 'automated science' processes remain safe and controllable, even after many rounds of self-directed data collection and training. In particular, we consider problems such as:
    Preventing self-training from amplifying undesirable behaviors
    Preventing semantic drift in concept representations during self-training
    Ensuring cross-modality actions (such as a generated image for a text-to-image model, or robot movement for a text-and-image-to-actuator-motion model) remain grounded in their natural language descriptions after self-training in a non-lingual modality
    Preventing value drift during multiple, iterated steps of self-retraining
    Currently, we're focusing on scalable methods of tracking behavioral drift in language models, as well as benchmarks for evaluating a language model's capacity for stable self-modification via self-training.
    Introduction
    So far, most improvements in AI capabilities fall into two categories:
    Algorithmic: better architectures, better loss functions, better optimizers, etc.
    Data-driven: more training data, higher quality data, domain-specific data, etc.
    We expect the future to resemble the past, and so we expect that future capabilities improvements will come from these two sources. However, it also seems likely that AIs will increasingly be the ones responsible for such capabilities advances. In fact, current work has already started using language models as part of the data curation process, or to generate future training data directly. Moreover, with GPT-3's widespread adoption, it is probable that GPT-4's training data contains content generated by its predecessor. This phenomenon extends to fine-tuning processes like RLHF, where earlier versions' output influences the cognition of subsequent iterations.
    Researchers are likely to use the most capable models available to them in whatever AI-driven improvement process they devise. Thus, such a process is iterative, with the first AI shaping the training of the second AI, shaping the training of the thir...
    38 min
  • EA - New open letter on AI — "Include Consciousness Research" by Jamie Harris
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New open letter on AI — "Include Consciousness Research", published by Jamie Harris on April 28, 2023 on The Effective Altruism Forum.
    Quick context:
    The potential development of artificial sentience seems very important; it presents large, neglected, and potentially tractable risks.
    80,000 Hours lists artificial sentience and suffering risks as "similarly pressing but less developed areas" than their top 8 "list of the most pressing world problems".
    There's some relevant work on this topic by Sentience Institute, Future of Humanity Institute, Center for Reducing Suffering, and others, but room for much more. Yesterday someone asked on the Forum "How come there isn't that much focus in EA on research into whether / when AI's are likely to be sentient?"
    A month ago, people got excited about the FLI open letter: "Pause giant AI experiments".
    Now, Researchers from the Association for Mathematical Consciousness Science have written an open letter emphasising the urgent need for accelerated research in consciousness science in light of rapid advancements in artificial intelligence. (I'm not affiliated with them in any way.)
    It's quite short, so I'll copy the full text here:
    This open letter is a wakeup call for the tech sector, the scientific community and society in general to take seriously the need to accelerate research in the field of consciousness science.
    As highlighted by the recent “Pause Giant AI Experiments” letter [1], we are living through an exciting and uncertain time in the development of artificial intelligence (AI) and other brain-related technologies. The increasing computing power and capabilities of the new AI systems are accelerating at a pace that far exceeds our progress in understanding their capabilities and their “alignment” with human values.
    AI systems, including Large Language Models such as ChatGPT and Bard, are artificial neural networks inspired by neuronal architecture in the cortex of animal brains. In the near future, it is inevitable that such systems will be constructed to reproduce aspects of higher-level brain architecture and functioning. Indeed, it is no longer in the realm of science fiction to imagine AI systems having feelings and even human-level consciousness. Contemporary AI systems already display human traits recognised in Psychology, including evidence of Theory of Mind [2].
    Furthermore, if achieving consciousness, AI systems would likely unveil a new array of capabilities that go far beyond what is expected even by those spearheading their development. AI systems have already been observed to exhibit unanticipated emergent properties [3]. These capabilities will change what AI can do, and what society can do to control, align and use such systems. In addition, consciousness would give AI a place in our moral landscape, which raises further ethical, legal, and political concerns.
    As AI develops, it is vital for the wider public, societal institutions and governing bodies to know whether and how AI systems can become conscious, to understand the implications thereof, and to effectively address the ethical, safety, and societal ramifications associated with artificial general intelligence (AGI).
    Science is starting to unlock the mystery of consciousness. Steady advances in recent years have brought us closer to defining and understanding consciousness and have established an expert international community of researchers in this field. There are over 30 models and theories of consciousness (MoCs and ToCs) in the peer-reviewed scientific literature, which already include some important pieces of the solution to the challenge of consciousness.
    To understand whether AI systems are, or can become, conscious, tools are needed that can be applied to artificial systems. In particular, science needs to further develop formal and mat...
    6 min