The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Decision Transformer Interpretability by Joseph Bloom
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Decision Transformer Interpretability, published by Joseph Bloom on February 6, 2023 on LessWrong.
    TLDR: We analyse how a small Decision Transformer learns to simulate agents on a grid world task, providing evidence that it is possible to do circuit analysis on small models which simulate goal-directedness. We think Decision Transformers are worth exploring further and may provide opportunities to explore many alignment-relevant deep learning phenomena in game-like contexts.
    Link to the GitHub Repository. Link to the Analysis App. I highly recommend using the app if you have experience with mechanistic interpretability. All of the mechanistic analysis should be reproducible via the app.
    Key Claims
    A 1-Layer Decision Transformer learns several contextual behaviours which are activated by a combination of Reward-to-Go/Observation combinations on a simple discrete task.
    Some of these behaviours appear localisable to specific components and can be explained with simple attribution and the transformer circuits framework.
    The specific algorithm implemented is strongly affected by the lack of a one-hot-encoding scheme (initially left out for simplicity of analysis) of the state/observations, which introduces inductive biases that hamper the model.
    If you are short on time, I recommend reading:
    Dynamic Obstacles Environment
    Black Box Model Characterisation
    Explaining Obstacle Avoidance at positive RTG using QK and OV circuits
    Alignment Relevance
    Future Directions
    I would welcome assistance with:
    Engineering tasks like app development, improving the model, training loop, wandb dashboard etc. and people who can help me make nice diagrams and write up the relevant maths/theory in the app).
    Research tasks. Think more about how to exactly construct/interpret circuit analysis in the context of decision transformers. Translate ideas from LLMs/algorithmic tasks.
    Communication tasks: Making nicer diagrams/explanations.
    I have a Trello board with a huge number of tasks ranging from small stuff to massive stuff.
    I’m also happy to collaborate on related projects.
    Introduction
    For my ARENA Capstone project, I (Joseph) started working on decision transformer interpretability at the suggestion of Paul Colognese. Decision transformers can solve reinforcement learning tasks when conditioned on generating high rewards via the specified “Reward-to-Go” (RTG). However, they can also generate agents of varying quality based on the RTG, making them simultaneously simulators, small transformers and RL agents. As such, it seems possible that identifying and understanding circuits in decision transformers would not only be interesting as an extension of current mechanistic interpretability research but possibly lead to alignment-relevant insights.
    Previous Work
    The most important background for this post is:
    The Decision Transformers paper showed how RL tasks can be solved with transformer sequence modelling. Figure 1 from their paper describes the critical components of a Decision Transformer.
    A Mathematical Framework for Transformer Circuits that describes how to think about transformers in the context of mechanistic interpretability. Important ideas include the ability to decompose the residual stream into the output of attention heads and MLPs, the QK circuits (decides if to write information to the residual stream), and OV circuits (decides what to write to the residual stream).
    The Understanding RL Vision, which analyses how an RL agent with a large CNN component responds to input features, attributing them as good or bad news in the value function and proposes the Diversity hypothesis - “Interpretable features tend to arise (at a given level of abstraction) if and only if the training distribution is diverse enough (at that level of abstraction).”
    Methods
    Environment - RL Environments.
    GridWor...
    36 min
  • EA - If Adult Insects Matter, How Much Do Juveniles Matter? by Bob Fischer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: If Adult Insects Matter, How Much Do Juveniles Matter?, published by Bob Fischer on February 6, 2023 on The Effective Altruism Forum.
    Key Takeaways
    There are major uncertainties about sentience, welfare ranges, and the taxa of interest that make it difficult to draw any firm conclusions.
    Nevertheless, it’s plausible that there are some differences in the probability of sentience and welfare ranges across insect life stages. However likely it is that adult insects are sentient, it's less likely that juveniles are sentient; however much welfare adult insects can realize, juveniles can probably realize less.
    These differences are probably best explained by the life requirements at those stages, which vary in species-specific ways.
    Insofar as we can compare the magnitudes of these differences, the best evidence currently available doesn’t support the conclusion that they’re large. For instance, we don't see a plausible argument for the view that adult insects matter 1,000x more than juveniles.
    We may be able to make some predictions about the probability of sentience based on two factors: first, degrees of independence at the immature life stages; second, the developmentally-relevant details of the rearing environments.
    Introduction
    This is the ninth and final post in the Moral Weight Project Sequence. The aim of the sequence is to provide an overview of the research that Rethink Priorities conducted between May 2021 and October 2022 on interspecific cause prioritization—i.e., making resource allocation decisions across species.
    The goal of this post is to help animal welfare grantmakers assess the relative value of improvements to the lives of some commercially-important insects: the yellow mealworm (Tenebrio molitor), black soldier fly (Hermetia illucens), western honey bee (Apis mellifera), house cricket (Acheta domesticus), and silkworm (Bombyx mori). The vast majority of yellow mealworms, black soldier flies, and silkworms do not reach adulthood in the commercial systems of interest here (the exceptions are the members of the breeding population). By contrast, honey bees and house crickets do reach adulthood. This raises the question of how to value improvements to the lives of insects in their immature versus adult life stages. If adult insects are more likely to be sentient than immature insects, then, all else equal, it’s more important to prevent harm to adult insects than immature insects.
    Similarly, if it’s likely that adult insects can suffer more intensely than immature insects (which is relevant to their respective welfare ranges, which are one dimension of their respective capacities for welfare), then, all else equal, it’s more important to prevent harm to adult insects than immature insects. So, insofar as it’s possible to identify differences in either the probability of sentience or welfare ranges, that information is action-relevant when all else is equal. And insofar as we can compare the magnitudes of any such differences, that information may be action-relevant when all else isn’t equal.
    This document unfolds as follows. In the next section, we provide some relevant biological information about the taxa of interest and introduce the reader to some important dimensions of the development and neurobiology of insects. After that, we explain how Rethink Priorities tried to estimate differences in the probability of sentience and welfare ranges for other taxa—which, among other things, involves surveying the literature for evidence regarding sentience- and welfare-range-relevant traits. We extend that approach here. Then, we summarize the literature we reviewed. Next, we argue for the key takeaways. Finally, we make some suggestions regarding future work in this area.
    Biological Background Information
    Insects develop in the larval (or nymphal) stage, un...
    49 min
  • AF - Decision Transformer Interpretability by Joseph Bloom
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Decision Transformer Interpretability, published by Joseph Bloom on February 6, 2023 on The AI Alignment Forum.
    TLDR: We analyse how a small Decision Transformer learns to simulate agents on a grid world task, providing evidence that it is possible to do circuit analysis on small models which simulate goal-directedness. We think Decision Transformers are worth exploring further and may provide opportunities to explore many alignment-relevant deep learning phenomena in game-like contexts.
    Link to the GitHub Repository. Link to the Analysis App. I highly recommend using the app if you have experience with mechanistic interpretability. All of the mechanistic analysis should be reproducible via the app.
    Key Claims
    A 1-Layer Decision Transformer learns several contextual behaviours which are activated by a combination of Reward-to-Go/Observation combinations on a simple discrete task.
    Some of these behaviours appear localisable to specific components and can be explained with simple attribution and the transformer circuits framework.
    The specific algorithm implemented is strongly affected by the lack of a one-hot-encoding scheme (initially left out for simplicity of analysis) of the state/observations, which introduces inductive biases that hamper the model.
    If you are short on time, I recommend reading:
    Dynamic Obstacles Environment
    Black Box Model Characterisation
    Explaining Obstacle Avoidance at positive RTG using QK and OV circuits
    Alignment Relevance
    Future Directions
    I would welcome assistance with:
    Engineering tasks like app development, improving the model, training loop, wandb dashboard etc. and people who can help me make nice diagrams and write up the relevant maths/theory in the app).
    Research tasks. Think more about how to exactly construct/interpret circuit analysis in the context of decision transformers. Translate ideas from LLMs/algorithmic tasks.
    Communication tasks: Making nicer diagrams/explanations.
    I have a Trello board with a huge number of tasks ranging from small stuff to massive stuff.
    I’m also happy to collaborate on related projects.
    Introduction
    For my ARENA Capstone project, I (Joseph) started working on decision transformer interpretability at the suggestion of Paul Colognese. Decision transformers can solve reinforcement learning tasks when conditioned on generating high rewards via the specified “Reward-to-Go” (RTG). However, they can also generate agents of varying quality based on the RTG, making them simultaneously simulators, small transformers and RL agents. As such, it seems possible that identifying and understanding circuits in decision transformers would not only be interesting as an extension of current mechanistic interpretability research but possibly lead to alignment-relevant insights.
    Previous Work
    The most important background for this post is:
    The Decision Transformers paper showed how RL tasks can be solved with transformer sequence modelling. Figure 1 from their paper describes the critical components of a Decision Transformer.
    A Mathematical Framework for Transformer Circuits that describes how to think about transformers in the context of mechanistic interpretability. Important ideas include the ability to decompose the residual stream into the output of attention heads and MLPs, the QK circuits (decides if to write information to the residual stream), and OV circuits (decides what to write to the residual stream).
    The Understanding RL Vision, which analyses how an RL agent with a large CNN component responds to input features, attributing them as good or bad news in the value function and proposes the Diversity hypothesis - “Interpretable features tend to arise (at a given level of abstraction) if and only if the training distribution is diverse enough (at that level of abstraction).”
    Methods
    Environment - RL Environm...
    36 min
  • LW - Why is Everyone So Boring? By Robin Hanson by Trevor1
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why is Everyone So Boring? By Robin Hanson, published by Trevor1 on February 6, 2023 on LessWrong.
    This is from the blog Overcoming Bias, which many people have heard of but don't know that it's still going and they can read it and recommend it to friends (especially to introduce acquaintances to rationality, which can be used to elevate your status in the average workplace).
    If posts from Overcoming Bias aren't supposed to be linkposted to Lesswrong, please message a mod, message me, and/or or delete this post immediately without asking me.
    From "Why is Everyone So Boring?":
    Centuries ago, while people could rest safe and show themselves at home, when traveling between towns they tried to look either look poor or well-defended, as bandits lay in wait. Even within towns, people without allies who acted unusually rich, assertive, and confident would induce others to try to trip them somehow. It’s the tall poppy that gets cut down, after all.
    I propose that the main reason that most of us look more boring in public is that social predators lie in wait there. With friends, family, and close co-workers, we are around people that mostly want to like us, and know us rather well. Yes, they want us to conform too, but they apply this pressure in moderation.
    Out in public, in contrast, we face bandits eager for chances to gain social credit by taking us down, often via accusing us of violating the sacred. And like townspeople traveling among the bandits, we are in public pretty vulnerable to the kinds of bandits that afflict us.
    If we act interesting, passionate, and opinionated in public, we are likely to seem to claim high status for ourselves, and to touch on sacred subjects, either by word or deed. And this makes us quite vulnerable to accusations of arrogance and violating the sacred.
    I see roughly three typical public stances: boring, lively, or outraged. Either you act boring, so the bandits will ignore you, you act lively, and invite bandit attacks, or you act outraged, and play a bandit yourself. Most big orgs and experts choose boring, and most everyone else who doesn’t pick boring picks bandit, especially on social media. It takes unusual art, allies, and energy, in a word “eliteness”, to survive while choosing lively.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - Gradient surfing: the hidden role of regularization by Jesse Hoogland
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gradient surfing: the hidden role of regularization, published by Jesse Hoogland on February 6, 2023 on The AI Alignment Forum.
    Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort
    In a previous post, I demonstrated that Brownian motion near singularities defies our expectations from "regular" physics. Singularities trap random motion and take up more of the equilibrium distribution than you'd expect from the Gibbs measure.
    In the computational probability community, this is a well-known pathology. Sampling techniques like Hamiltonian Monte Carlo get stuck in corners, and this is something to avoid. You typically don't want biased estimates of the distribution you're trying to sample.
    In deep learning, I argued, this behavior might be less a bug than a feature.
    The claim of singular learning theory is that models near singularities have lower effective dimensionality. From Occam's razor, we know that simpler models generalize better, so if the dynamics of SGD get stuck at singularities, it would suggest an explanation (at least in part) for why SGD works: the geometry of the loss landscape biases your optimizer towards good solutions.
    This is not a particularly novel claim. Similar versions of the claim been made before by Mingard et al. and Valle Pérez et al.. But from what I can tell, the proposed mechanism, of singularity "stickiness", is quite different.
    Moreover, it offers a new possible explanation for the role of regularization. If exploring the set of points with minimum training loss is enough to get to generalization, then perhaps the role of regularizer is not just to privilege "simpler" functions but also to make exploration possible.
    In the absence of regularization, SGD can't easily move between points of equal loss. When it reaches the bottom of a valley, it's pretty much stuck. Adding a term like weight decay breaks this invariance. It frees the neural network to surf the loss basin, so it can accidentally stumble across better generalizing solutions.
    So could we improve generalization by exploring the bottom of the loss basin in other ways — without regularization or even without SGD? Could we, for example, get a model to grok through random drift?
    .No. We can't.
    That is to say I haven't succeeded yet. Still, in the spirit of "null results are results", let me share the toy model that motivated this hypothesis and the experiments that have (as of yet) failed to confirm it.
    The inspiration: a toy model
    First, let's take a look at the model that inspired the hypothesis.
    Let's begin by modifying the example of the previous post to include an optional regularization term controlled by λ:
    We deliberately center the regularization away from the origin at c=(−1,−1) so it doesn't already privilege the singularity at the origin.
    Now, instead of viewing U(x) as a potential and exploring it with Brownian motion, we'll treat it as a loss function and use stochastic gradient descent to optimize for x.
    We'll start our optimizer at a uniformly sampled random point in this region and take T=100 steps down the gradient (with optional momentum controlled by β). After each gradient step, we'll inject a bit of Gaussian noise to simulate the "stochasticity." Altogether, the update rule for x is as follows:
    with momentum updated according to:
    and noise given by,
    If we sample the final obtained position, x(T) over independent initializations, then, in the absence of regularization and in the presence of a small noise term, we'll get a distribution that looks like the figure on the left.
    Unlike the case of random motion, the singularity at the origin is now repulsive. Good luck finding those simple solutions now.
    However, as soon as we turn on the regularization (middle figure) or increase the noise term (figure on the right), the singulari...
    9 min
  • LW - Why Are Bacteria So Simple? by aysja
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Are Bacteria So Simple?, published by aysja on February 6, 2023 on LessWrong.
    As far as we can tell, bacteria were the first lifeforms on Earth. Which means they’ve had a full four billion years to make something of themselves. And yet, despite their long evolutionary history, they mostly still look like this:
    Bacteria belong to one major class of cells—prokaryotes. The other major class of cells, eukaryotes, arrived about one billion years after bacteria. But despite their late start, they are vastly more complex.
    Prokaryotes mostly only contain DNA, and DNA translation machinery. Eukaryotes, on the other hand, contain a huge variety of internal organelles that run all kinds of specialized processes—lysosomes digest, vesicles transport, cytoskeletons offer structural support, etc.
    Not only that, but all multicellular life is eukaryotic. Every complex organism evolution has produced—eukaryotic. Trees, humans, worms, giant squid, dogs, insects—eukaryotic. Somehow, eukaryotes managed to blossom into all of these complex forms, while bacteria steadfastly remained single-celled, simple, and small. Why?
    The short answer is that prokaryotes have vastly less DNA than eukaryotes—four to five orders of magnitude less, on average—and hence can’t do nearly as much stuff. The long answer is the rest of this post, which investigates two related questions: first, why are eukaryotic genomes so long? And second, how exactly does more DNA allow for more complexity?
    Why Are Eukaryotic Genomes So Long?
    Scalable Energy Production
    Using DNA—replicating, transcribing, and translating it into proteins—isn’t free. Cells need energy (such as ATP) to power these reactions and, all else equal, longer genomes will require more of it.
    Both prokaryotes and eukaryotes pay similar energetic costs to maintain genes. The difference is that eukaryotes have way more energy and hence can afford to have longer genomes. But why this disparity?
    Prokaryotes generate ATP along their cell membrane. Which means that as they increase in size, their surface area—and hence their energy production—will scale sublinearly with their volume. So a prokaryote that doubles in size, for example, will only end up producing half as much ATP per unit volume. Because prokaryotes become less metabolically efficient as they get bigger, most are quite small—six orders of magnitude smaller than eukaryotes, on average.
    There are some exceptions. For instance, individual bacteria in the species Thiomargarita can reach up to one centimeter in size, visible to the naked eye! But its cell structure suggests the exception proves the rule—80% of its volume is a vacuole, essentially empty space. So in effect, evolution expanded its surface area without concomitantly expanding its functional volume—a neat trick!
    But how do eukaryotes avoid this surface area constraint? Well, eukaryotes generate energy using mitochondria, which are inside the cells. As a result, their number of mitochondria—and hence their energy production—scales with their volume. This allows them to afford both larger cell sizes than prokaryotes, and also longer genomes.
    Tolerance for Junk
    But bioenergetic constraints aren't the whole story. Even leaving aside the direct energy costs, prokaryotes face way more selection pressure toward having short genomes.
    Empirically, bacteria are very quick to rid themselves of genes once they're no longer useful. For example, if you insert DNA into a bacteria that affords antibiotic resistance, it will keep those genes as long as antibiotics are around. But once you remove the antibiotics, it will jettison that DNA within a few hours.
    Eukaryotic DNA, on the other hand, is much more weakly selected against. While bacteria are sensitive to additions of DNA fewer than ten base pairs in length, eukaryotes will keep additions of ove...
    23 min
  • EA - Pandemic Prediction Checklist: H5N1 by DirectedEvolution
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pandemic Prediction Checklist: H5N1, published by DirectedEvolution on February 5, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - Effective Altruism Deconfusion, Part 2: Causes, Philosophy, and Social Constraints by Davidmanheim
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Effective Altruism Deconfusion, Part 2: Causes, Philosophy, and Social Constraints, published by Davidmanheim on February 5, 2023 on The Effective Altruism Forum.
    This is part 2 of my attempt to disentangle and clarify some parts of the overall set of claims that comprise effective altruism, in this case, the set of philosophical positions and cause areas - not to evaluate them, but to enable clearer discussion of the claims, and disagreements, to both opponents and proponents. In my previous post, I made a number of claims about Effective Altruism as a philosophical position. I claimed that there is a nearly universally accepted normative claim that doing good things is important, and a slightly less universal but widely agreed upon claim that one should do those things effectively.
    My central claim in this post is that the notion of “impartial,” when determining “how to maximize the good with a given unit of resources, in impartial welfarist terms” is hiding almost all of the philosophical complexity and debate that occurs. In the previous post, I said that Effective Altruism as a philosophy was widely shared. Now I’m saying that the specific goals are very much not shared. Unsurprisingly, this mostly appears in discussions of cause area prioritization. But the set of causes that could be prioritized are, I claim, far larger than the set effective altruists typically assume - and embed lots of assumptions and claims that aren’t getting questioned clearly.
    Causes and Philosophy
    To start, I’d like to explore the compatibility or lack of compatibility of Effective Altruism with other philosophical positions. There are many different philosophical areas and positions, and most of them aren’t actually limited to philosophers. Without going into the different areas of philosophy in detail, I’ll say that I think all of axiology, which includes both aesthetics and ethics, and large parts of metaphysics, are all actually pretty central to the questions Effective Altruism addresses. These debates are central to any discussion of how to pursue cause-neutrality, but are often, in fact, nearly always, ignored by the community.
    Aesthetics and EA, or Aesthetics versus EA?
    For example, aesthetics, the study of beauty and joy, could be central to the question of maximizing welfare. According to some views, joy and beauty tell us what welfare is. Many point out that someone can derive great pleasure from something physically painful or unpleasant directly - working out, or sacrificing themselves for a cause, whether it be their children, or their religious beliefs. Similarly, many actually value art and music highly personally. Given a choice between, say, losing their hearing and never again getting to listen to music, or living another decade, many would choose music. Preference utilitarianism (or the equivalent preference beneficentrism,) would say that people benefit from getting what they want.
    Similarly, many people place great value on aesthetics, and think that music and the arts are an important part of benefiting others. On the other hand, a thought experiment that is sometimes used to argue about consequentialism is to imagine a museum on fire, and weigh saving, say, the Mona Lisa against saving a patron who was there to look at it. Typically, the point being made is that the Mona Lisa has a monetary value that far exceeds the cost of saving a life, and so a certain type of person, say, an economist, might say to save the painting. (An EA might then sell it, and use the proceeds to save many lives.) But a different viewpoint is that there is a reason the Mona Lisa is valued so highly - aesthetics matters to people so much that, when considering public budgeting between fighting homelessness and funding museums, they think the correct moral tradeoff is to spend some money ...
    22 min
  • LW - Are short timelines actually bad? by joshc
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are short timelines actually bad?, published by joshc on February 5, 2023 on LessWrong.
    Sam Altman recently posted the following:
    I have seen very little serious discussion about whether short timelines are actually bad. This is surprising given that nearly everyone I talk to in the AI risk community seems to think that they are.
    Of course, the question "was the founding of OpenAI net positive?" and "would it be good to accelerate capabilities in 2023?" are different questions. I'm leaning towards yes on the first and no on the second. I’ve listed arguments that factor into these questions below.
    Reasons one might try to accelerate progress
    Avoid/delay a race with China. If the language model boom happened 10 years from now, China might be a bigger player. Global coordination seems harder than domestic coordination. A lot harder. Perhaps the U.S. will have to shake hands with China eventually, but the more time we have to experiment with powerful systems before then the better. That corresponds to time demonstrating dangers and iterating on solutions, which is way more valuable than "think about things in front of a white board" time.
    Smooth out takeoff. FLOPS get cheaper over time. Data is accumulating. Architectures continue to improve. The longer it takes for companies to invest ungodly amounts of money, the greater the potential overhang. Shortening timelines in 2015 may have slowed takeoff, which again corresponds to more time of the type that matters most.
    Keep the good guys in the lead. We're lucky that the dominant AGI companies respect safety as much as they do. Sam Altman recently commented that "the bad case — and I think this is important to say — is, like, lights out for all of us." I'm impressed that he said this given how bad this sort of thing could be for business -- and this doesn't seem like a PR move. AI x-risk isn't really in the overton window yet.
    The leading companies set an example. Maybe OpenAI’s hesitance to release GPT-4 has set a public expectation. It might now be easier to shame companies who don’t follow suit for being too fast and reckless.
    Reasons to hit the breaks
    There are lots of research agendas that don’t particularly depend on having powerful systems in front of us. Transparency research is the cat’s meow these days and has thus far been studied for relatively tiny models; in the extreme case, two-layer attention only transformers that I can run on my laptop. It takes time to lay foundations. In general, research progress is quite serial. Results build on previous results. It might take years before any of the junior conceptual researchers could get into an argument with Eleizer without mostly eliciting cached responses. The type of work that can be done in advance is also typically the type of work that requires the most serial time.
    Language models are already sophisticated for empirical work to be useful. GPT-3-level models have made empirically grounded investigations of deception possible. Soon it might also make sense to say that a large language model has a goal or is situationally aware. If slowing down takeoff is important, we should hit the breaks. Takeoff has already begun.
    We need time for field-building. The technical AI safety research community is growing very rapidly right now (28% per year according to this post). There are two cruxes here:
    How useful is this field building in expectation?
    Is this growth mostly driven by AGI nearness?
    I’m pretty confused about what’s useful right now so I’ll skip the first question.
    As for the second, I’ll guess that most of the growth is coming from increasing the number of AI safety research positions and university field building. The growth in AI safety jobs is probably contingent on the existence of maybe-useful empirical research. The release of GPT-3 may have been an imp...
    5 min
  • LW - Empathy as a natural consequence of learnt reward models by beren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Empathy as a natural consequence of learnt reward models, published by beren on February 4, 2023 on LessWrong.
    Epistemic Status: Pretty speculative but built on scientific literature. This post builds off my previous post on learnt reward models. Crossposted from my personal blog.
    Empathy, the ability to feel another's pain or to 'put yourself in their shoes' is often considered to be a fundamental human cognitive ability, and one that undergirds our social abilities and moral intuitions. As so much of human's success and dominance as a species comes down to our superior social organization, empathy has played a vital role in our history. Whether we can build artificial empathy into AI systems also has clear relevance to AI alignment. If we can create empathic AIs, then it may become easier to make an AI be receptive to human values, even if humans can no longer completely control it. Such an AI seems unlikely to just callously wipe out all humans to make a few more paperclips. Empathy is not a silver bullet however. Although (most) humans have empathy, human history is still in large part a history of us waging war against each other, and there are plenty of examples of humans and other animals perpetuating terrible cruelty on enemies and outgroups.
    A reasonable literature has grown up in psychology, cognitive science, and neuroscience studying the neural bases of empathy and its associated cognitive processes. We now know a fair amount about the brain regions involved in empathy, what kind of tasks can reliably elicit it, how individual differences in empathy work, as well as the neuroscience underlying disorders such as psychopathy, autism, and alexithmia which result in impaired empathic processing. However, much of this research does not grapple with the fundamental question of why we possess empathy at all. Typically, it seems to be tacitly assumed that, due to its apparent complexity, empathy must be some special cognitive module which has evolved separately and deliberately due to its fitness benefits. From an evolutionary theory perspective, empathy is often assumed to have evolved because of its adaptive function in promoting reciprocal altruism.
    The story goes that animals that are altruistic, at least in certain cases, tend to get their altruism reciprocated and may thus tend to out-reproduce other animals that are purely selfish. This would be of especial importance in social species where being able to form coalitions of likeminded and reciprocating indivudals is key to obtaining power and hence reproductive opportunities. If they could, such coalitions would obviously not include purely selfish animals who never reciprocated any benefits they received from other group members. Nobody wants to be in a coalition with an obviously selfish freerider.
    Here, I want to argue a different case. Namely that the basic cognitive phenomenon of empathy -- that of feeling and responding to the emotions of others as if they were your own, is not a special cognitive ability which had to be evolved for its social benefit, but instead is a natural consequence of our (mammalian) cognitive architecture and therefore arises by default. Of course, given this base empathic capability, evolution can expand, develop, and contextualize our natural empathic responses to improve fitness. In many cases, however, evolution actually reduces our native empathic capacity -- for instance, we can contextualize our natural empathy to exclude outgroup members and rivals.
    The idea is that empathy fundamentally arises from using learnt reward models to mediate between a low-dimensional set of primary rewards and reinforcers and the high dimensional latent state of an unsupervised world model. In the brain, much of the cortex is thought to be randomly initialized and implements a general purp...
    23 min