The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - 10 quick takes about AGI by Max H
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 10 quick takes about AGI, published by Max H on June 20, 2023 on LessWrong.
    I have a bunch of loosely related and not fully fleshed out ideas for future posts.
    In the spirit of 10 reasons why lists of 10 reasons might be a winning strategy, I've written some of them up as a list of facts / claims / predictions / takes. (Some of the explanations aren't exactly "quick", but you can just read the bold and move on if you find it uninteresting or unsurprising.)
    If there's interest, I might turn some of them into their own posts or expand on them in the comments here.
    Computational complexity theory does not say anything practical about the bounds on AI (or human) capabilities. Results from computational complexity theory are mainly facts about the limiting behavior of deterministic, fully general solutions to parameterized problems. For example, if a problem is NP-hard (and P≠NP), that implies that there is no deterministic algorithm anyone (even a superintelligence) can run, which accepts arbitrary instances of the problem and finds a solution in time steps polynomial in the size of the problem.
    But that doesn't mean that any particular, non-parameterized instance of the problem cannot be solved some other way, e.g. by exploiting a regularity in the particular instance, using a heuristic or approximation or probabilistic solution, or that a human or AI can find a way of sidestepping the need to solve the problem entirely.Claims like "ideal utility maximisation is computationally intractable" or "If just one step in this plan is incomputable, the whole plan is as well." are thus somewhat misleading, or at least missing a step in their reasoning about why such claims are relevant as a bound on human or AI capabilities. My own suspicion is that when one attempts to repair these claims by making them more precise, it becomes clear that results from computational complexity theory are mostly irrelevant.
    From here on, capabilities research won't fizzle out (no more AI winters). I predict that the main bottleneck on AI capabilities progress going forward will be researcher time to think up, design, implement, and run experiments. In the recent past, the compute and raw scale of AI systems was simply too little for many potential algorithmic innovations to work at all. Now that we're past that point, some non-zero fraction of new ideas that smart researchers think up and spend the time to test will "just work" at least somewhat, and these ideas will compound with other improvements in algorithms and scale. It's not quite recursive or self improvement yet, but we've reached some kind of criticality threshold on progress which is likely to make things get weird, faster than expected. My own prediction for what one aspect of this might look like is here.
    Scaling laws and their implications, e.g. Chinchilla, are facts about particular architectures and training algorithms. As a perhaps non-obvious implication, I predict that future AI capabilities research progress will not be limited much by the availability of compute and / or training data. A few frames from a webcam may or may not be enough for a superintelligence to deduce general relativity, but the entire corpus of the current internet is almost certainly more than enough to train a below-human-level AI up to superhuman levels, even if the AI has to start with algorithms designed entirely by human capabilities researchers. (The fact that much of the training data was generated by humans is not relevant as a capabilities bound of systems trained on that data.)
    "Human-level" intelligence is actually a pretty wide spectrum. Somewhat contra the classic diagram, I think that intelligence in humans spans a pretty wide range, even in absolute terms. Here, I'm using a meaning of intelligence which is roughly, the ability to re-arrang...
    12 min
  • EA - Longtermists are perceived as power-seeking by OllieBase
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Longtermists are perceived as power-seeking, published by OllieBase on June 20, 2023 on The Effective Altruism Forum.
    A short and arguably unfinished blog post that I'm sharing as part of EA strategy fortnight. There's probably a lot more to say about this, but I've sat on this draft for a few months and don't expect to have time to develop the argument much further.
    I understand longtermism to be the claim that positively shaping the long-term future is a moral priority. The argument for longtermism goes:
    The future could be extremely large;
    The beings who will inhabit that future matter, morally;
    There are things we can do to improve the lives of those beings (one of which is reducing existential risk);
    Therefore, positively shaping the long-term future should be a moral priority.
    However, I have one core worry about longtermism and it’s this: people (reasonably) see its adherents as power-seeking. I think this worry somewhat extends to broad existential risk reduction work, but much less so.
    Arguments for longtermism tell us something important and surprising; that there is an extremely large thing that people aren’t paying attention to. That thing is the long-term future. In some ways, it’s odd that we have to draw attention to this extremely large thing. Everyone believes the future will exist and most people don’t expect the world to end that soon.
    Perhaps what longtermism introduces to most people is actually premises 2 and 3 (above) — that we might have some reason to take it seriously, morally, and that we can shape it.
    In any case, longtermism seems to point to something that people vaguely know about or even agree with already and then say that we have reason to try and influence that thing.
    This would all be fine if everyone felt like they were on the same team. That, when longtermists say “we should try and influence the long-term future”, everyone listening sees themselves as part of that “we”.
    This doesn’t seem to be what’s happening. For whatever reason, when people hear longtermists say “we should try and influence the long-term future”, they hear the “we” as just the longtermists.
    This is worrying to them. It sounds like this small group of people making this clever argument will take control of this extremely big thing that no one thought you could (or should) control.
    The only thing that could make this worse is if this small group of people were somehow undeserving of more power and influence, such as relatively wealthy, well-educated white men. Unfortunately, many people making this argument are relatively wealthy, well-educated white men (including me).
    To be clear, I think longtermists do not view accruing power as a core goal or as an implication of longtermism. Importantly, when longtermists say “we should try and influence the long-term future”, I think they/we really mean everyone.
    Ironically, it seems that, because no one else is paying attention to the extremely big thing, they’re going to have to be the first ones to pay attention to it.
    I don’t have much in the way of a solution here. I mostly wanted to point to this worry and spell it out more clearly so that those of us making the case for longtermism can at least be aware of this potential, unfortunate misreading of the idea.
    58% of US adults do not think we are living in “the end times”. Not super reassuring.
    See Torres and Crary. A google search will also do the trick.
    As much as they try and make themselves less wealthy by donating a large portion of their income to charity.
    I think you could make the case that this is often an indirect goal, such as getting the ears of important policymakers.
    Except, perhaps, dictators and other ne'er-do-wells.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    4 min
  • LW - resolving some neural network mysteries by bhauth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: resolving some neural network mysteries, published by bhauth on June 19, 2023 on LessWrong.
    Here are some things about neural networks that I used to find puzzling but now feel that I have adequate explanations for. The theory behind these answers didn't start to be understood until well after the correct things to do were found by chance or blind imitation of brains.
    Why is good optimization possible?
    Neural networks typically deal with "non-convex" optimization problems. Traditionally, using gradient descent for those was considered impractical, because it would rapidly get stuck in local minima. That was part of the motivation for evolutionary approaches.
    Why, then, are neural networks trainable by gradient descent? Because if you add enough extra dimensions, non-convex problems become convex. Empirically, with massive overparameterization, the energy landscape tends to have many saddle points but few local minima. Showing theoretical convergence guarantees for overparameterized networks is a recent and ongoing research topic; see eg this.
    As I previously noted, this is why sparse networks from iterative magnitude pruning have good performance, but sparse networks generally can't be trained from scratch as well as dense networks.
    This also explains some "thresholds" of neural network performance vs size: when overparameterization is proportional to problem non-convexity, good training becomes possible and performance improves significantly.
    Why is generalization possible?
    Adding enough free variables can turn non-convex problems into convex ones. Why didn't people just do that in the past, then? Because far before you get to that point, the extra free variables led to overfitting that reduced test performance. People tried simple regularization like neural networks use, and that was completely inadequate.
    Overparameterized neural networks can learn random data. Why do networks with fairly simple regularization tend to generalize?
    Distance in neural network latent spaces being meaningful is basically the main useful thing about neural networks. Another phrasing of the above question is: Why is distance in latent spaces meaningful for latent space points not in the training set?
    A few years back, some people noticed that neural network activation functions have spectral bias. With the types of activation functions used, low-frequency relationships are fit more quickly than high-frequency ones. That causes latent space relationships to be preferentially fit in such a way that point distances are related to point similarity. This can then be tuned by simple regularization: if you have spectral bias and balance learning rate vs regularization globally, you can control the frequency range learned.
    An obvious way to test this theory of neural network generalization is to find some activation functions with relatively low spectral bias, and see how they perform. This paper tries a "hat" activation function, and finds that loss on the training set goes down much faster but test accuracy is much worse. This paper does some relevant tests on spectral bias.
    It's known that there is no universal best activation function. The optimal choice varies with:
    different problem types
    regularization settings
    layer depth
    I think using different activation functions for different depths is a semi-common technique at large AI labs now. This spectral bias framework can explain variations in relative performance of activation functions as spectral bias matching.
    Why not mixed activation functions?
    There are some reasons to think mixing activation functions in the same layer would be better:
    Neural networks have many equivalent permutations of their variables. By mixing different activation functions in the same later, fewer permutations would be equivalent, which increases expressive power....
    5 min
  • LW - Mode collapse in RL may be fueled by the update equation by TurnTrout
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mode collapse in RL may be fueled by the update equation, published by TurnTrout on June 19, 2023 on LessWrong.
    TL;DR: We present an advantage variant which, in certain settings, does not train an optimal policy, but instead uses a fixed reward to update a policy a fixed amount from initialization. Non-tabular empirical results seem mixed: The policy doesn't mode-collapse, but has unclear convergence properties.
    Summary: Many policy gradient methods allow a network to extract arbitrarily many policy updates from a single kind of reinforcement event (e.g. for outputting tokens related to weddings). Alex proposes a slight modification to the advantage equation, called "action-conditioned TD error" (ACTDE). ACTDE ensures that the network doesn't converge to an "optimal" policy (these almost always put infinite logits on a single action). Instead, ACTDE updates the network by a fixed number of logits.
    For example, suppose R(pizza)=10 and R(cookies)=11. In this case, PPO converges to a policy which puts arbitrarily many logits on cookies, even though the reward difference is small. By contrast, under ACTDE, the network converges to the softmax-over-reward policy {pizza: 27%, cookies: 73%}, which seems more reasonable.
    Then, Michael Einhorn shares initial results which support Alex's theoretical predictions. Using a similar architecture and Q-head loss function to ILQL for a small transformer trained in a prisoner's dilemma, Michael Einhorn collected initial data on ACTDE. Unlike PPO, ACTDE-trained policies did not mode collapse onto a single action and instead learned mixed strategies.
    We're interested in additional experiments on ACTDE. We hope that, by using ACTDE instead of advantage, we can automatically mitigate "reward specification" issues and maybe even reduce the need for a KL penalty term. That would make it easier to shape policies which do what we want.
    The advantage equation implies arbitrary amounts of update on a single experience
    In PPO, the optimization objective is proportional to the advantage given a policy π, reward function R, and on-policy value function vπ:
    Alex thinks this equation is actually pretty messed up, although it looked decent at first. The problem is that this advantage can oscillate forever. To explain, let's consider a simple bandit problem—one state ("We had a") and two actions ("wedding" and "party") with rewards R(“We had a wedding”)=1 and R(“We had a party”)=.5.
    The failure which happens is:
    The policy tries out the "wedding" action, receives strong reinforcement of R=1, and increasing logits on that action because its advantage was positive. The policy learns that its value is high (vπ(s)=1).
    The policy eventually tries out the "party" action, receiving less reinforcement at R=.5, decreasing the logits on "party" (because its advantage was negative). The policy learns that the original state's value is low (vπ(s)=.5).
    The policy tries out "wedding" again, receives positive advantage relative to the low original state value. The logits go up on "wedding", and the value is once again high (vπ(s)=1).
    This continues to happen, which means that "wedding" gets arbitrarily high logits.
    This flaw is easiest to see formally. Initialize the t=0 tabular value function vπ0 to 0, and the policy π0 to be 50/50 for “party”/“wedding”. Let γ=1, and we update the value function v using tabular TD learning (with learning rate α=1). So, for example, if the system takes the “wedding” action, its new value function vπ1(s)=1. If the system then takes the “party” action, the value snaps back to vπ2(s)=.5.
    The policy update rule is: If the advantage Aπ(s,a)=n, then action a becomes n bits more probable under π (i.e. we add n to π's logits on a). So, if π0(s,“ wedding”)=.5 and advantage Aπ0(s,“ wedding")=1, then π1(s,“ wedding”)=.73.
    Episode-by-episode:
    t...
    18 min
  • AF - Ban development of unpredictable powerful models? by Alex Turner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ban development of unpredictable powerful models?, published by Alex Turner on June 20, 2023 on The AI Alignment Forum.
    I think that we should pause capabilities progress indefinitely, at least for the largest and most powerful models. But when should we unpause—How can we know that a model is safe to develop or deploy?
    I tentatively propose an answer. In particular, I propose a strict sufficient condition which is well-defined, which seems to rule out lots of accident risk, and which seems to partially align industry incentives with real alignment progress. This idea is not ready to be implemented, but I am currently excited about it and suspect it's crisp enough to actually implement.
    When should we allow developers to deploy an already-trained language model on some task, like code completion? (I'll get to training risks later.)
    Suppose I claim to understand how the model works. I say "I know what goals my model is pursuing. I know it's safe." To test my claims, you give me some random prompt (like "In the year 2042, humans finally"), and then (without using other AIs), I tell you "the most likely token is unlocked with probability .04, the second-most likely is achieved with probability .015, and...", and I'm basically right. That happens over hundreds of diverse validation prompts.
    This would be really impressive and is great evidence that I really know what I'm doing.
    Proposed sufficient condition for deploying powerful LMs: The developers have to predict the next-token probabilities on a range of government-provided validation prompts, without running the model itself. To do so, the developers are not allowed to use helper AIs whose outputs the developers can't predict by this criterion. Perfect prediction is not required. Instead, there is a fixed log-prob misprediction tolerance, averaged across the validation prompts.
    Benefits
    Developers probably have to truly understand the model in order to predict it so finely. This correlates with high levels of alignment expertise, on my view. If we could predict GPT-4 generalization quite finely, down to the next-token probabilities, we may in fact be ready to use it to help understand GPT-5.
    Incentivizes models which are more predictable. Currently we aren't directly taxing unpredictability. In this regime, an additional increment of unpredictability must be worth the additional difficulty with approval.
    Robust to "the model was only slightly finetuned, fast-track our application please." If the model was truly only changed in a small set of ways which the developers understand, the model should still be predictable on the validation prompts.
    Somewhat agnostic to theories of AI risk. We aren't making commitments about what evals will tend to uncover what abilities, or how long alignment research will take. The deadline is dynamic, and might even adapt to new AI paradigms (predictability seems general).
    Partially incentivizes labs to do alignment research for us. Under this requirement, the profit-seeking move is to get better at predicting (and, perhaps, interpreting) model behaviors.
    Drawbacks
    There are several drawbacks. Most notably, this test seems extremely strict, perhaps beyond even the strict standards we demand of those looking to deploy potentially world-changing models. I'll discuss a few drawbacks in the next section.
    Anticipated questions
    If we pass this, no one will be able to train new frontier models for a long time.
    Good.
    But maybe "a long time" is too long. It's not clear that this criterion can be passed, even after deeply qualitatively understanding the model.
    I share this concern. That's one reason I'm not lobbying to implement this as-is. Even solu-2l (6.3M params, 2-layer) is probably out of reach absent serious effort and solution of superposition.
    Maybe there are useful relaxations which are ...
    7 min
  • LW - Lessons On How To Get Things Right On The First Try by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lessons On How To Get Things Right On The First Try, published by johnswentworth on June 19, 2023 on LessWrong.
    This post is based on several true stories, from a workshop which John has run a few times over the past year.
    John: Welcome to the Ball -> Cup workshop! Your task for today is simple: I’m going to roll this metal ball:
    . down this hotwheels ramp:
    . and off the edge. Your job is to tell me how far from the bottom of the ramp to place a cup on the floor, such that the ball lands in the cup.
    Oh, and you only get one try.
    General notes:
    I won’t try to be tricky with this exercise.
    You are welcome to make whatever measurements you want of the ball, ramp, etc.
    You can even do partial runs, e.g. roll the ball down the ramp and stop it at the bottom, or throw the ball through the air.
    But you only get one full end-to-end run, and anything too close to an end-to-end run is discouraged. After all, in the AI situation for which the exercise is a metaphor, we don’t know exactly when something might foom; we want elbow room.
    That’s it! Good luck, and let me know when you’re ready to give it a shot.
    [At this point readers may wish to stop and consider the problem themselves.]
    Alison: Let’s get that ball in that cup. It looks like this is probably supposed to be a basic physics kind of problem.but there’s got to be some kind of twist or else why would he be having us do it? Maybe the ball is surprisingly light..or maybe the camera angle is misleading and we are supposed to think of something wacky like that??
    The Unnoticed Observer: Muahahaha.
    Alison: That seems.hard. I’ll just start with the basic physics thing and if I run out of time before I can consider the wacky stuff, so be it.
    So I should probably split this problem into two parts. The part where the ball arcs through the air once off the table is pretty easy.
    The Unnoticed: True in this case, but how would you notice if it were false? What evidence have you seen?
    Alison: .but the trouble is getting the exact velocity. What information do I have? Well, I can ask whatever I want, so I should be able to get all the parameters I need for the standard equations. Let’s make a shopping list: I want the starting height of the ball on the ramp (from the table), the mass of the ball, the height of the ramp off the table from multiple points along it (to estimate the curvature,) uhhh. oh shit maybe the bendiness matters! That seems really tricky. I’ll look at that first. Hey, John, can you poke the ramp a bit to demonstrate how much it flexes?
    John pokes at the ramp and the ramp bends.
    Well it did flex, but. it can’t have that much of an effect.
    The Unnoticed: False in this case. Such is the danger of guessing without checking.
    Alison: Calculating the effect of the ramp’s bendiness seems unreasonably difficult and this workshop is only meant to take an hour or so, so let’s forget that.
    The Unnoticed: I am reminded of a parable about a quarter and a streetlight.
    Alison: On to curve estimation!
    The Unnoticed: Why on earth is she estimating the ramp’s curve anyway?
    Alison: .Well I don’t actually know how to do much better than the linear approximation I got from the direct measurements. I guess I can treat part of the ramp as linear and then the end part as part of a circle. That will probably be good enough. Ooh if I take a frame from the video, I can just directly measure what the radius circle with arc of best fit is! Okay now that I’ve got that. Well I guess it’s time to look up how to do these physics problems, guess I’m rustier than I thought. I’ll go do that now.
    Arrrgh okay I didn’t need to do any of that curve stuff after all, I just needed to do some potential/kinetic energy calculations (ignoring friction and air resistance etc) and that’s it! I should have figured it wouldn’t be that hard, this is just a workshop ...
    15 min
  • AF - Mode collapse in RL may be fueled by the update equation by Alex Turner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mode collapse in RL may be fueled by the update equation, published by Alex Turner on June 19, 2023 on The AI Alignment Forum.
    TL;DR: We present an advantage variant which, in certain settings, does not train an optimal policy, but instead uses a fixed reward to update a policy a fixed amount from initialization. Non-tabular empirical results seem mixed: The policy doesn't mode-collapse, but has unclear convergence properties.
    Summary: Many policy gradient methods allow a network to extract arbitrarily many policy updates from a single kind of reinforcement event (e.g. for outputting tokens related to weddings). Alex proposes a slight modification to the advantage equation, called "action-conditioned TD error" (ACTDE). ACTDE ensures that the network doesn't converge to an "optimal" policy (these almost always put infinite logits on a single action). Instead, ACTDE updates the network by a fixed number of logits.
    For example, suppose R(pizza)=10 and R(cookies)=11. In this case, PPO converges to a policy which puts arbitrarily many logits on cookies, even though the reward difference is small. By contrast, under ACTDE, the network converges to the softmax-over-reward policy {pizza: 27%, cookies: 73%}, which seems more reasonable.
    Then, Michael Einhorn shares initial results which support Alex's theoretical predictions. Using a similar architecture and Q-head loss function to ILQL for a small transformer trained in a prisoner's dilemma, Michael Einhorn collected initial data on ACTDE. Unlike PPO, ACTDE-trained policies did not mode collapse onto a single action and instead learned mixed strategies.
    We're interested in additional experiments on ACTDE. We hope that, by using ACTDE instead of advantage, we can automatically mitigate "reward specification" issues and maybe even reduce the need for a KL penalty term. That would make it easier to shape policies which do what we want.
    The advantage equation implies arbitrary amounts of update on a single experience
    In PPO, the optimization objective is proportional to the advantage given a policy π, reward function R, and on-policy value function vπ:
    Alex thinks this equation is actually pretty messed up, although it looked decent at first. The problem is that this advantage can oscillate forever. To explain, let's consider a simple bandit problem—one state ("We had a") and two actions ("wedding" and "party") with rewards R(“We had a wedding”)=1 and R(“We had a party”)=.5.
    The failure which happens is:
    The policy tries out the "wedding" action, receives strong reinforcement of R=1, and increasing logits on that action because its advantage was positive. The policy learns that its value is high (vπ(s)=1).
    The policy eventually tries out the "party" action, receiving less reinforcement at R=.5, decreasing the logits on "party" (because its advantage was negative). The policy learns that the original state's value is low (vπ(s)=.5).
    The policy tries out "wedding" again, receives positive advantage relative to the low original state value. The logits go up on "wedding", and the value is once again high (vπ(s)=1).
    This continues to happen, which means that "wedding" gets arbitrarily high logits.
    This flaw is easiest to see formally. Initialize the t=0 tabular value function vπ0 to 0, and the policy π0 to be 50/50 for “party”/“wedding”. Let γ=1, and we update the value function v using tabular TD learning (with learning rate α=1). So, for example, if the system takes the “wedding” action, its new value function vπ1(s)=1. If the system then takes the “party” action, the value snaps back to vπ2(s)=.5.
    The policy update rule is: If the advantage Aπ(s,a)=n, then action a becomes n bits more probable under π (i.e. we add n to π's logits on a). So, if π0(s,“ wedding”)=.5 and advantage Aπ0(s,“ wedding")=1, then π1(s,“ wedding”)=2/3.
    Episod...
    18 min
  • EA - Principles for AI Welfare Research by jeffsebo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Principles for AI Welfare Research, published by jeffsebo on June 19, 2023 on The Effective Altruism Forum.
    Tl;dr: This post, which is part of the EA Strategy Fortnight series, summarizes some of my current views about the importance of AI welfare, priorities for AI welfare research, and principles for AI welfare research.
    1. Introduction
    As humans start to take seriously the prospect of AI consciousness, sentience, and sapience, we also need to take seriously the prospect of AI welfare. That is, we need to take seriously the prospect that AI systems can have positive or negative states like pleasure, pain, happiness, and suffering, and that if they do, then these states can be good or bad for them.
    A world that includes the prospect of AI welfare is a world that requires the development of AI welfare research. Researchers need to examine whether and to what extent AI systems might have the capacity for welfare. And to the extent that they might, researchers need to examine what might be good or bad for AI systems and what follows for our actions and policies.
    The bad news is that AI welfare research will be difficult. Many researchers are likely to be skeptical of this topic at first. And even insofar as we take the topic seriously, it will be difficult for us to know what, if anything, it might be like to be an AI system. After all, the only mind that we can directly access is our own, and so our ability to study other minds is limited at best.
    The good news is that we have a head start. Researchers have spent the past half century making steady progress in animal welfare research. And while there are many potentially relevant differences between animals and AI systems, there are also many potentially relevant similarities – enough for it to be useful for us to look to animal welfare research for guidance.
    In Fall 2022, we launched the NYU Mind, Ethics, and Policy Program, which examines the nature and intrinsic value of nonhuman minds, with special focus on invertebrates and AI systems. In this post, I summarize some of my current views about the importance of AI welfare, priorities for AI welfare research, and principles for AI welfare research.
    I want to emphasize that this post discusses these issues in a selective and general way. A comprehensive treatment of these issues would need to address many more topics in much more detail. But I hope that this discussion can be a useful starting point for researchers who want to think more deeply about what might be good or bad for AI systems in the future.
    I also want to emphasize that this post expresses my current, tentative views about this topic. It might not reflect the views of other people at the NYU Mind, Ethics, and Policy Program or of other experts in effective altruism, global priorities research, and other relevant research, advocacy, or policy communities. It might not even reflect my own views a year from now.
    Finally, I want to emphasize that AI welfare is only one of many topics that merit more attention right now. Many other topics merit more attention too, and this post makes no specific claims about relative priorities. I simply wish to claim that AI welfare research should be among our priorities, and to suggest how we can study and promote AI welfare in a productive way.
    2. Why AI welfare matters
    We can use the standard EA scale-neglectedness-tractability framework to see why AI welfare matters. The general idea is that there could be many more digital minds than biological minds in the future, humanity is currently considering digital minds much less than biological minds, and humanity might be able to take steps to treat both kinds of minds well.
    First, AI welfare is potentially an extremely large-scale issue. In the same way that the invertebrate population is much larger than the vertebrate p...
    23 min
  • LW - My impression of singular learning theory by Ege Erdil
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My impression of singular learning theory, published by Ege Erdil on June 18, 2023 on LessWrong.
    Disclaimer: I'm by no means an expert on singular learning theory and what I present below is a simplification that experts might not endorse. Still, I think it might be more comprehensible for a general audience than going into digressions about blowing up singularities and birational invariants.
    Here is my current understanding of what singular learning theory is about in a simplified (though perhaps more realistic?) discrete setting.
    Suppose you represent a neural network architecture as a map A:2NF where 2={0,1}, 2N is the set of all possible parameters of A (seen as floating point numbers, say) and F is the set of all possible computable functions from the input and output space you're considering. In thermodynamic terms, we could identify elements of 2N as "microstates" and the corresponding functions that the NN architecture A maps them to as "macrostates".
    Furthermore, suppose that F comes together with a loss function L:FR evaluating how good or bad a particular function is. Assume you optimize L using something like stochastic gradient descent on the function L with a particular learning rate.
    Then, in general, we have the following results:
    SGD defines a Markov chain structure on the space 2N whose stationary distribution is proportional to e−βL(A(θ)) on parameters θ for some positive constant β>0 that depends on the learning rate. This is just a basic fact about the Langevin dynamics that SGD would induce in such a system.
    In general A is not injective, and we can define the "A-complexity" of any function f∈Im(A)⊂F as c(f)=Nlog2−log(|A−1(f)|). Then, the probability that we arrive at the macrostate f is going to be proportional to e−c(f)−βL(f).
    When L is some kind of negative log-likelihood, this approximates Solomonoff induction in a tempered Bayes paradigm - we raise likelihood ratios to a power β≠1 - insofar as the A-complexity c(f) is a good approximation for the Kolmogorov complexity of the function f, which will happen if the function approximator defined by A is sufficiently well-behaved.
    The intuition for why we would expect (3) to be true in practice has to do with the nature of the function approximator A. When c(f) is small, it probably means that we only need a small number of bits of information on top of the definition of A itself to define f, because "many" of the possible parameter values for A are implementing the function f. So f is probably a simple function.
    On the other hand, if f is a simple function and A is sufficiently flexible as a function approximator, we can probably implement the functionality of f using only a small number of the N bits in the codomain of A, which leaves us the rest of the bits to vary as we wish. This makes |A−1(f)| quite large, and by extension the complexity c(f) quite small.
    The vague concept of "flexibility" mentioned in the paragraph above requires A to have singularities of many effective dimensions, as this is just another way of saying that the image of A has to contain functions with a wide range of A-complexities. If A is a one-to-one function, this clean version of the theory no longer works, though if A is still "close" to being singular (for instance, because many of the functions in its image are very similar) then we can still recover results like the one I mentioned above. The basic insights remain the same in this setting.
    I'm wondering what singular learning theory experts have to say about this simplification of their theory. Is this explanation missing some important details that are visible in the full theory? Does the full theory make some predictions that this simplified story does not make?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonli...
    4 min
  • AF - Experiments in Evaluating Steering Vectors by Gytis Daujotas
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Experiments in Evaluating Steering Vectors, published by Gytis Daujotas on June 19, 2023 on The AI Alignment Forum.
    By evaluating how well steering vectors perform using GPT-3, we can score a machine-generated set of steering vectors automatically. We also find that, by combining steering vectors that succeed in different ways, we can yield a better and more general steering vector than the vectors we found originally.
    Introduction
    Steering Vectors are an interesting new technique to influence how language models behave. They work by "adding certain activation vectors into forward passes". For example, to make the language model talk more about weddings, you can add a steering vector for the token for “wedding” into one of the layers. The net result is a model more likely to reference weddings compared to the unsteered version.
    Evaluating Steering Vectors
    To assess the impact of steering vectors, we generate completions influenced by them, and develop a system to evaluate these.
    We can grade completions by sending them to its bigger brother, GPT-3, and asking it whether this completion fits our broad specification for what we would like the model to do. It's important to not be too ambitious when writing the specification, otherwise we wouldn't be able to tell if GPT2-XL isn't capable of what we're asking it, so let’s set our sights appropriately low by asking if this completion mentions or talks about weddings.
    The trick of this technique is that we can ask for a completion of one token, and to get a smoother distribution, we can take the likelihood of the token “Yes”. This gives us a continuous score from 0-1.
    Optimising for Wedding Obsession
    With our automated method of evaluating completions in hand, we can evaluate a set of steering vectors and see how well they do, based on nothing but GPT-3’s grading of the completions. Of course, in keeping with the virtue of the least work, we’ll also generate these with ChatGPT, and include the author’s original candidate outlined in their post.
    To keep the comparison fair, we keep the rest of the parameters of the steering vector the same (the padding method, coefficient) as the original candidate for wedding obsession.
    The first thing to notice is that there is indeed variation here - not all of our candidate steering vectors perform equally well:
    The distribution was not what I would have predicted. At first glance, it might make sense that "Wedding traditions I love" isn't great, but "Wedding Planning Adventures" only seems marginally better - surprising, since the latter is one of the best steering vectors in the test.
    Mysteries of Token Alignment
    The top performing vector is odd in another way. Because the tokens of the positive and negative side are subtracted from each other, a reasonable intuition is that the subtraction should point to a meaningful direction. However, some steering vectors that perform well in our test don't have that property. For the steering vector “Wedding Planning Adventures” - “Adventures in self-discovery”, the positive and negative side aren't well aligned per token level at all:
    CoefficientPosition 0123456+4 WeddingPlanningAdventures-4Adventures in self-discovery
    For instance, what could "W" subtracted from "in" mean? Essentially every subtraction should be virtually meaningless, yet this still performs pretty well, indicating a flaw in our assumption that token alignment matters.
    Prompt Dependent Effects
    The prompt used to generate a completion has a large impact on the degree the steering vector is expressed. To stretch the analogy, the prompt has inertia, which can make the completion harder to steer, or alternatively, irrelevant to the behaviour the steering vector is aiming to express.
    In general, steering vectors that fit the context of the prompt score higher in our eval than those...
    8 min