The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Thoughts on Loss Landscapes and why Deep Learning works by beren
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on Loss Landscapes and why Deep Learning works, published by beren on July 25, 2023 on LessWrong.
    Crossposted from my personal blog.
    Epistemic status: Pretty uncertain. I don't have an expert level understanding of current views in the science of deep learning about why optimization works but just read papers as an amateur. Some of the arguments I present here might be already either known or disproven. If so please let me know!
    There are essentially two fundamental questions in the science of deep learning: 1.) Why are models trainable? And 2.) Why do models generalize? The answer to both of these questions relates to the nature and basic geometry of the loss landscape which is ultimately determined by the computational architecture of the model. Here I present my personal and fairly idiosyncratic and speculative answers to these questions and present what I think are fairly novel answers for both of these questions. Let's get started.
    First let's think about the first question: Why are deep learning models trainable at all? A-priori, they shouldn't be. Deep neural networks are fundamentally solving incredibly high dimensional problems with extremely non-convex loss-landscape. The entirety of optimization theory should tell you that, in general, this isn't possible. You should get stuck in exponentially many local minima before getting close to any decent optimum, and also you should probably just explode because you are using SGD, the stupidest possible optimizer.
    So why does it, in fact, work? The fundamental reason is that we are very deliberately not in the general case. Much of the implicit work of deep learning is fiddling with architectures, hyperparameters, initializations, and so forth to ensure that, in the specific case, the loss landscape is benign. To even get SGD working in the first place, there are two fundamental problems which neural networks solve:
    First, vanilla SGD assumes unit variance. This can be shown straightforwardly from a simple Bayesian interpretation of SGD and is also why second order optimizers like natural gradients are useful. In effect, SGD fails whenever the variance and curvature of the loss landscape is either too high (SGD will explode) or too small (SGD will grind to a halt and take forever to traverse the landscape). This problem has many names such as 'vanishing/exploding gradients', 'poor conditioning', 'poor initialization', and so on. Many of the tricks to get neural networks to work address this initializing SGD in, and forcing it to stay in the regime of unit variance. Examples of this include:
    a.) Careful scaling of initializations by width to ensure that the post-activation distribution is as close to a unit variance Gaussian as possible.
    b.) Careful scaling of initialization by depth to ensure that the network stays close to an identity function at initialization and variance is preserved (around 1). This is necessary to prevent either exploding activations with depth or vanishing activations (rank collapse).
    c.) Explicit normalization at every layer (layer norm, batch norm) to force activations to a 0-mean unit variance gaussian, which then shapes the gradient distribution.
    d.) Gradient clipping to ensure that large updates do not happen which temporarily cuts off the effects of bad conditioning.
    e.) Weight decay which keeps weights close to a 0-mean unit variance Gaussian and hence SGD in the expected range.
    f.) Careful architecture design. We design neural networks to be as linear as possible while still having enough nonlinearities to represent useful functions. Innovations like residual connections allow for greater depth without exploding or vanishing as easily.
    g.) The Adam optimizer performs a highly approximate (but cheap) rescaling of the parameters with their empirical variance, thus approximatin...
    16 min
  • EA - [Linkpost] Can we confidently dismiss the existence of near aliens? Probabilities and implications by Magnus Vinding
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] Can we confidently dismiss the existence of near aliens? Probabilities and implications, published by Magnus Vinding on July 25, 2023 on The Effective Altruism Forum.
    An earlier post of mine reviewed the most credible evidence I have managed to find regarding seemingly anomalous UFOs. My aim in this post is to mostly set aside the purported UFO evidence and to instead explore whether we can justify placing an extremely low probability on the existence of near aliens, irrespective of the alleged UFO evidence. (By "near aliens", I mean advanced aliens on or around Earth.)
    Specifically, after getting some initial clarifications out of the way, I proceed to do the following:
    I explore three potential justifications for a high level of confidence (>99.99 percent) regarding the absence of near aliens: (I) an extremely low prior, (II) technological impossibility, and (III) expectations about what we should observe conditional on advanced aliens being here.
    I review various considerations that suggest that these potential justifications, while they each have some merit, are often overstated.
    For example, in terms of what we should expect to observe conditional on advanced aliens having reached Earth, I argue that it might not look so different from what we in fact observe.
    In particular, I argue that near aliens who are entirely silent or only occasionally visible are more plausible than commonly acknowledged. The motive of gathering information about the evolution of life on Earth makes strategic sense relative to a wide range of goals, and this info gain motive is not only compatible with a lack of clear visibility, but arguably predicts it.
    I try to give some specific probability estimates - priors and likelihoods on the existence of near aliens - that seem reasonable to me in light of the foregoing considerations.
    Based on these probability estimates, I present Bayesian updates of the probability of advanced aliens around Earth under different assumptions about our evidence.
    I argue that, regardless of what we make of the purported UFO evidence, the probability of near aliens seems high enough to be relevant to many of our decisions, especially those relating to large-scale impact and risks.
    Lastly, I consider the implications that a non-negligible probability of near aliens might have for our future decisions, including the possibility that our main influence on the future might be through our influence on near aliens.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • EA - 2022 Effective Animal Advocacy Forum Survey: Results and analysis by Rethink Priorities
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2022 Effective Animal Advocacy Forum Survey: Results and analysis, published by Rethink Priorities on July 25, 2023 on The Effective Altruism Forum.
    Introduction
    In September 2022, the Effective Animal Advocacy (EAA) Coordination Forum (now titled the Animal Advocacy Strategy Forum) was held with the purpose of bringing together key decision-makers in the animal advocacy community to connect, coordinate, and strategize. The attendees represented approximately 20 key groups in the effective animal advocacy space.
    At the end of the forum, 25 participants filled out a survey that sought to better understand the future needs of effective animal advocacy groups and the perceptions of animal advocates about the most important areas to focus on in the future. This report analyzes the results of that survey.
    Key Takeaways
    On average, respondents to the EAA Survey believe that:
    The largest share (29%) of effective animal advocacy resources should be spent in Asia and the Pacific, followed by Western Europe, the U.S., Canada, Australia, and New Zealand (26%).
    Farmed fish and farmed invertebrates received the highest allocations of resources among respondents (16.5% and 17.1%, respectively), shortly followed by egg-laying hens and broiler chickens (12.8% and 13.1%, respectively).
    The plurality of resources should be spent targeting businesses (34%), followed by government institutions (28%).
    There's about a 60% chance that an area that should receive over 20% of the EAA funding currently receives less than 5% of it.
    In addition, a plurality of respondents believes that:
    EAA needs more people who are experts on the developing world/populous-yet-neglected countries (17/25 votes), government and policy (16/25), and/or figuring out what matters most and setting priorities (13/25).
    Their EAA organization is sometimes (9/25 votes) or often (10/25) funding-constrained and it is sometimes hard (11/25) to find outstanding candidates for roles.
    When asked about issues facing the effective animal advocacy movement:
    The lack of a strong evidence base (11/25 votes) and ability to appeal to the people most able to contribute to EAA cause areas (10/25) were the most commonly cited problems for EAA.
    Epistemic uncertainty regarding interventions (10/25 votes) and a lack of influence over the public, donors, and others with power (10/25) were generally cited as the most pressing problems in EAA.
    Acknowledgments
    This report is a project of Rethink Priorities-a think tank dedicated to informing decisions made by high-impact organizations and funders across various cause areas. It was written by Laura Duffy. Thanks to William McAuliffe for helpful feedback and to Adam Papineau for copy-editing.
    If you are interested in RP's work, please visit our research database and subscribe to our newsletter.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    4 min
  • LW - Anthropic Observations by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anthropic Observations, published by Zvi on July 25, 2023 on LessWrong.
    Dylan Matthews had in depth Vox profile of Anthropic, which I recommend reading in full if you have not yet done so. This post covers that one.
    Anthropic Hypothesis
    The post starts by describing an experiment. Evan Hubinger is attempting to create a version of Claude that will optimize in part for a secondary goal (in this case 'use the word "paperclip" as many times as possible') in the hopes of showing that RLHF won't be able to get rid of the behavior. Co-founder Jared Kaplan warns that perhaps RLHF will still work here.
    Hubinger agrees, with a caveat. "It's a little tricky because you don't know if you just didn't try hard enough to get deception," he says. Maybe Kaplan is exactly right: Naïve deception gets destroyed in training, but sophisticated deception doesn't. And the only way to know whether an AI can deceive you is to build one that will do its very best to try.
    The problem with this approach is that an AI that 'does its best to try' is not doing the best that the future dangerous system will do.
    So by this same logic, a test on today's systems can only show your technique doesn't work or that it works for now, it can never give you confidence that your technique will continue to work in the future.
    They are running the test because they think that RLHF is so hopeless we can likely already prove, at current optimization levels, that it is doomed to failure.
    Also, the best try to deceive you will sometimes be, of course, to make you think that the problem has gone away while you are testing it.
    This is the paradox at the heart of Anthropic: If the thesis that you need advanced systems to do real alignment work is true, why should we think that cutting edge systems are themselves currently sufficiently advanced for this task?
    But Anthropic also believes strongly that leading on safety can't simply be a matter of theory and white papers - it requires building advanced models on the cutting edge of deep learning. That, in turn, requires lots of money and investment, and it also requires, they think, experiments where you ask a powerful model you've created to deceive you.
    "We think that safety research is very, very bottlenecked by being able to do experiments on frontier models," Kaplan says, using a common term for models on the cutting edge of machine learning. To break that bottleneck, you need access to those frontier models. Perhaps you need to build them yourself.
    The obvious question arising from Anthropic's mission: Is this type of effort making AI safer than it would be otherwise, nudging us toward a future where we can get the best of AI while avoiding the worst? Or is it only making it more powerful, speeding us toward catastrophe?
    If we could safely build and work on things as smart and capable as the very models that we will later need to align, then this approach would make perfect sense. Given we thankfully cannot yet build such models, the 'cutting edge' is importantly below the capabilities of the future dangerous systems. In particular they are on different sides of key thresholds where we would expect current techniques to stop working, such as the AI becoming smarter than those evaluating its outputs.
    For Anthropic's thesis to be true, you need to thread a needle. Useful work now must require cutting edge models, and those cutting edge models must be sufficiently advanced to do useful work.
    In order to pursue that thesis and also potentially to build an AGI or perhaps transform the world economy, Anthropic plans on raising $5 billion over the next two years. Their investor pitch deck claims that whoever gets out in front of the next wave can generate an unstoppable lead. Most of the funding will go to development of cutting edge frontier models. That certainly seems ...
    16 min
  • EA - Updates to the flow of funding in EA movement building post by Vaidehi Agarwalla
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Updates to the flow of funding in EA movement building post, published by Vaidehi Agarwalla on July 25, 2023 on The Effective Altruism Forum.
    This is an summary of updates made to my previous post, The flow of funding in EA movement building.
    Overall Changes
    Total funding tracked in the data increased to $290M (from $245M). New data is from:
    Several private donors and Longview Philanthropy who shared (previously non-public) donation & grant recommendation data
    Global health & wellbeing spending e.g. GiveWell, ACE and some animal orgs (at a discounted rate since these organizations aren't explicitly focused on EA movement building but did contribute to the growth of the EA movement)
    The inclusion of some longtermist research organizations such as FHI which have helped do field building (also at a discounted rate)
    Changes to proportions and funding over time
    During the 2012-2016 period, funding tracked in my data roughly doubled from ~$4M to ~$8.9M (quick estimate) including $4M in funding to GiveWell and $0.5M from other donors. During 2017-2023 period, funding tracked roughly increased from $241 to $281M, from other donors and the inclusion of some cause-area specific organizations that contributed to movement building.
    The table below summarizes the changes to the proportions of funding coming from different sources:
    Funder CategoryChange in % New %Original %Other donorsUp ~8% 9.6%1.5%FTX Future FundDown ~3%14.8%17.5%EAIF (non-OP donors), LTFF & Jaan Tallinn (incl. SFF)Down ~1% EA Animal FundUp ~1%1.1%0%
    Open Philanthropy
    OP LT: Down 9.5%(~10.1% w. EAIF)
    OP GH&W: Down 0.4%
    OP Other: Up 5.9%
    Overall: Down ~3%
    OP LT: 50.4% (~54.5% w. EAIF)
    OP GH&W: 2.6%
    OP Other: 5.9%
    Overall: 63%
    OP LT: 59.8% (~64.6% w. EAIF)
    OP GH&W: 2.2%
    OP Other: 0%
    Overall: 66%
    Here's the new % data in a pie chart:
    What data is still missing?
    Total funding: I estimate total funding from 2012 to June 2023 is likely $300-350M (medium confidence). I previously estimated $250-280M (significant underestimate).
    Individual donors: I estimate that $1-20M since 2012 is probably still missing, since I haven't included donors who work with Effective Giving, Generation Pledge or Founders' Pledge.
    Allocation of cause-specific efforts: You may disagree with the discounting I've done towards different cause-specific projects (in either direction). If you think I'm underweighting those efforts, then you could consider that "missing" data.
    The most accurate way to do these estimates would be to ask movement building organizations for their annual expenses and to break down the sources of their funding. This information is not publicly available, and some organizations do not publish annual expenses publicly from where you might make initial guesses. I'd encourage organizations to share their numbers to give us a fuller picture of the landscape.
    Mistakes & reflections
    I didn't expect this post to be read by as many people as it was. If I'd known this in advance, I think it's likely I would have delayed publication and seeked more external feedback because concrete numbers can be sticky and hard to update people's views on.
    I noted that this was a preliminary analysis in the opening, but the data may have been seen as more final than it was. In the future I would spend more time hedging numbers and stating ranges of possible values and encourage people to cite those instead of exact numbers.
    I didn't add enough uncertainty estimates to the numbers throughout the post. For example, I mentioned that the data was incomplete, and provided an estimate on the total amount of funding ($250-280M) - this was a moderately large underestimate (the total new total tracked data now stands at $290M).
    I missed several sources of global health & wellbeing spending, which significantly increased total spend between 2012-2016. This ...
    5 min
  • LW - How LLMs are and are not myopic by janus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How LLMs are and are not myopic, published by janus on July 25, 2023 on LessWrong.
    Thanks to janus, Nicholas Kees Dupuis, and Robert Kralisch for reviewing this post and providing helpful feedback. Some of the experiments mentioned were performed while at Conjecture.
    TLDR: The training goal for LLMs like GPT is not cognitively-myopic (because they think about the future) or value myopic (because the transformer architecture optimizes accuracy over the entire sequence, not just the next-token). However, training is consequence-blind, because the training data is causally independent of the models actions. This assumption breaks down when models are trained on AI generated text.
    Summary
    Myopia in machine learning models can be defined in several ways. It could be the time horizon the model considers when making predictions (cognitive myopia), the time horizon the model takes into account when assessing its value (value myopia), or the degree to which the model considers the consequences of its decisions (consequence-blindness).
    Both cognitively-myopic and consequence-blind models should not pursue objectives for instrumental reasons. This could avoid some important alignment failures, like power-seeking or deceptive alignment. However, these behaviors can still exist as terminal values, for example when a model is trained to predict power-seeking or deceptively aligned agents.
    LLM pretraining is not cognitively myopic because there is an incentive to think about the future to improve immediate prediction accuracy, like when predicting the next move in a chess game.
    LLM pretraining is not value/prediction myopic (does not maximize myopic prediction accuracy) because of the details of the transformer architecture. Training gradients flow through attention connections, so past computation is directly optimized to be useful when attended to by future computation. This incentivizes improving prediction accuracy over the entire sequence, not just the next token. This means that the model can and will implicitly sacrifice next-token prediction accuracy for long horizon prediction accuracy.
    You can modify the transformer architecture to remove the incentive for non-myopic accuracy, but as expected, the modified architecture has worse scaling laws.
    LLM pretraining on human data is consequence-blind as the training data is causally independent from the model's actions. This implies the model should predict actions without considering the effect of its actions on other agents, including itself. This makes the model miscalibrated, but likely makes alignment easier.
    When LLMs are trained on data which has been influenced or generated by LLMs, the assumptions of consequence-blindness partially break down. It's not clear how this affects the training goal theoretically or in practice.
    A myopic training goal does not ensure the model will learn myopic computation or behavior because inner alignment with the training goal is not guaranteed
    Introduction
    The concept of myopia has been frequently discussed as a potential solution to the problem of deceptive alignment. However, the term myopia is ambiguous and can refer to multiple different properties we might want in an AI system, only some of which might rule out deceptive alignment. There's also been confusion about the extent to which Large language model (LLM) pretraining and other supervised learning methods are myopic and what this implies about their cognition and safety properties. This post will attempt to clarify some of these issues, mostly by summarizing and contextualizing past work.
    Types of Myopia
    1. Cognitive Myopia
    One natural definition for myopia is that the model doesn't think about or consider the future at all. We will call this cognitive myopia. Myopic cognition likely comes with a significant capabilities handicap...
    14 min
  • AF - How LLMs are and are not myopic by janus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How LLMs are and are not myopic, published by janus on July 25, 2023 on The AI Alignment Forum.
    Thanks to janus, Nicholas Kees Dupuis, and Robert Kralisch for reviewing this post and providing helpful feedback. Some of the experiments mentioned were performed while at Conjecture.
    TLDR: The training goal for LLMs like GPT is not cognitively-myopic (because they think about the future) or value myopic (because the transformer architecture optimizes accuracy over the entire sequence, not just the next-token). However, training is consequence-blind, because the training data is causally independent of the models actions. This assumption breaks down when models are trained on AI generated text.
    Summary
    Myopia in machine learning models can be defined in several ways. It could be the time horizon the model considers when making predictions (cognitive myopia), the time horizon the model takes into account when assessing its value (value myopia), or the degree to which the model considers the consequences of its decisions (consequence-blindness).
    Both cognitively-myopic and consequence-blind models should not pursue objectives for instrumental reasons. This could avoid some important alignment failures, like power-seeking or deceptive alignment. However, these behaviors can still exist as terminal values, for example when a model is trained to predict power-seeking or deceptively aligned agents.
    LLM pretraining is not cognitively myopic because there is an incentive to think about the future to improve immediate prediction accuracy, like when predicting the next move in a chess game.
    LLM pretraining is not value/prediction myopic (does not maximize myopic prediction accuracy) because of the details of the transformer architecture. Training gradients flow through attention connections, so past computation is directly optimized to be useful when attended to by future computation. This incentivizes improving prediction accuracy over the entire sequence, not just the next token. This means that the model can and will implicitly sacrifice next-token prediction accuracy for long horizon prediction accuracy.
    You can modify the transformer architecture to remove the incentive for non-myopic accuracy, but as expected, the modified architecture has worse scaling laws.
    LLM pretraining on human data is consequence-blind as the training data is causally independent from the model's actions. This implies the model should predict actions without considering the effect of its actions on other agents, including itself. This makes the model miscalibrated, but likely makes alignment easier.
    When LLMs are trained on data which has been influenced or generated by LLMs, the assumptions of consequence-blindness partially break down. It's not clear how this affects the training goal theoretically or in practice.
    A myopic training goal does not ensure the model will learn myopic computation or behavior because inner alignment with the training goal is not guaranteed
    Introduction
    The concept of myopia has been frequently discussed as a potential solution to the problem of deceptive alignment. However, the term myopia is ambiguous and can refer to multiple different properties we might want in an AI system, only some of which might rule out deceptive alignment. There's also been confusion about the extent to which Large language model (LLM) pretraining and other supervised learning methods are myopic and what this implies about their cognition and safety properties. This post will attempt to clarify some of these issues, mostly by summarizing and contextualizing past work.
    Types of Myopia
    1. Cognitive Myopia
    One natural definition for myopia is that the model doesn't think about or consider the future at all. We will call this cognitive myopia. Myopic cognition likely comes with a significant capabili...
    14 min
  • EA - The Productivity Fallacy by Deena Englander
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Productivity Fallacy, published by Deena Englander on July 25, 2023 on The Effective Altruism Forum.
    When I find myself saying the same thing multiple times, it's time to write it up in an article.
    Recently, in my coaching sessions primarily with EA clients, I've found myself giving the same advice multiple times - cut down on what you're doing, spend time on yourself, and try to be unproductive for short periods of time.
    The problem is that people want to work at their maximum capacity in order to be the most impactful. And that is the productivity fallacy. So let's take a look at what happens when you are constantly working at full capacity.
    To start with, I'm going to use the analogy of a computer since most of us are pretty familiar with the basic mechanics of how it works. When a computer is going slowly, there are a few basic troubleshooting steps:
    Check your computer's resources
    Your computer only has a limited amount of resources to apply to the tasks set to it. When it exceeds hardware limitations, it will not be able to perform the required functions. Even worse, when it's exceeding its resource availability (running at >80%) for an extended period of time, it can have unintended and negative consequences such as shortened lifespan, slower performance, and software errors.
    I don't think anyone would argue with that - that's pretty much basic knowledge.
    So let's apply that to you and productivity:
    We only have a limited amount of energy. You can definitely argue that it's a design flaw with humans. When we exceed that amount of energy without replenishing it properly, you start running at "max capacity". When you're running at max capacity (being highly productive and efficient with your time without the restorative components to balance it), there are 3 big problems you'll encounter:
    You're at a much greater risk of burnout, getting sick, and harming your long-term ability to be impactful. The stress on your system has damaging consequences for both your physical and mental health, and they're not easy to recover from.
    Humans aren't built to do too much at once. If you take too much on, it will likely take necessary energy away from the things that matter most.
    You're much more likely to make mistakes. Mistakes can often be prevented by having the presence, calm, and headspace to focus properly. When you have too much going on, mistakes should be expected. You're also less likely to be able to come up with creative solutions since our creativity flows much more when we're not in a stressed state.
    2. Close unnecessary applications
    If your computer is running too many applications, it slows everything else down. So you have to make a choice - which are the ones that are critical to have running, and which ones can you live without, are consuming too many resources, or you didn't even realize were consuming resources?
    Applying that back to you, take an honest look at the activities that consume your resources. Which ones are critical to keep going? Which ones are less essential? Letting go of something isn't a failure - it's redirecting your resources to excel in your top priorities. Sometimes it helps to use a "monitoring program" like time tracking to see where your time and energy is going.
    3. Optimize your settings
    Sometimes there are some applications that you need, but they consume a lot of resources. So the next recommended step is to optimize your settings. Sometimes it's deleting the backlog, or changing the refresh rate, or having it not run in the background, or run at lower intensity. There are lots of potential solutions, and they differ based on your unique set of programs, available resources, and objectives.
    In your life, there may be some things that are high-resource consuming. But they don't need to be that way. How can you adjust these ...
    6 min
  • LW - Open problems in activation engineering by TurnTrout
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open problems in activation engineering, published by TurnTrout on July 24, 2023 on LessWrong.
    Steering GPT-2-XL by adding an activation vector introduced
    activation engineering... techniques which steer models by modifying their activations. As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime.
    These results were recently complemented by Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, which doubled TruthfulQA performance by adding a similarly computed activation vector to forward passes!
    We think that activation engineering has a bunch of low-hanging fruit for steering and understanding models. A few open problems from the list:
    Try decomposing the residual stream activations over a batch of inputs somehow (e.g. PCA). Using the principal directions as activation addition directions, do they seem to capture something meaningful?
    Take a circuit studied from existing literature on GPT2, or find another one using ACDC. Targeting the nodes in these circuits, can you learn anything more about them and generally about how activation additions interact with circuits?
    What's the mechanism by which adding a steering vector with too large a coefficient breaks the model? (Credit: Thomas Kwa; see also @Ulisse Mini's initial data/explanation.)
    If you want to work on activation engineering, come by the Slack server to coordinate research projects and propose new ideas.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Could someone help me understand why it's so difficult to solve the alignment problem? by Jadon Schmitt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Could someone help me understand why it's so difficult to solve the alignment problem?, published by Jadon Schmitt on July 24, 2023 on The Effective Altruism Forum.
    AGI will be able to model human langauge and psychology very accurately. Given that, wouldn't alignment be easy if you trained the AGI to interpret linguistic prompts in the way that the "average" human would? (I know language doesn't encode an exact meaning, but for any chunk of text, there does exist a distribution of ways that humans interpret it.)
    Thus, on its face, inner alignment seems fairly doable. But apparently, according to RobBesinger, "We don't know how to get an AI system's goals to robustly 'point at' objects like 'the American people' ... [or even] simpler physical systems." Why is this so difficult? Is there an argument that it is impossible?
    Outer alignment doesn't seem very difficult to me, either. Here's a prompt I thought of: "Do not do an action if anyone in a specified list of philosophers, intellectuals, members of the public, etc. would prefer you not do it, if they had all relevant knowledge of the action and its effects beforehand, consistent with the human legal standard of informed consent." Wouldn't this prompt (in its ideal form, not exactly as I wrote it) guard against many bad actions, including power-seeking behavior?
    Thank you for the help!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min