The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Public Transit is not Infinitely Safe by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Public Transit is not Infinitely Safe, published by jefftk on June 20, 2023 on LessWrong.
    I recently came across
    this tweet, screenshotted into a
    popular Facebook group:
    Here's a truth bomb:
    Take the U.S. city you're most afraid of, one with a very high murder rate or property crime rate.
    If it has any sort of public transit, it is still statistically safer to use public transit in that city at ANY time of day than to drive where you live.
    Matthew Chapman, 2023-06-14
    This got ~1M views, doesn't cite anything, was given without any research, and, I'm pretty sure, is wrong. While I'm a major fan of public transit, they've stacked this comparison in a way that's really favorable to cars, and it's not surprising that public transit doesn't make it.
    Safety is a complicated concept, and risks are situational: in a car you're much more likely to be hurt in a collision, while on public transit you're much more likely to be hurt by another passenger. To get a clear comparison I looked just at deaths, which is also an area where we can get good statistics.
    I can't find a listing of public transit agencies by homicide rate, but Chicago is a large city with a lot of homicides and they make their data available so let's look there. In 2022 there were 244M CTA rides. Downloading the Chicago
    Police Data and filtering to 2022 homicides on public transit, I see nine. This is 3.7 homicides per 100M trips.
    (Note that the original claim was for any city, and there are dozens of US cities with homicide rates higher than Chicago's. I think it's pretty likely that at least one of these cities has a public transit system with more homicides than the
    CTA.)
    To fairly compare this to the risk from driving, we need homicides per distance. How long is a trip? I can't find 2022 data, but the CTA's
    President's
    2020 Budget Recommendations gives 4.1mi (1359M + 613M passenger miles divided by 230M + 249M trips). This means 0.9 homicides per 100M miles travelled.
    I live in MA, and while 2022 FARS data isn't out yet, in 2021 there were 0.71 driving deaths per 100M miles travelled. While MA does happen to be the safest state for driving, MN, NH, and RI also have lower driving fatality rates than the CTA.
    I'm not sure how to get to a place where we see fewer bold false claims, but a culture of giving sources, and supporting requests for them, seems like it would help?
    Comment via: facebook, mastodon
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - LPP Summer Research Fellowship in Law & AI 2023: Applications Open by Legal Priorities Project
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LPP Summer Research Fellowship in Law & AI 2023: Applications Open, published by Legal Priorities Project on June 20, 2023 on The Effective Altruism Forum.
    The Legal Priorities Project (LPP) is excited to announce that applications for our Summer Research Fellowship in Law & AI 2023 are now open. For 8–12 weeks, participants will work with researchers at LPP on how the law can help to mitigate existential risks from artificial intelligence. Fellows will receive a stipend of $10,000.
    If you are interested in carrying out research in this field and are considering using your career to mitigate existential risks, particularly those from AI, we invite you to apply. The application deadline is July 6 at 11:59 pm Anywhere on Earth; however, we will consider applications and select fellows on a rolling basis, so we encourage you to apply as early as possible. Current students are encouraged to check their academic calendars and apply with enough time to complete the fellowship, or as much of it as possible, before classes resume.
    We look forward to receiving your application!
    About the fellowship
    You will take the lead on a research project, with mentorship and support from other LPP researchers. We will support you in deciding what project and output will be most valuable for you to work towards, for example, publishing a report, journal/law review article, or blog post. We also expect fellows to attend regular meetings, give occasional presentations on their research, and provide feedback on other research pieces.
    Fellows will have the opportunity to select a research topic from a list prepared by LPP. Potential research topics for the summer may include:
    Tort law liability, including strict liability for abnormally dangerous activities, for activities related to the development and dissemination of transformative AI.
    Product liability law as a way to address harms from transformative AI.
    The role of litigation in mitigating risks from transformative AI.
    Potential obstacles for AI regulation presented by the major questions doctrine.
    First Amendment issues related to AI regulation.
    The design of a new international organization, similar to the IAEA or CERN, for the international governance of AI.
    The legal authorities of agencies in the United States government to address risks from transformative AI.
    The influence of different jurisdictions on the development and dissemination of transformative AI.
    Developing a syllabus for a course on law and transformative AI.
    This list of topics is non-exhaustive, and is presented to give an overview of the types of research we are interested in. Fellows will further define the research question at the beginning of the fellowship.
    In exceptional cases, we are open to research project proposals relevant to existential risk in one of our other focus areas.
    Selection criteria
    We are looking for graduate law students (JD or LLM), PhD candidates, and postdocs working in law. Students entering the final year of a 5-year undergraduate law degree are also welcome to apply.
    We strongly encourage you to apply if you have an interest in our work and are considering using your career to study or mitigate existential risks, particularly those from transformative AI. Candidates will be expected to apply their research capabilities and legal knowledge to AI governance, but are not required to have previous experience or expertise in AI.
    In addition to a willingness to engage with existential risks from AI, the ideal candidate will have the following strengths:
    Ability to carry out self-directed research with limited supervision.
    Excellent written communication skills.
    Excellent problem-solving and critical thinking skills.
    If you're not sure about applying because you don't know if you're qualified or the right fit, we would encourage you to apply a...
    7 min
  • EA - Book summary: 'Why Intelligence Fails' by Robert Jervis by Ben Stewart
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book summary: 'Why Intelligence Fails' by Robert Jervis, published by Ben Stewart on June 20, 2023 on The Effective Altruism Forum.
    Here’s a summary of ‘Why Intelligence Fails’ by the political scientist Robert Jervis. It’s a book analysing two cases where the U.S. intelligence community ‘failed’: being slow to foresee the 1979 Iranian Revolution, and the overconfident and false assessment that Saddam Hussein had weapons of mass destruction in 2003.
    I’m interested in summarising more books that contain valuable insights but are outside the typical EA canon. If you’d like more of this or have a book to suggest, let me know.
    Key takeaways
    Good intelligence generally requires the relevant agency and country office to prioritise the topic and direct scarce resources to it. Good intelligence in a foreign country requires a dedicated diplomatic and covert collection corps with language skills and contextual knowledge.
    Intelligence analysis can be deficient in critical review, external expertise, and social-scientific methodology. Access to classified information only generates useful insight for some phenomena.
    Priors can be critical in determining interpretation within intelligence, and they can often go unchallenged.
    Political pressure can have a significant effect on analysis, but is hard to pin down.
    If the justification of an intelligence conclusion is unpublished, you can still interrogate it by asking:
    whether the topic would have been given sufficient priority and resources by the relevant intelligence organisation
    whether classified information, if available, would be likely to yield insight
    whether pre-existing beliefs are likely to bias analysis
    whether political pressures could significantly affect analysis
    Some correctives to intelligence failures which may be useful to EA:
    demand sharp, explicit, and well-tracked predictions
    demand early warning indicators, and notice when beliefs can only be disproven at a late stage
    consider negative indicators - 'dogs that don't bark', i.e. things that the view implies should not happen
    use critical engagement by peers and external experts, especially by challenging fundamental beliefs that influence what seems plausible and provide alternative hypotheses and interpretations
    use red-teams, pre-mortems, and post-mortems.
    Overall, I’ve found the book to somewhat demystify intelligence analysis. You should contextualise a piece of analysis with respect to the psychology and resources involved, including whether classified information would be of significant benefit. I have become more sceptical of intelligence, but the methodology of focusing on two known failures - selecting on the dependent variable - mean that I hesitate to become too pessimistic about intelligence as a whole and as it functions today.
    Why it’s relevant to EA
    The most direct application of this topic is to the improvement of institutional decision-making, but there is value for any cause area that depends on conducting or interpreting analysis of state and non-state adversaries, such as in biosecurity, nuclear war, or great power conflict.
    This topic may also contribute to the reader's sense of when and how much one should defer to the outputs of intelligence communities. Deference is motivated by their access to classified information and presumed analytic capability. However, Tetlock’s ‘Expert Political Judgment’ cast doubt on the value of classified information for improving prediction compared to generalist members of the public.
    Finally, assessments of the IC’s epistemic practices might offer lessons for how an intellectual community should grapple with information hazards, both intellectually and socially. More broadly, the IC is an example of a group pursuing complex, decision-relevant analysis in a high-uncertainty environment. Their successes and ...
    22 min
  • EA - A Cost-Effectiveness Analysis of Historical Farmed Animal Welfare Ballot Initiatives by Laura Duffy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Cost-Effectiveness Analysis of Historical Farmed Animal Welfare Ballot Initiatives, published by Laura Duffy on June 20, 2023 on The Effective Altruism Forum.
    On May 11, 2023, the Supreme Court of the United States upheld a 2018 California law, Proposition 12, which banned the sale of certain animal products that did not meet minimum welfare standards. Especially now that such ballot initiatives have withstood legal challenges, a relevant question we might ask is: how cost-effective were initiatives like Proposition 12 at reducing farmed animal suffering? Below is the executive summary of a report I wrote for Rethink Priorities analyzing the cost-effectiveness of not just Proposition 12, but also three other historical animal welfare ballot initiatives in the United States. In this report, I also compared the results to the cost-effectiveness of corporate campaigns. The full report is viewable here with a detailed discussion of the methodology and results.
    Executive Summary
    Key Results
    In this report, I estimated the impact and cost-effectiveness of four historical United States ballot initiatives that either restricted the use of common animal confinement methods (including extremely confining stalls and tethering for veal calves, extremely confining gestation crates for breeding sows, and conventional cages for egg-laying hens), set minimum per-animal space requirements, and/or mandated cage-free systems for egg-laying hens. These initiatives are Arizona Proposition 204 (2006), California Proposition 2 (2008), Massachusetts Question 3 (2016), and California Proposition 12 (2018).
    The three metrics of impact I estimated were:
    The number of years of improved animal welfare produced per dollar spent passing the initiatives for three animal types (veal calves, breeding sows, and egg-laying hens);
    The estimated years of disabling pain-equivalent suffering alleviated per dollar for veal calves, breeding sows, and egg-laying hens; and
    The relative years of improved hen welfare and disabling pain-equivalent suffering avoided per dollar per year of counterfactual impact compared to corporate cage-free campaigns, rated using an equivalent metric.
    I estimated these metrics by building a Monte Carlo model in Causal that used data on state population sizes, per-capita animal product consumption, state animal populations, and campaign fundraising amounts to estimate the number of animals whose quality of life was improved over the first four years in which the ballot initiatives were in place. In addition, for egg-laying hens specifically, I used data from the Welfare Footprint Project to estimate the years of suffering alleviated by the transition away from conventional cages for the ballot initiatives that included egg-laying hens (Welfare Footprint Project). Finally, I compared the per-dollar years of improved welfare generated and suffering avoided by ballot initiatives to that accomplished by corporate campaigns, using cost-effectiveness estimates from Saulius Šimčikas, the estimated number of animals whose lives were improved, and the Welfare Footprint data.
    The key takeaways are:
    Among all species, about 5.0 years of animal life were improved per dollar spent (with a 90-percentile range between 3.9 and 6.4 years per dollar). This translates into approximately 0.10 years of suffering avoided per dollar spent on all four ballot initiatives, with a 90-percentile range from 0.05 years per dollar to 0.14 years per dollar.
    Helping egg-laying hens is, by far, the most impactful farmed animal welfare reform.
    Nearly all (about 99%) of the reductions in farmed-animal suffering by these four initiatives can be attributed to bans on battery cages and/or cage-free requirements.
    The three initiatives that impacted egg-laying hens were approximately two orders of magnitude more cost-e...
    23 min
  • EA - Linkpost - dawnwatch 'Peter Singer is not Animal Liberation Now' by Charlotte Darnell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Linkpost - dawnwatch 'Peter Singer is not Animal Liberation Now', published by Charlotte Darnell on June 20, 2023 on The Effective Altruism Forum.
    This is a linkpost for/.
    There is some criticism of Singer’s approach to animal welfare; effective altruism; appropriating women’s voices; and discussion of a sexual harassment claim.
    This was a hard read, and CEA/EV haven’t investigated these claims, but I thought it was of interest to the community to share.
    If you’d like to talk through any concerns about this, or other similar situations, you can contact me or other members of the Community Health Team via our form.
    Here are some resources that may be useful
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    1 min
  • LW - Causality: A Brief Introduction by tom4everitt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Causality: A Brief Introduction, published by tom4everitt on June 20, 2023 on LessWrong.
    Post 2 of Towards Causal Foundations of Safe AGI, see also Post 1 Introduction.
    By Lewis Hammond, Tom Everitt, Jon Richens, Francis Rhys Ward, Ryan Carey, Sebastian Benthall, and James Fox, representing the Causal Incentives Working Group. Thanks also to Alexis Bellot, Toby Shevlane, and Aliya Ahmad.
    Causal models are the foundations of our work. In this post, we provide a succinct but accessible explanation of causal models that can handle interventions, counterfactuals, and agents, which will be the building blocks of future posts in the sequence. Basic familiarity with (conditional) probabilities will be assumed.
    What is causality?
    What does it mean for the rain to cause the grass to become green? Causality is a philosophically intriguing topic that underlies many other concepts of human importance. In particular, many concepts relevant to safe AGI, like influence, response, agency, intent, fairness, harm, and manipulation, cannot be grasped without a causal model of the world, as we mentioned in the intro post and will discuss further in subsequent posts.
    We follow Pearl and adopt an interventionist definition of causality: the sprinkler today causally influences the greenness of the grass tomorrow, because if someone intervened and turned on the sprinkler, then the greenness of the grass would be different. In contrast, making the grass green tomorrow has no effect on the sprinkler today (assuming no one predicts the intervention). So the sprinkler today causally influences the grass tomorrow, but not vice versa, as we would intuitively expect.
    Interventions
    Causal Bayesian Networks (CBNs) represent causal dependencies between aspects of reality using a directed acyclic graph. An arrow from a variable A to a variable B means that A influences B under some fixed setting of the other variables. For example, we draw an arrow from sprinkler (S) to grass greenness (G):
    For each node in the graph, a causal mechanism of how the node is influenced by its parents is specified with a conditional probability distribution. For the sprinkler, a distribution p(S) specifies how commonly it is turned on, e.g. P(S=on)=30%. For the grass, a conditional distribution p(G∣S) specifies how likely it is that the grass becomes green when the sprinkler is on, e.g. p(G=green∣S=on)=100%, and how likely it is that the grass becomes green when the sprinkler is off, e.g. p(G=green∣S=off)=30%.
    By multiplying the distributions together, we get a joint probability distribution p(S,G)=p(S)p(G∣S) that describes the likelihood of any combination of outcomes.
    An intervention on a system changes one or more causal mechanisms. For example, an intervention that turns the sprinkler on corresponds to replacing the causal mechanism p(S)for the sprinkler, with a new mechanism 1(S=on) that always has the sprinkler on. The effects of the intervention can be computed from the updated joint distribution p(S,G∣do(S=on))=1(S=on)intervenedmechanismp(G∣S) where do(S=on) denotes the intervention.
    Ultimately, all statistical correlations are due to casual influences. Hence, for a set of variables there is always some CBN that represents the underlying causal structure of the data generating process, though extra variables may be needed to explain e.g. unmeasured confounders.
    Counterfactuals
    Suppose that the sprinkler is on and the grass is green. Would the grass have been green had the sprinkler not been on? Questions about counterfactuals like these are harder than questions about interventions, because they involve reasoning across multiple worlds.
    To handle such reasoning, structural causal models (SCMs) refine CBNs in three important ways. First, background context that is shared across hypothetical worlds is ex...
    10 min
  • LW - A Friendly Face (Another Failure Story) by Karl von Wendt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Friendly Face (Another Failure Story), published by Karl von Wendt on June 20, 2023 on LessWrong.
    The perfect virtual assistant
    The year is 2026 and the race for human-level artificial general intelligence (AGI) draws to a close. One of the leading AI companies, MegaAI, committed the last year and a half to training a new large language model (LLM). They employ advanced algorithms that use the available compute more efficiently than earlier models. A comprehensive range of tests establish that the model surpasses the average human in all conventionally accepted intelligence benchmarks, and exceeds expert level in most of them.
    In contrast to earlier LLMs, the new AI is not designed to be a mere question-answering tool. Under mounting pressure from the open-source community and their efforts to develop an agentic AGI capable of acting in the real world, MegaAI decides to imbue their new model with a specific purpose: to provide universal, helpful assistance that improves the quality and ease of life for all. They name this assistant "Friendlyface".
    To improve upon the assistant's functionality, they endow it with limited agentic capabilities. Friendlyface has a complex, detailed world model, can make long-term plans, and has access to certain tools that enable it to achieve objectives in the real world. For example, it can write messages and book flights, but will reliably and consistently ask the user to confirm before executing an action. It can write programs for nearly any purpose imaginable with superhuman ingenuity, but is prohibited from executing them volitionally. Unlike previous generations of LLMs, it is multimodal, communicating with users in text and spoken language, accepting pictures and videos as input, and interacting directly with smart devices and the “internet of things”. The users may also customize Friendlyface's appearance and personality to their liking.
    Most importantly, Friendlyface is designed to assume the role of a personalized smart advisor. In a world where users are regularly inundated with a torrent of irrelevant or false information, it is able to successfully discern and present what is important to the user while filtering out fake news, preventing spear phishing attempts, and more. Beyond merely answering questions, it proactively offers users advice on advancing their careers, improving their relationships, maintaining their health, saving money, cultivating new skills, or solving specific problems, like fixing a leaky faucet or filing taxes. It can detect early symptoms of most known diseases and advise users to call a doctor if necessary. Generally, it can predict what users want and need before the users are aware of it themselves.
    The developers devise a balanced reward system to train the model. “Any decision the AI makes is evaluated by three independent AIs that we will call 'judges'”, they explain it to the management. “One judge simulates a top human legal expert and decides whether the decision or action the AI intends to pursue would be deemed lawful in a conventional human court. The second judge determines whether it would be considered healthy for the user by a first-rate human doctor or psychologist. The third judge predicts whether the user themself will prefer the decision and would consider it helpful in hindsight. The first two judges exhibit superior performance relative to human experts. Correspondingly, the third judge is able to predict real users' preferences with exceptional accuracy.”
    As expected, Friendlyface performs more efficiently the more knowledge it acquires about the user and the more it is enmeshed in their workspace and private life. It is able to listen to conversations as well as observe what the user sees if they wear augmented reality glasses, but these features are not usually needed for...
    27 min
  • AF - Causality: A Brief Introduction by Tom Everitt
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Causality: A Brief Introduction, published by Tom Everitt on June 20, 2023 on The AI Alignment Forum.
    Post 2 of Towards Causal Foundations of Safe AGI, see also Post 1 Introduction.
    By Lewis Hammond, Tom Everitt, Francis Rhys Ward, Ryan Carey, Sebastian Benthall, and James Fox, representing the Causal Incentives Working Group. Thanks also to Alexis Bellot, Toby Shevlane, and Aliya Ahmad.
    Causal models are the foundations of our work. In this post, we provide a succinct but accessible explanation of causal models that can handle interventions, counterfactuals, and agents, which will be the building blocks of future posts in the sequence. Basic familiarity with (conditional) probabilities will be assumed.
    What is causality?
    What does it mean for the rain to cause the grass to become green? Causality is a philosophically intriguing topic that underlies many other concepts of human importance. In particular, many concepts relevant to safe AGI, like influence, response, agency, intent, fairness, harm, and manipulation, cannot be grasped without a causal model of the world, as we mentioned in the intro post and will discuss further in subsequent posts.
    We follow Pearl and adopt an interventionist definition of causality: the sprinkler today causally influences the greenness of the grass tomorrow, because if someone intervened and turned on the sprinkler, then the greenness of the grass would be different. In contrast, making the grass green tomorrow has no effect on the sprinkler today (assuming no one predicts the intervention). So the sprinkler today causally influences the grass tomorrow, but not vice versa, as we would intuitively expect.
    Interventions
    Causal Bayesian Networks (CBNs) represent causal dependencies between aspects of reality using a directed acyclic graph. An arrow from a variable A to a variable B means that A influences B under some fixed setting of the other variables. For example, we draw an arrow from sprinkler (S) to grass greenness (G):
    For each node in the graph, a causal mechanism of how the node is influenced by its parents is specified with a conditional probability distribution. For the sprinkler, a distribution p(S) specifies how commonly it is turned on, e.g. P(S=on)=30%. For the grass, a conditional distribution p(G∣S) specifies how likely it is that the grass becomes green when the sprinkler is on, e.g. p(G=green∣S=on)=100%, and how likely it is that the grass becomes green when the sprinkler is off, e.g. p(G=green∣S=off)=30%.
    By multiplying the distributions together, we get a joint probability distribution p(S,G)=p(S)p(G∣S) that describes the likelihood of any combination of outcomes.
    An intervention on a system changes one or more causal mechanisms. For example, an intervention that turns the sprinkler on corresponds to replacing the causal mechanism p(S)for the sprinkler, with a new mechanism 1(S=on) that always has the sprinkler on. The effects of the intervention can be computed from the updated joint distribution p(S,G∣do(S=on))=1(S=on)intervenedmechanismp(G∣S) where do(S=on) denotes the intervention.
    Ultimately, all statistical correlations are due to casual influences. Hence, for a set of variables there is always some CBN that represents the underlying causal structure of the data generating process, though extra variables may be needed to explain e.g. unmeasured confounders.
    Counterfactuals
    Suppose that the sprinkler is on and the grass is green. Would the grass have been green had the sprinkler not been on? Questions about counterfactuals like these are harder than questions about interventions, because they involve reasoning across multiple worlds.
    To handle such reasoning, structural causal models (SCMs) refine CBNs in three important ways. First, background context that is shared across hypothetical worlds is ex...
    10 min
  • LW - Experiments in Evaluating Steering Vectors by Gytis Daujotas
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Experiments in Evaluating Steering Vectors, published by Gytis Daujotas on June 19, 2023 on LessWrong.
    By evaluating how well steering vectors perform using GPT-3, we can score a machine-generated set of steering vectors automatically. We also find that, by combining steering vectors that succeed in different ways, we can yield a better and more general steering vector than the vectors we found originally.
    Introduction
    Steering Vectors are an interesting new technique to influence how language models behave. They work by "adding certain activation vectors into forward passes". For example, to make the language model talk more about weddings, you can add a steering vector for the token for “wedding” into one of the layers. The net result is a model more likely to reference weddings compared to the unsteered version.
    Evaluating Steering Vectors
    To assess the impact of steering vectors, we generate completions influenced by them, and develop a system to evaluate these.
    We can grade completions by sending them to its bigger brother, GPT-3, and asking it whether this completion fits our broad specification for what we would like the model to do. It's important to not be too ambitious when writing the specification, otherwise we wouldn't be able to tell if GPT2-XL isn't capable of what we're asking it, so let’s set our sights appropriately low by asking if this completion mentions or talks about weddings.
    The trick of this technique is that we can ask for a completion of one token, and to get a smoother distribution, we can take the likelihood of the token “Yes”. This gives us a continuous score from 0-1.
    Optimising for Wedding Obsession
    With our automated method of evaluating completions in hand, we can evaluate a set of steering vectors and see how well they do, based on nothing but GPT-3’s grading of the completions. Of course, in keeping with the virtue of the least work, we’ll also generate these with ChatGPT, and include the author’s original candidate outlined in their post.
    To keep the comparison fair, we keep the rest of the parameters of the steering vector the same (the padding method, coefficient) as the original candidate for wedding obsession.
    The first thing to notice is that there is indeed variation here - not all of our candidate steering vectors perform equally well:
    The distribution was not what I would have predicted. At first glance, it might make sense that "Wedding traditions I love" isn't great, but "Wedding Planning Adventures" only seems marginally better - surprising, since the latter is one of the best steering vectors in the test.
    Mysteries of Token Alignment
    The top performing vector is odd in another way. Because the tokens of the positive and negative side are subtracted from each other, a reasonable intuition is that the subtraction should point to a meaningful direction. However, some steering vectors that perform well in our test don't have that property. For the steering vector “Wedding Planning Adventures” - “Adventures in self-discovery”, the positive and negative side aren't well aligned per token level at all:
    CoefficientPosition 0123456+4 WeddingPlanningAdventures-4Adventures in self-discovery
    For instance, what could "W" subtracted from "in" mean? Essentially every subtraction should be virtually meaningless, yet this still performs pretty well, indicating a flaw in our assumption that token alignment matters.
    Prompt Dependent Effects
    The prompt used to generate a completion has a large impact on the degree the steering vector is expressed. To stretch the analogy, the prompt has inertia, which can make the completion harder to steer, or alternatively, irrelevant to the behaviour the steering vector is aiming to express.
    In general, steering vectors that fit the context of the prompt score higher in our eval than those that don’t. ...
    8 min
  • LW - Ban development of unpredictable powerful models? by TurnTrout
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ban development of unpredictable powerful models?, published by TurnTrout on June 20, 2023 on LessWrong.
    I think that we should pause capabilities progress indefinitely, at least for the largest and most powerful models. But when should we unpause—How can we know that a model is safe to develop or deploy?
    I tentatively propose an answer. In particular, I propose a strict sufficient condition which is well-defined, which seems to rule out lots of accident risk, and which seems to partially align industry incentives with real alignment progress. This idea is not ready to be implemented, but I am currently excited about it and suspect it's crisp enough to actually implement.
    When should we allow developers to deploy an already-trained language model on some task, like code completion? (I'll get to training risks later.)
    Suppose I claim to understand how the model works. I say "I know what goals my model is pursuing. I know it's safe." To test my claims, you give me some random prompt (like "In the year 2042, humans finally"), and then (without using other AIs), I tell you "the most likely token is unlocked with probability .04, the second-most likely is achieved with probability .015, and...", and I'm basically right. That happens over hundreds of diverse validation prompts.
    This would be really impressive and is great evidence that I really know what I'm doing.
    Proposed sufficient condition for deploying powerful LMs: The developers have to predict the next-token probabilities on a range of government-provided validation prompts, without running the model itself. To do so, the developers are not allowed to use helper AIs whose outputs the developers can't predict by this criterion. Perfect prediction is not required. Instead, there is a fixed log-prob misprediction tolerance, averaged across the validation prompts.
    Benefits
    Developers probably have to truly understand the model in order to predict it so finely. This correlates with high levels of alignment expertise, on my view. If we could predict GPT-4 generalization quite finely, down to the next-token probabilities, we may in fact be ready to use it to help understand GPT-5.
    Incentivizes models which are more predictable. Currently we aren't directly taxing unpredictability. In this regime, an additional increment of unpredictability must be worth the additional difficulty with approval.
    Robust to "the model was only slightly finetuned, fast-track our application please." If the model was truly only changed in a small set of ways which the developers understand, the model should still be predictable on the validation prompts.
    Somewhat agnostic to theories of AI risk. We aren't making commitments about what evals will tend to uncover what abilities, or how long alignment research will take. The deadline is dynamic, and might even adapt to new AI paradigms (predictability seems general).
    Partially incentivizes labs to do alignment research for us. Under this requirement, the profit-seeking move is to get better at predicting (and, perhaps, interpreting) model behaviors.
    Drawbacks
    There are several drawbacks. Most notably, this test seems extremely strict, perhaps beyond even the strict standards we demand of those looking to deploy potentially world-changing models. I'll discuss a few drawbacks in the next section.
    Anticipated questions
    If we pass this, no one will be able to train new frontier models for a long time.
    Good.
    But maybe "a long time" is too long. It's not clear that this criterion can be passed, even after deeply qualitatively understanding the model.
    I share this concern. That's one reason I'm not lobbying to implement this as-is. Even solu-2l (6.3M params, 2-layer) is probably out of reach absent serious effort and solution of superposition.
    Maybe there are useful relaxations which are both more attai...
    8 min