The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Some common confusion about induction heads by Alexandre Variengien
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some common confusion about induction heads, published by Alexandre Variengien on March 28, 2023 on LessWrong.
    Epistemic status: conceptual discussion and opinions informed by doing 6 months of interpretability research at Redwood Research and exchanging with other researchers, but I’m just speaking for myself.
    Induction heads are defined twice by Anthropic.
    The first time as a mechanism in 2L attention-only transformers
    A second time as a behavioral description on repeated random sequences of tokens
    However, these two definitions rely on distinct sources of evidence and create confusion, as their difference is not always acknowledged when people cite these papers. The mechanistic definition applies to toy language models, while the behavioral definition is a useful yet incomplete characterization of attention heads.
    I think that many people are in fact confused by this: I have talked to many people who aren’t clear on the fact that these two concepts are different, and incorrectly believe that (e.g.) the mechanism of induction heads in larger language models has been characterized.
    More specifically, the two Anthropic papers introduce the following two distinct definitions of induction heads:
    Mechanistic: The first definition, introduced by Elhage et al., describes a behavior in a 2 layer attention-only model (copying a token given a matching prefix) and a minimal mechanism to perform this behavior (a set of paths in the computational graph and a human interpretation of the transformation along those paths). Empirically, this mechanism seems to be the best possible short description of what those heads are doing (i.e. if you have to choose a subgraph made of a single path as input for the keys, queries, and values of these heads, the induction circuit is likely to be the one that recovers the most loss). But this explanation does not encompass everything these heads do. In reality, many more paths are used than the one described (see Redwood’s causal scrubbing results on induction heads) and the function of the additional paths is unclear.
    I don’t know whether the claims about the behavior and mechanisms of these heads are best described as “mostly true but missing details” or “only a small part of what’s going on”. See also Buck’s comment for more discussion on the interpretation of causal scrubbing recovered loss.
    Behavioral: The second definition, introduced by Olsson et al., relies on head activation evaluators (measuring attention patterns and head output) on out-of-distribution sequences made of Repeated Random Tokens (RRT). Two scores are used to characterize induction heads: i) Prefix matching: attention probability to the first occurrence of the token [A] on patterns like [A][B] . [A] ii) Copying: how much the head output increases the logit of [A] compared to the other logits. The RRT distribution was chosen so that fully abstracted induction behavior is one of the few useful heuristics to predict the next token.
    In this post, I’ll use mechanistic or behavioral induction heads to differentiate between the definitions.
    I’ll present three points that — I think — are important to keep in mind when using these definitions.
    1 - The two-head mechanism (induction head and previous token head) described in Elhage et al. is the minimal way to implement an induction head
    As noted in the paper, induction heads can use more complicated mechanisms. For instance, instead of relying on a previous token head to match only one token as a prefix (the token [A] in the example above), they could rely on a head that attends further back to match longer prefixes (e.g. patterns like [X][A][B] . [X][A]). Empirically, evidence for induction heads using some amount of longer prefix matching has been observed in the causal scrubbing experiments on induction.
    The two-head mechanism i...
    10 min
  • LW - Nobody’s on the ball on AGI alignment by leopold
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nobody’s on the ball on AGI alignment, published by leopold on March 29, 2023 on LessWrong.
    Far fewer people are working on it than you might think, and even the alignment research that is happening is very much not on track. (But it’s a solvable problem, if we get our act together.)
    Observing from afar, it's easy to think there's an abundance of people working on AGI safety. Everyone on your timeline is fretting about AI risk, and it seems like there is a well-funded EA-industrial-complex that has elevated this to their main issue. Maybe you've even developed a slight distaste for it all—it reminds you a bit too much of the woke and FDA bureaucrats, and Eliezer seems pretty crazy to you.
    That’s what I used to think too, a couple of years ago. Then I got to see things more up close. And here’s the thing: nobody’s actually on the friggin’ ball on this one!
    There’s far fewer people working on it than you might think. There are plausibly 100,000 ML capabilities researchers in the world (30,000 attended ICML alone) vs. 300 alignment researchers in the world, a factor of ~300:1. The scalable alignment team at OpenAI has all of ~7 people.
    Barely anyone is going for the throat of solving the core difficulties of scalable alignment. Many of the people who are working on alignment are doing blue-sky theory, pretty disconnected from actual ML models. Most of the rest are doing work that’s vaguely related, hoping it will somehow be useful, or working on techniques that might work now but predictably fail to work for superhuman systems.
    There’s no secret elite SEAL team coming to save the day. This is it. We’re not on track.
    If timelines are short and we don’t get our act together, we’re in a lot of trouble. Scalable alignment—aligning superhuman AGI systems—is a real, unsolved problem. It’s quite simple: current alignment techniques rely on human supervision, but as models become superhuman, humans won’t be able to reliably supervise them.
    But my pessimism on the current state of alignment research very much doesn’t mean I’m an Eliezer-style doomer. Quite the opposite, I’m optimistic. I think scalable alignment is a solvable problem—and it’s an ML problem, one we can do real science on as our models get more advanced. But we gotta stop fucking around. We need an effort that matches the gravity of the challenge.
    Alignment is not on track
    A recent post estimated that there were 300 full-time technical AI safety researchers (sounds plausible to me, if we’re counting generously). By contrast, there were 30,000 attendees at ICML in 2021, a single ML conference. It seems plausible that there are ≥100,000 researchers working on ML/AI in total. That’s a ratio of ~300:1, capabilities researchers:AGI safety researchers.
    That ratio is a little better at the AGI labs: ~7 researchers on the scalable alignment team at OpenAI, vs. ~400 people at the company in total (and fewer researchers). But 7 alignment researchers is still, well, not that much, and those 7 also aren’t, like, OpenAI’s most legendary ML researchers. (Importantly, from my understanding, this isn’t OpenAI being evil or anything like that—OpenAI would love to hire more alignment researchers, but there just aren’t many great researchers out there focusing on this problem.)
    But rather than the numbers, what made this really visceral to me is. actually looking at the research. There’s very little research where I feel like “great, this is getting at the core difficulties of the problem, and they have a plan for how we might actually solve it in <5 years.”
    Let’s take a quick, stylized, incomplete tour of the research landscape.
    Paul Christiano / Alignment Research Center (ARC).
    Paul is the single most respected alignment researcher in most circles. He used to lead the OpenAI alignment team, and he has made useful conceptual contri...
    22 min
  • AF - Othello-GPT: Reflections on the Research Process by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Othello-GPT: Reflections on the Research Process, published by Neel Nanda on March 29, 2023 on The AI Alignment Forum.
    This is the third in a three post sequence about interpreting Othello-GPT. See the first post for context.
    This post is a detailed account of what my research process was, decisions made at each point, what intermediate results looked like, etc. It's deliberately moderately unpolished, in the hopes that it makes this more useful!
    The Research Process
    This project was a personal experiment in speed-running doing research, and I got the core results in in ~2.5 days/20 hours. This post has some meta level takeaways from this on doing mech interp research fast and well, followed by a (somewhat stylised) narrative of what I actually did in this project and why - you can see the file tl_initial_exploration.py in the paper repo for the code that I wrote as I went (using VSCode's interactive Jupyter mode).
    I wish more work illustrated the actual research process rather than just a final product, so I'm trying to do that here. This is approximately just me converting my research notes to prose, see the section on process-level takeaways for a more condensed summary of my high-level takeaways.
    The meta level process behind everything below is to repeatedly be confused, plot stuff a bunch, be slightly less confused, and iterate. As a result, there's a lot of pictures!
    Takeaways on doing mech interp research
    Warning: I have no idea if following my advice about doing research fast is actually a good idea, especially if you're starting out in the field! It's much easier to be fast and laissez faire when you have experience and an intuition for what's crucial and what's not, and it's easy to shoot yourself in the foot. And when you skimp on rigour, you want to make sure you go back and check! Though in this case, I got strong enough results with the probe that I was fairly confident I hadn't entirely built a tower of lies. And generally, beware of generalising from one example - in hindsight I think I got pretty lucky on how fruitful this project was!
    Be decisive: Subjectively, by far the most important change was suppressing my perfectionism and trying to be bold and decisive - make wild guesses and act on them, be willing to be less rigorous, etc.
    If I noticed myself stuck on doing the best or most principled thing, I'd instead try to just do something.
    Eg I wanted to begin by patching between two similar sequences of moves - I couldn't think of a principled way to change a move without totally changing the downstream game, so I just did the dumb thing of patching by changing the final move.
    Eg when I wanted to try intervening with the probe, I couldn't think of a principled way to intervene on a bunch of games or to systematically test that this worked, or exactly how best to intervene, so I decided to instead say "YOLO, let's try intervening in the dumbest possible way, by flipping the coefficient at a middle layer, on a single move, and see what happens"
    Pursue the hypothesis that seems "big if true"
    Eg I decided to try training a linear probe on just black moves after a hunch that this might work given some suggestive evidence from interpreting neuron L5N1393
    Notice when I get stuck in a rabbit hole/stop learning things and move on
    Eg after training a probe I found it easy to be drawn into eg inspecting more and more neurons, or looking at head attention patterns, and it worked much better to just say
    Be willing to make quick and dirty hacks
    Eg when I wanted to look at the max activating dataset examples for neurons, I initially thought I'd want to run the model on thousands to millions of games, to get a real sample size. But in practice, just running the model on a batch of 100 games and taking the top 1% of moves by neuron act in there, worked totally fine...
    23 min
  • AF - Othello-GPT: Future Work I Am Excited About by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Othello-GPT: Future Work I Am Excited About, published by Neel Nanda on March 29, 2023 on The AI Alignment Forum.
    This is the second in a three post sequence about interpreting Othello-GPT. See the first post for context.
    This post covers future directions I'm excited to see work on, why I care about them, and advice to get started. Each section is self-contained, feel free to skip around.
    Look up unfamiliar terms here
    Future work I am excited about
    The above sections leaves me (and hopefully you!) pretty convinced that I've found something real and dissolved the mystery of whether there's a linear vs non-linear representation. But I think there's a lot of exciting mysteries left to uncover in Othello-GPT, and that doing so may be a promising way to get better at reverse-engineering LLMs (the goal I actually care about). In the following sections, I try to:
    Justify why I think further work on Othello-GPT is interesting
    (Note that my research goal here is to get better at transformer mech interp, not to specifically understand emergent world models better)
    Discuss how this unlocks finding modular circuits, and some preliminary results
    Rather than purely studying circuits mapping input tokens to output logits (like basically all prior transformer circuits work), using the probe we can study circuits mapping the input tokens to the world model, and the world model to the output logits - the difference between thinking of a program as a massive block of code vs being split into functions and modules.
    If we want to reverse-engineer large models, I think we need to get good at this!
    Discuss how we can interpret Othello-GPT's neurons - we're very bad at interpreting transformer MLP neurons, and I think that Othello-GPT's are simple enough to be tractable yet complex enough to teach us something!
    Discuss how, more broadly, Othello-GPT can act as a laboratory to get data on many other questions in transformer circuits - it's simple enough to have a ground truth, yet complex enough to be interesting
    My hope is that some people reading this are interested enough to actually try working on these problems, and I end this section with advice on where to start.
    Why and when to work on toy models
    This is a long and rambly section about my research philosophy of mech interp, and you should feel free to move on to the next section if that's not your jam
    At first glance, playing legal moves in Othello (not even playing good moves!) has nothing to do with language models, and I think this is a strong claim worth justifying. Can working on toy tasks like Othello-GPT really help us to reverse-engineer LLMs like GPT-4? I'm not sure! But I think it's a plausible bet worth making.
    To walk through my reasoning, it's worth first thinking on what's holding us back - why haven't we already reverse-engineered the most capable models out there? I'd personally point to a few key factors (though note that this is my personal hot take, is not comprehensive, and I'm sure other researchers have their own views!):
    Conceptual frameworks: To reverse-engineer a transformer, you need to know how to think like a transformer. Questions like: What kinds of algorithms is it natural for a transformer to represent, and how? Are features and circuits the right way to think about it? Is it even reasonable to expect that reverse-engineering is possible? How can we tell if a hypothesis or technique is principled vs hopelessly confused? What does it even mean to have truly identified a feature or circuit?
    I personally thought A Mathematical Framework significantly clarified my conceptual frameworks for transformer circuits!
    This blog post is fundamentally motivated by forming better conceptual frameworks - do models form linear representations?
    Practical Knowledge/Techniques: Understanding models is hard, and being ab...
    51 min
  • AF - Actually, Othello-GPT Has A Linear Emergent World Representation by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Actually, Othello-GPT Has A Linear Emergent World Representation, published by Neel Nanda on March 29, 2023 on The AI Alignment Forum.
    Epistemic Status: This is a write-up of an experiment in speedrunning research, and the core results represent ~20 hours/2.5 days of work (though the write-up took way longer). I'm confident in the main results to the level of "hot damn, check out this graph", but likely have errors in some of the finer details.
    Disclaimer: This is a write-up of a personal project, and does not represent the opinions or work of my employer
    This post may get heavy on jargon. I recommend looking up unfamiliar terms in my mechanistic interpretability explainer
    Thanks to Chris Olah, Martin Wattenberg, David Bau and Kenneth Li for valuable comments and advice on this work, and especially to Kenneth for open sourcing the model weights, dataset and codebase, without which this project wouldn't have been possible! Thanks to ChatGPT for formatting help.
    Overview
    Context: A recent paper trained a model to play legal moves in Othello by predicting the next move, and found that it had spontaneously learned to compute the full board state - an emergent world representation.
    This could be recovered by non-linear probes but not linear probes.
    We can causally intervene on this representation to predictably change model outputs, so it's telling us something real
    I find that actually, there's a linear representation of the board state!
    But that rather than "this cell is black", it represents "this cell has my colour", since the model plays both black and white moves.
    We can causally intervene with the linear probe, and the model makes legal moves in the new board!
    This is evidence for the linear representation hypothesis: that models, in general, compute features and represent them linearly, as directions in space! (If they don't, mechanistic interpretability would be way harder)
    The original paper seemed at first like significant evidence for a non-linear representation - the finding of a linear representation hiding underneath shows the real predictive power of this hypothesis!
    This (slightly) strengthens the paper's evidence that "predict the next token" transformer models are capable of learning a model of the world.
    Part 2: There's a lot of fascinating questions left to answer about Othello-GPT - I outline some key directions, and how they fit into my bigger picture of mech interp progress
    Studying modular circuits: A world model implies emergent modularity - many early circuits together compute a single world model, many late circuits each use it. What can we learn about what transformer modularity looks like, and how to reverse-engineer it?
    Prior transformer circuits work focuses on end-to-end circuits, from the input tokens to output logits. But this seems unlikely to scale!
    I present some preliminary evidence reading off a neuron's function from its input weights via the probe
    Neuron interpretability and Studying Superposition: Prior work has made little progress on understanding MLP neurons. I think Othello GPT's neurons are tractable to understand, yet complex enough to teach us a lot!
    I further think this can help us get some empirical data about the Toy Models of Superposition paper's predictions
    I investigate max activating dataset examples and find seeming monosemanticity, yet deeper investigation show it seems more complex.
    A transformer circuit laboratory: More broadly, the field has a tension between studying clean, tractable yet over-simplistic toy models and studying the real yet messy problem of interpreting LLMs - Othello-GPT is toy enough to be tractable yet complex enough to be full of mysteries, and I detail many more confusions and conjectures that it could shed light on.
    Part 3: Reflections on the research process
    I did the bulk of this proje...
    31 min
  • EA - Linkpost: Italy introduces bill to ban lab-grown meat by Matt Goodman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Linkpost: Italy introduces bill to ban lab-grown meat, published by Matt Goodman on March 29, 2023 on The Effective Altruism Forum.
    This is a linkpost for a Reuters article about a new bill proposed in Italy's parliament to ban lab-grown meat. I don't have much to say about it, except that I hope it fails, and I hope this isn't the start of a culture war around lab-grown meat.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - The billionaires’ philanthropy index by brb243
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The billionaires’ philanthropy index, published by brb243 on March 29, 2023 on The Effective Altruism Forum.
    Summary
    I made a spreadsheet that allows anyone to find who of the about 2,500 billionaires donates most and least consistently with the user’s personal values (roughly, records might be inaccurate or incomplete).
    The billionaires’ philanthropy index indexes billionaires’ philanthropy based on the causes they donated to and the amounts they pledged. It displays top donors using up to 52 user’s inputs of their relative valuation of contribution to/impact in 12 cause areas divided into subcategories. It is possible to account for developed/developing country impact cost differential.
    This post displays billionaires’ philanthropy summary statistics. I also discuss the limitations of the indexing method and suggest improvements. Information about possible partnerships with billionaires’ foundations in EA-popular fields is included. For inspiration, I am providing personal value inputs that an individual involved in Effective Altruism can have, alongside with these values’ ITN-based calculation and reasoning.
    The three documents shared are:
    The billionaires’ philanthropy index
    User manual to The billionaires’ philanthropy index
    Value examples for The billionaires’ philanthropy index
    Superlinear contest
    This submission is for the Superlinear Create 'The Good Billionaires Index' contest.
    I was awarded ⅔ of the prize for partially completed work. ⅓ could be remaining (if the prize is listed on Superlinear) for someone who makes the user interface more friendly. Feel free to use the material for this purpose, but do not use fear-, power-, or impulse-based marketing.
    It was suggested that the material is open for comments.
    Notes on terminology and purpose
    I use “donate,” “pledge,” and “donate or pledge” (and similar expressions) interchangeably.
    I use “subcategory” and “minor category” interchangeably.
    This piece, the index, the manual, and the value examples are for entertainment purposes only. Nothing in any of these resources constitutes a donation, career, investment/divestment, or any other advice.
    Data sources
    The main sources of data are:
    Forbes Real-Time Billionaires
    Giving Pledge
    The full list of data sources for each column/cell can be found in the “Calculations” tab of the index.
    Research methodology
    Copied and pasted the Real-Time Billionaires Forbes list into Sheets
    Recorded each billionaire’s donations based on their Forbes profile and additional research, if the profile indicated or hinted philanthropic interests but did not specify donations
    Added pledge information based on the Giving Pledge Signatory letters
    Checked whether any major donations are omitted on top donor lists, including those of Wikipedia, Forbes (2, 3), The Borgen Project, and The New Humanitarian. Added these donations.
    Donation categories
    I categorized each billionaire's donations, if any.
    I used 13 major categories and 4 different subcategory lists.
    The total in all major categories and any subcategory list is 100.
    The value assigned to each category or subcategory represents the number of percent of the billionaire’s total donations provided to that category or subcategory (see the example further below).
    Major categories
    Education
    Health
    Economic/social empowerment
    Environmental causes
    Animal welfare
    Art/culture
    Religious causes
    Political causes
    Disaster relief
    Local causes
    Meta charity
    Fraud
    Other
    Minor category lists
    Geographical location list
    Each major category, except Meta charity, Fraud, and Other, is subdivided into 3 geographical locations:
    US
    Non-US developed
    Developing
    Education subcategory list
    In addition, education is categorized as either:
    Higher
    K-12
    Health subcategory list
    Health donations can support either:
    Physical health
    Mental health
    Animal welfare subcategory ...
    49 min
  • EA - Nathan A. Sears (1988-2023) by HaydnBelfield
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nathan A. Sears (1988-2023), published by HaydnBelfield on March 29, 2023 on The Effective Altruism Forum.
    Nathan Sears was one of seven to die in a fire in Montreal on the 16 March 2023. He was 35.
    Nathan was becoming a leading figure at the intersection of existential risk and international relations (IR).
    Indeed, he was in Montreal to attend the 2023 International Studies Association (ISA) conference, the leading conference on international relations (IR). The day before on March 15th, he presented a paper on "Great Power Rivalry and Human Survival: Why States Fail to “Securitize” Existential Threats to Humanity" at a panel on 'Catastrophic-Existential Risks and World Orders'
    After his undergrad at Western University and his Masters in IR at Carleton University Nathan moved to Quito, Ecuador. For four years he taught IR at the Universidad de Las Américas. He then came back to Canada in 2016 to earn his PhD.
    During that time he was a 2017-2018 Trudeau Centre Fellow in Peace, Conflict and Justice at the Munk School of Global Affairs. He also took a year out to serve his country as a 2019-2020 Cadieux-Léger Fellow in the Foreign Policy Research and Foresight Division of Global Affairs Canada.
    Nathan was already an important scholar in the field of existential risk, making groundbreaking & much-discussed contributions at the intersection with international relations. He was also a really friendly, supportive and engaging guy. I was so excited about what he was going to accomplish.
    Five of his most important papers are:
    International Politics in the Age of Existential Threats
    Humans in the twenty-first century live under the specter of anthropogenic existential threats to human civilization and survival. What is the significance of humanity’s capacity for self-destruction to the meaning of “security” and “survival” in international politics? The argument is that it constitutes a material “revolution” in international politics—that is, the growing spectrum of anthropogenic existential threats represents a radical transformation in the material context of international politics that turns established truths about security and survival on their heads. The paper develops a theoretical framework based in historical security materialism, especially the theoretical proposition that the material circumstances of the “forces of destruction” determine the security viability of different “modes of protection”, political “units” and “structures”, and “security ideologies” in international politics.
    The argument seeks to demonstrate the growing disjuncture (or "contradiction") between the material context of anthropogenic existential threats ("forces of destruction"); and the security practices of war, the use of military force, and the balance-of-power ("modes of protection"); the political units of nation-states and structure of international anarchy ("political superstructure"); and the primacy of "national security" and doctrines of "self-help" and "power politics" in international politics ("security ideologies"). Specifically, humanity's survival interdependence with respect to an-thropogenic existential threats calls into question the centrality of national security and survival in international politics. In an age of existential threats, "security" is better understood as about the survival of humanity.
    Existential Security: Towards a Security Framework for the Survival of Humanity
    Humankind faces a growing spectrum of anthropogenic existential threats to human civilization and survival. This article therefore aims to develop a new framework for security policy – ‘existential security’ – that puts the survival of humanity at its core. It begins with a discussion of the definition and spectrum of ‘anthropogenic existential threats’, or those threats that have their origins i...
    9 min
  • EA - Want to win the AGI race? Solve alignment. by leopold
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Want to win the AGI race? Solve alignment., published by leopold on March 29, 2023 on The Effective Altruism Forum.
    Society really cares about safety. Practically speaking, the binding constraint on deploying your AGI could well be your ability to align your AGI. Solving (scalable) alignment might be worth lots of $$$ and key to beating China.
    Look, I really don't want Xi Jinping Thought to rule the world. If China gets AGI first, the ensuing rapid AI-powered scientific and technological progress could well give it a decisive advantage (cf potential for >30%/year economic growth with AGI). I think there's a very real specter of global authoritarianism here.
    Or hey, maybe you just think AGI is cool. You want to go build amazing products and enable breakthrough science and solve the world’s problems.
    So, race to AGI with reckless abandon then? At this point, people get into agonizing discussions about safety tradeoffs. And many people just mood affiliate their way to an answer: "accelerate, progress go brrrr," or "AI scary, slow it down."
    I see this much more practically. And, practically, society cares about safety, a lot. Do you actually think that you’ll be able to and allowed to deploy an AI system that has, say, a 10% chance of destroying all of humanity?
    Society has started waking up to AGI; like covid, the societal response will probably be a dumpster-fire, but it’ll also probably be quite intense. In many worlds, to deploy your AGI systems, people will need to be quite confident that your AGI won’t destroy the world.
    Right now, we’re very much not on track to solve the alignment problem for superhuman AGI systems (“scalable alignment”)—but it’s a solvable problem, if we get our act together. I discuss this in my main post today (“Nobody’s on the ball on AGI alignment”). On the current trajectory, the binding constraint on deploying your AGI could well be your ability to align your AGI—and this alignment solution being unambiguous enough that there is consensus that it works.
    Even if you just want to win the AGI race, you should probably want to invest much more heavily in solving this problem.
    Things are going to get crazy, and people will pay attention
    A mistake many people make when thinking about AGI is imagining a world that looks much like today, except for adding in a lab with a super powerful model. They ignore the endogenous societal response.
    I and many others made this mistake with covid—we were freaking out in February 2020, and despairing that society didn’t seem to be even paying attention, let alone doing anything. But just a few weeks later, all of America went into an unprecedented lockdown. If we're actually on our way to AGI, things are going to get crazy. People are going to pay attention.
    The wheels for this are already in motion. Remember how nobody paid any attention to AI 6 months ago, and now Bing chat/Sydney going awry is on the front page of the NYT, US senators are getting scared, and Yale econ professors are advocating $100B/year for AI safety? Well, imagine that, but 100x as we approach AGI.
    AI safety is going mainstream. Everyone has been primed to be scared about rogue AI by science fiction; all the CEOs have secretly believed in AI risk for years but thought it was too weird to talk about it; and the mainstream media loves to hate on tech companies. Probably there will be further, much scarier wakeup calls (not just misalignment, but also misuse and scary demos in evals). People already freaked out about GPT-4 using a TaskRabbit to solve a captcha—now imagine a demo of AI systems designing a new bioweapon or autonomously self-replicating on the internet, or people using AI coders to hack major institutions like the government or big banks. Already, a majority of the population says they fear AI risk and want FDA-style regulation ...
    9 min
  • EA - Some updates to my thinking in light of the FTX collapse by Owen Cotton Barratt [Link Post] by Nathan Young
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some updates to my thinking in light of the FTX collapse by Owen Cotton Barratt [Link Post], published by Nathan Young on March 29, 2023 on The Effective Altruism Forum.
    Some quick notes
    Owen Cotton Barratt has written a post of reflections about FTX
    He didn’t want to post them
    These seem obviously valuable for people to read, if they wish
    Personally I don’t think revelations about Owen’s actions much affect my opinion of the value of the interesting things he might say about FTX
    But neither do interesting things he has to say about FTX affect his actions (more below)
    So this is a link post. This is my choice, with his permission.
    On balance I think it's better to link to the post, if the consensus is that I should post the whole post (it's quite long) I will
    Equally there is a discussion to be had about whether Owen’s content should be on the forum or indeed further discussion of the whole situation, feel free to have that discussion here. The mod team has suggested (and I cautiously endorse) having a dedicated comment thread on this post for meta-discussion about Owen, details below.
    I think this could be seen as soft rehabilitation. I don’t endorse that
    Personally I think there is at least one more large discussion necessary about Owen’s actions. If the manner this piece was posted was used to avoid that, I’d be upset
    You may disagree with either my or the moderation team’s choices here. I am happy to discuss the point - perhaps I’m wrong
    As elsewhere I think it may be helpful to split up thoughts and feelings, personally I think my feelings do not automatically translate into a need for action. Feelings are important, but they are only part of good arguments.
    [A note from the moderation team] We realize that some people might want to discuss how to process this post in light of Owen's recent statement and apology. But we also want to give space to object-level discussion of the contents of the post, and separate those out somewhat. So we ask that you avoid commenting on Owen's recent apology anywhere but in this thread. New top-level comments (and responses to them) should focus on the contents of the post; if they don't, we'll move them to said thread.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min