The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - So, geez there's a lot of AI content these days by Raemon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: So, geez there's a lot of AI content these days, published by Raemon on October 6, 2022 on LessWrong.
    Since April this year, there's been a huge growth in the the number of posts about AI, while posts about rationality, world modeling, etc. have remained constant. The result is that much of the time, the LW frontpage is almost entirely AI content.
    Looking at the actual numbers, we can see that during 2021, no core LessWrong tags represented more than 30% of LessWrong posts. In 2022, especially starting around April, AI has started massively dominating the LW posts.
    Here's the total posts for each core tag each month for the past couple years. On April 2022, most tags' popularity remains constant, but AI-tagged posts spike dramatically:
    Even people pretty involved with AI alignment research have written to say "um, something about this feels kinda bad to me."
    I'm curious to hear what various LW users think about the situation. Meanwhile, here's my own thoughts.
    Is this bad?
    Maybe this is fine.
    My sense of what happened was that in April, Eliezer posted MIRI announces new "Death With Dignity" strategy, and a little while later AGI Ruin: A List of Lethalities. At the same time, PaLM and DALL-E 2 came out. My impression is that this threw a brick through the overton window and got a lot of people going "holy christ AGI ruin is real and scary". Everyone started thinking a lot about it, and writing up their thoughts as they oriented.
    Around the same time, a lot of alignment research recruitment projects (such as SERI MATS or Refine) started paying dividends, and resulting in a new wave of people working fulltime on AGI safety.
    Maybe it's just fine to have a ton of people working on the most important problem in the world?
    Maybe. But it felt worrisome to Ruby and me. Some of those worries felt easier to articulate, others harder. Two major sources of concern:
    There's some kind of illegible good thing that happens when you have a scene exploring a lot of different topics. It's historically been the case that LessWrong was a (relatively) diverse group of thinkers thinking about a (relatively) diverse group of things. If people show up and just see the All AI All the Time, people who might have other things to contribute may bounce off. We probably wouldn't lose this immediately
    AI needs Rationality, in particular. Maybe AI is the only thing that matters. But, the whole reason I think we have a comparative advantage at AI Alignment is our culture of rationality. A lot of AI discourse on the internet is really confused. There's such an inferential gulf about what sort of questions are even worth asking. Many AI topics deal with gnarly philosophical problems, while mainstream academia is still debating whether the world is naturalistic. Some AI topics require thinking clearly about political questions that tend to make people go funny in the head.
    Rationality is for problems we don't know how to solve, and AI is still a domain we don't collectively know how to solve.
    Not everyone agrees that rationality is key, here (I know one prominent AI researcher who disagreed). But it's my current epistemic state.
    Whispering "Rationality" in your ear
    Paul Graham says that different cities whisper different ambitions in your ear. New York whispers "be rich". Silicon Valley whispers "be powerful." Berkeley whispers "live well." Boston whispers "be educated."
    It seems important for LessWrong to whisper "be rational" in your ear, and to give you lots of reading, exercises, and support to help you make it so.
    As a sort of "emergency injection of rationality", we asked Duncan to convert the CFAR handbook from a PDF into a more polished sequence, and post it over the course of a month. But commissioning individual posts is fairly expensive, and over the past couple months the LessWrong team's foc...
    11 min
  • AF - A shot at the diamond-alignment problem by Alex Turner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A shot at the diamond-alignment problem, published by Alex Turner on October 6, 2022 on The AI Alignment Forum.
    I think that relatively simple alignment techniques can go a long way. In particular, I want to tell a plausible-to-me story about how simple techniques can align a proto-AGI so that it makes lots of diamonds.
    But why is it interesting to get an AI which makes lots of diamonds? Because we avoid the complexity of human value and thinking about what kind of future we want, while still testing our ability to align an AI. Since diamond-production is our goal for training, it’s actually okay (in this story) if the AI kills everyone. The real goal is to ensure the AI ends up acquiring and producing lots of diamonds, instead of just optimizing some weird proxies that didn’t have anything to do with diamonds. It’s also OK if the AI doesn’t maximize diamonds, and instead just makes a whole lot of diamonds.
    Someone recently commented that I seem much more specifically critical of outer and inner alignment, than I am specifically considering alternatives. So, I had fun writing up a very specific training story for how I think we can just solve diamond-alignment using extremely boring, non-exotic, simple techniques, like “basic reward signals & reward-data augmentation.” Yes, that’s right. As I’ve hinted previously, I think many arguments against this working are wrong, but I’m going to lay out a positive story in this post. I’ll reserve my arguments against certain ideas for future posts.
    Can we tell a plausible story in which we train an AI, it cares about diamonds when it’s stupid, it gets smart, and still cares about diamonds? I think I can tell that story, albeit with real uncertainties which feel more like normal problems (like “ensure a certain abstraction is learned early in training”) than impossible-flavored alignment problems (like “find/train an evaluation procedure which isn’t exploitable by the superintelligence you train”).
    Before the story begins:
    Obviously I’m making up a lot of details, many of which will turn out to be wrong, even if the broad story would work. I think it’s important to make up details to be concrete, highlight the frontiers of my ignorance, and expose new insights. Just remember what I’m doing here: making up plausible-sounding details.
    This story did not actually happen in reality. It’s fine, though, to update towards my models if you find them compelling.
    This is not my best guess for how we get AGI. In particular, I think chain of thought / language modeling is more probable than RL, but I’m still more comfortable thinking about RL for the moment, so that’s how I wrote the story.
    The point of the following story is not “Gee, I sure am confident this story goes through roughly like this.” Rather, I am presenting a training story template I expect to work for some foreseeably good design choices. I would be interested and surprised to learn that this story template is not only unworkable, but comparably difficult to other alignment approaches as I understand them.
    This story is set in Evan Hubinger’s training stories format. I'll speak in the terminology of shard theory.
    A diamond-alignment story which doesn’t seem fundamentally blocked
    Training goal
    I want an AI which makes lots of diamonds. In particular, the AI should secure its future diamond production against non-diamond-aligned AI.
    Training rationale
    Use a very large (future) multimodal self-supervised learned (SSL) initialization to give the AI a latent ontology for understanding the real world and important concepts. Combining this initialization with a recurrent state and an action head, train an embodied AI to do real-world robotics using imitation learning on human in-simulation datasets and then sim2real. Since we got a really good pretrained initialization, there's re...
    36 min
  • EA - Invisible impact loss (and why we can be too error-averse) by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Invisible impact loss (and why we can be too error-averse), published by Lizka on October 6, 2022 on The Effective Altruism Forum.
    TL;DR: It’s hard to really feel sad about impact that doesn’t happen, but it’s easy to feel sad about mistakes in existing work.
    This is bad because we sometimes have a choice between making a good-but-small thing or a big-but-slightly-worse thing, and the mismatch in our emotional reactions to the downsides of the two options turns into a cognitive bias that harms our decision-making. This can make us too error-averse and err too much on the side of just not doing things at all.
    For instance, if I’m choosing between making five imperfect cookies or one perfect cookie, I might notice the imperfections more than the four missing cookies, whereas it’s often better to just make more imperfect cookies.
    In some cases, we should be quite wary of mistakes. I outline where I think we’re too error-averse and where error-aversion is appropriate below.
    Summary of my concrete suggestions:
    To counteract fear of criticism, foster a culture of celebrating successes and exciting work.
    To notice invisible impact loss, have the phrase “invisible impact loss” rattling inside your head, and try to quantify decisions that are prone to such errors.
    Fight perfectionism; lean into agile project management, start half-assing, and try to determine what your real goals are.
    Preamble: What sparked this thought
    When I was on the Events Team at CEA, we had a conversation about whether to approximately double EA Global capacity at the last minute (a few weeks before the actual event) in order to admit more people who met the admissions bar. (This was the most recent conference for which capacity was a problem.)
    I remember starting off skeptical about the idea. After all,
    There would be more logistics issues; lunch lines would be long, etc.
    We’d have to move to two venues, which is awkward. People would have to walk from one venue to another to get to a different meeting or session.
    Doubling would put a lot of strain on the team, which would mean we’d be worse at responding to people, communicating things like the schedule or important COVID information, catching awesome ways to improve the conference.
    And in general, the experience of the average attendee would probably be worse.
    But someone else on the team made me realize a big thing I wasn’t fully tracking:
    About 500 more people who met the admissions bar would get to experience the conference. However much impact the default conference would produce — sparked collaborations, connections for years to come, inspirations for new projects, positive career changes — there would be about double that.
    Although the whole conversation was sparked by the idea that we had way more people we wanted to admit than slots for attendees, I wasn’t internalizing the benefit we would miss out on if we didn’t double. I couldn’t viscerally feel the loss of impact from a smaller but near-perfect conference the way I could feel my aversion to the various issues I could imagine with the bigger conference. The impact loss was invisible to me.
    I now actively try to notice invisible impact loss. I’ve noticed discussions on the Forum that miss this, and worry that it’s stunting work that could be extremely valuable, so I decided to write this post.
    Note: EA Global conferences have changed since then. You can see some discussion of this here. I should also note that the conference didn’t end up facing many of the issues I worried about while we were deciding whether to double.
    What causes this phenomenon? How can we push back? (Suggestions)
    Here are some factors I think are at play, and ideas for what we can do about them. (Please add to this in the comments!) See the summary at the top.
    1. Fear of criticism
    We’re much, much more likely to be c...
    15 min
  • AF - More Recent Progress in the Theory of Neural Networks by jylin04
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More Recent Progress in the Theory of Neural Networks, published by jylin04 on October 6, 2022 on The AI Alignment Forum.
    Thanks to Dan Roberts and Sho Yaida for comments on a draft of this post.
    In this post, I would like to draw attention to the book Principles of Deep Learning Theory (PDLT), which I think represents a significant advance in our understanding of how neural networks work . Among other things, this book explains how to write a closed-form formula for the function learned by a realistic, finite-width neural network at the end of training to an order of approximation that suffices to describe representation learning, and how that formula can be interpreted as the solution to a regression model. This makes manifest the intuition that NNs are doing something like regression, but where they learn the features appropriate for a given dataset rather than having them be hand-engineered from the start.
    I've condensed some main points from the 400-page book into an 8-page summary here:
    Review of select results from PDLT
    (Other good places to learn about the book, though perhaps with less of a focus on AI-safety-relevant parts, include this series of five lectures given by the authors at a deep learning summer school or this one-hour lecture for a non-expert audience.)
    For those who have been following the discussions of ML theory on this forum, the method used in the book is to go to the next-to-leading order in a 1/width expansion. It thus builds on recent studies of infinitely wide NNs that were reviewed in the AF post Recent Progress in the Theory of Neural Networks . However, by going beyond the leading order, the authors of PDLT are able to get around a key qualitative shortcoming of the earlier work in that infinitely wide NNs can't learn features. The next-to-leading order formula also introduces a sum over many steps of gradient descent, getting around an objection that the NTK/infinite width limit may not be applicable to realistic models since in that limit, we can land on the fully trained model after just one fine-tuned training step.
    I think that this work could have significant implications for AGI forecasting and safety (via interpretability), and deserves to be better appreciated in this community. For example,
    In AGI forecasting, an important open question is whether the strong scaling hypothesis holds for any modern architectures. (For example, the forecasts in Ajeya Cotra's Bio-Anchors report are conditioned on assuming that 2020 algorithms can scale to TAI.) A longstanding challenge for this field is that as long as we treat neural networks as black boxes or random program search, it's hard to reason about this question in a principled way. But I think that by identifying a space of functions that realistic NNs end up learning in practice ( the space of all neural networks with finely-tuned weights!), the approach of PDLT gives us a way to start to reason about it. For example, despite the existence of the universal approximation theorem, I think the results of PDLT can be used to rule out the (strawmannish) hypothesis that feedforward MLPs can scale to AGI (see my review of the Bio-Anchors report for more on this point).
    As such, it could be really interesting to generalize PDLT to other architectures.
    In mechanistic interpretability, a basic open question is what the fundamental degrees of freedom are that we should be trying to interpret. A lot of work has been done under the assumption that we should look at the activations of individual neurons, but there's naively no reason that semantically meaningful properties of a dataset must align with individual neurons after training, and even some interesting counterexamples . By finding a dual description of a trained NN as a trained regression model, PDLT seems to hint that a (related, but)...
    7 min
  • EA - Research Deprioritizing External Communication by Jeff Kaufman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Research Deprioritizing External Communication, published by Jeff Kaufman on October 6, 2022 on The Effective Altruism Forum.
    This was originally written in December 2021. I think it's good practice to run posts by orgs before making them public, and I did in this case. This has some benefits: orgs aren't surprised, they can prepare a response in advance if they want to, they can point out errors before things become public, etc, and I think it's generally worth doing. In this case Leverage folks pointed out some errors, omissions, and bad phrasing in my post, which I've fixed, and I'm thankful for their help. Pre-publication review does also have downsides, however, and in this case as the email conversations grew to 10k+ words over three weeks I ran out of time and motivation.
    A month ago I came across this in my list of blog drafts and decided to publish it as-is with a note at the top explaining the situation. This means that it doesn't cover any more recent Leverage developments, including their Experiences
    Inquiry Report and On
    Intention Research paper, both published in April 2022. I shared this post again with Leverage, and while I've made edits in response to their feedback they continue to disagree with my conclusion.
    In the original pre-publication discussion with Geoff, one of the topics was whether we could make our disagreement more concrete with a bet. For example, research that launches a new subfield generally gets lots of citations, such as the the Concrete Problems paper (1,644 citations at 6 years), and if Leverage 1.0's research ends up having this kind of foundational impact this could be a clear way to tell. When I gave Leverage a second pre-publication heads up, Geoff and I talked more and we were able to nail down some terms: if a
    Leverage paper drawing primarily on their pre-2019 research has 100+ citations from people who've never worked for Leverage by 2032-10-01, then I'll donate $100 to a charity of Geoff's choosing; if not then Geoff will do the same for a charity of my choosing. I've listed this on my bets page.
    In 2011, several people I knew through Boston-area effective altruism and rationality meetups started an organization called Leverage Research. Their
    initial goal was to "make the world a much better place, using the most effective means we can", and they worked on a wide range of projects, but they're probably best known for trying to figure out how to make people more productive/capable/successful by better understanding how people think and interact. They initially lived and worked in a series of shared houses, first in Brooklyn and then in Oakland; I visited the latter for an evening in early 2014. The core project ("Leverage
    1.0") disintegrated in 2019, with some portions continuing, including
    Paradigm (training/coaching) and Leverage 2.0 (early stage science). In this post I'm only looking at Leverage 1.0, and specifically at their psychology research program.
    In mid-December 2021, Leverage's former head of operations, Cathleen, wrote In
    Defense of Attempting Hard Things: and my story of the Leverage ecosystem (LW comments), giving a detailed history with extensive thoughts on many aspects of the project. I remember her positively from my short 2014 visit, and I'm really glad she took the time to write this up.
    There are many directions from which people could approach
    Leverage 1.0, but the one that I'm most interested in is lessons for people considering attempting similar things in the future.
    My overall read of Cathleen's post is that she (and many other ex-Leverage folks) view the project as one where a group of people took an unorthodox approach to research, making many deep and important discoveries about how people think and relate to each other. I've read the Connection Theory paper and the four research reports
    G...
    14 min
  • LW - A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox by Henrik Karlsson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox, published by Henrik Karlsson on October 5, 2022 on LessWrong.
    I was born in July 1989, which means I am of the last generation who will remember the time before the internet. The cables and data centers and hyperlinks grew up around me; they grew with me. I find it hard to disentangle the evolution of my psyche from that of the internet.
    Explaining it to my daughter, who was born in 2017, a year when the world’s largest economy had begun tearing itself apart from the tension of this ever-evolving network, I tell her that the internet is like an alien intelligence. We don’t know exactly what it is; it has just landed, and only the first ship. We are trying to figure out how to talk to it. The first generation of explorers have noted that by making certain finger motions you can make the aliens show you images of cats and clothes, or tell you all the ways the world is falling apart.
    For a long time, I thought this was all there was to it. I could tap the keyboard in a particular way and the screen would show me the weather, or tell me which translation of the Iliad to read and then make someone jump in a truck and drive it to our house. I preferred the Iliad to the screen.
    But then, late 2021, after I had been making intricate finger movements again, I woke up in our guesthouse before sunrise and noticed that something had changed.
    During the night, the internet had been set in motion. Tossing hither and thither in silence—as the fields lay frozen and waiting and the hedgehog slept in its pile of leaves—the internet had rearranged itself around me.
    I had written an essay about Ivan Illich and systems thinking, a topic I had never found anyone else intrigued by, and which magazines thought below a rejection letter—and the internet had suddenly reshaped itself so that my keyboard hooked up to the screens of a bunch of people who wanted to talk about these topics, and a little later, their keyboards hooked up to mine.
    I had written for 15 years, but never before had this happened. I had conjured a minor conference! And I hadn’t even known that you could do that.
    This gave me a first glimpse of the social mechanics of the internet. Looking at the traffic data, and talking to readers, I could retrace how my words had traveled through the network, and I got a sense of why. I didn’t fully understand it; I don’t think anyone does. But like a scientist who’s got hold of an alien artifact, I proceeded by gleefully and semi-randomly pushing every button I could find to see what happened. I would think of a series of funny finger movements and then I’d say to myself, LOL I wonder what this combination does? And then I’d try.
    The way the machine seemed to work was:
    The more precise and niche the words I input, the better the internet would match me with people I could forge meaningful relationships with. This precision was hard for me, partly because my sense for how communication is supposed to work is shaped by reading mass media. Writing for a general public, you need to be broad and a bit bland. I didn’t want a general public. I wanted a specific set of people, the people who could help me along as a human being obsessed with certain intellectual problems. I didn’t know who these people were. I only knew that they existed. Hence my writing was a search query. It needed to be phrased in such a way that it found these people and, if necessary, filtered others.
    The pleasant parts of the internet seemed to be curated by human beings, not algorithms. For my writing to find its way in this netherworld, I needed to have a rough sense of how information flowed down there. The pattern was this: words flowed from the periphery to the centers. This was a surp...
    17 min
  • EA - Warning Shots Probably Wouldn't Change The Picture Much by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Warning Shots Probably Wouldn't Change The Picture Much, published by So8res on October 6, 2022 on The Effective Altruism Forum.
    One piece of advice I gave to EAs of various stripes in early 2021 was: do everything you can to make the government sane around biorisk, in the wake of the COVID pandemic, because this is a practice-run for AI.
    I said things like: if you can't get the world to coordinate on banning gain-of-function research, in the wake of a trillions-of-dollars tens-of-millions-of-lives pandemic "warning shot", then you're not going to get coordination in the much harder case of AI research.
    Biolabs are often publicly funded (rather than industry-funded). The economic forces arrayed behind this recklessly foolish and impotent research consists of “half-a-dozen researchers thinking it’s cool and might be helpful”. (While the work that would actually be helpful—such as removing needless bureaucracy around vaccines and investing in vaccine infrastructure—languishes.) Compared to the problem of AI—where the economic forces arrayed in favor of “ignore safety and rush ahead” are enormous and the argument for expecting catastrophe much murkier and more abstract—the problem of getting a sane civilizational response to pandemics (in the wake of a literal pandemic!) is ridiculously easier.
    And—despite valiant effort!—we've been able to do approximately nothing.
    We're not anywhere near global bans on gain-of-function research (or equivalent but better feats of coordination that the people who actually know what they're talking about when it comes to biorisk would tell you are better targets than gain-of-function research).
    The government continues to fund research that is actively making things worse, while failing to put any serious funding towards the stuff that might actually help.
    I think this sort of evidence has updated a variety of people towards my position. I think that a variety of others have not updated. As I understand the counter-arguments (from a few different conversations), there are two main reasons that people see this evidence and continue to hold out hope for sane government response:
    1. Perhaps the sorts of government interventions needed to make AI go well are not all that large, and not that precise.
    I confess I don't really understand this view. Perhaps the idea is that AI is likely to go well by default, and all the government needs to do is, like, not use anti-trust law to break up some corporation that's doing a really good job at AI alignment just before they succeed? Or perhaps the idea is that AI is likely to go well so long as it's not produced first by an authoritarian regime, and working against authoritarian regimes is something governments are in fact good at?
    I'm not sure. I doubt I can pass the ideological Turing test of someone who believes this.
    2. Perhaps the ability to cause governance to be sane on some issue is tied very directly to the seniority of the government officials advising sanity.
    EAs only started trying to affect pandemic policy a few years ago, and aren't very old or recognized among the cacophony of advisors. But if another pandemic hit in 20 years, the sane EA-ish advisors would be much more senior, and a lot more would get done. Similarly, if AI hits in 20 years, sane EA-ish advisors will be much more senior by then. The observation that the government has not responded sanely to pandemic near-misses, is potentially screened-off by the inexperience of EAs advising governance.
    I have some sympathy for the second view, although I'm skeptical that sane advisors have significant real impact. I'd love a way to test it as decisively as we've tested the "government (in its current form) responds appropriately to warning shots" hypotheses.
    On my own models, the "don't worry, people will wake up as the cliff-edge com...
    5 min
  • AF - confusion about alignment requirements by Tamsin Leake
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: confusion about alignment requirements, published by Tamsin Leake on October 6, 2022 on The AI Alignment Forum.
    for now, let's put aside the fact that we can't decide whether we're trying to achieve sponge coordination or FAS, and merely consider what it takes to build an aligned AI — regardless of whether it has the capability of saving the world as a singleton, or merely to be a useful but safe tool.
    the question this post is about is: what requirements do we want such a solution to satisfy?
    let's say three groups have each built an AI which they think is aligned, and before they press the start button on it, they're trying to convince the other two that their design is safe and leads to good worlds. however, their designs are actually very different from one another.
    maybe one is an advanced but still overall conventional text-predicting simulator, another is a clever agentic neural net with reinforcement learning and access to a database and calculator, and the third is a novel kind of AI whose core doesn't really relate to current machine learning technology.
    so, they start talking about why they think they AI is aligned. however, they run into an issue: they don't even agree on what it takes to be sure an AI is safe, let alone aligned!
    maybe one of them has a proof that their AI is resistant to a reasonable class of acausal attacks, another has reasons to think their approach probly avoids the issue altogether somehow, and the third has a model of the world that fails to understand acausal attacks and rejects their possibility altogether.
    maybe one of them has developed a world-modeling system that is general enough to support embedded agency, another has patched theirs to support it as a special case, and the third think their AI will simply modify itself that way because it's instrumenally convergent.
    maybe one of them has gone out and built a decision system which implements FDT, another counts on CDT turning itself into FDT, and the third has no idea how to determine how their system fits into decision theories.
    maybe one of them has built something that is confidently eventually aligned and hopefully enough continuously aligned, another has built something that is acting safely now and has a bunch of ad-hoc corrigibility devices which hopefully prevent it from taking a sharp left turn, and the third expects their AI to robustly keep being safe in the long run for reasons that seem hard to understand.
    maybe one of them has built their AI to respect the values of its creator, another has made the AI care about the part of its model that they believe to be pointing to an abstraction of human values, and the third has an AI that simply takes orders from what it interacts with and can hopefully be ordered to self-modify in a way that makes it resistant to alien superintelligences by the time it meets them.
    and those are optimistic cases! many alignment approaches would simply:
    fail to consider that their design might fail in ways they haven't thought of
    not think to ask the alignment community at large whether their design is safe
    ask the community, but then only select criticisms they take into account based on their ability to understand those criticisms, rather than based on their importance
    assume away the possible failure modes that would be brought up
    accidentally kill everyone way before any of this happens
    i've noticed this pattern of confusion in myself after trying to explain alignment ideas i've found promising to some people, and the nature of their criticism — "wait, where's the part that makes this lead to good worlds? why do you think it would work?" — seems to be of a similar nature to my criticism of people who think "alignment is easy, just do X": the proposal is failing to answer some fundamental concerns that the person proposing has a hard ...
    5 min
  • LW - AI Timelines via Cumulative Optimization Power: Less Long, More Short by jacob cannell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Timelines via Cumulative Optimization Power: Less Long, More Short, published by jacob cannell on October 6, 2022 on LessWrong.
    TLDR: We can best predict the future by using simple models which best postdict the past (ala Bayes/Solomonoff). A simple model based on net training compute postdicts the relative performance of successful biological and artificial neural networks. Extrapolation of this model into the future leads to short AI timelines: ~75% chance of AGI by 2032.
    Cumulative Optimization Power[1]: a Simple Model of Intelligence
    A simple generalized scaling model predicts the emergence of capabilities in trained ANNs(Artificial Neural Nets) and BNNs(Biological Neural Nets):
    perf ~= P = CT
    For sufficiently flexible and efficient NN architectures and learning algorithms, the relative intelligence and capabilities of the best systems are simply proportional to net training compute or intra-lifetime cumulative optimization power P, where P = CT (compute ops/cycle training cycles), assuming efficient allocation of (equivalent uncompressed) model capacity bits N roughly proportional to data size bits D.
    Intelligence Rankings
    Imagine ordering some large list of successful BNNs(brains or brain modules) by intelligence (using some committee of experts), and from that deriving a relative intelligence score for each BNN. Obviously such a scoring will be noisy in its least significant bits: is a bottlenose dolphin more intelligent than an american crow? But the most significant bits are fairly clear: C. Elegans is less intelligent than Homo Sapiens.
    Now imagine performing the same tedious ranking process for various successful ANNs. Here the task is more challenging because ANNs tend to be far more specialized, but the general ordering is still clear: char-RNN is less intelligent than GPT-3. We could then naturally combine the two lists, and make more fine-grained comparisons by including specialized sub-modules of BNNs (vision, linguistic processing, etc).
    The initial theory is that P - intra-lifetime cumulative optimization power (net training compute) - is a very simple model which explains a large amount of the entropy/variance in a rank order intelligence measure: much more so than any other simple proposed candidates (at least that I'm aware of). Since P follow a predictable temporal trajectory due to Moore's Law style technological progress, we can then extrapolate the trends to predict the arrival of AGI.
    This simple initial theory has a few potential flaws/objections, which we will then address.
    Initial Exemplars
    I've semi-randomly chosen 15 exemplars for more detailed analysis: 8 BNNs, and 9 ANNs. Here are the 8 BNNs
    (6 whole brains and 2 sub-systems) in randomized order:
    Honey Bee
    Human
    Raven
    Human Linguistic Cortex
    Cat
    C. Elegans
    Lizard
    Owl Monkey Visual Cortex
    The ranking of the 6 full brains in intelligence is rather obvious and likely uncontroversial. Ranking all 8 BNNs in terms of P (net training compute) is still fairly obvious.
    Here are the 9 ANNs, also initially in randomized order:
    AlphaGo: First ANN to achieve human pro-level play in Go
    Deepspeech 2: ANN speech transcription system
    VPT: Diamond-level minecraft play
    Alexnet: Early CNN imagenet milestone, subhuman performance
    6-L MNIST MLP: Early CNN milestone on MNIST, human level
    Chinchilla: A 'Foundation' Large Language Model
    GPT-3: A 'Foundation' Large Language Model
    DQN Atari: First strong ANN for Atari, human level on some games
    VIT L/14@336px: OpenAI CLIP 'Foundation' Large Vision Model
    Most of these systems are specialists in non-overlapping domains, such that direct performance comparison is mostly meaningless, but the ranking of the 3 vision systems should be rather obvious based on the descriptions.
    The DQN Atari and VPT agents are somewhat comparable to animal brains. How would you ran...
    31 min
  • AF - Warning Shots Probably Wouldn't Change The Picture Much by Nate Soares
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Warning Shots Probably Wouldn't Change The Picture Much, published by Nate Soares on October 6, 2022 on The AI Alignment Forum.
    One piece of advice I gave to EAs of various stripes in early 2021 was: do everything you can to make the government sane around biorisk, in the wake of the COVID pandemic, because this is a practice-run for AI.
    I said things like: if you can't get the world to coordinate on banning gain-of-function research, in the wake of a trillions-of-dollars tens-of-millions-of-lives pandemic "warning shot", then you're not going to get coordination in the much harder case of AI research.
    Biolabs are often publicly funded (rather than industry-funded). The economic forces arrayed behind this recklessly foolish and impotent research consists of “half-a-dozen researchers thinking it’s cool and might be helpful”. (While the work that would actually be helpful—such as removing needless bureaucracy around vaccines and investing in vaccine infrastructure—languishes.) Compared to the problem of AI—where the economic forces arrayed in favor of “ignore safety and rush ahead” are enormous and the argument for expecting catastrophe much murkier and more abstract—the problem of getting a sane civilizational response to pandemics (in the wake of a literal pandemic!) is ridiculously easier.
    And—despite valiant effort!—we've been able to do approximately nothing.
    We're not anywhere near global bans on gain-of-function research (or equivalent but better feats of coordination that the people who actually know what they're talking about when it comes to biorisk would tell you are better targets than gain-of-function research).
    The government continues to fund research that is actively making things worse, while failing to put any serious funding towards the stuff that might actually help.
    I think this sort of evidence has updated a variety of people towards my position. I think that a variety of others have not updated. As I understand the counter-arguments (from a few different conversations), there are two main reasons that people see this evidence and continue to hold out hope for sane government response:
    1. Perhaps the sorts of government interventions needed to make AI go well are not all that large, and not that precise.
    I confess I don't really understand this view. Perhaps the idea is that AI is likely to go well by default, and all the government needs to do is, like, not use anti-trust law to break up some corporation that's doing a really good job at AI alignment just before they succeed? Or perhaps the idea is that AI is likely to go well so long as it's not produced first by an authoritarian regime, and working against authoritarian regimes is something governments are in fact good at?
    I'm not sure. I doubt I can pass the ideological Turing test of someone who believes this.
    2. Perhaps the ability to cause governance to be sane on some issue is tied very directly to the seniority of the government officials advising sanity.
    EAs only started trying to affect pandemic policy a few years ago, and aren't very old or recognized among the cacophony of advisors. But if another pandemic hit in 20 years, the sane EA-ish advisors would be much more senior, and a lot more would get done. Similarly, if AI hits in 20 years, sane EA-ish advisors will be much more senior by then. The observation that the government has not responded sanely to pandemic near-misses, is potentially screened-off by the inexperience of EAs advising governance.
    I have some sympathy for the second view, although I'm skeptical that sane advisors have significant real impact. I'd love a way to test it as decisively as we've tested the "government (in its current form) responds appropriately to warning shots" hypotheses.
    On my own models, the "don't worry, people will wake up as the cliff-edge come...
    5 min