The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - What improvements should be made to improve EA discussion on heated topics? by Ozzie Gooen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What improvements should be made to improve EA discussion on heated topics?, published by Ozzie Gooen on January 16, 2023 on The Effective Altruism Forum.
    We've recently went through a series of intense EA controversies. I get the sense that many EAs have found EA discussion on these to be exhausting and disappointing.
    Whatever the details of the current controversies, I think it's clear there's improvement to be made. This could be a good time to reflect on what would be useful.
    I don't want to comment on the heated topics of the day, so let's assume that these changes will only be applies to future topics, no matter what those might be.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - Book Review: Worlds of Flow by remember
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book Review: Worlds of Flow, published by remember on January 16, 2023 on LessWrong.
    This work was written at Conjecture.
    “Worlds of Flow,” a history of 19th and early 20th-century hydrodynamics by Oliver Darrigol, concludes:
    What distinguishes the history of hydrodynamics from that of other physical theories is not so much the tremendous effect of challenges from phenomenal worlds, but rather it is the slowness with which these challenges were successfully met. Nearly two centuries elapsed between the first formulation of the fundamental equations of the theory and the deductions of laws of fluid resistance in the most important case of large Reynolds numbers.The reasons for this extraordinary delay are easily identified a posteriori. They are the infinite number of degrees of freedom and the nonlinear character of the fundamental equations, both of which present formidable obstacles to obtaining solutions in concrete cases. Moreover, instability often deprives the few known exact solutions of any physical relevance.
    These difficulties have barred progress along purely mathematical lines. They have also made physical intuition a poor guide, and a source of numerous paradoxes. Hydrodynamicists therefore sought inspiration in concrete phenomena. Engagement with and challenges from the real worlds of flow were essential to the development of the above-mentioned strategies. The challenged theorists strove to find new solutions and to develop new methods of approximation. Experience indicated some general properties of the motion, such as the existence of boundary layers, the random character of turbulence, the sudden character of the Reynolds transition, or the formation of trailing vortices.Altogether, there were many ways in which practical concerns oriented theorists in the conceptual maze of fluid dynamics. The evolution from a paper theory to an engineering tool thus depended on transgressions of the limits between academic hydrodynamics and applied hydrodynamics.
    This quote captures one of the most significant lessons in the book: the study of concrete phenomena was critical in overcoming many of the difficulties in hydrodynamics. Roughly two upstream problems required interacting with concrete phenomena to solve. The first is summarized nicely by the quote above: theorizing and abstract thinking alone was not enough to solve the problems posed by hydrodynamics. The second is a subtler point: early mathematical and theoretical tools weren’t adapted to understanding hydrodynamics. Much of the necessary mathematical machinery existed quite early in the 1800s, but people hadn’t built the physico-mathematical tools or intuitions to tell us what they physically meant. Contact with reality forced scientists to confront the inadequacies of their theories while guiding the adaption of physico-mathematical tools.
    While the book is organized along rough problems that hydrodynamics faced (waves, viscosity, vortices, instability, etc.), this review will focus on broader scientific lessons. First on the two big themes I think are most important, then briefly on the other themes at the end.
    Practice and theory in hydrodynamics
    The first problem was that abstract thinking and theorizing proved unable to solve many of the problems of hydrodynamics. A great example of this comes from the discovery of Reynold’s number, which predicts whether flow is turbulent or laminar. Reynold’s number could have potentially been hypothesized as a consequence of Navier-Stokes, which describes viscous flow behavior. But Navier-Stokes is not analytically solvable, so Reynold’s number doesn’t come automatically. Making this more difficult is that turbulent flow, such as after submerged propellers, is generally invisible. Instead of reasoning his way there from first principles, Osborne Reynolds firs...
    15 min
  • EA - Replace Neglectedness by Indra Gesink
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Replace Neglectedness, published by Indra Gesink on January 16, 2023 on The Effective Altruism Forum.
    for example with Leverage, as featured in Will MacAskill’s What We Owe the Future.
    The second bullet point featured in the website introduction to effective altruism is the ITN framework. This exists to prioritize problems. The framework does so by considering the Importance — or scale, S — of a problem, as the number of people or quality-adjusted life years (QALYs) affected, multiplied with the Tractability, as the potential that this problem can be addressed, and Neglectedness, as the number of people already working to address this problem (ITN-framework, including Leverage). Tractibility is sometimes also called Solvability, and non-neglectedness crowdedness.
    Some criticisms and difficulties in interpreting the framework (1, 2, 3, 4) have preceded this forum post. The ITN framework can be interpreted - as also in the final paragraph of (1) - such that IT represents the potential that a problem can be addressed, while ITN considers the difference that any one individual can make to that problem, particularly the next individual. How much impact can the next individual make, choosing to work on this problem, on average? Why do I add “on average”? We are still ignoring the person’s unique qualities, and instead more abstractly consider an average person. Adding “personal fit” as another multiplicative factor would make it personal as well.
    So “How much impact can the next individual make on this problem?” really asks for the marginal counterfactual impact. Respectively this is the amount of impact that this one individual adds to the total impact so far, which would not happen otherwise. The ITN-factor Neglectedness assumes that this marginal counterfactual impact is declining — strictly — as more individuals join the endeavor of addressing the particular problem. If this is true, then — indeed — a more neglected problem ceteris paribus — i.e. not varying factors I, T (or personal fit) simultaneously — always yields more impact when fewer individuals are already addressing it. This is however not always true, as also already pointed out in the criticisms referenced above.
    Consider the following string of examples. Suppose a partial civilizational collapse has occurred, and you consider whether it would be good to go and repopulate the now barren lands. The ITN-framework says that as the first person to do so you make the biggest difference. However, alone you cannot procreate, at least not without far-reaching technological assistance. In fact a sizable group of people deciding to do so might very well still be ineffective, by not bringing in sufficient genetic diversity. This is captured by a well-known term in population biology: the critical or minimally viable population size (to persist). Something similar operates to a lesser extent in the effectiveness of teams. I for example once found the advice to better not join a company as the sole data scientist, as you would not have a team to exchange ideas with. Working together, you become more effective, and develop more.
    Advocating for policies is another area that is important and where you need teams. Consider there being multiple equally worthwhile causes to protest for, but by the logic of the ITN-framework you always join the least populated protest. And no critical mass is obtained. Doesn’t that seem absurd? See also (5). (And the third image in (3), depicting a one-time significant increase in marginal counterfactual impact, as with a critical vote to establish a majority. This graph is also called an indicator function). Effective altruists might similarly often find themselves advocating for policies which are neglected and that are thus not well known to the recipient of such advocacy. As opposed to max...
    8 min
  • LW - Consequentialists: One-Way Pattern Traps by David Udell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consequentialists: One-Way Pattern Traps, published by David Udell on January 16, 2023 on LessWrong.
    Generated during MATS 2.1.
    A distillation of my understanding of Eliezer-consequentialism.
    Thanks to Jeremy Gillen, Ben Goodman, Paul Colognese, Daniel Kokotajlo, Scott Viteri, Peter Barnett, Garrett Baker, and Olivia Jimenez for discussion and/or feedback; to Eliezer Yudkowsky for briefly chatting about relevant bits in planecrash; to Quintin Pope for causally significant conversation; and to many others that I've bounced my thoughts on this topic off of.
    Introduction
    What is Eliezer-consequentialism? In a nutshell, I think it's the way that some physical structures monotonically accumulate patterns in the world. Some of these patterns afford influence over other patterns, and some physical structures monotonically accumulate patterns-that-matter in particular -- resources. We call such a resource accumulator a consequentialist -- or, equivalently, an "agent," an "intelligence," etc.
    A consequentialist understood in this way is (1) a coherent profile of reflexes (a set of behavioral reflexes that together monotonically take in resources) plus (2) an inventory (some place where accumulated resources can be stored with better than background-chance reliability.)
    Note that an Eliezer-consequentialist is not necessarily a consequentialist in the normative ethics sense of the term. By consequentialists we'll just mean agents, including wholly amoral agents. I'll freely use the terms 'consequentialism' and 'consequentialist' henceforth with this meaning, without fretting any more about this confusion.
    Path to Impact
    I noticed hanging around the MATS London office that even full-time alignment researchers disagree quite a bit about what consequentialism involves. I'm betting here that my Eliezer-model is good enough that I've understood his ideas on the topic better than many others have, and can concisely communicate this better understanding.
    Since most of the possible positive impact of this effort lives in the fat tail of outcomes where it makes a lot of Eliezerisms click for a lot of alignment workers, I'll make this an effortpost.
    The Ideas to be Clarified
    I've noticed that Eliezer seems to think the von Neumann-Morgenstern (VNM) theorem is obviously far reaching in a way that few others do.
    Understand the concept of VNM rationality, which I recommend learning from the Wikipedia article... Von Neumann and Morgenstern showed that any agent obeying a few simple consistency axioms acts with preferences characterizable by a utility function.
    MIRI Research Guide (2015)
    Can you explain a little more what you mean by "have different parts of your thoughts work well together"? Is this something like the capacity for metacognition; or the global workspace; or self-control; or...?
    No, it's like when you don't, like, pay five apples for something on Monday, sell it for two oranges on Tuesday, and then trade an orange for an apple.
    I have still not figured out the homework exercises to convey to somebody the Word of Power which is "coherence" by which they will be able to look at the water, and see "coherence" in places like a cat walking across the room without tripping over itself.
    When you do lots of reasoning about arithmetic correctly, without making a misstep, that long chain of thoughts with many different pieces diverging and ultimately converging, ends up making some statement that is... still true and still about numbers! Wow! How do so many different thoughts add up to having this property? Wouldn't they wander off and end up being about tribal politics instead, like on the Internet?
    And one way you could look at this, is that even though all these thoughts are taking place in a bounded mind, they are shadows of a higher unbounded structure which is the model identifie...
    25 min
  • EA - Announcing aisafety.training by JJ Hepburn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing aisafety.training, published by JJ Hepburn on January 17, 2023 on The Effective Altruism Forum.
    Crossposted to Less WrongTo help people find what to apply to, aisafety.training acts as a well-maintained living document of AI safety programs, conferences, and other events. This will smooth the experience of people working on and joining AI safety and reduce the burden on word-of-mouth transmission of available programs. This can also be helpful for field builders planning events to see when other things are happening to plan around. We at AI Safety Support have been internally maintaining this document for some time and using it in our free career coaching calls. We now have a public-facing version, a form to add anything we’ve missed, and an email to alert us to corrections.
    For example, below, you will find static / will-soon-be-outdated images without clicking on the website link.
    Application Deadlines
    Program Timeline
    Program table
    If you’re interested in helping to maintain this and can consistently dedicate a few hours per week to reading places where things might have been announced, accepting additions from the form and corrections by email, or if you have feedback or feature requests, leave us a comment or drop by the Alignment Ecosystem Development Discord, where this is worked on along with related projects to improve the information ecosystem such as aisafety.community and the upcoming aisafety.world and ea.domains.
    Additionally, if you wish to have a monthly reminder of upcoming events sent directly to your inbox, subscribe to AI Safety Support’s newsletter here.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Löbian emotional processing of emergent cooperation: an example by Andrew Critch
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Löbian emotional processing of emergent cooperation: an example, published by Andrew Critch on January 17, 2023 on The AI Alignment Forum.
    Related to: A Löbian argument pattern for implicit reasoning in natural language: Löbian party invitations
    Epistemic status: my opinion based on a mix of math, reflection, and speculation; not backed up by any systematic psychological studies.
    Summary: Since my 2019 paper generalizing Löb's Theorem, a couple dozen people have asked me if the way humans naturally cooperate might be well-described by Löb's Theorem. In short, my answer is probably sometimes, and in this post I'll try using an example to convey what that means. Importantly, Löb's Theorem is not a theorem when applied to real-world humans and emotions — i.e., when its hypotheses are met, its conclusion is only sometimes true. Nonetheless, the reasoning pattern in its proof (I claim) sometimes genuinely occurs at the level of intuition in real people, whether or not they know any math or Löb's Theorem.
    Introduction
    There are at least two real-world patterns that could reasonably be called Löbian cooperation in humans, which I'll name here:
    Functionally Löbian cooperation. Sometimes people become aware that they're anticipating (predicting) cooperation from each other, and then that anticipation causes them to cooperate, rendering the anticipation itself valid. In this pattern, the fact that anticipation of cooperation will cause cooperation is analogous to the hypothesis (main assumption) of Löb's Theorem, and the fact that the cooperation in fact emerges is analogous to the conclusion of Löb's Theorem. I call this pattern "functionally" Löbian because its input and output resemble the input (hypothesis) and output (conclusion) of Löb's Theorem.
    Procedurally Löbian cooperation. Sometimes the mental procedure a person follows to anticipate and decide upon cooperation can resemble an entire proof of Löb's Theorem, as I'll describe below. In other words, instead of just the hypothesis and conclusion of Löb's Theorem matching reality, the structure in the intermediate steps of the proof also match reality, at least somewhat. I call this "procedurally" Löbian cooperation, and it's a special case of functionally Löbian cooperation because it demands a stronger analogy between the theorem and the real world. Illustrating how this might work constitutes is the bulk of content in this post.
    What functionally Löbian cooperation feels like
    For those who recognize the symbols involved, Löb's Theorem says that if ⊢□cc then ⊢c. I don't plan to use these symbols with their normal meanings in the rest of this post, so don't worry if you don't recognize them.
    In words, functional Löbian cooperation happens when anticipation of future or unobserved cooperation causes present cooperation. So if you're interacting with someone, and you feel like they're probably going to be nice to you in the future, and that fact makes you decide to be nice to them now, I call that functional Löbian cooperation.
    What procedurally Löbian cooperation feels like
    Most human cooperation is probably not procedurally Löbian, and maybe not even functionally Löbian. However, I'm confident that human cooperation is sometimes procedurally Löbian, and I can even point to experiences of my own that fit the bill. To explain this, I'll be talking a lot more about feelings, because I think most unconscious processing is carried out by and/or experienced as feelings. I'll write
    Feeling("Pigs can probably fly.")
    for the feeling that pigs can probably fly. Such a feeling can be true or false, according to whether it correctly anticipates the real world.
    In procedurally Löbian cooperation, part of the mental process will involve first feeling something uncertain to do with cooperation, then believing it, and then feeling lik...
    13 min
  • AF - Preparing for AI-assisted alignment research: we need data! by CBiddulph
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Preparing for AI-assisted alignment research: we need data!, published by CBiddulph on January 17, 2023 on The AI Alignment Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • AF - Experiment Idea: RL Agents Evading Learned Shutdownability by Leon Lang
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Experiment Idea: RL Agents Evading Learned Shutdownability, published by Leon Lang on January 16, 2023 on The AI Alignment Forum.
    Preface
    Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort. Thanks to Erik Jenner who explained to me the basic intuition for why an advanced RL agent may evade the discussed corrigibility measure. I also thank Alex Turner, Magdalena Wache, and Walter Laurito for detailed feedback on the proposal and Quintin Pope and Lisa Thiergart for helpful feedback in the last December SERI-MATS shard theory group meeting.
    This text was part of my deliverable for the shard theory stream of SERI-MATS. In it, I present an idea for an experiment that tests the convergent drive of modern model-based RL agents to evade shutdownability. If successful, I expect the project could serve as a means to communicate the problem of corrigibility to the machine learning community. As such, I also consider this project idea a submission of the Shutdown Problem Contest.
    I do not personally want to work on the project since doing experiments does not seem like my comparative advantage. Thus, with posting this project, I mainly seek collaborators and feedback. More on that in the conclusion.
    Introduction
    At some point in the future, we will train very advanced AI, possibly by some version of model-based reinforcement learning. Since it is hard to correctly specify the reward function or steer the inner learning dynamics of the agent, it may be desirable to make sure that the agent is corrigible, including shutdownability:
    “(1) A corrigible reasoner must at least tolerate and preferably assist the programmers in their attempts to alter or turn off the system.”
    Next to a literal shutdown button, one could also try to use the inner functions of the agent to effectively get the ability to shut it down. The idea would be to design a certain type of input such that, as a reaction to that input, a computation in the agent itself will cause it to perform a null action that doesn’t do anything — which effectively equates to a shutdown. Scott Aaronson proposed this idea in a recent talk (timestamp: 57:10) in the form of a planted cryptographic backdoor, an idea he currently works on during his sabbatical at OpenAI.
    For this proposal, I consider a variation of that shutdown approach: instead of directly “planting” the shutdown behavior, I consider the possibility to train the behavior into the agent. Concretely, there is a specific input — here, an alert sound — that the agent receives at random times during training. Whenever that happens, the agent will receive a negative reward unless it performs the null action. Over time, the agent will learn to stop acting in these situations. During deployment, the input can then be shown to the agent whenever it performs harmful actions, which will ideally cause it to stop immediately.
    This shutdown approach has the following philosophical problem: if the agent performs world modeling, it will know that the alert sound will make it stop performing, and this may be counter to the more consequentialist motivations of the agent. Thus, we should expect that the agent will perform an action that makes it never hear the sound again — for example, by “putting in earplugs” — once it is advanced enough and has the ability to do so. This experimental proposal is meant to show this convergent behavior using contemporary model-based reinforcement learning.
    The remaining sections will discuss the following:
    A three-part training story for an advanced AI that evades corrigibility, including details for how to model the situation with contemporary reinforcement learning;
    More experimental details based on using MuZero;
    Caveats, what might go wrong;
    A short comparison to the original shutdown problem and how the proposal...
    29 min
  • EA - Some intuitions about fellowship programs by Joel Becker
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some intuitions about fellowship programs, published by Joel Becker on January 16, 2023 on The Effective Altruism Forum.
    Epistemic status: not sure how reliable my intuition is, but I do have more experience with these programs than most.
    TL;DR
    Making sure >30 participants have regular opportunities to spontaneously gather, active programming, basic food and medical amenities, and common knowledge about visit dates hugely increases the benefit of residential fellowship programs.
    For transparency, I should note a stronger, less confident belief of mine that I will not defend here. My instinct is that all of the above factors are not merely beneficial but necessary in order for these programs to be worth their cost (assuming that their goals are indeed as I describe below). I am aware that some factors come with large time or money costs. If these costs are prohibitive, so be it.
    Background
    I ran one of these programs and participated in two others. So what follows is a post-mortem of my own mistakes as much as it is feedback for organizers and recommendations for funders and organizers considering future initiatives.
    It’s worth emphasizing that I got lots out of each of these experiences, and that I feel very grateful to fellow participants and organizers!
    Fellowship programs
    I take the goals of these programs to be some combination of:
    Increasing short-term productivity,
    Generating counterfactual collaborations/relationships, and
    Moving participants towards more impactful careers at an accelerated rate.
    I take the method of these programs to be some combination of:
    Hosting people who largely do not know each other in a new physical environment for a small number of months,
    Covering the cost of accommodation, co-working space, and travel to-and-from, and
    Organizing social and professional activities.
    Things that I am not talking about include:
    Communities where many people already know one another, and
    Themed retreats, or other temporary communities organized around a professional aptitude/cause area/etc.
    Number of participants
    My intuition is that hosting 35 participants is much, much better than hosting 25. Not only in total, but on a per-participant basis.
    Evidence for the directional claim:
    In the program I ran, the overwhelming majority of reported positive impact anecdotes came during the ~50% of the time we hosted >30 participants.
    In fact, I need to get to the joint-30th most subjectively impressive anecdote before finding one I think came about during the time we hosted <=30 participants.
    A possible exception is the first few weeks of the program, when participants came across many novel ideas, relationships, and physical spaces even with smaller numbers of participants.
    With >30 participants, shared spaces were often packed in evenings, participants participated in and ran a wider variety of well-attended activities, the office felt alive, and work felt urgent.
    Among other ~vibes~ based on discussions with participants, my time as a participant, and my time as an organizer.
    30 might feel like an arbitrary cut-off: readers might think that I don’t mean this literally. In fact, I am tempted to defend this close-to-literally. One piece of evidence: in my experience, there are almost no spontaneous gatherings of most participants below 25; these gatherings become somewhat more likely beyond this, then shoot up around 30. They continue increasing beyond 35, albeit more slowly.
    My guess is that the effect runs through two mechanisms:
    The chance that someone's professional experiences are highly complementary with your own increases sharply around this point.
    Maybe because some small clusters of people with shared interests choose to participate at the same time — although I think this was largely not the case.
    The chance that someone will join you if you spontaneo...
    8 min
  • LW - Confusing the ideal for the necessary by adamShimi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Confusing the ideal for the necessary, published by adamShimi on January 16, 2023 on LessWrong.
    I had many meetings that day, and a lot of managing issues to attend, so I found myself thinking “I just can’t get any research done today”.
    It was not the first time I had thought that thought; but it might have been the first time I applied my main epistemologist’s tool to it: I asked “Why?” For I did have time to work, not that much, yes, splintered, yes, but enough to make progress.
    So why was I discarding the very possibility of making progress and working on my research?
    Because I had internalized a false requirement: that I needed large chunks of time with no distractions to do research. That it was a precondition to me doing anything worth my time, and so investing little pockets of time here and there was a mistake.
    Of course, long sessions of deep work are reliably more productive than 15 minute sprints peppered through a day of meeting. It’s a reliable and reproducible finding, and I’ve experienced it again and again in my own life.
    Yet the point here is different. Instead of simply recognizing my lack of ideal condition, and deciding whether it was still worth it for me to do research despite the discrepancy, I automatically discarded the possibility of doing research. I turned ideal conditions into necessary conditions.
    And it’s even worse than that. Because I could not even consider doing research despite distractions, I never exercised this specific attention muscle. I never trained myself to go back quickly and smoothly to my train of thought after an interruption. Which means that when I had to make progress despite the constraints and distractions (for example when I had both an important research deadline and documentation/posts to write), my research suffered far more than it needed to because I was not trained to get the best out of the little time I had.
    Which naturally leads to the more disturbing question: how many instances of this kind of thinking am I not noticing? How much am I costing myself by subscribing to necessary conditions that aren’t really necessary, just ideal?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min