The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Does descaling a kettle help? Theory and practice by philh
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Does descaling a kettle help? Theory and practice, published by philh on May 2, 2023 on LessWrong.
    I've heard that descaling a kettle makes it more efficient, and can save me time and money.
    Okay, but how much? For some reason my my intuition says it'll be basically unnoticable. Let's try to figure it out, and then actually try it.
    The Models
    There's a first-order approximation which says this should be impossible: heating things up is 100% efficient. What's the limescale going to do, convert some electricity into forms of energy other than heat?
    No, but it might cause the heat to go to the wrong places. I think there are two ways that could happen. The first is if the limescale itself has a high heat capacity, meaning it takes a lot of energy to heat up. That doesn't seem likely to me; I think there's likely less than 10 g of it, and I think it probably has less heat capacity than the same mass of water (because I think water's is higher than most other things). I don't think adding 10 ml water (weighing 10 g) to my kettle will significantly effect the boiling time.
    Spot check: water's specific heat capacity is about 4 J/K·g. So to heat 10 g water by 100 K needs about 4000 J. My kettle is 2200 W according to the email I got when I bought it, so an extra 10 ml should take about 2 s longer to boil.
    Also, I normally boil about 500 ml of water, so we'd expect 10 ml extra to make about a 2% difference. (I don't have a strong intuition for how long my kettle normally takes to boil. 1-3 minutes? The above calculation suggests it should be around 100 s.)
    The second way the limescale could matter is if it's a good thermal insulator. Then the metal heating element gets significantly above 100°C before the water boils. And from my googling, it seems like this is the reason people give that descaling a kettle is helpful. How much effect would this have?
    This page says the thermal conductivity is 2.2 W/m·K. I don't remember studying this in school, and I don't find that wikipedia page very clear. But I think maybe this means: the rate of energy transfer (W) from the heating element is 2.2 W/m·K, multiplied by the surface area of the heating element (m²), divided by the thickness of the limescale (m), multiplied by the temperature difference (K) between the heating element and the water. The units check out at least.
    This implies, if there's no limescale, that the power transfer should be infinite, regardless of the size of the heating element. That's clearly not how things work. Still, it seems fine to have as a model, it just means that the water and heating element will stay the same temperature.
    But doing some field research (i.e. looking at my kettle) makes me think it's a bad model in this case. Here's what it looks like:
    (This kettle was last descaled around August 2020. I know because I bought vinegar for it, and thanks to online shopping the receipt is in my email.)
    It seems that I have some limescale over some of the heating element (bits of the metal plate at the bottom, and the tube around where it passes through the plate), and zero limescale over some of it. The "no limescale gives ∞ W/K" model is going to give ∞ W/K no matter how small the no-limescale area is. So scratch that model.
    I don't feel like trying to model the rate of power transfer from an un-limescaled heating element to water. I don't know what material it's made of (some kind of steel I suppose?), and I don't know what property of that material to look up. Plus it probably depends on how fast water convects.
    Here's another way to think about it: assume that the amount of metal the manufacturer used is roughly as small as possible without compromising too much on boiling speed.
    When we turn the kettle on, we start putting 2200 W into the heating element, and the heating element starts putting some of ...
    14 min
  • LW - Alignment Research @ EleutherAI by Curtis Huebner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Research @ EleutherAI, published by Curtis Huebner on May 3, 2023 on LessWrong.
    The past and future of AI alignment at Eleuther
    Initially, EleutherAI focused mainly on supporting open source research. AI alignment was something that was acknowledged by many of the core members as important, but it was not the primary focus. We mainly had discussions about the topic in the #alignment channel and other parts of our discord while we worked on other projects.
    As EAI grew, AI alignment started to get taken more seriously, especially by its core members. What started off as a single channel turned into a whole host of channels about different facets of alignment. We also hosted several reading groups related to alignment, such as the modified version of Richard Ngo’s curriculum and an interpretability reading group. Eventually alignment became the central focus for a large segment of EAIs leadership, so much so that all our previous founders went off to do full time alignment research at Conjecture and OpenAI.
    Right now, the current leadership believes making progress in AI alignment is very important. The organization as a whole is involved in a mix of alignment research, interpretability work, and other projects that we find interesting.
    Moving forward, EAI remains committed to facilitating and enabling open source research, and plans to ramp up its alignment and interpretability research efforts. We want to increase our understanding and control of modern ML systems and minimize existential risks posed by artificial intelligence.
    Our meta-level approach to alignment
    It is our impression that AI alignment is still a very pre-paradigmatic field. Progress in the field often matches the research pattern we see in the ELK report, where high level strategies are proposed, and problems or counterexamples are found. Sometimes these issues can be fixed, but oftentimes fundamental issues are identified that make an initially promising approach less interesting.
    A consequence of this is that it’s difficult to commit to an object level strategy to make progress on AI alignment, and even harder to commit to any grand strategic plan to solve the problem. Instead it makes more sense to have a meta level strategy that makes us better able to leverage our unique position within the AI research ecosystem, and pivot when we get new information.
    Going forward, this means we want to pursue interesting projects that meet a few general desiderata.
    Our volunteers, partners, and collaborators are enthusiastic about the project.
    We believe that pursuing the project won’t lead to a net increase in existential risks from AI. We’ll check for this even if the project is ostensibly a project that will greatly increase our understanding of AI alignment.
    The project is something that EAI is better equipped to do than anyone else in the space, or the project seems interesting or important, but neglected by the broader community.
    In order to pull this off, we aim to stay on top of both the latest developments in AI and alignment research. We’ll also carefully consider new projects before we embark on or allocate resources for them.
    Problems we are interested in and research we are doing right now
    Given the current state AI landscape, there are a few directions that we find especially interesting. We’d love to collaborate with others to make progress on these issues.
    Interpretability work
    Interpretability work, especially with current models, seems like a very tractable and scalable research direction. It seems especially easy for current ML researchers to pick up and make progress on it. EAI is well equipped to enable this kind of research, especially for larger language models that more closely resemble the ones we see in modern production systems.
    This is arguably where most of our recent eff...
    6 min
  • LW - «Boundaries» and AI safety compilation by Chipmonk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: «Boundaries» and AI safety compilation, published by Chipmonk on May 3, 2023 on LessWrong.
    In this post I outline every post I could find that meaningfully connects the concept of «Boundaries» with AI safety. This seems to be a booming subtopic: interest has picked up substantially within the past year.
    Perhaps most notably, Davidad includes the concept in his Open Agency Architecture for Safe Transformative AI alignment paradigm. For a preview of the salience of this approach, see this comment by Davidad (2023 Jan):
    “defend the boundaries of existing sentient beings,” which is my current favourite. It’s nowhere near as ambitious or idiosyncratic as “human values”, yet nowhere near as anti-natural or buck-passing as corrigibility.
    This post also compiles recent work from Andrew Critch, Scott Garrabrant, John Wentworth, and others. But first I will recap what «Boundaries» are:
    «Boundaries» definition recap:
    You can see «Boundaries» Sequence for a longer explanation, but I will excerpt from a more recent post by Andrew Critch, 2023 March:
    By boundaries, I just mean the approximate causal separation of regions in some kind of physical space (e.g., spacetime) or abstract space (e.g., cyberspace). Here are some examples from my «Boundaries» Sequence:
    a cell membrane (separates the inside of a cell from the outside);
    a person's skin (separates the inside of their body from the outside);
    a fence around a family's yard (separates the family's place of living-together from neighbors and others);
    a digital firewall around a local area network (separates the LAN and its users from the rest of the internet);
    a sustained disassociation of social groups (separates the two groups from each other)
    a national border (separates a state from neighboring states or international waters).
    Also, beware:
    When I say boundary, I don't just mean an arbitrary constraint or social norm.
    Posts & researchers that link «Boundaries» and AI safety
    All bolding in the excerpts below is mine.
    Davidad’s OAA
    Saliently, Davidad uses «Boundaries» for one of the four hypotheses he outlines in An Open Agency Architecture for Safe Transformative AI (2022 Dec)
    All Principles That Human CEV Would Endorse⇒Q⇒Don't Kill Everyone
    Deontic Sufficiency Hypothesis: There exists a human-understandable set of features of finite trajectories in such a world-model, taking values in (−∞,0], such that we can be reasonably confident that all these features being near 0 implies high probability of existential safety, and such that saturating them at 0 is feasible[2] with high probability, using scientifically-accessible technologies.
    I am optimistic about this largely because of recent progress toward formalizing a natural abstraction of boundaries by Critch and Garrabrant. I find it quite plausible that there is some natural abstraction property Q of world-model trajectories that lies somewhere strictly within the vast moral gulf of
    Further explanation of this can be found in Davidad's Bold Plan for Alignment: An In-Depth Explanation (2023 Apr) by Charbel-Raphaël and Gabin:
    Getting traction on the deontic feasibility hypothesis
    Davidad believes that using formalisms such as Markov Blankets would be crucial in encoding the desiderata that the AI should not cross boundary lines at various levels of the world-model. We only need to “imply high probability of existential safety”, so according to davidad, “we do not need to load much ethics or aesthetics in order to satisfy this claim (e.g. we probably do not get to use OAA to make sure people don't die of cancer, because cancer takes place inside the Markov Blanket, and that would conflict with boundary preservation; but it would work to make sure people don't die of violence or pandemics)”. Discussing this hypothesis more thoroughly seems important.
    Also see:
    () Elicitors: Langua...
    12 min
  • LW - [New] Rejected Content Section by Ruby
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [New] Rejected Content Section, published by Ruby on May 4, 2023 on LessWrong.
    tl:dr;
    The LessWrong reviews all first time submissions from new users
    In the last month, we've built a new process where we reject some of those submissions (with explanation) – you can now see what was rejected from the main site on the new Rejected Content section
    New users shouldn't be very afraid to submit, the bar could well be lower than you think (and now you can check based on what we actually reject). The worst that happens now is you get some feedback from us, rather than getting downvoted by many people.
    Though maybe write something shorter as your first thing. Don't write thousands of words only for us to say "this has serious problem X".
    What is the Rejected Content section?
    When new users submit their first post or comment, it is reviewed by a LessWrong team member before it goes live on the site. Many times, especially with an increased influx of users to LessWrong, we will sometimes reject these first posts/comments and not let them go live.
    To increase transparency and accountability, we've made a new section of the site where you can view everything we've rejected, and usually with some explanation of why. This means that anyone who is motivated can both go fishing for things they think were actually good but were mistakenly classified as reject-worthy, or in general just verify that the LessWrong team is (or isn't) exercising good and fair judgment in what we're rejecting.
    The Rejected Content section is part of the Moderation Log, viewable at
    and
    What to do if you disagree with a rejection decision?
    Message us directly about it – maybe we didn't pay close enough attention and it'll be no big deal to change it. See the contact us page.
    Mention in the monthly Open Thread.
    Write a top-level post if you want to start a larger conversation about our moderation philosophy or approach.
    The point of the feature is precisely so people can weigh in, so don't feel afraid to question our choices here. (This isn't a promise to change in response to feedback, but is a good amount of willingness to listen to feedback.)
    Why build it?
    As I wrote in a recent post of of mine about moderation, the moderation team has historically faced a tricky choice. Either we:
    Approve and let userbase evaluate: When we view low quality content being submitted by new users, we nonetheless approve it (perhaps with downvote) and allow the userbase to evaluate and vote accordingly.
    Block using our own judgment: We block that content from going live (and send some message to the author explaining why)
    Each choice has costs and benefits. Letting userbase evaluate decreases reliance on the mod team's judgment. We will make mistakes, so seems safer to have more people look at things. Also I feel bad if behind the scenes the LessWrong team is filtering what goes live and people don't know even know what choices we're making or why.
    At the same time, letting the userbase decide means often:
    a whole bunch of users will view a bad post that they won't benefit from. This seems true even if a moderator gives it an initial downvote.
    you get negative karma posts showing up in different places that feels pretty ughy, e.g. in the All Posts page
    they can end up in search results
    negative karma comments (even if they're collapsed) still make comments sections feel more ughy
    Theoretically, visibility of negative karma content could be fixed in other ways, but it's kind of a weird situation to have this content if you're going to to a lot of trouble to prevent it being seen by people anyway.On the other hand, when us moderators decide to block content going live, we're relying on our judgment, which is fallible. And heck, we often disagree even among ourselves. Plus I generally dislike moderation actions aren't visible to p...
    9 min
  • LW - How MATS addresses “mass movement building” concerns by Ryan Kidd
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How MATS addresses “mass movement building” concerns, published by Ryan Kidd on May 4, 2023 on LessWrong.
    Recently, many AI safety movement-building programs have been criticized for attempting to grow the field too rapidly and thus:
    Producing more aspiring alignment researchers than there are jobs or training pipelines;
    Driving the wheel of AI hype and progress by encouraging talent that ends up furthering capabilities;
    Unnecessarily diluting the field’s epistemics by introducing too many naive or overly deferent viewpoints.
    At MATS, we think that these are real and important concerns and support mitigating efforts. Here’s how we address them currently.
    Claim 1: There are not enough jobs/funding for all alumni to get hired/otherwise contribute to alignment
    How we address this:
    Some of our alumni’s projects are attracting funding and hiring further researchers. Three of our alumni have started alignment teams/organizations that absorb talent (Vivek’s MIRI team, Leap Labs, Apollo Research), and more are planned (e.g., a Paris alignment hub).
    With the elevated interest in AI and alignment, we expect more organizations and funders to enter the ecosystem. We believe it is important to install competent, aligned safety researchers at new organizations early, and our program is positioned to help capture and upskill interested talent.
    Sometimes, it is hard to distinguish truly promising researchers in two months, hence our four-month extension program. We likely provide more benefits through accelerating researchers than can be seen in the immediate hiring of alumni.
    Alumni who return to academia or industry are still a success for the program if they do more alignment-relevant work or acquire skills for later hiring into alignment roles.
    Claim 2: Our program gets more people working in AI/ML who would not otherwise be doing so, and this is bad as it furthers capabilities research and AI hype
    How we address this:
    Considering that the median MATS scholar is a Ph.D./Masters student in ML, CS, maths, or physics and only 10% are undergrads, we believe most of our scholars would have ended up working in AI/ML regardless of their involvement with the program. In general, mentors select highly technically capable scholars who are already involved in AI/ML; others are outliers.
    Our outreach and selection processes are designed to attract applicants who are motivated by reducing global catastrophic risk from AI. We principally advertise via word-of-mouth, AI safety Slack workspaces, AGI Safety Fundamentals and 80,000 Hours job boards, and LessWrong/EA Forum. As seen in the figure below, our scholars generally come from AI safety and EA communities.
    MATS Summer 2023 interest form: “How did you hear about us?” (370 responses)
    We additionally make our program less attractive than comparable AI industry programs by introducing barriers to entry. Our grant amounts are significantly less than our median scholar could get from an industry internship, and the application process requires earnest engagement with complex AI safety questions. We additionally require scholars to have background knowledge at the level of AGI Safety Fundamentals, which is an additional barrier to entry that e.g. MLAB didn't require.
    We think that ~1 more median MATS scholar focused on AI safety is worth 5-10 more median capabilities researchers (because most do pointless stuff like image generation, and there is more low-hanging fruit in safety). Even if we do output 1-5 median capabilities researchers per cohort (which seems very unlikely), we likely produce far more benefit to alignment with the remaining scholars.
    Claim 3: Scholars might defer to their mentors and fail to critically analyze important assumptions, decreasing the average epistemic integrity of the field
    How we address this:
    Our scholars are enc...
    5 min
  • LW - Prizes for matrix completion problems by paulfchristiano
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prizes for matrix completion problems, published by paulfchristiano on May 3, 2023 on LessWrong.
    Here are two self-contained algorithmic questions that have come up in our research. We're offering a bounty of $5k for a solution to either of them—either an algorithm, or a lower bound under any hardness assumption that has appeared in the literature.
    Question 1 (existence of PSD completions): given m=Ω(n) entries of an n×n matrix, can we tell in time ~O(nm) whether it has any (real, symmetric) positive semidefinite completion? Proving that this task is at least as hard as dense matrix multiplication or PSD testing would count as a resolution.
    Question 2 (fast “approximate squaring”): given A∈Rn×n and a set of m=Ω(n) entries of AAT, can I find some PSD matrix that agrees with AAT in those m entries in time ~O(nm)?
    We'll pay $5k for a solution to either problem. The offer is open for each problem for 3 months or until the problem gets solved (whichever happens first). Winners are welcome to publish solutions independently. Otherwise, if the result ends up being a significant part of a paper, we’ll invite them to be a coauthor.
    We’ll also consider smaller prizes for partial progress, or anything that we find helpful for either solving the problem or realizing we should give up on it.
    To understand the motivation for these questions, you can read our paper on Formalizing the presumption of independence and in particular Appendix D.7.2. ARC is trying to find efficient heuristic estimators as a formalization of defeasible reasoning about quantities like the variance of a neural network's output. These two questions are very closely related to one of the simplest cases where we haven't yet found any reasonable linear time heuristic estimator.
    We don’t expect to receive many incorrect proposals, but if we receive more than 5 we may start applying a higher standard in order to save our time. If we can’t understand a solution quickly, we may ask you to provide more details, and if we still can’t understand it we may reject it. We expect a correct solution to be about as clear and easy to verify as a paper published at STOC.
    For both problems, it’s OK if we incorrectly treat a matrix as PSD as long as all of its eigenvalues are at least −ε for a small constant ε. ~O hides polylogarithmic factors in n, ε, and the max matrix entry. Feel free to ask for other clarifications on our question on Math Overflow, on Facebook, or by email.
    To submit a solution, send an email to [email protected].
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - Prizes for matrix completion problems by Paul Christiano
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prizes for matrix completion problems, published by Paul Christiano on May 3, 2023 on The AI Alignment Forum.
    Here are two self-contained algorithmic questions that have come up in our research. We're offering a bounty of $5k for a solution to either of them—either an algorithm, or a lower bound under any hardness assumption that has appeared in the literature.
    Question 1 (existence of PSD completions): given \(m\) entries of an \(n \times n\) matrix, including the diagonal, can we tell in time \(\tilde{O}(nm)\) whether it has any (real, symmetric) positive semidefinite completion? Proving that this task is at least as hard as dense matrix multiplication or PSD testing would count as a resolution.
    Question 2 (fast “approximate squaring”): given \(A \in \;\mathbb{R}^{n \times n}\) and a set of \(m = \Omega(n)\) entries of \(AA^T\), can I find some PSD matrix that agrees with \(AA^T\) in those m entries in time \(\tilde{O}(nm)\)?
    We'll pay $5k for a solution to either problem. The offer is open for each problem for 3 months or until the problem gets solved (whichever happens first). Winners are welcome to publish solutions independently. Otherwise, if the result ends up being a significant part of a paper, we’ll invite them to be a coauthor.
    We’ll also consider smaller prizes for partial progress, or anything that we find helpful for either solving the problem or realizing we should give up on it.
    To understand the motivation for these questions, you can read our paper on Formalizing the presumption of independence and in particular Appendix D.7.2. ARC is trying to find efficient heuristic estimators as a formalization of defeasible reasoning about quantities like the variance of a neural network's output. These two questions are very closely related to one of the simplest cases where we haven't yet found any reasonable linear time heuristic estimator.
    We don’t expect to receive many incorrect proposals, but if we receive more than 5 we may start applying a higher standard in order to save our time. If we can’t understand a solution quickly, we may ask you to provide more details, and if we still can’t understand it we may reject it. We expect a correct solution to be about as clear and easy to verify as a paper published at STOC.
    For both problems, it’s OK if we incorrectly treat a matrix as PSD as long as all of its eigenvalues are at least \(-\varepsilon\) for a small constant \(\varepsilon\). \(\tilde{O}\) hides polylogarithmic factors in \(n\), \(\varepsilon\), and the max matrix entry. Feel free to ask for other clarifications on our question on Math Overflow, on Facebook, or by email.
    To submit a solution, send an email to [email protected].
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - How Engineers can Contribute to Civilisation Resilience by Jessica Wen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How Engineers can Contribute to Civilisation Resilience, published by Jessica Wen on May 3, 2023 on The Effective Altruism Forum.
    Cross-posted from the High Impact Engineers Resource Portal. You can view the most up-to-date version on the Portal.
    Summary
    Civilisation resilience is concerned with both reducing the risk of civilisation collapse and increasing the capability for humanity to recover from such a collapse. A collapse of civilisation would likely cause a great deal of suffering and may jeopardise the future of the human race. We can defend against such risks by reducing the chances that a localised catastrophe starts, that it scales up to a global catastrophe, or that it triggers irreversible civilisation collapse. Many facets of our defence layers are physical, meaning there are many opportunities for engineers to contribute to improving humanity's resilience.
    Uncertainty
    The content of this article is largely based on research by 80,000 Hours, the Future of Humanity Institute, the Centre for the Study of Existential Risk, and ALLFED. We feel somewhat confident in the recommendations in this article.
    What is civilisation resilience?
    The industrial revolution gave humanity access to unprecedented amounts of valuable and lifesaving technologies and improved the lives we are able to live immensely. However, a global catastrophe could put unprecedented strain on the infrastructure — global agriculture, energy, industry, intercontinental shipping, communications, etc. — that enables civilisation as we know it today. If these systems were to collapse, would we be able to recover and return to the state of civilisation we have today?
    Could we re-industrialise? Would this be possible without easy access to fossil fuels, minerals, and chemicals? Could we rebuild flourishing global societies and infrastructure if there was a breakdown of international relations? Questions such as these fall under the purview of civilisation resilience.
    Civilisation resilience focuses on how we can buttress civilisation against collapse and increase our ability to recover from a collapse if it did occur.
    A framework for thinking about civilisation resilience
    Having a framework with which to analyse the risks and prioritise the strengthening of our defences is useful to sharpen our focus and direct our efforts to bolstering civilisation resilience. The paper Defence in Depth Against Human Extinction: Prevention, Response, Resilience, and Why They All Matter(Cotton-Barratt, Daniel and Sandberg) introduces a framework that breaks down protection against extinction risk into three layers of defence (figure 1). This framework is equally applicable to civilisation collapse given civilisation collapse is a precursor to human extinction.
    In evaluating extinction risk, the defence layers protect against an event becoming a catastrophe, scaling to a global catastrophe, and then wiping out the human race. When considering a given catastrophe, the following three defence layers are proposed:
    Prevention — how can we stop a catastrophe from starting?
    Response — how do we stop it from scaling up to a global catastrophe?
    Resilience — how does a global catastrophe get everyone?
    Figure 1: The three layers of defence against extinction risk (Cotton-Barratt, Daniel, Sandberg)
    One advantage of this characterisation framework is that it can be used to evaluate where the weaknesses are in humanity’s defence against a given catastrophe. If we consider a given catastrophic risk, , we can define an extinction probability,
    where:
    is the probability that risk is not prevented.
    is the probability that the risk gets past the response stage, given that it was not prevented.
    is the probability that the risk causes human extinction, given that it got past the response stage and became a global catastrophe.
    Within th...
    16 min
  • LW - Are Emergent Abilities of Large Language Models a Mirage? [linkpost] by Matthew Barnett
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are Emergent Abilities of Large Language Models a Mirage? [linkpost], published by Matthew Barnett on May 2, 2023 on LessWrong.
    Recent work claims that large language models display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, one can choose a metric which leads to the inference of an emergent ability or another metric which does not. Thus, our alternative suggests that existing claims of emergent abilities are creations of the researcher's analyses, not fundamental changes in model behavior on specific tasks with scale.
    We present our explanation in a simple mathematical model, then test it in three complementary ways: we (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities, (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on BIG-Bench; and (3) show how similar metric decisions suggest apparent emergent abilities on vision tasks in diverse deep network architectures (convolutional, autoencoder, transformers). In all three analyses, we find strong supporting evidence that emergent abilities may not be a fundamental property of scaling AI models.
    This result seems important for two reasons:
    If AI abilities are predictable, then we can forecast when we'll get dangerous capabilities ahead of time, rather than being taken by surprise. This result strengthens the case for a research program of devising a ton of interesting benchmarks to measure how capabilities are improving as a function of scale.
    It provides some evidence against the idea that "understanding is discontinuous", and that important AI abilities will suddenly click together at some level, which is a very loose description of what I understood to be one of the primary intuitions behind AI foom.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Test fit for roles / job types / work types, not cause areas by freedomandutility
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Test fit for roles / job types / work types, not cause areas, published by freedomandutility on May 3, 2023 on The Effective Altruism Forum.
    I often see university EAs aiming to do research projects to test their fit for specific cause areas. I don't think this is a good idea.
    I think if you felt you were good or bad fit for a research project, either you were a good or bad fit for research generally or a specific style of research (qualitative, quantitative, philosophical, primary, secondary, wet-lab, programming, focus groups, interviews, questionnaires, clinical trials).
    For example, it seems very unlikely to me that someone who disliked wet-lab research in biosecurity will enjoy wet-lab research in alternative proteins, but it seems less unlikely that someone who disliked wet-lab research in biosecurity will enjoy dry-lab research in biosecurity.
    Similarly, if you enjoyed literature review based research in one cause areas, I think you are likely to enjoy the same type of research across a range of different cause areas (provided you consider the cause areas to be highly impactful).
    I think decisions on cause areas should be made primarily on your views on what is most impactful (whilst avoiding single player thinking and considering any comparative advantages your background may give you), but decisions on roles / job types / work types should heavily consider what you have enjoyed and have done well.
    I think rather than testing fit for particular cause areas, students should test fit for different roles / job types / work types, such as entrepreneurship / operations, policy / advocacy and a range of different types of research.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min