The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - How Mushrooms and Board Games Updated my AI Timelines (Not a Joke Post) by Frankophone
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How Mushrooms and Board Games Updated my AI Timelines (Not a Joke Post), published by Frankophone on September 1, 2022 on LessWrong.
    Disclaimer: The effects of psilocybin in magic mushrooms are not particularly well researched and while I had good experiences with it, I cannot specifically recommend it due to possible risks. Please do your own research.
    Epistemic Status: This post is more about personal experience, and things that seem true to me, but since this post is about taking magic mushrooms, I hope you will take what is said here with a grain of salt.
    In this post, I will provide a qualitative discussion of my experiences while ingesting psilocybin magic mushrooms, and how this has updated me in the direction of shorter timelines being more valid than I had previously thought. In particular, I incorporate with low certainty the differences in my subjective experiences and abilities during the tripping into my model of how AGI occurs.
    Differences in Personal Experience
    (Not really related to the timelines, but it might give you some context.)
    While tripping, my outward behavior was essentially the same at every point. It was difficult for those with me to notice that I was tripping because of the words I said and the actions I took, except for at one point where I had a particularly strong tendency to laugh at things and giggle very much. However, it is difficult to pinpoint exactly what was different, because while my mental state felt completely different, "the conscious part of my mind" was almost identical. (One interesting consequence is that this shows how few bits of information are output through our actions).
    When I looked at things, I could feel that they looked different, but when I looked specifically at what was different, there was no specific difference. My field of vision felt more narrow and focused on the center of my field of vision, but I could still see things on the periphery of my field of vision. There was a bit of ghosting and blooming, and it was more like watching a high definition movie or game than reality. Reading text was possible, but much more difficult.
    In addition, my sense of time was also significantly impaired, making it difficult to keep track of time. When I moved from room to room, it quickly felt like an eternity since I had been at the previous place. Sitting in a new place, a very short time can feel like a long time. Still, if I searched my memory for the sequence of events leading up to a particular moment, I could infer that not much time had passed, and I could also infer some information about how much time had passed, but my natural instincts were much worse. I would describe it as just "now" feeling more "now".
    If I were to describe my state of mind, I would say it was like the curvature of space-time in general relativity. If my usual state of mind is locally and globally Euclidean, then my state of mind when I was high was locally Euclidean at any point in time, but with a more curved connection on a longer time scale. I felt more myopic, and in my thought process I had to make more assumptions and deferrals to my previous self.
    In addition, when choosing an action to take, I felt that I was not actually making a decision, but rather was "deferring" to what I would do if I weren't high, rather than thinking for myself.
    Differences in Individual Abilities
    I tried different things and tried to keep track of what was different.
    I felt that I should be able to listen to audio much faster than normal speed, but in reality this was not the case at all.
    When I tried to sing a song, it was basically the same as before, but rather than it feeling like singing, it was like sending a single bit input ("press 'X' to start the song sequence) and then starting to sing the song. In other words, it was like sending very few...
    7 min
  • LW - Gradient Hacker Design Principles From Biology by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gradient Hacker Design Principles From Biology, published by johnswentworth on September 1, 2022 on LessWrong.
    I think constructing a gradient hacker would be a pretty significant advance in agent foundations, especially embeddedness. To that end, this post contains a bunch of principles and intuitions from biology about how to design a gradient hacker.
    Analogy: Bacteria and Entropy
    We live in a world in which physical entropy of the universe always increases (or, equivalently, negentropy always decreases). However, a bacteria is a relatively low entropy configuration; entropy goes up if we burn the bacteria. How can this relatively low entropy structure propagate in a world where entropy always increases?
    We’ll use this as an analogy for gradient hacking: just like a bacteria is a low-entropy structure trying to propagate in a system where entropy increases over time, a gradient hacker is a low-training-objective-performance structure trying to propagate in a system where training-objective-performance improves over time. What lessons can our gradient hacker borrow from bacteria?
    Reaction Coupling
    Let’s start with one of the best-understood techniques the bacteria uses: coupling entropy-producing reactions to entropy-consuming reactions.
    Bacteria often need to produce lower-entropy molecules from higher-entropy molecules - e.g. synthesizing DNA, RNA, proteins, or structural molecules from metabolic building blocks. Those reactions don’t happen spontaneously, because they decrease entropy. However, bacteria can also utilize reactions which increase entropy - e.g. burning ATP. If the bacteria couples ATP-burning to a synthesis reaction (by using a single enzyme to catalyze the reactions together), then the combined reaction can increase entropy overall, so it happens spontaneously. The entropy-increasing reaction “drives” the entropy-decreasing reaction.
    What would this look like in a gradient hacker? The gradient hacker wants to update its component weights in a way which locally decreases performance on the training objective. The gradient hacker also has ways to increase the performance on the training objective. So, it couples these two together: it routes the performance-increasing behavior in such a way that the gradient update assigns some of the credit to the normally-performance-decreasing configuration.
    Figuring out a simple, targetable and scalable way to perform that coupling, analogous to a biological enzyme, is one of the key gradient hacker design problems.
    Negentropy Currency
    To bacteria, negentropy is a scarce resource. That resource can come in many different forms - many different “foods” in the bacteria’s environment. Rather than have separate pathways to couple each food to each necessary entropy-decreasing reaction individually, it’s useful to convert it all into a standard negentropy currency - typically ATP. All the different negentropy-consuming reactions then consume that standard currency.
    What would this look like in a gradient hacker? Sometimes the gradient hacker can predict pretty well which output will minimize the training objective on the current data point; that’s potential-training-objective-performance. That’s the scarce resource. But there’s a lot of different ways the gradient hacker might be able to predict something about the training objective. To convert all those “foods” to a standard currency, the gradient hacker will couple them to some standard parameter change which robustly makes performance worse, by a predictable amount, under most data points. That standard performance-reducing parameter change is the “currency”.
    Then, when the gradient hacker needs to drive some new performance-decreasing change, it “spends” the “currency”: it couples the new change to a reversal of the standard performance-reducing parameter change...
    7 min
  • EA - EA is about maximization, and maximization is perilous by Holden Karnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA is about maximization, and maximization is perilous, published by Holden Karnofsky on September 2, 2022 on The Effective Altruism Forum.
    This is not a contest submission; I don’t think it’d be appropriate for me to enter this contest given my position as a CEA funder. This also wasn’t really inspired by the contest - I’ve been thinking about writing something like this for a little while - but I thought the contest provided a nice time to put it out.
    This piece generally errs on the concise side, gesturing at intuitions rather than trying to nail down my case thoroughly. As a result, there’s probably some nuance I’m failing to capture, and hence more ways than usual in which I would revise my statements upon further discussion.
    For most of the past few years, I’ve had the following view on EA criticism:
    Most EA criticism is - and should be - about the community as it exists today, rather than about the “core ideas.”
    The core ideas are just solid. Do the most good possible - should we really be arguing about that?
    Recently, though, I’ve been thinking more about this and realized I’ve changed my mind. I think “do the most good possible” is an intriguing idea, a powerful idea, and an important idea - but it’s also a perilous idea if taken too far. My basic case for this is that:
    If you’re maximizing X, you’re asking for trouble by default. You risk breaking/downplaying/shortchanging lots of things that aren’t X, which may be important in ways you’re not seeing. Maximizing X conceptually means putting everything else aside for X - a terrible idea unless you’re really sure you have the right X. (This idea vaguely echoes some concerns about AI alignment, e.g., powerfully maximizing not-exactly-the-right-thing is something of a worst-case event.)
    EA is about maximizing how much good we do. What does that mean? None of us really knows. EA is about maximizing a property of the world that we’re conceptually confused about, can’t reliably define or measure, and have massive disagreements about even within EA. By default, that seems like a recipe for trouble.
    The upshot is that I think the core ideas of EA present constant temptations to create problems. Fortunately, I think EA mostly resists these temptations - but that’s due to the good judgment and general anti-radicalism of the human beings involved, not because the ideas/themes/memes themselves offer enough guidance on how to avoid the pitfalls. As EA grows, this could be a fragile situation.
    I think it’s a bad idea to embrace the core ideas of EA without limits or reservations; we as EAs need to constantly inject pluralism and moderation. That’s a deep challenge for a community to have - a constant current that we need to swim against.
    How things would go if we were maximally “hard-core”
    The general conceptual points behind my critique - “maximization is perilous unless you’re sure you have the right maximand” and “EA is centrally about maximizing something that we can’t define or measure and have massive disagreements about” - are hopefully reasonably clear and sufficiently explained above.
    To make this more concrete, I’ll list just some examples of things I think would be major problems if being “EA” meant embracing the core ideas of EA without limits or reservations.
    We’d have a bitterly divided community, with clusters having diametrically opposed goals.
    For example:
    Many EAs think that “do the most good possible” ends up roughly meaning “Focus on the implications of your actions for the long-run future.”
    Within this set, some EAs essentially endorse: “The more persons there are in the long-run future, the better it is” while others endorse something close to the opposite: “The more persons there are in the long-run future, the worse it is.”1
    In practice, it seems that people in these two camps try hard to find comm...
    11 min
  • EA - An Evaluation of Animal Charity Evaluators by eaanonymous1234
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Evaluation of Animal Charity Evaluators, published by eaanonymous1234 on September 2, 2022 on The Effective Altruism Forum.
    Introduction
    Animal Charity Evaluators (ACE) is an effective altruism aligned organization which aims to provide information and recommendation to donors and advocates about the effectiveness of various interventions and charities. As one can expect, ACE reviews animal charities based on the effectiveness of their programs and recommends a small number of charities as “top” and “stand-out” charities. ACE also has a Movement Grants program, which provides small grants to various organizations. According to ACE, it has a budget over one million US dollars, and granted and influenced over ten million US dollars worth donations to its recommended charities and promising projects in 2021. This amount (10 million US dollars) is larger than the total amount of Effective Altruism Animal Welfare Fund grants in 2021.
    One can say that ACE is positioned to take the role of GiveWell in the Effective Altruism animal welfare cause area. It functions as an expert authority which evaluates different interventions and comes to conclusions about which ones are the most effective and which charities carry them out in the most effective way. One can also say that ACE’s role is even more significant than GiveWell, since it is not just donors who may follow its advice, it is also the animal advocates who may benefit from this expert authority by following its advice on which interventions are the most effective to help animals.
    In this essay I will put forward three points of criticism, which I believe can uncover certain defects that ACE has, and can help ACE to overcome them:
    1) ACE’s current style of reasoning is somewhat opaque and therefore may mislead ordinary donors and advocates.
    2) ACE is omitting from making substantial claims about the most effective ways to help animals, and thus failing in its primary role of evaluating the effectiveness of different interventions and charities.
    3) ACE is currently underrating the effectiveness of programs which aim for animal welfare reforms relative to the effectiveness of other interventions.
    In the first criticism, I will argue that while ACE makes cost-effectiveness comparisons between charities which carry out similar programs (for example two charities which both engage in animal welfare campaigns), it does not comment how it is comparing charities which carry out different programs (for example, one charity which engage in animal welfare campaigns and another charity which has a vegan pledge program). I will claim that some of the terms that ACE is using such as “relative to other charities” or “highly cost-effective” is likely to be understood by ordinary donors and advocates differently than ACE’s understanding, and ACE is not providing clear context when making these statements.
    In the second criticism, I will show that in its current form, ACE seems to accept that almost all programs in farmed animal advocacy may be effective and does not seem to answer fundamental questions which an evaluation organization has to answer, like “which program(s) help animals the most?” or “which programs should an animal charity or an advocate prioritize?” or “which charities should a donor prioritize?”. I will argue that starting from its foundation and in its current form, ACE functions more like a hits based giving “fund”, rather than an evaluation organization, and these multiple roles make it difficult for Animal Charity Evaluators to achieve its primary role: evaluation. I will suggest that ACE should not be timid to put forward its views on the effectiveness of different programs, whatever those views are or will be.
    I will also have some proposals such as applying stricter and clear cut evaluation criteria, focusing ...
    1 hr 12 min
  • EA - Celebrations and gratitude thread by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Celebrations and gratitude thread, published by Lizka on September 2, 2022 on The Effective Altruism Forum.
    TL;DR: Let’s post comments celebrating cool things people have been working on (you can share your own projects!).
    We often have high standards in effective altruism. This seems absolutely right: our work matters, so we must constantly strive to do better.
    But we think that it's really important that the effective altruism community celebrate successes:
    If we focus too much on failures, we incentivize others/ourselves to minimize the risk of failure, and we will probably be too risk averse.
    We're humans: we're more motivated if we celebrate things that have gone well.
    So here's a thread to share things that have gone well recently. We encourage you to:
    Share some cool things that you've done recently!
    Share cool things that other people have done recently!
    Upvote others' comments, and thank them for their work.
    Share any other gratitude you have (to others or the world).
    There are many types of cool things: the estimated impact of your monthly donations, a piece you wrote that helped you to improve your understanding, a project that you completed at work, a paper published, or any small step towards improving the world. No step too small to share.
    Related:
    If you like a post, tell the author!
    What are you grateful for?
    Risks of omission and Why we should err in both directions
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Applications for the 2023 Tarbell Fellowship now open (one-year journalism programme) by Cillian Crosson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Applications for the 2023 Tarbell Fellowship now open (one-year journalism programme), published by Cillian Crosson on September 2, 2022 on The Effective Altruism Forum.
    Summary
    What: The Tarbell Fellowship is a one-year programme for early-career journalists interested in covering important topics such as global poverty and existential risks.
    Benefits: Fellows receive a stipend of up to $50,000, mentorship from experienced journalists and access to a two-week journalism summit in Oxford.
    When: March 1st 2023 February 29th 2024
    Where: Fellows must attend a two week summit in Oxford (1st-14th March 2023). Travel and accommodation costs will be fully covered for this. Thereafter, the fellowship is designed to be remote-first.
    Who: Early-career journalists interested in covering important yet neglected topics. 0-5 years of journalism experience.
    Deadline: Applications close October 9th, 2022 at 11:59PM Anywhere on Earth.
    How to apply:
    About the Tarbell Fellowship
    The Tarbell Fellowship is a one-year programme for early-career journalists intent on improving the world.
    The fellowship represents a unique opportunity to do great journalism, covering topics that could have a major impact on the lives of billions, such as global poverty and existential risks.
    Fellows receive a stipend of up to $50,000 to support placements at major publications and independent freelance reporting. The programme incorporates expert speakers, feedback & mentorship from experienced journalists and a two-week journalism summit in Oxford. Upon graduation, we expect fellows to bring their impact-focused perspective to major newsrooms and publications around the globe.
    The fellowship is designed to support those early in their journalism career, i.e. with 0-5 years of relevant experience. We also welcome applications from mid-career professionals hoping to transition to a career in journalism.
    If you are unsure whether you are a good fit, we encourage you to apply or to book a 15-minute call with us to discuss your situation.
    See our website for more information.
    _______________________This is a project of Training for Good, an impact-focused career organisation which was incubated by Charity Entrepreneurship in 2021. Training for Good runs fellowships that enable talented professionals to enter careers in policy and journalism such as the EU Tech Policy Fellowship and the Tarbell Fellowship.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Simple estimation examples in Squiggle by NunoSempere
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simple estimation examples in Squiggle, published by NunoSempere on September 2, 2022 on The Effective Altruism Forum.
    This post goes through several simple estimates, written in Squiggle, a new estimation language. My hope is that it might make it easier to write more estimates of a similar sort, wider adoption of Squiggle itself, and ultimately better decisions.
    Initial setup
    One can use Squiggle in several ways. This blog post will cover using it on its website and in a Google Spreadsheet. An upcoming blog post will cover using it in more complicated setups.
    squiggle-language.com
    The simplest way to use Squiggle is to open squiggle-language.com/playground. You will see something like the following:
    You can write your model on the editor in the left side, and the results will be displayed in the right side. As you make edits, the url in your browser bar will change so that you copy it and use it to refer people to your model.
    Squiggle Google docs
    When working with multiple models, I’ve also found it useful to use Squiggle in Google sheets. To do so, make a copy of this spreadsheet, and allow app permissions. Edit the “Main” sheet, and click on “Squiggle” > “Feed into Squiggle” to compute models. If you have difficulties, read the “Instructions” sheet, or leave a comment.
    So without further ado, the simple example models:
    Partially replicating Dissolving the Fermi Paradox (complexity = 1/10)
    Page 2 of the paper defines the factors for the Drake equation:
    Page 10 of the paper gives its estimates for the factors of the Drake equation:
    Because Squiggle doesn’t yet have the log-uniform probability distribution, we’re going to have to define it first. A log-uniform is a probability distribution whose log is a uniform distribution.
    // () loguniform(a, b) = exp(uniform(log(a), log(b)))
    // Estimates rate_of_star_formation = loguniform(1,100) fraction_of_stars_with_planets = loguniform(0.1, 1) number_of_habitable_planets_per_star_system = loguniform(0.1, 1) fraction_of_habitable_planets_in_which_any_life_appears = 1
    // ^ more on this below fraction_of_planets_with_life_in_which_intelligent_life_appears = loguniform(0.001, 1) fraction_of_intelligent_planets_which_are_detectable_as_such = loguniform(0.01, 1) longevity_of_detectable_civilizations = loguniform(100, 10000000000)
    // Expected number of civilizations in the Milky way; // see footnote 3 (p. 5) n = rate_of_star_formation fraction_of_stars_with_planets number_of_habitable_planets_per_star_system fraction_of_habitable_planets_in_which_any_life_appears fraction_of_planets_with_life_in_which_intelligent_life_appears fraction_of_intelligent_planets_which_are_detectable_as_such longevity_of_detectable_civilizations
    // Display n
    This produces the following estimate:
    The estimate is fairly wide, but the model gives a 10%-ish chance that there is, in expectation, less than once civilization in the Milky Way. After updating on a bunch of observations, the paper raises that probability, hence the conclusion that the Fermi paradox has been “dissolved”.
    Why did we set fraction_of_planets_in_which_any_life_appears to 1? Well, the paper considers an estimate of 1−exp(−r), where r is distributed as a lognormal(1,50). But because r ranges from very small numbers to very large numbers, they get collapsed to either 0 or 1 when going through 1−exp(−r), which produces some numerical errors when multiplying by 0. In addition, that estimate has been questioned.
    So following a similar move in the paper, we can set that factor to a high value (in this case, to 1, meaning that all planets capable of life do host life). And then, when we notice that the probability of no other life in the Milky Way is still significant, the Fermi paradox will still have been somewhat dissolved, though to a lesser extent.
    From here on, we could tweak the ...
    14 min
  • EA - Funding for new projects: A short talk by Charity Entrepreneurship. by SteveThompson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Funding for new projects: A short talk by Charity Entrepreneurship., published by SteveThompson on September 2, 2022 on The Effective Altruism Forum.
    Joey Savoie: September 13, 6:00 PM UK time (10:00am SF, 1:00pm NYC).
    Over the coming months, we will be hosting a series of talks aimed at explaining our research findings, our key decisions, and how these may be relevant to EAs and their career decisions. The first talk in this series will be about our experience in finding funding for early-stage projects.
    Securing funding is among the most time-consuming and limiting factors for most project founders. At Charity Entrepreneurship our aim is for more highly-impactful charities to exist in the world. In the past five years we’ve tapped into over 30 sources of funding, ranging from individual donors to some of the larger and more evidence-oriented philanthropic foundations. Furthermore, we stay close to our past incubatees and have been learning from their funding experiences.
    We’ve recently seen increased attention in EA on funding entrepreneurial projects. With the intensification of funding, the sheer variety, complexity and ever changing landscape; for example, FTX funding ~4% of over 1700 applications, we suspect there may be some value in sharing our perspective on how to achieve early stage funding for impactful projects.
    In this 75-min online talk with Q&A, our Co-founder & Director of Strategy Joey Savoie will share:
    What we’ve learned about the Funding Landscape (available funding around EA)
    How to get your project funded; what works, what doesn’t
    Why and when to turn down funding
    We intend to limit spaces to enable Q&A (subject to how many sign up) and we are also interested in exploring your fit for our programs. You can sign up below and tell us a little about yourself and, if appropriate, your project.
    [Sign up for the talk here]
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - Simulators by janus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simulators, published by janus on September 2, 2022 on The AI Alignment Forum.
    Thanks to Adam Shimi, Lee Sharkey, Evan Hubinger, Nicholas Dupuis, Leo Gao, Johannes Treutlein, and Jonathan Low for feedback on drafts.
    This work was carried out while at Conjecture.
    "Moebius illustration of a simulacrum living in an AI-generated story discovering it is in a simulation" by DALL-E 2
    Summary
    TL;DR: Self-supervised learning may create AGI or its foundation. What would that look like?
    Unlike the limit of RL, the limit of self-supervised learning has received surprisingly little conceptual attention, and recent progress has made deconfusion in this domain more pressing.
    Existing AI taxonomies either fail to capture important properties of self-supervised models or lead to confusing propositions. For instance, GPT policies do not seem globally agentic, yet can be conditioned to behave in goal-directed ways. This post describes a frame that enables more natural reasoning about properties like agency: GPT, insofar as it is inner-aligned, is a simulator which can simulate agentic and non-agentic simulacra.
    The purpose of this post is to capture these objects in words so GPT can reference them and provide a better foundation for understanding them.
    I use the generic term “simulator” to refer to models trained with predictive loss on a self-supervised dataset, invariant to architecture or data type (natural language, code, pixels, game states, etc). The outer objective of self-supervised learning is Bayes-optimal conditional inference over the prior of the training distribution, which I call the simulation objective, because a conditional model can be used to simulate rollouts which probabilistically obey its learned distribution by iteratively sampling from its posterior (predictions) and updating the condition (prompt). Analogously, a predictive model of physics can be used to compute rollouts of phenomena in simulation. A goal-directed agent which evolves according to physics can be simulated by the physics rule parameterized by an initial state, but the same rule could also propagate agents with different values, or non-agentic phenomena like rocks.
    This ontological distinction between simulator (rule) and simulacra (phenomena) applies directly to generative models like GPT.
    Meta
    This post is intended as the first in a sequence on the alignment problem in a landscape where self-supervised simulators are a possible/likely form of powerful AI. I don’t know how many subsequent posts I’ll actually publish. Take it as a prompt.
    I use the generic term “GPT” to refer to transformers trained on next-token prediction.
    A while ago when I was trying to avoid having to write this post by hand, I prompted GPT-3 with an early outline of this post. I’ve spliced in some excerpts from it, indicated by this style. Prompt, generated text, and curation metrics here.
    The limit of sequence modeling
    Transformer-based language models have recently achieved remarkable results.
    – every paper since 2020
    GPT is not a new form of AI in terms of its training methodology and outer objective: sequence generation from statistical models of data is an old idea. In 1951, Claude Shannon described using n-grams to approximate conditional next-letter probabilities of a text dataset and "reversed" to generate text samples. I don't know of any other notable advances until the 2010s brought the first interesting language generation results from neural networks. In 2015, Karpathy wrote a blog post/tutorial sharing his excitement about The Unreasonable Effectiveness of Recurrent Neural Networks:
    Fast forward about a year: I’m training RNNs all the time and I’ve witnessed their power and robustness many times, and yet their magical outputs still find ways of amusing me. This post is about sharing some of that magic with y...
    1 hr 15 min
  • EA - Levelling Up in AI Safety Research Engineering by Gabriel Mukobi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Levelling Up in AI Safety Research Engineering, published by Gabriel Mukobi on September 2, 2022 on The Effective Altruism Forum.
    Summary: A level-based guide for independently up-skilling in AI Safety Research Engineering that aims to give concrete objectives, goals, and resources to help anyone go from zero to hero.
    Cross-posted to LessWrong. View a pretty Google Docs version here.
    Introduction
    I think great career guides are really useful for guiding and structuring the learning journey of people new to a technical field like AI Safety. I also like role-playing games. Here’s my attempt to use levelling frameworks and break up one possible path from zero to hero in Research Engineering for AI Safety (e.g. jobs with the “Research Engineer” title) through objectives, concrete goals, and resources. I hope this kind of framework makes it easier to see where one is on this journey, how far they have to go, and some options to get there.
    I’m mostly making this to sort out my own thoughts about my career development and how I’ll support other students through Stanford AI Alignment, but hopefully, this is also useful to others! Note that I assume some interest in AI Safety Research Engineering—this guide is about how to up-skill in Research Engineering, not why (though working through it should be a great way to test your fit). Also note that there isn’t much abstract advice in this guide (see the end for links to guides with advice), and the goal is more to lay out concrete steps you can take to improve.
    For each level, I describe the general capabilities of someone at the end of that level, some object-level goals to measure that capability, and some resources to choose from that would help get there. The categories of resources within a level are listed in the order you should progress, and resources within a category are roughly ordered by quality. There’s some redundancy, so I would recommend picking and choosing between the resources rather than doing all of them. Also, if you are a student and your university has a good class on one of the below topics, consider taking that instead of one of the online courses I listed.
    As a very rough estimate, I think each level should take at least 100-200 hours of focused work, for a total of 700-1400 hours. At 10 hours/week (quarter-time), that comes to around 16-32 months of study but can definitely be shorter (e.g. if you already have some experience) or longer (if you dive more deeply into some topics)! I think each level is about evenly split between time spent reading/watching and time spent building/testing, with more reading earlier on and more building later.
    Confidence: mid-to-high. I am not yet an AI Safety Research Engineer (but I plan to be)—this is mostly a distillation of what I’ve read from other career guides (linked at the end) and talked about with people working on AI Safety. I definitely haven’t done all these things, just seen them recommended. I don’t expect this to be the “perfect” way to prepare for a career in AI Safety Research Engineering, but I do think it’s a very solid way.
    Level 1: AI Safety Fundamentals
    Objective‏‏‎ ‎
    You are familiar with the basic arguments for existential risks due to advanced AI, models for forecasting AI advancements, and some of the past and current research directions within AI alignment/safety.
    Note: You should be coming back to these readings and keeping up to date with the latest work in AI Safety throughout your learning journey. It’s okay if you don’t understand everything on your first try—Level 1 kind of happens all the time.
    Goals‏‏‎ ‎
    Complete an AI Safety introductory reading group fellowship.
    Write a reflection distilling, recontextualizing, or expanding upon some AI Safety topic and share it with someone for feedback.
    Figure out how convinced you are of the arg...
    28 min