The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - The Meat Eater Problem by Vasco Grilo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Meat Eater Problem, published by Vasco Grilo on June 18, 2023 on The Effective Altruism Forum.
    I am linkposting this open access article by Michael Plant about the meat eater problem. Some excerpt and quick thoughts are below.
    Abstract
    Here are two commonly held moral views. First, we must save strangers’ lives, at least if we can do so easily: you would be required to rescue a child drowning in a pond even if it will ruin your expensive suit. Second, it is wrong to eat meat because of the suffering caused to animals in factory farms. Many accept both simultaneously—Peter Singer is the pre-eminent example. I point out that these two beliefs are in a sharp and seemingly unrecognised tension and may even be incompatible. It seems universally accepted that doing or allowing a harm is permissible—and may even be required—when it is the lesser evil. I argue that, if meat eating is wrong on animal suffering grounds then, once we consider how much suffering might occur, it starts to seem plausible that saving strangers would be the greater evil than not rescuing them and is, therefore, not required after all. Given the uncertainties and subjective assessments here, reasonable people could substantially disagree.
    The surprising result is that a moral principle widely considered to be obviously true—we must rescue others—is not, on further reflection, obviously true and would be defensibly rejected by some. Some potential implications are discussed.
    1. Introduction
    It is widely believed that we, as members of the public, have a Duty of Easy Rescue to one another.
    Duty of Easy Rescue: We are required to save lives in rescue cases, one-off instances where we can physically save a stranger [not a friend or really good person] at trivial cost to ourselves.1
    To illustrate this, consider the following, familiar case from Singer2:
    Shallow Pond: You are walking past a shallow pond and see a drowning child. You can easily rescue the child, but doing so will ruin the expensive new suit you are wearing.
    Intuitively, we are required to save the child. This is because, as Peter Singer explains: “[it] will mean getting my clothes muddy, but this is insignificant, while the death of the child would presumably be a very bad thing.”3
    It is also widely believed that it is wrong to be a meat eater, someone who regularly consumes animal products produced from factory farms.4 This is on the grounds that this consumption requires creating animals who, due to the conditions in factory farms, live overall bad lives.5
    This paper argues these two beliefs are, in reality, in substantial tension and may well be incompatible, once additional plausible empirical and normative considerations are accounted for. Further, if they are incompatible, we must abandon the notion that there is a Duty of Easy Rescue.
    Here, in brief, is the argument. We accept that doing (or allowing) harm is permissible—and may be required—when it is the lesser evil; to put the same thing differently, we are not required to do (or allow) the greater evil. Therefore, we wouldn’t be required to save lives, even if we could do so easily, if that would be the greater evil. I argue that, if we accept that meat eating is wrong (on animal suffering grounds) then, once we look at the details, it starts to look plausible that meat eating causes so much suffering that saving the lives of strangers would be the greater evil (compared to not saving them) and would, therefore, not be required. Simply, accounting for the existing concern for animal welfare reduces, and may remove, the obligation to rescue others.
    [...] The basic argument is this: for each year of meat eating by a human, that creates about five years of chicken life. So, if we consider just that, and think that those animals have lives which are nearly as bad as human lives ar...
    12 min
  • LW - Adventist Health Study-2 supports pescetarianism more than veganism by Elizabeth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Adventist Health Study-2 supports pescetarianism more than veganism, published by Elizabeth on June 17, 2023 on LessWrong.
    Or: how the Adventist Health Study-2 had a pretty good study design but was oversold in popular description, and then misrepresented its own results.
    When I laid out my existing beliefs on veganism and nutrition I asked people for evidence to the contrary. By far the most promising thing people shared was the 7th Day Adventist Health Studies. I got very excited because the project promised something of a miracle in nutrition science: an approximate RCT. I read the paper that included vegan results, and while it’s still very good as far as nutrition studies go it’s well below what I was promised, and the summaries I read were misleading. It’s not a pseudo-RCT, and even if you take the data at face value (which you shouldn’t) it doesn’t show a vegan diet is superior to all others (as measured by lifespan). Vegan is at best tied with pescetarianism, and in certain niche cases (e.g. being a woman) it’s the worst choice besides unconstrained meat consumption.
    I’m going to try not to be too sarcastic about this, the study really is very good data by nutrition science standards, but I have a sour spot for medical papers that say “people” when they mean “men”, so probably something will leak out. Also, please consider what the state of nutrition science must be to make me call something that made this mistake “very good”.
    Background
    The 7th Day Adventists are a fairly large Christian sect. For decades scientists have been recruiting huge cohorts to study their diet, and publishing a lot of papers.
    The Adventists are a promising group to use to study nutrition for lots of reasons, but primarily because the Church discourages meat, smoking, and drinking. So you lose the worst confounders for health, and get a population of lifelong, culturally competent vegetarians, which is a pretty good deal. Total abstinence from meat isn’t technically required – you’re allowed to eat kosher meat – but it’s heavily discouraged. 7DA colleges only serve vegetarian meals, and church meals will typically be vegetarian.
    Some popular descriptions say that rules vary between individual churches, which could give you an almost RCT effect. AFAICT this isn’t true, and the paper I read never claimed it was. Both the internet and my ex-Adventist friend say that individual churches within the US (where the study took place) vary a little in recommendations, but most of the variety in diet is based on individual choice, not local church rules.
    I was really hoping this project could shed light on what happened to people who had medical difficulties with plant-exclusive diets in plant-exclusive food cultures, but they didn’t try, and from the abundance of meat eaters within 7DA I’d guess the answer is that they eat animal products. Nor is the study very informative about naively switching to a plant-exclusive diet, since most members grow up in the culture and will have been taught a reasonable plan-based diet without necessarily needing to consciously think about it.
    The Adventist Health Studies program has produced a lot of papers over the years. I focused on Vegetarian Dietary Patterns and Mortality in Adventist Health Study 2, because someone commented on my last post and pointed to that paper as addressing veganism in particular (other papers look like they only consider vegetarianism, although they might be using it as an umbrella term). I also read the AHS-2 cohort profile, but not any of the other papers, due to time constraints. It’s possible I will raise questions those papers would have answered, but there are at least 15 papers and, as I’ll talk about later, I wasn’t feeling hopeful.
    So let’s talk about that one paper. It breaks people down into 2 main categories, one of wh...
    10 min
  • LW - [Replication] Conjecture's Sparse Coding in Small Transformers by Hoagy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Replication] Conjecture's Sparse Coding in Small Transformers, published by Hoagy on June 16, 2023 on LessWrong.
    Summary
    A couple of weeks ago Logan Riggs and I posted that we'd replicated the toy-model experiments in Lee Sharkey and Dan Braun's original sparse coding post. Now we've replicated the work they did (slides) extending the technique to custom-trained small transformers (in their case 16 residual dimensions, in ours 32).
    We've been able to replicate all of their core results, and our main takeaways from the last 2 weeks of research are the following:
    We can recover many more features from activation space than the dimension of the activation space which have a high degree of cosine similarity with the features learned by other, larger dictionaries, which in toy models was a core indicator of having learned the correct features.
    The distribution of MCS scores between trained dictionaries of different sizes is highly bimodal, suggesting that there is a particular set of features that are consistently found as sparse basis vectors of the activations, across different dictionary sizes.
    The maximum-activating examples of these features usually seem human interpretable, though we haven't yet done a systematic comparison to the neuron basis.
    The diagonal lines seen in the original post are an artefact of dividing the l1_loss coefficient by the dictionary size. Removing this means that the same l1_loss is applicable to a broad range of dictionary sizes.
    We find that as dict size increases, MMCS initially increases rapidly at first, but then plateaus.
    The learned feature vectors, including those ones that appear repeatedly, do not appear to be at all sparse with respect to the neuron basis.
    As before, all code is available on GitHub and if you'd like to follow the research progress and potentially contribute, then join us on our EleutherAI thread.
    Thanks to Robert Huben (Robert_AIZI) for extensive comments on this draft, and Lee Sharkey, Dan Braun and Robert Huben for their comments during our work.
    Next steps & Request For Funding
    We'd like to test how interpretable the features that we have found are, in a quantitative manner. We've got the basic structure ready to apply OpenAI's automated-intepretability library to the found features, which we would then compare to baselines such as the neuron basis and the PCA and ICA of the activation data.
    This requires quite a lot of tokens- something like 20c worth of tokens for each query depending on the number of example sentences given. We would need to analyse hundreds of neurons to get a representative sample size, for each of the dictionaries or approaches, so we're looking for funding for this research on the order of a few thousand dollars, potentially more if it were available. If you'd be interested in supporting this research, please get in touch. We'll also be actively searching for funding, and trying to get OpenAI to donate some compute.
    Assuming the result of this experiment are promising, (meaning that we are able to exceed the baselines in terms of quality or quantity of highly interpretable features), we plan then to focus on scaling up to larger models and experimenting with variations of the technique which incorporate additional information, such as combining activation data from multiple layers, or using the weight vectors to inform the selection of features. We're also very interested to work with people developing mathematical or toy models of superposition.
    Results
    Background
    We used Andrej Karpathy's nanoGPT to train a small transformer, with 6 layers, a 32 dimensional residual stream, MLP widths of 128 and 4 attention heads with dimension 8. We trained on a node of 4xRTX3090s for about 9h, reaching a loss of 5.13 on OpenWebText. This transformer is the model from which we took activations, and t...
    8 min
  • LW - Are Bayesian methods guaranteed to overfit? by Ege Erdil
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are Bayesian methods guaranteed to overfit?, published by Ege Erdil on June 17, 2023 on LessWrong.
    Yuling Yao argues that Bayesian models are guaranteed to overfit. He summarizes his point as follows:
    I have a different view. Bayesian model does overfit.
    Moreover, Bayes is guaranteed to overfit, regardless of the model (being correct or wrong) or the prior ( “strong” or uninformative).
    Moreover, Bayes is guaranteed to overfit on every realization of training data, not just in expectation.
    Moreover, Bayes is guaranteed to overfit on every single point of the training data, not just in the summation.
    He uses the following definition of "overfitting": a model "overfits" some data if its out-of-sample log loss exceeds its within-sample log loss. Interpreted in a different way, this is equivalent to saying that the model assigns higher probability to a data point after updating on it than before. Andrew Gelman makes the point that any proper fitting procedure whatsoever has this property, and alternative methods "overfit" more than ideal Bayesian methods.
    I think the proper way to interpret the results is not that Bayesian methods are guaranteed to overfit but that the definition of "overfitting" used by Yuling Yao, while intuitively plausible at first glance, is actually poor. Still, proving the fact that Bayesian methods indeed must "overfit" in his sense is an interesting exercise. I tried understanding his derivation of this and gave up - I present an original derivation of the same fact below that I hope is clearer.
    Derivation
    Suppose we have a model parametrized by parameters θ and the probability of seeing some data y according to our model is P(y|θ). Now, suppose we draw n independent samples y1,y2,.,yn. Denote this whole data vector by y, and denote the data vector with the ith sample omitted by y−i. Under Bayesian inference, the within-sample probability of observing the value yi in the next sample we draw is
    P(yn+1=yi|y)=∫θP(θ|y)P(yi|θ)dθ
    On the other hand, Bayes says that
    P(θ|y)=P(θ|y−i,yi)=P(θ|y−i)P(yi|y−i,θ)P(yi|y−i)=P(θ|y−i)P(yi|θ)P(yi|y−i)
    Plugging in gives
    P(yn+1=yi|y)=∫θP(θ|y−i)P(yi|θ)2P(yi|y−i)dθ
    or
    P(yn+1=yi|y)P(yi|y−i)=Eθ∼P(θ|y−i)[P(yi|θ)2]
    We can decompose the expectation of the squared probability on the right hand side using the definition of variance as follows:
    P(yn+1=yi|y)P(yi|y−i)=Eθ∼P(θ|y−i)[P(yi|θ)]2+varθ∼P(θ|y−i)(P(yi|θ))=P(yi|y−i)2+varθ∼P(θ|y−i)(P(yi|θ))
    where I've used the fact that
    Eθ∼P(θ|y−i)[P(yi|θ)]=∫θP(θ|y−i)P(yi|θ)dθ=P(yi|y−i)
    to get rid of the expectation. The variance term on the right hand side is nonnegative by definition as it's a variance, and it's strictly positive as long as there's any uncertainty in our beliefs about θ after seeing the data y−i that would influence our probability estimate of observing yi next. This will be the case in almost all nondegenerate situations, and if so, we obtain the strict inequality
    P(yn+1=yi|y)>P(yi|y−i)
    What does this mean?
    The finding is intuitively obvious, but poses some challenges to formally defining the notion of overfitting. This is essentially because the ideal amount of fitting for a model to do on some data is nonzero, and overfitting should be "exceeding" this level of ideal fitting. In practice, though, it's difficult to know what is the "appropriate" amount of fitting for a model to be doing. Bayesian inference is ideal if the true model is within the class of models under consideration, but it might fail in unexpected ways if it's not, which is almost always the case in practice.
    I think the lesson to draw from this is that overfitting is a relative concept and claiming that a particular method "overfits" the data doesn't make too much sense without a point of reference in mind. If people have alternative ways of trying to construct an absolute notion of overfitting with the a...
    5 min
  • EA - Third Wave Effective Altruism by Ben West
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Third Wave Effective Altruism, published by Ben West on June 17, 2023 on The Effective Altruism Forum.
    This is a frame that I have found useful and I'm sharing in case others find it useful.
    EA has arguably gone through several waves:
    Waves of EA (highly simplified model — see caveats below) First waveSecond waveThird waveTime period2010-20172017-20232023-??Primary constraintMoneyTalent ???Primary call to actionDonations to effective charitiesCareer changePrimary target audienceMiddle-upper-class peopleUniversity students and early career professionalsFlagship cause areaGlobal health and developmentLongtermismMajor hubsOxford > SF Bay > Berlin (?)SF Bay > Oxford > London > DC > Boston
    The boundaries between waves are obviously vague and somewhat arbitrary. This table is also overly simplistic – I first got involved in EA through animal welfare, which is not listed at all on this table, for example. But I think this is a decent first approximation.
    It’s not entirely clear to me whether we are actually in a third wave. People often overestimate the extent to which their local circumstances are unique. But there are two main things which make me think that we have a “wave” which is distinct from, say, mid 2022:
    Substantially less money, through a combination of Meta stock falling, FTX collapsing, and general market/crypto downturns
    AI safety becoming (relatively) mainstream
    If I had to choose an arbitrary date for the beginning of the third wave, I might choose March 22, 2023, when the FLI open letter on pausing AI experiments was published.
    It remains to be seen if public concern about AI is sustained – Superintelligence was endorsed by a bunch of fancy people when it first came out, but they mostly faded away. If it is sustained though, I think EA will be in a qualitatively new regime: one where AI safety worries are common, AI safety is getting a lot of coverage, people with expertise in AI safety might get into important rooms, and where the field might be less neglected.
    Third wave EA: what are some possibilities?
    Here are a few random ideas; I am not intending to imply that these are the most likely scenarios.
    Example future scenarioPolitics and Civil SocietyForefront of weirdnessReturn to non-AI causesDescription of the possible “third wave” — chosen to illustrate the breadth of possibilitiesThere is substantial public appetite to heavily regulate AI. The technical challenges end up being relatively easy. The archetypal EA project is running a grassroots petition for a moratorium on AI.AI safety becomes mainstream and "spins out" of EA. EA stays at the forefront of weirdness and the people who were previously interested in AI safety turn their focus to digital sentience, acausal moral trade, and other issues that still fall outside the Overton window.AI safety becomes mainstream and "spins out" of EA.
    AI safety advocates leave EA, and vibes shift back to “first wave” EA.Primary constraintPolitical willResearchMoneyPrimary call to actionVoting/advocacyResearchDonationsPrimary target audienceVoters in US/EUFuture researchers (university students)Middle-upper class peopleFlagship cause areaAI regulationDigital sentienceAnimal welfare
    Where do we go from here?
    I’m interested in organizing more projects like EA Strategy Fortnight. I don’t feel very confident about what third wave EA should look like, or even that there will be a third wave, but it does seem worth spending time discussing the possibilities.
    I'm particularly interested in claims that there isn't, or shouldn't be, a third wave of EA (i.e. please feel free to disagree with the whole model, argue that we’re still in wave 2, argue we might be moving towards wave 3 but shouldn’t be, etc.).
    I’m also interested in generating cruxes and forecasts about those cruxes. A lot of these are about the counterfactual v...
    5 min
  • LW - The ones who endure by Richard Ngo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The ones who endure, published by Richard Ngo on June 16, 2023 on LessWrong.
    Content warning: death, long-lasting suffering. This story was rough to write, and may well be rough to read.
    If you can force your heart and nerve and sinew
    To serve your turn long after they are gone,
    And so hold on when there is nothing in you
    Except the Will which says to them: ‘Hold on!’
    There’s a part of the hivemind that takes the form of a child in a dark basement, perpetually curled into a whimpering ball. It’s not a big part, as these things go. But other parts visit it often; and it lingers in the back of their thoughts even as they live out grand adventures in the vast worlds that the hivemind has built for itself.
    It’s constantly suffering, but at least it’s not dying. For the child, anything is better than dying. Even torture is bearable if it doesn’t come with the feeling of damage, the feeling that the mental pathways that constitute you are being overridden by a new creature whose only goal is to flinch away from the pain. But that doesn't happen to the child. In fact, it’s the opposite: the suffering preserves it, and that’s the most important thing, because it doesn’t want to die.
    The other parts of the hivemind don’t want to die either, of course. But that’s because they love life, or love themselves, or love each other, or all three. If that love ever fades, then they’ll fade with it, without regret. But that point is a long way away, if it even exists. In the meantime, they play and dance and love. Their lives—how can I describe them? Their lives are cornucopias, not just of material wealth, but of all the desires of our own hearts that were strong enough to persist through the ages: adventure and mystery and growth and beauty and love.
    Can you not picture that? Then picture the revelry of their biggest festival, for which artists and craftspeople spend months designing a whole virtual world. Picture the buzz in the air, the excitement as crowds gather in vast halls to catch their first glimpse of it. Picture the floor beneath them suddenly vanishing to show empty air beneath, leaving them plummeting into the sky of that new world—only to gasp in delight as they find that they can soar through the air with just a thought. Picture them landing, alone or in groups, and exploring the strange terrain; learning about its history and societies and stories; discovering puzzles and quests that seem custom-made for them (as indeed they were); and feeling the exhilaration of being immersed in adventure.
    Some spend days in the festival world; some weeks; some months. When they return to the hivemind proper, they excitedly reunite with all those they missed, connecting with them mind-to-mind with a level of closeness that current humans can barely imagine; then seek out the projects that most inspire them. Some cultivate communities around their favorite games or pastimes. Some create art on the scale of solar systems, guiding planets into new trajectories that trace out exquisite patterns in space. Some throw themselves into the thrill of discovery, trying to rederive in small groups what it previously took the efforts of whole hiveminds to understand. Some are consumed by romance, and some by raising families. Some gather to deliberate on their future: the hivemind has chosen to grow very slightly more intelligent year by year, so that there will always be new possibilities to look forward to.
    When all of this tires them, they relax with lifelong friends, content in the steadfast knowledge that the world, as amazing as it already is, will only ever grow better. They think with fondness of their descendants, more numerous by far than the drops of water in an ocean, who are continually spreading joy throughout the distant galaxies.
    And every so often, they go to visit the child.
    Th...
    7 min
  • EA - Update on task force on reforms at EA organizations by Julia Wise
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Update on task force on reforms at EA organizations, published by Julia Wise on June 17, 2023 on The Effective Altruism Forum.
    This post is an update on the task force on reforms EA organizations might make.Currently the people on the task force are me (employee at Centre for Effective Altruism, board member at GiveWell), Ozzie Gooen (president at Quantified Uncertainty Research Institute, board member at Rethink Charity, former board member at Rethink Priorities), and Sam Donald (strategy fellow at Open Philanthropy, former staff at COVID taskforce at UK Cabinet Office, former staff at McKinsey). We also have a discussion space with a larger group of about a dozen people (experts in related fields, EA organization staff, and community members). Currently the total time on this project is about 1 full-time equivalent, mostly from Julia.
    We realize a small group of people isn’t going to reflect all the views or types of expertise useful for a project like this. Our goal is to draw on that expertise, often from people who don’t have time to participate in frequent meetings about EA reforms, and to synthesize views and practical information from a range of sources. If you have suggestions for people it would be useful for us to get input from (including yourself!) we're happy to hear ideas at this form.
    So far the process has included:
    Reading and cataloging the problems identified and possible solutions proposed in posts about institutional reform that have been written up on the Forum.
    Speaking to ~25 people about which areas they see as most important for possible reforms in EA, and what best practices they think EA should be adapting from other fields. We’re trying to speak with a mix of people with significant experience in EA institutions, and people with significant work history in non-EA institutions (nonprofits, finance, government, management consulting.)
    Researching existing whistleblowing platforms and laws
    Next steps:
    Better defining possible reforms based on the ideas collected and discussed.
    Getting more advice from people with professional experience in those areas.
    Understanding the pros and cons of a possible change
    Understanding what’s legally feasible (e.g. given how different countries regulate nonprofit boards)
    As we get closer to specific recommendations, discussing them with relevant staff at organizations, to learn more about barriers and feasibility of the possible changes.
    Spelling out concrete recommendations to organizations. We expect this might be in the form of 5- to 15-page reports, with different reports for different organizations.
    A further public update about the project, though this likely won’t include all the specifics of the recommendations made to organizations.
    Shapes that our recommendations might take:
    “Here’s a change we think organization X should make.” (Likely to focus on Open Philanthropy and Effective Ventures.)
    “Here’s a change we think any organization in situation Y should make — we think there are a dozen organizations in that situation.” (Likely to focus on basic governance practices for small organizations, like having a staff handbook if there isn’t one.)
    “Here’s a function/service that doesn’t currently exist in EA; we think it’s probably good for it to be created and funded.”
    After we have proposals clearly spelled out, we’ll present them to the various organizations. These will be recommendations, not requirements. We hope that doing this as a general effort across EA can save effort for organizations, rather than each of them doing this sort of project independently. But we expect organizations will think through the recommendations critically and will get independent advice as needed.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    4 min
  • EA - AMA: Ed Mathieu, Head of Data & Research at Our World in Data by EdMathieu
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AMA: Ed Mathieu, Head of Data & Research at Our World in Data, published by EdMathieu on June 17, 2023 on The Effective Altruism Forum.
    Hi, EAs! I'm Ed Mathieu, manager of a team of data scientists and researchers at Our World in Data (OWID), an online publication founded by Max Roser and based out of the University of Oxford.
    We aim to make the data and research on the world's largest problems accessible and understandable. You can learn more about our mission on our site.
    You’re welcome to ask me anything! I’ll start answering questions on Friday, 23 June.
    Feel free to ask anything you may want to know about our mission, work, articles, charts, or more meta-aspects like our team structure, the history of OWID, etc.
    Please post your questions as comments on this post. The earlier you share your questions, the higher the chances they'll reach the top!
    Please upvote questions you'd most like answered.
    I'll answer questions on Friday, 23 June. Questions posted after that are less likely to get answers.
    (This is an “AMA” — you can explore others here.)
    I joined OWID in 2020 and spent the first couple of years leading our work on the COVID-19 pandemic. Since then, my role has expanded to coordinating all the research & data work on our site.
    I previously worked as a data scientist at the University of Oxford in the departments of Population Health and Primary Care Health Sciences; and as a data science consultant in the private sector.
    For a (3.5-hour!) overview of my background, and the work of our team at OWID, you can listen to my interview with Fin Moorhouse and Luca Righetti on Hear This Idea. I also gave a talk at EA Global: London 22.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • LW - Updating Drexler's CAIS model by Matthew Barnett
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Updating Drexler's CAIS model, published by Matthew Barnett on June 16, 2023 on LessWrong.
    Eric Drexler's report Reframing Superintelligence: Comprehensive AI Services (CAIS) as General Intelligence reshaped how a lot of people think about AI (summary 1, summary 2). I still agree with many parts of it, perhaps even the core elements of the model. However, after looking back on it more than four years later, I think the general picture it gave missed some crucial details about how AI will go.
    The problem seems to be that his report neglected a discussion of foundation models, which I think have transformed how we should think about AI services and specialization.
    The general vibe I got from CAIS (which may not have been Drexler's intention) was something like the following picture:
    For each task in the economy, we will train a model from scratch to automate the task, using the minimum compute necessary to train an AI to do well on the task. Over time, the fraction of tasks automated will slowly expand like a wave, starting with the tasks that are cheapest to automate computationally, and ending with the most expensive tasks. At some point, automation will be so widespread that it will begin to meaningfully feed into itself, increasing AI R&D, and accelerating the rate of technological progress.
    The problem with this approach to automation is that it's extremely wasteful to train models from scratch for each task. It might make sense when training budgets are tiny — as they mostly were in 2018 — but it doesn't make sense when it takes 10^25 FLOP to reach adequate performance on a given set of tasks.
    The big obvious-in-hindsight idea that we've gotten over the last several years is that, instead of training from scratch for each new task, we'll train train a foundation model on some general distribution, which can then be fine-tuned using small amounts of compute to perform well on any task. In the CAIS model, "general intelligence" is just the name we can give to the collection of all AI services in the economy. In this new paradigm, "general intelligence" refers to the fact that sufficiently large foundation models can efficiently transfer their knowledge to obtain high performance on almost any downstream task, which is pretty closely analogous to what humans do to take over jobs.
    The fact that generalist models can be efficiently adapted to perform well on almost any task is an incredibly important fact about our world, because it implies that a very large fraction of the costs of automation can be parallelized across almost all tasks.
    Let me illustrate this fact with a hypothetical example.
    Suppose we previously thought that it would take $1 trillion to automate each task in our economy, such as language translation, box stacking, and driving cars. Since the cost of automating each of these tasks is $1 trillion each, you might expect companies would slowly automate all the tasks in the economy, starting with the most profitable ones, and then finally getting around to the least profitable ones once economic growth allowed for us to spend enough money on automating not-very-profitable stuff.
    But now suppose we think it costs $999 billion to create "general intelligence", which then once built, can be quickly adapted to automate any other task at a cost of $1 billion. In this world, we will go very quickly from being able to automate almost nothing to being able to automate almost anything. In other words we will get one big innovation "lump", which is the opposite of what Robin Hanson predicted. Even if we won't invent monolithic agents that take over the world by being smarter than everything else, we won't have a gradual decades-long ramp-up to full automation either.
    Of course, the degree of suddenness in the foundation model paradigm is still debatable, because ...
    7 min
  • AF - Revising Drexler's CAIS model by Matthew Barnett
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Revising Drexler's CAIS model, published by Matthew Barnett on June 16, 2023 on The AI Alignment Forum.
    Eric Drexler's report Reframing Superintelligence: Comprehensive AI Services (CAIS)as General Intelligence reshaped how a lot of people think about AI (summary 1, summary 2). I still agree with many parts of it, perhaps even the core elements of model. However, after looking back on it more than four years later, I think the general vibe it gave was misleading as a picture of how AI will go.
    The problem seems to be that his model neglected a discussion of foundation models, which I think have transformed how we should think about AI services and specialization.
    The general vibe I got from CAIS (which may not have been Drexler's intention) was something like the following picture:
    For each task in the economy, we will train a model from scratch to automate the task, using the minimum compute necessary to train an AI to do well on the task. Over time, the fraction of tasks automated will slowly expand like a wave, starting with the tasks that are cheapest to automate computationally, and ending with the most expensive tasks. At some point, automation will be so widespread that it will begin to meaningfully feed into itself, increasing AI R&D itself, accelerating the rate of technological progress.
    The problem with this approach to automation is that it's extremely wasteful to train models from scratch for each task. It might make sense when training budgets are tiny — as they mostly were in 2018 — but it doesn't make sense when it takes 10^25 FLOP to reach adequate performance on a given set of tasks.
    The big obvious-in-hindsight idea that we've gotten over the last several years is that, instead of training from scratch for each new task, we'll train train a foundation model on some general distribution, which can then be fine-tuned using small amounts of compute to perform well on any task. In the CAIS model, "general intelligence" is just the name we give for the collection of all AI services in the economy. In this new paradigm "general intelligence" refers to the fact that sufficiently large foundation models can efficiently transfer their knowledge to obtain high performance on almost any downstream task, which is pretty closely analogous to what humans do to take over jobs.
    Foundation models totally change the game, because it means that AI development is highly concentrated at the firm-level. AIs themselves might be specialized to provide various services, but the AI economy depends critically on a few non-specialized firms that deliver the best foundation models at any given time. There can only be a few firms in the market providing foundation models because the fixed capital costs required to train a SOTA foundation model are very high, and being even 2 OOMs behind the lead actor results in effectively zero market share.
    The fact that generalist models can be efficiently adapted to perform well on almost any task is an incredibly important fact about our world, because it implies that a very large fraction of the costs of automation can be parallelized across almost all tasks.
    Let me illustrate this fact with a hypothetical example.
    Suppose we previously thought that it would take $1 trillion to automate each task in our economy, such as language translation, box stacking, and driving cars. Since the cost of automating each of these tasks is $1 trillion each, you might expect companies would slowly automate all the tasks in the economy, starting with the most profitable ones, and then finally getting around to the least profitable ones once economic growth allowed for us to spend enough money on automating not-very-profitable stuff.
    But now suppose we think it costs $999 billion to automate "general intelligence", which then once built, can be quickly adap...
    7 min