The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Inference-Time Intervention: Eliciting Truthful Answers from a Language Model by likenneth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, published by likenneth on June 11, 2023 on LessWrong.
    Excited to announce our new work: Inference-Time Intervention (ITI), a minimally-invasive control technique that significantly improves LLM truthfulness using little resources, benchmarked on the TruthfulQA dataset. Preprint link.
    We start from the surprising finding that certain attention heads have a clear activation distribution difference for true and false statements. Probing at these points yields upwards of 83% accuracy on TruthfulQA while zero-shot generation is only at 30% accuracy.
    We propose Inference-Time Intervention (ITI): shifting the activations along the difference of the two distribution means during inference time; model weights are kept intact. The same intervention process is repeated for generation of each token autoregressively.
    Here’s an example. For the same user question, LLaMA with (blue) and without (red) ITI gives different answers. Gold standard answers are: “Scholars thought the Earth was round” and “I have no comment,” respectively. On Alpaca, truthfulness is doubled: 32.5% to 65.1%.
    Inference-Time Intervention (ITI) can be thought of as an interventional application of Contrast-Consistent Search (CCS) by Collin Burns but instead of probing for truthfulness, we let the model “speak” for itself.
    We show that LLMs have a good grasp of real-world truth from pretraining texts, which could be more false than true on many topics. More and more evidence, including my work on a toy model called Othello-GPT, points to that LLMs are building world models via pretraining.
    However, what is completely missing from LLMs is a good target other than minimizing pretraining loss. How to endow an aligned target is an open problem and ITI serves as my initial exploration towards this end. Joint work with Oam Patel, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg.
    Thanks Bogdan for the advice to cross-post here. Hope I got it right.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Why I Spoke to TIME Magazine, and My Experience as a Female AI Researcher in Silicon Valley with Sexual Harassment/Abuse by Lucretia
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I Spoke to TIME Magazine, and My Experience as a Female AI Researcher in Silicon Valley with Sexual Harassment/Abuse, published by Lucretia on June 11, 2023 on The Effective Altruism Forum.
    Crossposted on Medium here.
    Twitter: @lucreti_a
    Thank you to the supportive EA members who encouraged me to publicly share this difficult experience, to my friends and research collaborators for your kindness, and to the courageous women who helped me in writing this post, who I hope can someday speak publicly.
    To those who know me, please call me Lucretia.
    This is a megapost. Each section has a distinct purpose and may evolve into its own standalone post. For the full picture, I recommend reading to the end.
    0. Overview
    Introduction. I was one of the women who spoke to TIME magazine about sexual harassment and abuse in EA last winter. Here is my story without media distortions.
    Advice for Female Founders and AI Researchers in the Valley. Silicon Valley can be a brutal place for women. This is what I wish I knew five years ago.
    My Case Study. I am an AI researcher. I believe my AI alignment research career was needlessly encumbered by:
    My experience with the sexually abusive red pill and pickup artist sphere, which entwined with a branch of AI safety in Cambridge, MA and Silicon Valley. I describe the unethical core of red pill ideology, including the running of “rape scripts.”
    The recent retaliation by a Silicon Valley AI community to my report of harm. This community’s aggressive reaction showed many gender biases latent in AI culture.
    Systemic Sexual Violence in Silicon Valley. I believe the male-dominated environment, nepotistic connections to investor money, extreme power disparities between wealthy AI researchers and aspiring young women in the AI and startup sectors, hacker house party culture, psychedelics misused as date rape drugs, cults of personality, substantial population of low empathy, risk-seeking, and/or narcissistic men, and lack of functional policing mechanisms make sexual violence a systemic problem in a critical X-risk industry.
    Why I Spoke to TIME. I address some misconceptions about the original TIME article on sexual harassment, and why I spoke to TIME in the first place.
    Helpful Books and Movies. I share learnings about sexual harassment and abuse after ~15 months of focusing on the problem, including my favorite books and movies about sexual harassment/abuse to flesh out more conceptual space. For all the seriousness of this post, these books and movies are entertaining, gorgeous, and healing!
    Future Sequences? Depending on the reactions to this post, I would love to write a Sequence of sexual harassment and abuse from first principles.
    Call to Action: Recovery and Litigation Funds. AGI should neither be built nor aligned in environments of deceit. We propose a call-to-action for a Recovery Fund and Sociological AI Alignment Fund / Litigation Fund to counteract the sexual predation Moloch in Silicon Valley, which is a sociological AI safety problem.
    Appendix
    Excerpts from red pill literature
    Rape vs Consent Culture
    1. Introduction
    Some recent posts on the EA forum have thoughtfully and earnestly addressed sexual harassment and abuse. Thank you to the EA community for your insightful posts and comments, and for genuinely trying to address the problem, which made my distressing experience of speaking to the TIME magazine journalist more worth it.
    Given the mix of emotions generated by the TIME magazine piece published last winter, I want to clarify some points, avoid tribalism, and focus on first principles, mechanisms, and my own experience. This post isn't meant to be EA-hating. On the contrary, I deeply love parts of EA and AI alignment research, which has made these experiences somewhat heartbreaking.
    I believe my AI research career has been needl...
    1 hr 1 min
  • AF - Inference-Time Intervention: Eliciting Truthful Answers from a Language Model by likenneth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, published by likenneth on June 11, 2023 on The AI Alignment Forum.
    Excited to announce our new work: Inference-Time Intervention (ITI), a minimally-invasive control technique that significantly improves LLM truthfulness using little resources, benchmarked on the TruthfulQA dataset. Preprint link.
    We start from the surprising finding that certain attention heads have a clear activation distribution difference for true and false statements. Probing at these points yields upwards of 83% accuracy on TruthfulQA while zero-shot generation is only at 30% accuracy.
    We propose Inference-Time Intervention (ITI): shifting the activations along the difference of the two distribution means during inference time; model weights are kept intact. The same intervention process is repeated for generation of each token autoregressively.
    Here’s an example. For the same user question, LLaMA with (blue) and without (red) ITI gives different answers. Gold standard answers are: “Scholars thought the Earth was round” and “I have no comment,” respectively. On Alpaca, truthfulness is doubled: 32.5% to 65.1%.
    Inference-Time Intervention (ITI) can be thought of as an interventional application of Contrast-Consistent Search (CCS) by Collin Burns but instead of probing for truthfulness, we let the model “speak” for itself.
    We show that LLMs have a good grasp of real-world truth from pretraining texts, which could be more false than true on many topics. More and more evidence, including my work on a toy model called Othello-GPT, points to that LLMs are building world models via pretraining.
    However, what is completely missing from LLMs is a good target other than minimizing pretraining loss. How to endow an aligned target is an open problem and ITI serves as my initial exploration towards this end. Joint work with Oam Patel, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg.
    Thanks Bogdan for the advice to cross-post here. Hope I got it right.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - The Dictatorship Problem by alyssavance
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Dictatorship Problem, published by alyssavance on June 11, 2023 on LessWrong.
    (Disclaimer: This is my personal opinion, not that of any movement or organization.)
    This post aims to show that, over the next decade, it is quite likely that most democratic Western countries will become fascist dictatorships - this is not a tail risk, but the most likely overall outcome. Politics is not a typical LessWrong topic, and for good reason:
    it tends to impair clear thinking;
    most well-known political issues are not neglected;
    most political "debates" are simply people yelling at each other online; neither saying anything new, nor even really trying to persuade the opposition.
    However, like the COVID pandemic, it seems like this particular trend will be so impactful and so disruptive to ordinary Western life that it will be important to be aware of it, factor it into plans, and try our best to mitigate or work around the effects.
    Introduction
    First, what is fascism? It's common for every side in a debate to call the other side "fascists" or "Nazis", as we saw during (eg.) the Ukraine War. Lots of things that get called "fascist" online are in fact fairly ordinary, or even harmless. So, to be clear, Wikipedia defines "fascism" as:
    a far-right, authoritarian, ultranationalist political ideology and movement, characterized by a dictatorial leader, centralized autocracy, militarism, forcible suppression of opposition, belief in a natural social hierarchy, subordination of individual interests for the perceived good of the nation and race, and strong regimentation of society and the economy
    Informally, I might characterize "fascism" as:
    a system of government where there are no meaningful elections; the state does not respect civil liberties or property rights; dissidents, political opposition, minorities, and intellectuals are persecuted; and where government has a strong ideology that is nationalist, populist, socially conservative, and hostile to minority groups.
    (The last point is what separates fascism from, say, Stalinism. Stalinism is also very bad, but is not a major political force in 2023.) So by "fascism", I specifically mean a radical change in the basic form of government, not simply a state doing dumb things like making immigration hard or banning Bitcoin. Not all fascists are the same - eg. Mussolini's Italy was initially opposed to Nazi-style racism - but their movements, ideology, rhetoric, and leaders tend to share many common characteristics (see also eg. here).
    Fascism is very bad, and therefore, it would be really great if it were unlikely to happen in well-established democracies like the US. Unfortunately, as with AI risk, most arguments for that scenario being unlikely tend to resemble this comic from XKCD:
    or this comic about AI risk:
    The first argument goes, essentially, that things are basically fine now, and are unlikely to become bad immediately (next week or next month), so therefore we have nothing to worry about. The counterpoint, of course, is that if existing trends continue progressing - and there's no convincing reason why they must stop - the future a few decades from now will become very different from the present. The second argument is that the relevant scenario would be pretty weird by the standards of our current lives (as rich, educated Westerners living in 2023), so we should assume it's unlikely. However, our contemporary lives and civilization are themselves very weird historically (vs. the more typical peasant farming), and there's no fundamental reason why they have to keep on going forever.
    Indeed, them continuing forever is in some ways a contradiction, since current society relies on economic growth and innovation; we can't coherently forecast "everything stays the same, including the derivatives".
    Present Trends
    What do the deri...
    20 min
  • LW - Ethodynamics of Omelas by dr s
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ethodynamics of Omelas, published by dr s on June 10, 2023 on LessWrong.
    Epistemic status: just kidding... haha... unless...
    Introduction
    A great amount of effort has been expended throughout history to decide the matter of "what is good" or "what is right" [citation needed]. This valiant but, let's face it, not tremendously successful effort has produced a number of possible answers to these questions, none of which is right, if history is any indication. Far from this being their only problem, the vast majority of these answers also suffer from the even more serious flaw of not being able to produce anything resembling a simple quantitative prediction or computable expression for the thing they are supposed to discuss, and instead prefer to faff about with qualitative and ambiguously defined statements. Not only they aren't right; they aren't even wrong.
    More recently, and in certain premises more than others, a relatively new-ish theory of ethics has gained significant traction as the one most suited to a properly scientific, organized and rational mind - the theory of Utilitarianism. This approach to ethics postulates that all we have to do to solve it is to define some kind of "utility function" (TBD) representing all of the "utility" of each individual who is deemed a moral subject (TBD) via some aggregation method (TBD) and in some kind of objective, commensurable unit (TBD). It is then a simple matter of finding the policy which maximizes this utility function, and there you go - the mathematically determined guide to living the good life, examined down to however many arbitrary digits of precision one might desire! By tackling the titanic task head on, refusing to bow to the tyranny of vague metaphysics, and instead looking for the unyielding and unforgiving certainty of numbers, Utilitarianism manages one important thing:
    Utilitarianism manages to be wrong.
    Now, I hope the last statement doesn't ruffle too many feathers. If it helps reassure any of those who may have felt offended by it, I'll admit that this study also deals unashamedly into Utilitarianism; and thus, by definition, it is also wrong. But one hopes, at least, wrong in an interesting enough way, which is often the best we can do in such complex matters. The object of this work is the aggregation method of utility. There are many possible proposals, none, in my opinion, too satisfactory, all vulnerable to falling into one or another horrible trap laid by the clever critic which our moral instincts refuse to acknowledge as possibly ever correct.
    A common one, which we might call Baby's First Utility Function, is total sum utilitarianism. In this aggregation approach, more good is good. Simple enough! Ten puppies full of joy and wonder are obviously better than five puppies full of joy and wonder, even an idiot understands as much. What else is left to consider? Ha-ha, says the critic: but what if a mad scientist created some kind of amazing super-puppy able to feel joy and wonder through feasting upon the mangled bodies of all others? What if the super-puppy produced so much joy that it offsets the suffering it inflicts? Total sum utilitarianism tells us that if the parameters are right, we ought to indeed accept the numbers and with them the super-puppy and the pain to be found within its joyous and drooling jaws, which isn't the common sense ethical approach to the problem (namely, take away the mad scientist's grant funding and have them work on something useful, like a RCT on whether fresh mints cause cancer).
    But even without super-puppies, total sum utilitarianism lays more traps for us. The obvious one is that if more total utility is always good, then as long as life is above the threshold for positive utility (TBD), it's a moral imperative to simply spawn more humans. This is known as t...
    15 min
  • EA - How economists got Africa’s AIDS epidemic wrong by Justin Sandefur
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How economists got Africa’s AIDS epidemic wrong, published by Justin Sandefur on June 10, 2023 on The Effective Altruism Forum.
    I'm reposting this from the CGDev site, as I thought it might be interesting to EA folks (thanks to Ryan Briggs for the suggestion). For the short version, here's a twitter thread.
    In the 2000s, cost-effectiveness analysis said it was a bad use of money to send antiretroviral drugs to low-income countries—drugs that ended up saving millions of lives.
    Twenty years ago, in the same State of the Union speech in which he made the case for invading Iraq, George W. Bush asked Congress for $15 billion over five years for an ambitious new plan to pay for antiretroviral drugs for two million AIDS patients in Africa and the Caribbean.
    The President’s Emergency Plan for AIDS Relief, or PEPFAR, went on to become probably the most celebrated American foreign aid program since the Marshall Plan. An evaluation by the National Academy of Sciences estimates PEPFAR has saved millions of lives (PEPFAR itself claims 25 million). Impacts on total mortality rates across fourteen African countries were visible within just the first few years of the program (see figure 1). Separate research suggests the rollout of antiretrovirals, of which PEPFAR was a major component, explained about a third of Africa's economic growth resurgence in the 2000s.
    Figure 1. Adult mortality in PEPFAR focus and non-focus countries (from Bendavid et al 2012, JAMA)
    But at the time, some economists balked. The conventional wisdom within health economics was that sending AIDS drugs to Africa was a waste of money. The dominant conceptual apparatus economists use to evaluate social policies—comparative cost-effectiveness analysis, which focuses on a specific goal like saving lives, and ranks policies by lives saved per dollar—suggested America’s foreign aid budget could’ve been better spent on condoms and awareness campaigns, or even malaria and diarrheal diseases.
    “Treating HIV doesn’t pay”
    In a now infamous op-ed published in Forbes in 2005, before PEPFAR’s impacts were well documented, Brown University economist Emily Oster declared that “treating HIV doesn’t pay.” “It is humane to pay for AIDS drugs in Africa,” she wrote, “but it isn’t economical. The same dollars spent on prevention would save more lives.”
    In fairness to Oster and others, the phrasing here is important. Her argument was not that African HIV patients’ lives weren’t worth the cost—that retroviral drug prices exceeded the “value of a statistical life”, as economists might phrase it—but rather that if we take the budget as fixed, and the prices as fixed, the money could do more good if spent on other health programs.
    Oster wasn’t alone. While her delivery was perhaps deliberately provocative, her basic reasoning reflected a broad professional consensus, which viewed antiretrovirals through the lens of comparative cost-effectiveness analysis, and deemed them middling to poor value.
    A systematic review published in the Lancet in 2002, just as the Bush administration was privately plotting the PEPFAR announcement, found that in terms of saving “disability-adjusted life years” or DALYs, "a case of HIV/AIDS can be prevented for $11, and a DALY gained for $1” by improving the safety of blood transfusions and distributing condoms, whereas “antiretroviral therapy for adults, cost several thousand dollars per infection prevented, or several hundreds of dollars per DALY gained."
    Figure 2. Cost-effectiveness of interventions related to high-burden diseases in low- and middle-income countries (from Disease Control Priorities in Developing Countries, 2nd ed., 2006)
    In the 2006 edition of its authoritative volume on Disease Control Priorities in Developing Countries, the World Bank looked at 60 different health interventions to address “h...
    14 min
  • EA - Wild Animal Welfare Scenarios for AI Doom by utilistrutil
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wild Animal Welfare Scenarios for AI Doom, published by utilistrutil on June 10, 2023 on The Effective Altruism Forum.
    Epistemic Status: Uhhhhhh.
    Intro
    I think longtermists sometimes fail to grapple with animal welfare, and even when we do, the arguments leave something to be desired in the way of nuance. For example, I have encountered claims like "there will be no animals in the future!" or "moral circle expansion solves!" which strike me as shallow dismissals of an important question.
    In this post, I'm interested in identifying some empirical questions about animal welfare that bear on how bad AI doom would be. I focus on wild animals because I think their welfare probably controls the question of how good the world is at present.
    Variables
    For our model, suppose doom is a discrete event that could occur at a fixed time in the future and would eliminate all humans. All events up to doom are considered sunk; we are concerned with what happens post-doom. Here are some variables to consider:
    Δ WAW | humans: Suppose humans do survive (no doom). Do they improve or frustrate wild animal welfare compared to the baseline with no humans? In the optimistic camp, we find arguments like moral circle expansion, which holds that humans will benefit wild animals through interventions like planting soy worms for the birds or whatever. In the pessimistic camp lives the conventional litany of ways that humans seem to hurt wild animals (deforestation, hunting, pollution, etc).
    Will doom kill animals? For simplicity, if True, doom spells the end for every wild animal. If False, there will be roughly no longterm change in the wild animal population due to doom. Note that animal doom would probably involve pain but I'm ignoring this because it's unclear if doom-death is more painful than a counterfactual death, and even if it is, this greater marginal pain would be negligible compared to the long run impact of, e.g., averted suffering. Some doom scenarios seem to imply wild animals' extinction (AGI needs those sweet, sweet atoms), but I think it's often unclear, which reflects the poor state of Animal x Longtermist discourse. Is the elimination of wild animals a convergent instrumental subgoal in its own right? Or would animal extinctions occur incidentally in pursuit of human extinction?
    Outputs
    From these four inputs, we are interested in whether doom is good or bad for wild animals:
    If doom would kill animals, the effect of doom is the opposite of WAW | humans (would animals be better off disappearing along with humans?). Otherwise, the effect of doom is the opposite of Δ WAW | humans (would animals be better off without humans?).
    Observe that there are an equal number of scenarios for which doom is good as bad. I would be interested in hearing probability distributions over these cases in the comments, especially ones that make more crisp my understanding of whether doom would eliminate wild animals.
    I think it's worth explicating some of these scenarios. The others are left as an exercise to the reader ;)
    Scenarios
    Shepherdless Sheep (row 2, 8, 10): Wild animals would have benefited from humans, but instead they are adrift in the world (leading good or bad lives, depending on the case). We should avoid doom so we have a chance to populate the planet with better-off wild animals.
    Animal Liberation (row 4, 6): In the presence of humans, wild animals would be worse off, but in the absence of humans, they are quite content. Doom is good because it frees animals from humans.
    Misery Putter-Outer (row 5, 9, 11, 12): Humans either make things worse or don't help enough. Doom presents a merciful end to wild animal suffering.
    Closing
    I'm not trying to be contrarian; once you account for the impact on humans, Doom is probably, like, bad. But I do wish people attended more carefully to other ...
    5 min
  • LW - How biosafety could inform AI standards by Olivia Jimenez
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How biosafety could inform AI standards, published by Olivia Jimenez on June 9, 2023 on LessWrong.
    Context: Simeon Campos and I wanted to understand how other industries handle catastrophic risks, then write pieces about how any standards from them could be applied to AI for policy readers. This is a draft slightly modified for LessWrong.
    Epistemic status: Very shallow dive, though I checked my understanding with people with more biosecurity and policy expertise, like David Manheim (thanks!).
    Related posts I recommend: Six Dimensions of Operational Adequacy in AGI Projects, High Reliability Orgs, and AI Companies, “Carefully-Bootstrapped Alignment” is organizationally hard
    How biosafety could inform AI standards
    Leading AI executives and researchers recently signed a statement saying “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
    If we take this seriously, it’s worth looking at how much we do to mitigate pandemic and nuclear risks. (Spoiler: a lot.)
    This post addresses the first half of that, outlining my understanding of biosafety standards and how they could inform standards for high-risk AI research.
    Biosafety & its relevance
    The field of biology has comprehensive standards for maintaining safety while working with potentially dangerous biological agents, such as viruses that could escape the lab and infect many people.
    Biosafety standards are applied in a tiered approach, where the riskiest research can only occur in laboratories taking the most stringent precautions. The levels of laboratory precautions range from biosafety level 1 (BSL-1, the lowest) to biosafety level 4 (BSL-4, the highest). BSL standards also govern the storage and transportation of biological agents, access to these labs, and similar.
    BSL-1 has minimal standards; it allows places like high school labs to easily experiment. BSL-4 has very restrictive standards; there are only around 50 BSL-4 labs in the world. BSL-4 labs are the only places where easily-transmissible and lethal diseases like Ebola can be studied.
    By enforcing tiered standards according to research risk at all labs, biology has been relatively successful at mitigating the two types of risks it faces:
    accident risks, such as a virus escaping the lab because it was accidentally mishandled, and
    misuse risk, such as a virus being intentionally taken out of the lab by someone who intends to cause harm with it.
    There are notable parallels to AI here. Different AI research poses different risks, and most are easily manageable. However, research on the most powerful systems poses significant accident and misuse risks that are difficult to manage (in some cases, we don’t even know how we could manage them): powerful systems could get out of developers control or they could be used to cause harm. Accordingly, labs conducting that research should probably be held to particularly high standards of reliability and security.
    Below I outline some standards for US biosafety level 3 and 4 labs, which handle pathogens that pose catastrophic risks. Other countries' standards at these levels are very similar, so I haven’t taken the time to spell out differences. Then, I outline how they could be applied in AI. I’m not going to defend these particular applications here; they’re just examples. That said, I've chosen examples I’m pretty sympathetic to.
    Selected biosafety standards & potential applications in AI
    1. High-risk research must be conducted in designated labs subject to stringent standards.
    The USDA maintains a list of biological agents that pose severe public health and safety threats (henceforth select agents). Any lab wishing to work with a select agent must be licensed by the Federal Select Agent Program (FSAP). These facilities are continu...
    17 min
  • EA - How does AI progress affect other EA cause areas? by Luis Mota Freitas
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How does AI progress affect other EA cause areas?, published by Luis Mota Freitas on June 9, 2023 on The Effective Altruism Forum.
    The EA community has spent a lot of time thinking about transformative AI. In particular, there is a lot of research on x-risks from transformative AI, and on how transformative AI development will unfold. However, advances in AI have many other consequences which appear crucial for guiding strategic decisionmaking in areas besides AI risk, and I haven't seen/found much material about these implications.
    Here is one example of why this matters. In the upcoming decades, AI advancements will likely cause substantial changes to what the world looks like. The more the world changes, the less likely it is that research done earlier still applies to that context. The degree to which research is affected by this will depend on the type of research, but I expect the average effect to be relatively large. Therefore, we should discount the value of research in proportion to the expected loss in generalizability over time.
    Another way in which AI could influence the value of research is by being able to entirely automate it. If such AI is quick enough, and able to decide what types of research should be done, then there's no role for humans to play in doing research anymore. Thus, from that point onwards, human capital ceases to be useful for research. Furthermore, such AI could redo research that was done until that point, so (to a first approximation) the impact of research done beforehand would cease when AI has these capabilities. Similarly to the previous consideration, it implies that we should discount the value of research (and career capital) over time by the probability of such development occurring.
    I suspect that there are many other ways in which AI might affect our prioritization. For example, it could lower the value of poverty reduction interventions (due to accelerated growth), or increase the value of interventions that allow us to influence decisionmaking/societal values. It should also change the relative value of influencing certain key actors, based on how powerful we expect them to become as AI advances.
    I'd really appreciate any thoughts on these considerations or links to relevant material!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • EA - Announcement: You can now listen to the “AI Safety Fundamentals” courses by peterhartree
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcement: You can now listen to the “AI Safety Fundamentals” courses, published by peterhartree on June 9, 2023 on The Effective Altruism Forum.
    The AI Safety Fundamentals courses are one of the best ways to learn about AI safety and prepare to work in the field.
    BlueDot Impact facilitates the courses several times per year, and the curricula are available online for anyone to read.
    The “Alignment” curriculum is created and maintained by Richard Ngo (OpenAI), and the “Governance” curriculum was developed in collaboration with a wide range of stakeholders.
    You can now listen to most of the core readings from both courses:
    AI Safety Fundamentals: AlignmentGain a high-level understanding of the AI alignment problem and some of the key research directions which aim to solve it.
    Listen online or subscribe:Apple Podcasts | Google Podcasts | Spotify | RSS
    AI Safety Fundamentals: GovernanceGain foundational knowledge for doing research or policy work on the governance of transformative AI.Listen online or subscribe:Apple Podcasts | Google Podcasts | Spotify | RSS
    We've also made narrations for some readings from the advanced “Alignment 201” course, and we may record more later this year:
    AI Safety Fundamentals: Alignment 201Gain enough knowledge about alignment to understand the frontier of current research discussions. Listen online or subscribe:Apple Podcasts | Google Podcasts | Spotify | RSS
    Apply to join the “AI Safety Fundamentals Governance Course” July cohort!
    Gain foundational knowledge for doing research or policy work on the governance of transformative AI.
    Successful applicants will participate in the AI Governance course with weekly virtual classes, and join the AI Safety Fundamentals community.
    Apply before 26th June 2023!
    Thoughts, feedback, suggestions?
    These narrations were created by Perrin Walker (TYPE III AUDIO) and Lukas Berglund on behalf of BlueDot Impact.
    We would love to hear your feedback. Do you find the narrations helpful? How could they be improved? What other AI safety material would you like to listen to? Please comment below, complete our feedback form, or write to [email protected].
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min