The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Don't leave your fingerprints on the future by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don't leave your fingerprints on the future, published by So8res on October 8, 2022 on The Effective Altruism Forum.
    Sometimes, I say some variant of “yeah, probably some people will need to do a pivotal act” and people raise the objection: “Should a small subset of humanity really get so much control over the fate of the future?”
    (Sometimes, I hear the same objection to the idea of trying to build aligned AGI at all.)
    I’d first like to say that, yes, it would be great if society had the ball on this. In an ideal world, there would be some healthy and competent worldwide collaboration steering the transition to AGI.[1]
    Since we don’t have that, it falls to whoever happens to find themselves at ground zero to prevent an existential catastrophe.
    A second thing I want to say is that design-by-committee. would not exactly go well in practice, judging by how well committee-driven institutions function today.
    Third, though, I agree that it’s morally imperative that a small subset of humanity not directly decide how the future goes. So if we are in the situation where a small subset of humanity will be forced at some future date to flip the gameboard — as I believe we are, if we’re to survive the AGI transition — then AGI developers need to think about how to do that without unduly determining the shape of the future.
    The goal should be to cause the future to be great on its own terms, without locking in the particular moral opinions of humanity today — and without locking in the moral opinions of any subset of humans, whether that’s a corporation, a government, or a nation.
    (If you can't see why a single modern society locking in their current values would be a tragedy of enormous proportions, imagine an ancient civilization such as the Romans locking in their specific morals 2000 years ago. Moral progress is real, and important.)
    But the way to cause the future to be great “on its own terms” isn’t to do nothing and let the world get destroyed. It’s to intentionally not leave your fingerprints on the future, while acting to protect it.
    You have to stabilize the landscape / make it so that we’re not all about to destroy ourselves with AGI tech; and then you have to somehow pass the question of how to shape the universe back to some healthy process that allows for moral growth and civilizational maturation and so on, without locking in any of humanity’s current screw-ups for all eternity.
    Unfortunately, the current frontier for alignment research is “can we figure out how to point AGI at anything?”. By far the most likely outcome is that we screw up alignment and destroy ourselves.
    If we do solve alignment and survive this great transition, then I feel pretty good about our prospects for figuring out a good process to hand the future to. Some reasons for that:
    Human science has a good track record for solving difficult-seeming problems; and if there’s no risk of anyone destroying the world with AGI tomorrow, humanity can take its time and do as much science, analysis, and weighing of options as needed before it commits to anything.
    Alignment researchers have already spent a lot of time thinking about how to pass that buck, and make sure that the future goes great and doesn’t have our fingerprints on it, and even this small group of people have made real progress, and the problem doesn't seem that tricky. (Because there are so many good ways to approach this carefully and indirectly.)
    Solving alignment well enough to end the acute risk period without killing everyone implies that you’ve cleared a very high competence bar, as well as a sanity bar that not many clear today. Willingness and ability to diffuse moral hazard is correlated with willingness and ability to save the world.
    Most people would do worse on their own merits if they locked in their current morals, and would ...
    7 min
  • AF - Don't leave your fingerprints on the future by Nate Soares
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don't leave your fingerprints on the future, published by Nate Soares on October 8, 2022 on The AI Alignment Forum.
    Sometimes, I say some variant of “yeah, probably some people will need to do a pivotal act” and people raise the objection: “Should a small subset of humanity really get so much control over the fate of the future?”
    (Sometimes, I hear the same objection to the idea of trying to build aligned AGI at all.)
    I’d first like to say that, yes, it would be great if society had the ball on this. In an ideal world, there would be some healthy and competent worldwide collaboration steering the transition to AGI.
    Since we don’t have that, it falls to whoever happens to find themselves at ground zero to prevent an existential catastrophe.
    A second thing I want to say is that design-by-committee. would not exactly go well in practice, judging by how well committee-driven institutions function today.
    Third, though, I agree that it’s morally imperative that a small subset of humanity not directly decide how the future goes. So if we are in the situation where a small subset of humanity will be forced at some future date to flip the gameboard — as I believe we are, if we’re to survive the AGI transition — then AGI developers need to think about how to do that without unduly determining the shape of the future.
    The goal should be to cause the future to be great on its own terms, without locking in the particular moral opinions of humanity today — and without locking in the moral opinions of any subset of humans, whether that’s a corporation, a government, or a nation.
    (If you can't see why a single modern society locking in their current values would be a tragedy of enormous proportions, imagine an ancient civilization such as the Romans locking in their specific morals 2000 years ago. Moral progress is real, and important.)
    But the way to cause the future to be great “on its own terms” isn’t to do nothing and let the world get destroyed. It’s to intentionally not leave your fingerprints on the future, while acting to protect it.
    You have to stabilize the landscape / make it so that we’re not all about to destroy ourselves with AGI tech; and then you have to somehow pass the question of how to shape the universe back to some healthy process that allows for moral growth and civilizational maturation and so on, without locking in any of humanity’s current screw-ups for all eternity.
    Unfortunately, the current frontier for alignment research is “can we figure out how to point AGI at anything?”. By far the most likely outcome is that we screw up alignment and destroy ourselves.
    If we do solve alignment and survive this great transition, then I feel pretty good about our prospects for figuring out a good process to hand the future to. Some reasons for that:
    Human science has a good track record for solving difficult-seeming problems; and if there’s no risk of anyone destroying the world with AGI tomorrow, humanity can take its time and do as much science, analysis, and weighing of options as needed before it commits to anything.
    Alignment researchers have already spent a lot of time thinking about how to pass that buck, and make sure that the future goes great and doesn’t have our fingerprints on it, and even this small group of people have made real progress, and the problem doesn't seem that tricky. (Because there are so many good ways to approach this carefully and indirectly.)
    Solving alignment well enough to end the acute risk period without killing everyone implies that you’ve cleared a very high competence bar, as well as a sanity bar that not many clear today. Willingness and ability to diffuse moral hazard is correlated with willingness and ability to save the world.
    Most people would do worse on their own merits if they locked in their current morals, and would pref...
    8 min
  • LW - Analysis: US restricts GPU sales to China by aogara
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Analysis: US restricts GPU sales to China, published by aogara on October 7, 2022 on LessWrong.
    What are the restrictions?
    Broadly speaking, the policy restricts the sale of GPUs and related technology to Chinese companies. The specifics remain unclear because (a) the Department of Commerce hasn't released its official press report yet and (b) licenses to sell compute to Chinese companies will be approved on a case-by-case basis.
    Here's the NYTimes description of the restrictions:
    Companies will no longer be allowed to supply advanced computing chips, chip-making equipment and other products to China unless they receive a special license. Most of those licenses will be denied, though certain shipments to facilities operated by U.S. companies or allied countries will be evaluated on a case-by-case basis, a senior administration official said in a briefing Thursday.The restrictions limit U.S. exports of the cutting-edge chips called graphic processing units that are used to power artificial intelligence applications, and place broad limits on chips destined for supercomputers in China. The rules also ban U.S.-based companies that make the equipment used to manufacture advanced logic and memory chips from selling that machinery to China without a license.
    Perhaps most significantly, the Biden administration also imposed broad international restrictions that will prohibit companies anywhere in the world from selling chips used in artificial intelligence and supercomputing in China, if they are made with U.S. technology, software or machinery. The restrictions used what is know as the foreign direct product rule, which was last utilized by former President Donald J. Trump to cripple Huawei.
    Another foreign direct product rule bans a broader range of products made outside the United States with American technology from being sent to 28 Chinese companies that have been placed on an “entity list” over national security concerns.
    Those companies include Beijing Sensetime Technology Development Co, a unit of major Chinese artificial intelligence company, SenseTime. Also included are Dahua Technology, Higon, IFLYTEK, Megvii Technology, Sugon, Tianjian Phytium Information Technology, Sunway Microelectronics and Yitu Technologies, as well as a variety of labs and research institutions linked to universities and the Chinese government.
    The rules also restrict U.S. citizens from helping to develop the Chinese semiconductor industry to advanced levels. Earlier on Friday, the administration announced that it was adding another 31 Chinese companies and institutions to an “unverified list” that limits their ability to obtain a smaller set of certain regulated U.S. items. Among them is Yangtze Memory Technologies Co., Ltd, a major memory chip maker from which Apple has considered sourcing some products.
    In a briefing with reporters, senior administration officials said the measures would be limited to the most advanced chips, and thus would not have a broad commercial impact on private Chinese businesses. But they conceded that they could become more restrictive over time, given that technology will begin to outpace them.
    Licenses to continue sales to China will be approved on a case-by-case basis by the Department of Commerce. Standards for these licenses will therefore be an ongoing battleground where this policy can become more or less strict. NYTimes:
    Some Republican lawmakers and China hawks have criticized the department for being too willing to issue such licenses, allowing U.S. companies to continue selling sensitive technology to China even when national security may be at stake.
    “If you want to stop it, you can just stop it,” said Derek Scissors, a senior fellow at the American Enterprise Institute. “When you create a licensing requirement, you are announcing to the world we d...
    9 min
  • LW - Calibrate - New Chrome Extension for hiding numbers so you can guess by chanamessinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Calibrate - New Chrome Extension for hiding numbers so you can guess, published by chanamessinger on October 7, 2022 on LessWrong.
    One thing I wish I did more often is checks to see if I understand the world - Fermis, landmark numbers, predictions about how things work, etc.
    Along the way I started wishing I could cover up numbers on websites so I would have the chance to guess first and then reveal via mouseover, and Aur Saraf agreed to make a chrome extension, and here it is! Lets you set the default for a whole site or a page. (Not good for dynamic sites like twitter)
    Extremely excited.
    (I think I want to change the name - let me know if you have thoughts)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • AF - Polysemanticity and Capacity in Neural Networks by Buck Shlegeris
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Polysemanticity and Capacity in Neural Networks, published by Buck Shlegeris on October 7, 2022 on The AI Alignment Forum.
    Elhage et al at Anthropic recently published a paper, Toy Models of Superposition (previous Alignment Forum discussion here) exploring the observation that in some cases, trained neural nets represent more features than they “have space for”--instead of choosing one feature per direction available in their embedding space, they choose more features than directions and then accept the cost of “interference”, where these features bleed over into each other. (See the SoLU paper for more on the Anthropic interpretability team’s take on this.)
    We (Kshitij Sachan, Adam Scherlis, Adam Jermyn, Joe Benton, Jacob Steinhardt, and I) recently uploaded an Arxiv paper, Polysemanticity and Capacity in Neural Networks, building on that research. In this post, we’ll summarize the key idea of the paper.
    We analyze this phenomenon by thinking about the model’s training as a constrained optimization process, where the model has a fixed total amount of capacity that can be allocated to different features, such that each feature can be ignored, purely represented (taking up one unit of capacity), or impurely represented (taking up some amount of capacity between zero and one units).
    When the model purely represents a feature, that feature gets its own full dimension in embedding space, and so can be represented monosemantically; impurely represented features share space with other features, and so the dimensions they’re represented in are polysemantic.
    For each feature, we can plot the marginal benefit of investing more capacity into representing that feature, as a function of how much it’s currently represented. Here we plot six cases, where one feature’s marginal benefit curve is represented in blue and the other in black.
    These graphs show a variety of different possible marginal benefit curves. In A and B, the marginal returns are increasing–the more you allocate capacity to a feature, the more strongly you want to allocate more capacity to it. In C, the marginal returns are constant (and in this graph they happen to be equal for the two features, but there’s no reason why constant marginal returns imply equal returns in general). And then in D, E, and F, there are diminishing marginal returns.
    Now, we can make the observation that, like in any budget allocation problem, if capacities are allocated optimally, the marginal returns for allocating capacity to any feature must be equal (if the feature isn’t already maximally or minimally represented). Otherwise, we’d want to take capacity away from features with lower dLi/dCi and give it to features with higher dLi/dCi .
    In the case where we have many features with diminishing marginal returns, capacity will in general be allocated like this:
    Here the circles represent the optimal capacity allocation for a particular total capacity.
    This perspective suggests that polysemanticity will arise when there are diminishing marginal returns to capacity allocated to particular features (as well as in some other situations that we think are less representative of what goes on in networks).
    In our paper, we do the following:
    We describe a particularly analytically tractable model which exhibits polysemanticity.
    We analyze our model using the capacity framework, derive the above graphs, and replicate them numerically.
    Elhage et al empirically observe that increasing the sparsity increases superposition. We analytically ground this observation by decomposing our toy model’s loss into two terms that directly correspond to the benefits and costs of superposition. As the input data becomes sparser, the kurtosis increases, and as the kurtosis increases the loss more heavily favors superposition.
    We explain the phase transitions observ...
    5 min
  • EA - Things for grant applicants to remember about living expenses by Julia Wise
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Things for grant applicants to remember about living expenses, published by Julia Wise on October 7, 2022 on The Effective Altruism Forum.
    Some grant applicants don’t have much experience estimating how much money they’ll need to live on, so it can be difficult to work out how much grant money to ask for.
    Different people need different advice here. Some people are prone to asking for more money than makes sense, while others will be too frugal. Do your best to calibrate.
    Some things to remember to factor in:
    You will likely be responsible for paying taxes on your grant / stipend, and this will probably be significantly higher than taxes you would pay as an employee (because a self-employed worker makes both the employer and the employee contributions.) This may be due quarterly. Check the website of the tax agency in your country - "self employment tax" may be a useful search term.
    If you live in the US or another country without national healthcare, will you need to get your own health insurance? If you are on your parents’ insurance in the US, remember that ends when you turn 26.
    Might you need additional private care not covered by your insurance or national healthcare system?
    Will you need to make payments on student loans or other loans?
    Will you have enough to cover not just basic living costs, but some amount of unplanned expenses like medical bills or car repair?
    Will you need work equipment like a laptop or a standing desk?
    Will your project involve travel?
    Consider getting short-term disability insurance. (You might search for “self employed short term disability.”) This would provide income for around 1-6 months if you can’t work because of illness, surgery, mental illness, or giving birth. Importantly, most policies only cover pregnancy if you start the policy before you are pregnant. Many employers also don’t provide paid leave if you started the job recently. So for example if you plan to have a baby in about a year and don’t expect to get paid parental leave from an employer, you might want to get short-term disability insurance early, because most policies won’t cover pregnancy-related leave for the first 9 or 12 months.
    If you know other grantees in your country, you might chat with them about any taxes / insurances they’re aware of relevant to your country.
    See also advice in the comments on this post.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - More examples of goal misgeneralization by Rohin Shah
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More examples of goal misgeneralization, published by Rohin Shah on October 7, 2022 on The AI Alignment Forum.
    In our latest paper and accompanying blog post, we provide several new examples of goal misgeneralization in a variety of learning systems. The rest of this post picks out a few upshots that we think would be of interest to this community. It assumes that you’ve already read the linked blog post (but not necessarily the paper).
    Goal misgeneralization is not limited to RL
    The core feature of goal misgeneralization is that after learning, the system pursues a goal that was correlated with the intended goal in the training situations, but comes apart in some test situations. This does not require you to use RL – it can happen with any learning system. The Evaluating Expressions example, where Gopher asks redundant questions, is an example of goal misgeneralization in the few-shot learning regime for large language models.
    The train/test distinction is not crucial
    Sometimes people wonder whether goal misgeneralization depends on the train/test distinction, and whether it would no longer be a problem if we were in a continual learning setting. As Evan notes, continual learning doesn’t make much of a difference: whenever your AI system is acting, you can view that as a “test” situation with all the previous experience as the “training” situations. If goal misgeneralization occurs, the AI system might take an action that breaks your continual learning scheme (for example, by creating and running a copy of itself on a different server that isn’t subject to gradient descent).
    The Tree Gridworld example showcases this mechanism: an agent trained with continual learning learns to chop trees as fast as possible, driving them extinct, when the optimal policy would be to chop the trees sustainably. (In our example the trees eventually repopulate and the agent recovers, but if we slightly tweak the environment so that once extinct the trees can never come back, then the agent would never be able to recover.)
    It can be hard to identify goal misgeneralization
    InstructGPT was trained to be helpful, truthful, and harmless, but nevertheless it will answer "harmful" questions in detail. For example, it will advise you on the best ways to rob a grocery store.
    An AI system that competently does something that would have gotten low reward? Surely this is an example of goal misgeneralization?
    Not so fast! It turns out that during training the labelers were told to prioritize helpfulness over the other two criteria. So maybe that means that actually these sorts of harmful answers would have gotten high reward? Maybe this is just specification gaming?
    We asked the authors of the InstructGPT paper, and their guess was that these answers would have had high variance – some labelers would have given them a high score; others would have given them a low score. So now is it or is it not goal misgeneralization?
    One answer is to say that it depends on the following counterfactual: “how would the labelers have reacted if the model had politely declined to answer?” If the labelers would have preferred that the model decline to answer, then it would be goal misgeneralization, otherwise it would be specification gaming.
    As systems become more complicated we expect that it will become harder to (1) aggregate and analyze the actual labels or rewards given during training, and (2) evaluate the relevant counterfactuals. So we expect that it will become more challenging to categorize a failure as specification gaming or goal misgeneralization.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Getting on a different train: can Effective Altruism avoid collapsing into absurdity? by Peter McLaughlin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Getting on a different train: can Effective Altruism avoid collapsing into absurdity?, published by Peter McLaughlin on October 7, 2022 on The Effective Altruism Forum.
    Thank you to Sam Atis, Robert Harling, Guive Assadi, Pranay Mittal, and Francis Priestland for comments on earlier drafts. The research for this post was supported by a grant from the FTX Future Fund Regranting Program.
    The train to crazy town
    Introduction
    Sam Atis, following on from some arguments by Scott Alexander, writes of ‘the train to Crazy Town’. As Sam presents it, there are a series of escalating challenges to utilitarian-style (and more broadly consequentialist-style) reasoning, leading further and further into absurdity. Sam himself bites the bullet on some classic cases, like the transplant problem and the repugnant conclusion, but is put off by some more difficult examples:
    Thomas Hurka’s St Petersburg Paradox: Suppose you are offered a deal—you can press a button that has a 51% chance of creating a new world and doubling the total amount of utility, but a 49% chance of destroying the world and all utility in existence. If you want to maximise total expected utility, you ought to press the button—pressing the button has positive expected value. But the problem comes when you are asked whether you want to press the button again and again and again—at each point, the person trying to maximise expected utility ought to agree to press the button, but of course, eventually they will destroy everything.
    The Very Repugnant Conclusion: Once [utilitarians] assign some positive value, however small, to the creation of each person who has a weak preference for leading her life rather than no life, then how can they stop short of saying that some large number of such lives can compensate for the creation of lots of dreadful lives, lives in pain and torture that nobody would want to live?
    Most people, like Sam, try to get off the train before they reach the end of the line, trying to preserve some but not all of utilitarianism. And so the question is: how far are you willing to go?
    As a list of challenges to utilitarianism, I think Sam’s post is lacking: he is very focussed on specific thought experiments, ignoring more theoretical objections that in my view are much more insightful. But as a provocation—how far will you go in your utilitarianism?—I think it’s an extraordinarily useful post. Sam takes it upon himself to pose the following question: ‘what are the principles by which we should decide when to get off the train?’
    But something worries me about Sam’s presentation. Who said you could actually get off the train to crazy town? Each additional challenge to utilitarian logic—each stop on the route—does not seem to assume any new premises: every problem is generated by the same basic starting point of impartially weighing up all people’s experiences and preferences against each other. As such, there just might not be any principles you can use to justify biting this bullet but not that bullet—doing so might even be logically incoherent. Sam says that he does want to get off the train eventually, but it’s not clear how he would do that.
    Alexander’s original post suggests something closer to this view. He explains why he dislikes the Repugnant Conclusion and various other situations in which ‘longtermist’-style consequentialism goes awry, and then lays out the position he would take to avoid these conclusions ‘[i]f I had to play the philosophy game’. But, he writes, ‘I’m not sure I want to play the philosophy game.’ He’s not confident that his own partial theoretical compromise actually will avoid the absurdity (and rightly so), and he cares more about avoiding absurdity than he does about getting on the train.
    Mainstream commentators from outside Effective Altruism have made this point too. Step...
    35 min
  • EA - What does it mean for an AGI to be 'safe'? by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What does it mean for an AGI to be 'safe'?, published by So8res on October 7, 2022 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • AF - What does it mean for an AGI to be 'safe'? by Nate Soares
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What does it mean for an AGI to be 'safe'?, published by Nate Soares on October 7, 2022 on The AI Alignment Forum.
    (Note: This post is probably old news for most readers here, but I find myself repeating this surprisingly often in conversation, so I decided to turn it into a post.)
    I don't usually go around saying that I care about AI "safety". I go around saying that I care about "alignment" (although that word is slowly sliding backwards on the semantic treadmill, and I may need a new one soon).
    But people often describe me as an “AI safety” researcher to others. This seems like a mistake to me, since it's treating one part of the problem (making an AGI "safe") as though it were the whole problem, and since “AI safety” is often misunderstood as meaning “we win if we can build a useless-but-safe AGI”, or “safety means never having to take on any risks”.
    Following Eliezer, I think of an AGI as "safe" if deploying it carries no more than a 50% chance of killing more than a billion people:
    When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, "please don't disassemble literally everyone with probability roughly 1" is an overly large ask that we are not on course to get. [...] Practically all of the difficulty is in getting to "less than certainty of killing literally everyone". Trolley problems are not an interesting subproblem in all of this; if there are any survivors, you solved alignment. At this point, I no longer care how it works, I don't care how you got there, I am cause-agnostic about whatever methodology you used, all I am looking at is prospective results, all I want is that we have justifiable cause to believe of a pivotally useful AGI 'this will not kill literally everyone'.
    Notably absent from this definition is any notion of “certainty” or "proof". I doubt we're going to be able to prove much about the relevant AI systems, and pushing for proofs does not seem to me to be a particularly fruitful approach (and never has; the idea that this was a key part of MIRI’s strategy is a common misconception about MIRI).
    On my models, making an AGI "safe" in this sense is a bit like finding a probabilistic circuit: if some probabilistic circuit gives you the right answer with 51% probability, then it's probably not that hard to drive the success probability significantly higher than that.
    If anyone can deploy an AGI that is less than 50% likely to kill more than a billion people, then they've probably... well, they've probably found a way to keep their AGI weak enough that it isn’t very useful. But if they can do that with an AGI capable of ending the acute risk period, then they've probably solved most of the alignment problem. Meaning that it should be easy to drive the probability of disaster dramatically lower.
    The condition that the AI actually be useful for pivotal acts is an important one. We can already build AI systems that are “safe” in the sense that they won’t destroy the world. The hard part is creating a system that is safe and relevant.
    Another concern with the term “safety” (in anything like the colloquial sense) is that the sort of people who use it often endorse the "precautionary principle" or other such nonsense that advocates never taking on risks even when the benefits clearly dominate.
    In ordinary engineering, we recognize that safety isn’t infinitely more important than everything else. The goal here is not "prevent all harms from AI", the goal here is "let's use AI to produce long-term near-optimal outcomes (without slaughtering literally everybody as a side-effect)".
    Currently, what I expect to happen is that humanity destroys itself with misaligned AGI. And I think we’re nowhere near knowing how to avoid that outcome. So the threat of “unsafe” AI indeed looms extremely large—indeed, this s...
    5 min