The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Clarifying wireheading terminology by leogao
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Clarifying wireheading terminology, published by leogao on November 24, 2022 on The AI Alignment Forum.
    See also: Towards deconfusing wireheading and reward maximization, Everett et al. (2019).
    There are a few subtly different things that people call "wireheading". This post is intended to be a quick reference for explaining my views on the difference between these things. I think these distinctions are sometimes worth drawing to reduce confusion.
    Specification gaming / reward hacking: The agent configures the world in such a way that makes its reward(/utility) function achieve a high value in a way that is not what the creators intended the reward function to incentivize.
    Examples: the boat race example, our recent work on reward model overoptimization
    Reward function input tampering: The agent tampers with the inputs to its reward function to make the reward function see the same inputs as it would when observing an actual desirable worldstate.
    Examples: sensor tampering, VR
    Reward function tampering: The agent changes its reward function.
    Examples: meditation, this baba-is-you-like gridworld
    Wireheading: The agent directly tampers with the reward signal going into the RL algorithm.
    Examples: dopamine agonists, setting the reward register
    Some crucial differences between these:
    1 is a problem of both embedded and non-embedded settings. 2 is a problem of partially-observedness, which is unavoidable in embedded settings, but also comes up in many non-embedded settings as well. 3 and 4 are problems of embeddedness; they do not show up in non-embedded settings.
    2, 3, 4 are all about "not caring about the real world" in various ways, whereas 1 is about "caring about the real world in the wrong way"
    4 is about as close as you can get to a "reward-maximizing policy" in an embedded setting.
    Note: this is not the same thing as changing a terminal goal! The reward function is not necessarily the terminal goal of the policy, because of inner misalignment.
    Here, by "embeddedness" I mean in the sense of Demski and Garrabrandt (2019); in particular, the fact that the agent is part of the environment, and not in a separate part of the universe that only interacts through well defined observation/action/reward channels. RL is the prototypical example of a non-embedded agency algorithm.
    That is, the reward function is unable to perfectly observe the ground truth state of the environment, which means that if there is another state the world cold be in that yields the same observation, the reward cannot distinguish between the two.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Thoughts on legal concerns surrounding the FTX situation: document preservation and communications by Molly
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on legal concerns surrounding the FTX situation: document preservation and communications, published by Molly on November 24, 2022 on The Effective Altruism Forum.
    Aside from concerns about bankruptcy clawbacks, the biggest repeat legal-oriented question I've been getting since the FTX fallout is: are my communications going to end up in court or in Bloomberg? Should I just delete everything? Should I save everything?
    It’s borderline impossible to give a large diffuse group of people clear guidance about their obligations in this regard. (And the usual disclaimers about not being able to provide legal advice to entities/people other than Open Phil still apply.) But I can lay out some principles and processes that should help people make better-informed decisions. Unfortunately this probably cannot be reliably exported to non-US contexts.
    Are My Records Going to Become Public?
    I think the most useful piece of information I’ve been able to give people who have asked about this kind of thing in the last two weeks is that being asked to turn things over to an investigation or discovery process does not mean you have to turn entire email servers or hard drives over. You only have to turn over what’s responsive to the inquiry. And having to turn over your materials does not automatically mean that they become public.
    The materials will probably go to some windowless legal office where dozens of junior attorneys (or interns) are listening to podcasts under fluorescent lights while combing through thousands of pages of documents trying to find stuff that’s relevant to the matter they’re working on. When they do notice something that could be relevant, they’ll stop and read more carefully. If it informs the matter, even in a small way, it could become an exhibit in a litigation proceeding, and that would make it accessible to the public, and therefore fair game for the media.
    It’s uncommon for these review processes to dig into issues that aren’t relevant to the proceeding that triggered the discovery. The peccadillos of day-to-day life are unlikely to draw additional investigative scrutiny.
    That said, if, for example, you’re regularly corresponding with Sam Bankman-Fried or Caroline Ellison, I would just assume all of that correspondence is going to be relevant and, eventually, public. The farther removed the correspondence is from anyone at the epicenter of any investigations, or from investigation-relevant subject-matter, the less likely it is to have to be turned over, or if turned over, made public.
    A final, minor, point: if you’re worried about leaks, I think it’s fair to say that most of those don’t come from legal teams, who would be risking their careers and credentials in leaking confidential material.
    Quick recap of the process:
    Discovery request or subpoena you turn over material responsive to the request junior lawyers look through massive quantity of information for relevant material relevant material may become an exhibit in legal proceedings, which are accessible to the public material accessible to the public can get picked up in the media
    Should I save/delete everything?
    Legal obligations to preserve documents generally hinge on whether you have a “reasonable anticipation of litigation.” This can mean you think you’re actually going to be a party to a lawsuit (either because you sue someone, or they sue you). It could also mean that you get a discovery demand or a subpoena or some other formal notice from a court.
    If something does trigger a legal obligation to preserve documents, you need to make sure that the material in your possession that could be relevant to the proceeding in question doesn’t get deleted. For example, if you use Signal a lot, and you have Signal messages set to delete after a week, you should screenshot any relevant texts ...
    6 min
  • LW - LW Beta Feature: Side-Comments by jimrandomh
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LW Beta Feature: Side-Comments, published by jimrandomh on November 24, 2022 on LessWrong.
    LessWrong now has side-comments. This feature is in beta; you can turn it on for yourself on individual posts using the triple-dot menu below the post title, or enable it for all posts by going to your user settings and checking the "Opt into experimental features" checkbox in the Site Customization section.
    Side-coments on LessWrong are conceptually similar to the side-comments you may be familiar with from Google Docs and other places, with one key difference: side-comments are placed automatically by lining up blockquotes. As a result, many historical posts already have side-comments on them! Side-comments are also still displayed in the comments section below the post as usual.
    Side-comments take the form of a comment icon in the right margin, which expands when you mouse over it to show a comment. Click on the icon to pin it open. It looks like this:
    To create a side-comment on a post, just write a normal comment which quotes an excerpt from the post, in blockquote format. (To create a blockquote, type > ). You can also highlight some text and click the button that appears to start a comment pre-seeded with a quote containing that text.
    By default, side-comments are filtered to comments with 10+ karma (plus comments by the post author). They aren't available on mobile and require that your screen is wide enough to have some space in the right margin.
    If a comment contains more than one blockquote that could be used to place it as a side-comment, it will be placed based on the first one. In order to appear as a side-comment, quotes should be an exact match, including formatting; there is some provision for "..." ellipses, but it's fairly limited.
    Please give feedback on how this affects the reading experience! If the feedback is positive, this will leave beta and be enabled (for comments above a karma threshold) for everyone.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Our recommendations for giving in 2022 by GiveWell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Our recommendations for giving in 2022, published by GiveWell on November 23, 2022 on The Effective Altruism Forum.
    Author: Miranda Kaplan, GiveWell Communications Associate
    We wrote back in July that we expected to be funding-constrained this year. That remains true as we approach the end of the year, putting us in the unusual position of leaving impact on the table.
    We've set a goal of raising $600 million in 2022, but our research team has identified $900 million in highly cost-effective funding gaps. That leaves $300 million in funding gaps unfilled. By donating this year, you can help us not only meet but exceed our goal—and say yes to more excellent opportunities to save and improve lives.
    Additionally, our giving guidance for donors has changed this year. For the first time, our top recommendation is to give to our new All Grants Fund, which we allocate to any need that meets our cost-effectiveness bar. We think it's the best bet for donors who want to support the most promising opportunities we've found to help people, regardless of program or location. And it reflects our current views on how we can best meet our goal of maximizing global well-being—by taking advantage of every path to impact, whether that's funding top charities, seeding and scaling newer programs, or funding research. See "Our giving funds, and our top recommendation" for more on all three of our giving funds.
    Why your support is so important
    We rely heavily on numbers to think through our funding decisions. But it’s important to remind ourselves what those numbers represent.[1] If we reach our goal of $600 million this year, we speculatively guess that that funding would save around 70,000 lives.[2] That's approximately the population of Portland, Maine.[3]
    To make the image a little more specific: we also expect most of the lives saved will be those of very young children, under five years old.[4] If they reach their fifth birthday, they'll have a much higher chance of surviving into adulthood.[5] We think about 49,000 of the lives these donations are expected to save will be those of children under five[6]—enough to fill more than 2,000 average US primary school classrooms.[7]
    But raising $600 million is not a given. We expect $350 million of our funding this year to come from Open Philanthropy, our single largest donor.[8] The rest will come from our broader community of supporters (like you!), and our projections for this category of our fundraising are fairly uncertain.
    What $600 million will enable
    Last year, our research team had tremendous success in identifying new room for more funding,[9] and those efforts have continued to bear fruit in 2022. We've found highly cost-effective funding opportunities in both the interventions implemented by our top charities—malaria prevention, incentives for vaccination, and vitamin A supplementation (VAS)—and newer-to-us areas, such as water treatment, iron fortification, and maternal syphilis screening and treatment.
    Below are just a few examples of what the money we raise this year will likely fund:
    $30.2 million to New Incentives for continued expansion of its program. New Incentives, a top charity since 2020, provides cash transfers to incentivize caregivers in northern Nigeria to get their infants vaccinated. Funding from GiveWell and others has enabled the program to grow rapidly over the past couple of years.[10] Prior to this grant, New Incentives had raised funding to reach approximately 3.2 million children across northern Nigeria; we estimate that this grant will allow it to reach approximately 1.4 million additional children.[11]
    $2.4 million to r.i.c.e., a nonprofit that has partnered with the government of Uttar Pradesh, India, to continue operating the Project on Breastfeeding and Newborn Care. The project focuses on kangaroo mot...
    10 min
  • LW - Against "Classic Style" by strawberry calm
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against "Classic Style", published by strawberry calm on November 23, 2022 on LessWrong.
    What is Classic Style?
    The Sense of Style is Steven Pinker's style guide informed by cognitive psychology and linguistics. The main idea is that the author should write in a particular mode of communication called "classic style".
    The guiding metaphor of classic style is seeing the world. The writer can see something that the reader has not yet noticed, and he orients the reader’s gaze so that she can see it for herself. The purpose of writing is presentation, and its motive is disinterested truth. It succeeds when it aligns with the truth, the proof of success being clarity and simplicity.
    Clear and Simple as the Truth, Francis-Noel Thomas and Mark Turner
    Here are the do's and don't of classic style:
    Eliminate meta-discourse.Don't write "in the next section I will..."
    Eliminate "hedging".Don't write "maybe", "it seems to me", or "I think that".
    Don't use concepts about concepts.Don't write "approach", "assumption", "concept", "condition", "context", "framework", "issue", "level", "model", "paradigm", "perspective", "process", "role", "strategy", "tendency", "variable".
    Talk about the subject, not about research about the subject.If you're writing about apples, then write "apples are X" rather than "Dr Smith first discovered that apples are X".
    Roughly speaking, Pinker recommends that the author should write about something as if neither the document nor the author nor the reader actually exists. If they're writing a book about apples, then the sentences should assert facts about apples, not about the study of apples and certainly not about the book itself.
    My opinion: I like that Pinker deduces his advice from an underlying theory of communication. This is better than many style guides which present an ad-hoc list of tips. However, I think the theory is wrong.
    What's wrong with Classic Style?
    I'm suspicious of "A Sense of Style" because it separates what you should write and what is true. To clarify, Pinker isn't saying there are things you should write that aren't true, but he is saying there are things that are true that you shouldn't write.
    Pinker contrasts "classic style" with what he calls "postmodern style" — where the author explicitly refers to the document itself, the readers, the authors, any uncertainties, controversies, errors, etc. I think a less pejorative name for "postmodern style" would be "self-aware style". This is the predominant communication style I see on LessWrong.
    In this post, I will list some advantages of self-aware style over classic style.
    Problem 1: Self-referentiality.
    The Capital-T-Truth is that the book does have an author, a reader, and exists in the same world as the objects the book discusses. Therefore there are certain facts about the subject that a classic-style document won't be able to assert — namely, these are facts about the relationship between the document and the subject.
    In map-territory framing: a map is "classic style" if it doesn't include a "you are here" marker. But often the map is part of the territory that is being mapped, and therefore a lack of a "you are here" marker is an omission of an important truth about the territory.
    This seems defective to me. If the writers in a community assert all truths about the subject, including self-referential truths, then the community is likely to have more accurate beliefs.
    Problem 2: Epistemic qualifiers.
    It's very difficult to state in classic-style anything like "the probability that X is p" because probabilities are statements about the author's beliefs about X. Recall that in classic style, the document does not know the author exists!
    This is far more restricting than it may first appear. If you can't use epistemic qualifiers, then you aren't allowed to discuss anything that is...
    7 min
  • EA - Rethink Priorities’ Leadership Statement on the FTX situation by abrahamrowe
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rethink Priorities’ Leadership Statement on the FTX situation, published by abrahamrowe on November 23, 2022 on The Effective Altruism Forum.
    From the Executive Team and Board of Directors of Rethink Priorities (Peter Wildeford, Marcus Davis, Abraham Rowe, Kieran Greig, David Moss, Ozzie Gooen, Cameron Meyer Shorb, and Vicky Bond).
    We were saddened and shocked to learn about the extremely serious alleged misdeeds and misconduct of Sam Bankman-Fried and FTX. While we are still trying to understand what happened and the consequences of these events, we are dismayed that customer funds may have been used improperly, and that, currently, many customers are unable to retrieve funds held by FTX. We unequivocally and in the strongest possible terms condemn any potential fraud or misuse of customer funds and trust that occurred at FTX. The actions that Bankman-Fried and FTX have been accused of are out of line with the values that we believe in and try to represent as an organization.
    At this time, Rethink Priorities remains in a stable financial and legal position. We do not plan on laying off staff or cutting salaries in response to these events or to the changed financial condition of the EA space. However, the strategies of our General Longtermism, Special Projects, and Surveys teams were partly based on the existence of FTX funding for Rethink Priorities and others in the EA community. For the time being, we've mainly paused further hiring for these programs and are revisiting our strategies for them going forward. We’ve decided that hiring for our Special Projects team, which was already in progress before we learned about the FTX situation, will proceed in order to evaluate and onboard new fiscal sponsees.
    Unfortunately, this situation does impact our long-term financial outlook and our ability to keep growing. Rethink Priorities continues to have large funding needs and we look forward to sharing more about our plans with the community in the next few days. We will need to address the funding gap left by these changed conditions for the coming years.
    In terms of legal exposure, Rethink Priorities’ legal counsel are looking into the possibility of clawbacks of funds previously donated to us by FTX-related sources. At this time, we are not aware of any other significant legal exposure for Rethink Priorities or its staff.
    Prior to the news breaking this month, we already had procedures in place intended to mitigate potential financial risks from relying on FTX or other cryptocurrency donors. Internally, we've always had a practice of treating pledged or anticipated cryptocurrency donations as less reliable than other types of donations for fundraising forecasting purposes, simply due to volatility in that sector. As a part of regular crisis management exercises, we also engaged in an internal simulation in August around the possibility of FTX funds no longer being available. We did this exercise due to the relative size and importance of the funding to us, and the base failure rates of cryptocurrency projects, not due to having non-public information about FTX or Bankman-Fried.
    In hindsight, we believe we could have done more to share these internal risk assessments with the rest of the EA community. Going forward, we are reevaluating our own approach to risk management and the assessment of donors, though we do not believe any changes we will make would have caught this specific issue.
    As mentioned above, Rethink Priorities is receiving legal advice on clawbacks, and we are happy to share resources with other organizations that are concerned about their exposure. We cannot provide legal advice, but we are able to provide information on our own response—please reach out to Abraham Rowe ([email protected]) for more information.
    Thanks for listening. To help u...
    4 min
  • LW - The Geometric Expectation by Scott Garrabrant
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Geometric Expectation, published by Scott Garrabrant on November 23, 2022 on LessWrong.
    A Suspicious Pattern
    There is a pattern that shows up in many of the toys we like to play with around here: the pattern of maximizing the expected logarithm.
    Nash bargaining is a method for aggregating preferences without a means to directly compare them. When Nash bargaining, you are maximizing the expected logarithm of utility, where the expectation is over uncertainty about which person you are.
    Kelly betting is an extremely useful tool for not putting all your future wealth in one basket. When Kelly betting, you are maximizing the expected logarithm of your wealth.
    The log scoring rule is a very natural way to extract beliefs. When maximizing your log score, you are maximizing the expectation of the logarithm of the probability you assign to the right answer. This is one example of a general pattern. Maximizations of expected logarithms show up all over information theory, often phrased as minimizing the negative of the expected logarithm.
    Why does maximization of the expected logarithm keep showing up?
    One answer is that all of the instances of it showing up are actually related. In my previous two posts, I made some connections between Nash bargaining and Kelly betting. The fact that Kelly betting can be used to model Bayesian updating illustrates its relationship with the information theory applications. To a certain extent, there is really only one instance of this pattern.
    However, I think that there is another argument for why you should expect this pattern to show up a lot, which is that the pattern is very simple. More simple than it looks on the surface. It only looks complicated because mathematicians have failed us.
    The Geometric Integral
    One of the most underrated concepts in mathematics is the geometric integral, given by ∏f(x)dx=e∫ln(f(x))dx. (The fact that I couldn't easily get a latex symbol that looks like an elongated P is a testament to its underratedness.) The geometric integral is just like the standard integral, but everywhere you would add, you multiply instead. Defining it in terms of the standard (arithmetic) integral with logs and exponents is insulting to its nature, and I don't recommend thinking of it that way. (You wouldn't define x×y as eln(x)+ln(y).) Instead, you should just think of it as the multiplicative version of the integral. However, using logs and exponentiation, it is the fastest way to get the definition across.
    I think people don't practice thinking multiplicatively enough, which causes them to throw inherently multiplicative things into logarithms, so they can think about them additively.
    I will use the phrase geometric expectation when I take a geometric integral over a probability distribution, and I will use the symbol G. Thus, we will write Gx∼Pf(x)=eEx∼Plnf(x).
    Discrete Geometric Expectations
    Luckily, most of the time, we will want to talk about discrete geometric expectations, where we can use (possibly infinite) sums rather than integrals and (possibly infinite) products rather than geometric integrals.
    Let us gain some intuition for discrete geometric expectations by going though some simple cases. We will start with a uniform distribution on a finite set.
    Let X={x1,.,xn} be a finite set with n elements. Let f:XR≥0 be a function that assigns a nonnegative value to each xi. Let P be the uniform probability distribution on X that assigns probability 1n to each element of X.
    We have that Ex∼Pf(x)=∑x∈XP(x)f(x)=∑ni=1f(xi)n=f(x1)+.+f(xn)n. This is just the average, or arithmetic mean of the f values.
    We can compute Gx∼Pf(x) using the above formula Gx∼Pf(x)=eEx∼Plnf(x). Here, we get
    Gx∼Pf(x)=eEx∼Plnf(x)=elnf(x1)+.+lnf(xn)n=n√elnf(x1).elnf(xn)=n√f(x1).f(xn).
    Thus, the geometric expectation of the uniform distribution is just the...
    8 min
  • EA - Timeline of FTX collapse by vipulnaik
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Timeline of FTX collapse, published by vipulnaik on November 23, 2022 on The Effective Altruism Forum.
    Sebastian Sanchez has been working on a timeline of FTX collapse on the Timelines Wiki; I'm paying for the work and also providing some input and direction. The timeline isn't quite done yet; you can see some pending (unchecked) items in the What the timeline is still missing section. And we may need to expand it further as we find more information or new stuff happens. Further proofreading and fact-checking are also pending.
    The timeline is focused largely on the collapse in November 2022; Sebastian plans to work on a timeline of
    FTX that will go more into the background. For more on what the collapse timeline is focused on, see the inclusion criteria section of the timeline.
    Thoughts welcome!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - When AI solves a game, focus on the game's mechanics, not its theme. by strawberry calm
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When AI solves a game, focus on the game's mechanics, not its theme., published by strawberry calm on November 23, 2022 on LessWrong.
    Epistemic status: This is a brief sketch of an idea I'm pretty sure about.
    A board game design consists of two things: mechanics and theme.
    The game mechanics are the abstract rules governing how the players interact with each other and the shared environment.
    The game theme is a fictional interpretation of the game elements.
    Consider Battleship — the theme is a naval battle, and the mechanics are a particular 2-player sequential discovery game. There is a correspondence between the ontology of the mechanics and the ontology of the theme.
    There is often little connection between mechanics and theme. For example, the Knight in chess has almost nothing to do with horses.
    When an AI solves a game, people sometimes overfocus on the theme of the game relative to the mechanics of the game.
    Maybe this is for psychological reasons:
    The theme is more interesting than the mechanics.
    The theme is in our pre-cached ontology. That is, my brain already has a pre-cached concept of "naval battle" but it doesn't have a pre-cached concept corresponding to the particular mechanics of Battleship. In fact, this is why games have themes in the first place — they serve partly as mnemonics for the rules.
    Did you know that people cooperate more in the Prisoner's Dilemma if the game is called "Community Game" than "Wall Street Game"?!
    Or maybe this is for rational reasons:
    Other people might think that there is a deeper connection between the theme and the mechanics of the particular game than I do. For example, they might think there is some genuine non-arbitrary connection between the mechanics of monopoly and the real estate market.
    See ludonarrative dissonance.
    If people overfocus on the theme, then they make incorrect predictions about AI.
    For example, they'll hear "AI has solved Full-Press Diplomacy" and extrapolate that AI will soon be able to solve other games of a similar theme (i.e. international military negotiations).
    Instead, they should extrapolate that AI will soon be able to solve other games with similar mechanics.
    Here's some practical advice: imagine the game had the same mechanics but a different theme. Sure, AI has solved Full-Press Diplomacy, which is scary because the theme is militaries negotiating which countries to invade. But what if the theme was gardeners negotiating which flowers to plant? Okay still pretty scary tbh.
    Could we use this bias to scare the public? "Oh, no — AI has just solved the Kill-All-the-Humans game."
    Appendix
    So anyway, what are game mechanics?
    Is time discrete (i.e. turn-based) or continuous?
    How many turns are there? Boundedly-many? Finite-but-unboundedly many? Infinitely many?
    Do the players move sequentially or simultaneously?
    How many possible moves are there each turn?
    How many players are there?
    Are players ever eliminated?
    What are the winning conditions?
    How many players can win?
    Do the players accumulate points?
    How sparsely are points allocated?
    Do players know how many points the other players have?
    Is it harder to accumulate points the more you have, or easier?
    Is there a bound on the number of points?
    Is the game purely cooperative, purely adversarial, or somewhere between?
    Can the players communicate publicly?
    Can the players communicate privately?
    Are the mechanics symmetric with respect to each player?
    Are the mechanics symmetric with respect to each pair of players? E.t.c.
    How unequal is the advantage between players begin?
    How random is the environment?
    How Kolmogorov-complex is the environment?
    How computationally-complex is the environment?
    Is there little interference between players (like the 400m dash), or much interference (like a football game)?
    Does one strategy dominate all others, or...
    5 min
  • LW - Against a General Factor of Doom by AI Impacts
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against a General Factor of Doom, published by AI Impacts on November 23, 2022 on LessWrong.
    Jeffrey Heninger, 22 November 2022
    I was recently reading the results of a survey asking climate experts about their opinions on geoengineering. The results surprised me: “We find that respondents who expect severe global climate change damages and who have little confidence in current mitigation efforts are more opposed to geoengineering than respondents who are less pessimistic about global damages and mitigation efforts.”1 This seems backwards. Shouldn’t people who think that climate change will be bad and that our current efforts are insufficient be more willing to discuss and research other strategies, including intentionally cooling the planet?
    I do not know what they are thinking, but I can make a guess that would explain the result: people are responding using a ‘general factor of doom’ instead of considering the questions independently. Each climate expert has a p(Doom) for climate change, or perhaps a more vague feeling of doominess. Their stated beliefs on specific questions are mostly just expressions of their p(Doom).
    If my guess is correct, then people first decide how doomy climate change is, and then they use this general factor of doom to answer the questions about severity, mitigation efforts, and geoengineering. I don’t know how people establish their doominess: it might be as a result of thinking about one specific question, or it might be based on whether they are more optimistic or pessimistic overall, or it might be something else. Once they have a general factor of doom, it determines how they respond to specific questions they subsequently encounter. I think that people should instead decide their answers to specific questions independently, combine them to form multiple plausible future pathways, and then use these to determine p(Doom). Using a model with more details is more difficult than using a general factor of doom, so it would not be surprising if few people did it.
    To distinguish between these two possibilities, we could ask people a collection of specific questions that are all doom-related, but are not obviously connected to each other. For example:
    How much would the Asian monsoon weaken with 1°C of warming?
    How many people would be displaced by a 50 cm rise in sea levels?
    How much carbon dioxide will the US emit in 2040?
    How would vegetation growth be different if 2% of incoming sunlight were scattered by stratospheric aerosols?
    If the answers to all of these questions were correlated, that would be evidence for people using a general factor of doom to answer these questions instead of using a more detailed model of the world.
    I wonder if a similar phenomenon could be happening in AI Alignment research.2
    We can construct a list of specific questions that are relevant to AI doom:
    How long are the timelines until someone develops AGI?
    How hard of a takeoff will we see after AGI is developed?
    How fragile are good values? Are two similar ethical systems similarly good?
    How hard is it for people to teach a value system to an AI?
    How hard is it to make an AGI corrigible?
    Should we expect simple alignment failures to occur before catastrophic alignment failures?
    How likely is human extinction if we don’t find a solution to the Alignment Problem?
    How hard is it to design a good governance mechanism for AI capabilities research?
    How hard is it to implement and enforce a good governance mechanism for AI capabilities research?
    I don’t have any good evidence for this, but my vague impression is that many people’s answers to these questions are correlated.
    It would not be too surprising if some pairs of these questions ought to be correlated. Different people would likely disagree on which things ought to be correlated. For example, Paul Christiano se...
    9 min