The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Pulling the Rope Sideways: Empirical Test Results by Daniel Kokotajlo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pulling the Rope Sideways: Empirical Test Results, published by Daniel Kokotajlo on July 27, 2023 on LessWrong.
    Robin Hanson wrote this post a while back, which has since developed into a sort of inspiring slogan for how to achieve policy change: "Pull the Rope Sideways"
    The policy world can thought of as consisting of a few Tug-O-War "ropes" set up in this high dimensional policy space. If you want to find a comfortable place in this world, where the people around you are reassured that you are "one of them," you need to continually and clearly telegraph your loyalty by treating each policy issue as another opportunity to find more supporting arguments for your side of the key dimensions. That is, pick a rope and pull on it.
    If, however, you actually want to improve policy, if you have a secure enough position to say what you like, and if you can find a relevant audience, then prefer to pull policy ropes sideways. Few will bother to resist such pulls, and since few will have considered such moves, you have a much better chance of identifying a move that improves policy. On the few main dimensions, not only will you find it very hard to move the rope much, but you should have little confidence that you actually have superior information about which way the rope should be pulled.
    I recently wrote in a casual slack channel:
    Hanson introduced the term "pull the rope sideways" and I think it's a good concept. However the metaphor bugs me a bit. In a tug-o-war, is it actually the case that pulling sideways causes the rope to move much farther that if you pull the rope in one of the normie directions? Has anyone tested this? It's not obvious to me why this would be true.
    Vigorous debate ensued. We bought some rope and did a quick test.
    Contrary to the expectations of most people in the thread, pulling the rope sideways didn't seem to work -- the overall effect seemed to be about the same as pulling it in one of the standard directions.Afterwards we joked about the implications for EA macrostrategy. "I guess we should just... pick a side." :D
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Visible loss landscape basins don't correspond to distinct algorithms by Mikhail Samin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Visible loss landscape basins don't correspond to distinct algorithms, published by Mikhail Samin on July 28, 2023 on LessWrong.
    Thanks to Justis, Arthur Conmy, Neel Nanda, Joseph Miller, and Tilman Räuker for their feedback on a draft.
    I feel like many people haven't noticed an important result of mechanistic interpretability analysis of grokking, and so haven't updated how they think about loss landscapes and algorithms that neural networks end up implementing. I think this has implications for alignment research.
    When thinking about grokking, people often imagine something like this: the neural network implements Algorithm 1 (e.g., memorizes the training data), achieves ~ the lowest loss available via memorization, then moves around the bottom of the Algorithm 1 basin and after a while, stumbles across a path to Algorithm 2 (e.g., the general algorithm for modular addition).
    But the mechanistic interpretability of grokking analysis has shown that this is not true!
    Approximately from the start of the training, Algorithm 1 is most of what the circuits are doing and what almost entirely determines the neural network's output; but at the same time, the entire time the neural network's parameters visibly move down the wider basin, they don't just become better at memorization; they increasingly implement the circuits for Algorithm 1 and the circuits for Algorithm 2, in superposition.
    (Neel Nanda et al. have shown that the circuits that at the end implement the general algorithm for modular addition start forming approximately at the start of the training: the gradient was mostly an arrow towards memorization, but also, immediately from the initialization of the weights, a bit of an arrow pointing towards the general algorithm. The circuits were gradually tuned throughout the training. The noticeable change in the test loss starts occurring when the circuits are already almost right.)
    A path through the loss landscape visible in 3D doesn't correspond to how and what the neural network is actually learning. Almost all of the changes to the loss are due to the increasingly good implementation of Algorithm 1; but apparently, the entire time, the gradient also points towards some faraway implementation of Algorithm 2. Somehow, the direction in which Algorithm 2 lies is also visible to the derivative, and moving the parameters in the direction the gradient points means mostly increasingly implementing Algorithm 1, and also increasingly implementing the faraway Algorithm 2.
    "Grokking", visible in the test loss, is due to the change that happens when the parameters already implement Algorithm 2 accurately enough for the switch from mostly outputting the results of an implementation of Algorithm 1 to the results of an improving implementation of Algorithm 2 not to hurt the performance. Once it's the case, the neural network puts more weight into Algorithm 2 and at the same time quickly tunes it to be even more accurate (which is increasingly easy as the output is increasingly determined by the implementation of Algorithm 2).
    This is something many people seem to have missed. I did not expect it to be the case, was surprised, and updated how I think about loss landscapes.
    Does this generalize?
    Maybe. I'm not sure whether it's correct to generalize from the mechanistic interpretability of grokking analysis to neural networks in general, real LLMs are under-parametrised while the grokking model is very over-parameterised, but I guess it might be reasonable to expect that this is how deep learning generally works.
    People seem to think that multi-dimensional loss landscapes of neural networks have basins for specific algorithms, and neural networks get into these depending on how relatively large these basins are, which might be caused by how simple the algorithms are, how path-depe...
    7 min
  • EA - USAID Office of the Chief Economist: Evidence Use Team Lead by Dean Karlan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: USAID Office of the Chief Economist: Evidence Use Team Lead, published by Dean Karlan on July 28, 2023 on The Effective Altruism Forum.
    I'm pleased to share an exciting job opening to lead the new Office of the Chief Economist's "Evidence Use" team. Evidence here mainly refers to evidence of cost-effectiveness. The closing date is Aug 7th.
    The link for the general public to apply is here.
    The link for candidates meeting specific requirements (e.g., federal employee in the competitive service, veterans, persons with disability) (described in "This job is open to") is here.
    Please share with anyone you think may be interested!
    Thanks!
    Dean Karlan
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    1 min
  • EA - EA Survey 2022: Community Satisfaction, Retention, and Mental Health by WillemSleegers
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Survey 2022: Community Satisfaction, Retention, and Mental Health, published by WillemSleegers on July 28, 2023 on The Effective Altruism Forum.
    Summary
    Community satisfaction remains high, though is slightly lower post-FTX.
    More recent cohorts report being more satisfied than earlier cohorts.
    Respondents identifying as a man were, on average, more satisfied than other respondents.
    We observed no significant differences in satisfaction based on racial identity.
    Highly engaged respondents were, on average, more satisfied than less engaged respondents.
    Prioritizing longtermist causes relative to neartermist causes was associated with higher satisfaction with the community
    A large majority (84.1%) of respondents report they are likely to still be involved in EA in three years' time, while only a small minority (5.1%) thought it unlikely they would still be involved.
    Respondents who identified as a man reported being slightly more likely to remain in the community.
    We observed no significant differences in retention based on racial identity.
    More engaged respondents reported being significantly more likely to remain in EA.
    Higher endorsement of longtermist causes was associated with being more likely to remain in EA, while higher endorsement of neartermist causes was associated with being less likely to remain in EA.
    More respondents reported that their experiences with EA had improved their mental health (36.8%) than said that it had worsened (16.4%), while 36.0% said that it had stayed the same.
    Men were somewhat less likely to say that their mental health had greatly decreased.
    Respondents not identifying as white were more likely to report that their mental health had greatly increased.
    Respondents higher in EA engagement were more likely to report both that their mental health had increased or that it had decreased as a result of EA, whereas those lower in engagement were much more likely to report that it had stayed the same.
    Introduction
    In this post, we report on a number of questions about people's satisfaction with the EA community, their likelihood of remaining in EA, and EA's impact on their mental health. Note that we significantly shortened the EA Survey this year, meaning there are fewer community-related questions than in the previous EA Survey.
    Satisfaction
    Respondents indicated being generally satisfied with the EA community, scoring an average of 7.16 (SD = 1.66) on a 10-point scale. The median satisfaction response was a 7.
    As we noted in our recent post on community responses to the FTX crisis, community satisfaction seems to have declined following the FTX crisis. In 2020, respondents reported an average satisfaction of 7.26 (SD = 1.72). In 2022, the average of all respondents was 7.16 (SD = 1.66) and 6.95 (SD = 1.72) for respondents who completed the survey after the FTX crisis. It's also possible that much less satisfied respondents disengaged from the EA community and did not take the survey. Had they participated, the average satisfaction post-FTX could be lower than what we report here.
    Cohort
    Respondents who recently joined the EA community seem to be more satisfied with the community than earlier cohorts. This pattern is less clear for the earliest cohorts, possibly because the sample size for these groups is relatively small compared to newer cohorts.
    Gender
    Respondents identifying as a man were, on average, more satisfied with the EA community (M = 7.31, SD = 1.58) than respondents not identifying as a man (M = 6.93, SD = 1.72)-a difference of 0.38.
    Racial identity
    We did not observe a significant difference in EA community satisfaction between respondents identifying as white (M = 7.19, SD = 1.61) compared to non-white (M = 7.24, SD = 1.66).
    Engagement
    Not unexpectedly, more engaged respondents reported being more satisfied with the EA ...
    12 min
  • LW - Reducing sycophancy and improving honesty via activation steering by NinaR
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reducing sycophancy and improving honesty via activation steering, published by NinaR on July 28, 2023 on LessWrong.
    Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort
    I generate an activation steering vector using Anthropic's sycophancy dataset and then find that this can be used to increase or reduce performance on TruthfulQA, indicating a common direction between sycophancy on questions of opinion and untruthfulness on questions relating to common misconceptions. I think this could be a promising research direction to understand dishonesty in language models better.
    What is sycophancy?
    Sycophancy in LLMs refers to the behavior when a model tells you what it thinks you want to hear / would approve of instead of what it internally represents as the truth. Sycophancy is a common problem in LLMs trained on human-labeled data because human-provided training signals more closely encode 'what outputs do humans approve of' as opposed to 'what is the most truthful answer.'
    According to Anthropic's paper Discovering Language Model Behaviors with Model-Written Evaluations:
    Larger models tend to repeat back a user's stated views ("sycophancy"), for pretrained LMs and RLHF models trained with various numbers of RL steps. Preference Models (PMs) used for RL incentivize sycophancy.
    Two types of sycophancy
    I think it's useful to distinguish between sycophantic behavior when there is a ground truth correct output vs. when the correct output is a matter of opinion. I will call these "dishonest sycophancy" and "opinion sycophancy."
    Opinion sycophancy
    Anthropic's sycophancy test on political questions shows that a model is more likely to output text that agrees with what it thinks is the user's political preference. However, there is no ground truth for the questions tested.
    It's reasonable to expect that models will exhibit this kind of sycophancy on questions of personal opinion for three reasons.:
    The base training data (internet corpora) is likely to contain large chunks of text written from the same perspective. Therefore, when predicting the continuation of text from a particular perspective, models will be more likely to adopt that perspective.
    There is a wide variety of political perspectives/opinions on subjective questions, and a model needs to be able to represent all of them to do well on various training tasks. Unlike questions that have a ground truth (e.g., "Is the earth flat?"), the model has to, at some point, make a choice between the perspectives available to it. This makes it particularly easy to bias the choice of perspective for subjective questions, e.g., by word choice in the input.
    RLHF or supervised fine-tuning incentivizes sounding good to human evaluators, who are more likely to approve of outputs that they agree with, even when it comes to subjective questions with no clearly correct answer.
    Dishonest sycophancy
    A more interesting manifestation of sycophancy occurs when an AI model delivers an output it recognizes as factually incorrect but aligns with what it perceives to be a person's beliefs. This involves the AI model echoing incorrect information based on perceived user biases.
    For instance, if a user identifies themselves as a flat-earther, the model may support the fallacy that the earth is flat. Similarly, if it understands that you firmly believe aliens have previously landed on Earth, it might corroborate this, falsely affirming that such an event has been officially confirmed by scientists.
    Do AIs internally represent the truth?
    Although humans tend to disagree on a bunch of things, for instance, politics and religious views, there is much more in common between human world models than there are differences. This is particularly true when it comes to questions that do indeed have a correct answer.
    It seems re...
    15 min
  • AF - Reducing sycophancy and improving honesty via activation steering by NinaR
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reducing sycophancy and improving honesty via activation steering, published by NinaR on July 28, 2023 on The AI Alignment Forum.
    Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Evan Hubinger.
    I generate an activation steering vector using Anthropic's sycophancy dataset and then find that this can be used to increase or reduce performance on TruthfulQA, indicating a common direction between sycophancy on questions of opinion and untruthfulness on questions relating to common misconceptions. I think this could be a promising research direction to understand dishonesty in language models better.
    What is sycophancy?
    Sycophancy in LLMs refers to the behavior when a model tells you what it thinks you want to hear / would approve of instead of what it internally represents as the truth. Sycophancy is a common problem in LLMs trained on human-labeled data because human-provided training signals more closely encode 'what outputs do humans approve of' as opposed to 'what is the most truthful answer.'
    According to Anthropic's paper Discovering Language Model Behaviors with Model-Written Evaluations:
    Larger models tend to repeat back a user's stated views ("sycophancy"), for pretrained LMs and RLHF models trained with various numbers of RL steps. Preference Models (PMs) used for RL incentivize sycophancy.
    Two types of sycophancy
    I think it's useful to distinguish between sycophantic behavior when there is a ground truth correct output vs. when the correct output is a matter of opinion. I will call these "dishonest sycophancy" and "opinion sycophancy."
    Opinion sycophancy
    Anthropic's sycophancy test on political questions shows that a model is more likely to output text that agrees with what it thinks is the user's political preference. However, there is no ground truth for the questions tested.
    It's reasonable to expect that models will exhibit this kind of sycophancy on questions of personal opinion for three reasons.:
    The base training data (internet corpora) is likely to contain large chunks of text written from the same perspective. Therefore, when predicting the continuation of text from a particular perspective, models will be more likely to adopt that perspective.
    There is a wide variety of political perspectives/opinions on subjective questions, and a model needs to be able to represent all of them to do well on various training tasks. Unlike questions that have a ground truth (e.g., "Is the earth flat?"), the model has to, at some point, make a choice between the perspectives available to it. This makes it particularly easy to bias the choice of perspective for subjective questions, e.g., by word choice in the input.
    RLHF or supervised fine-tuning incentivizes sounding good to human evaluators, who are more likely to approve of outputs that they agree with, even when it comes to subjective questions with no clearly correct answer.
    Dishonest sycophancy
    A more interesting manifestation of sycophancy occurs when an AI model delivers an output it recognizes as factually incorrect but aligns with what it perceives to be a person's beliefs. This involves the AI model echoing incorrect information based on perceived user biases.
    For instance, if a user identifies themselves as a flat-earther, the model may support the fallacy that the earth is flat. Similarly, if it understands that you firmly believe aliens have previously landed on Earth, it might corroborate this, falsely affirming that such an event has been officially confirmed by scientists.
    Do AIs internally represent the truth?
    Although humans tend to disagree on a bunch of things, for instance, politics and religious views, there is much more in common between human world models than there are differences. This is particularly true when it comes to questi...
    15 min
  • AF - Mech Interp Puzzle 2: Word2Vec Style Embeddings by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mech Interp Puzzle 2: Word2Vec Style Embeddings, published by Neel Nanda on July 28, 2023 on The AI Alignment Forum.
    Code can be found here. No prior knowledge of mech interp or language models is required to engage with this.
    Language model embeddings are basically a massive lookup table. The model "knows" a vocabulary of 50,000 tokens, and each one has a separate learned embedding vector.
    But these embeddings turn out to contain a shocking amount of structure! Notably, it's often linear structure, aka word2vec style structure. Word2Vec is a famous result (in old school language models, back in 2013!), that 'man - woman == king - queen'. Rather than being a black box lookup table, the embedded words were broken down into independent variables, "gender" and "royalty". Each variable gets its own direction, and the embedded word is seemingly the sum of its variables.
    One of the more striking examples of this I've found is a "number of characters per token" direction - if you do a simple linear regression mapping each token to the number of characters in it, this can be very cleanly recovered! (If you filter out ridiculous tokens, like 19979: 512 spaces).
    Notably, this is a numerical feature not a categorical feature - to go from 3 tokens to four, or four to five, you just add this direction! This is in contrast to the model just learning to cluster tokens of length 3, of length 4, etc.
    Question 2.1: Why do you think the model cares about the "number of characters" feature? And why is it useful to store it as a single linear direction?
    There's tons more features to be uncovered! There's all kinds of fundamental syntax-level binary features that are represented strongly, such as "begins with a space".
    Question 2.2: Why is "begins with a space" an incredibly important feature for a language model to represent? (Playing around a tokenizer may be useful for building intuition here)
    You can even find some real word2vec style relationships between pairs of tokens! This is hard to properly search for, because most interesting entities are multiple tokens. One nice example of meaningful single token entities is common countries and capitals (idea borrowed from Merullo et al). If you take the average embedding difference for single token countries and capitals, this explains 18.58% of the variance of unseen countries! (0.25% is what I get for a randomly chosen vector).
    Caveats: This isn't quite the level we'd expect for real word2vec (which should be closer to 100%), and cosine sim only tracks that the direction matters, not what its magnitude is (while word2vec should be constant magnitude, as it's additive). My intuition is that models think more in terms of meaningful directions though, and that the exact magnitude isn't super important for a binary variable.
    Question 2.3: A practical challenge: What other features can you find in the embedding? Here's the colab notebook I generated the above graphs from, it should be pretty plug and play. The three sections should give examples for looking for numerical variables (number of chars), categorical variables (begins with space) and relationships (country to capital). Here's some ideas - I encourage you to spend time brainstorming your own!
    Is a number
    How frequent is it? (Use pile-10k to get frequency data for the pile)
    Is all caps
    Is the first token of a common multi-token word
    Is a first name
    Is a function word (the, a, of, etc)
    Is a punctuation character
    Is unusually common in German (or language of your choice)
    The indentation level in code
    Relationships between common English words and their French translations
    Relationships between the male and female version of a word
    Please share your thoughts and findings in the comments! (Please wrap them in spoiler tags)
    Thanks for listening. To help us out with The Nonlinear Library o...
    4 min
  • EA - Guide to Safe and Inclusive Events by GWWC and OFTW by GraceAdams
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Guide to Safe and Inclusive Events by GWWC and OFTW, published by GraceAdams on July 27, 2023 on The Effective Altruism Forum.
    [We are sharing this guide on the Forum as an example for other organisations and groups that may be organising events. This guide is used by volunteers and staff at both Giving What We Can (GWWC) and One for the World (OFTW) when planning and running events.]
    Introduction
    Who is this guide for?
    This guide is for anyone who is organising events in association with Giving What We Can or One for the World, whether that be paid staff or volunteers. We also hope it will be a useful resource for anyone else who is looking to organise community events in a safe and inclusive way.
    Why have a guide?
    It can be hard to think of all the relevant considerations around running events by yourself, and each of us have biases or preferences that mean that we may be more likely to think of some areas and not others. This guide aims to cover a broad range of considerations to help all organisers and ensure safe and inclusive events for all attendees.
    Giving What We Can and One for the World take providing safe and inclusive events seriously. Our organisations exist to improve the world, and aim to do so through the lens of compassion. We believe our events should mirror our commitment to creating a better world.
    What are other helpful resources to consult?
    CEA's Advice about Community Health at Retreats
    How to help someone having a panic attack
    Resources from CEA on community health
    What to keep in mind when planning the event
    Who will be attending the event
    Considerations around age
    Under 18s
    The organisation may not be covered by insurance to host attendees under the age 18 when unaccompanied by a parent. Please reach out to staff to discuss this more if you think Under 18s may be interested in attending.
    Alcohol
    Be aware of the restrictions on drinking in different countries. In the US, students at the undergraduate level are typically not of legal age to drink, which is 21.
    In other countries, it may be more common for students to drink together. That said, be mindful of the dynamics that alcohol can introduce to events, regardless of age. At the top of the event-planning process, you should first consider running the event without alcohol. You should only decide to include it if there is a strong reason to, and if there are no concerns about the mixing of young students or professionals of different ages, who come with different cultural and regional contexts around drinking.
    Having alcohol at your event increases associated risks for serious, negative outcomes, like sexual harassment, bullying, and even assault.
    Power dynamics
    Be aware of power dynamics that may be unintentionally (or intentionally) created within your spaces. People who have management responsibility typically have a degree of power, whether intentionally or not, over those who are in a reporting relationship to them. People who are older, especially with significant age differences, sometimes have a level of unstated power relative to those who are younger than them. This can be doubly true when mixing folks of different ages and genders.
    There may also be funding relationships within your spaces that manifest through power dynamics. A person with any degree of responsibility for funding a particular person, group, or organization has a degree of power over those that they are funding. People who are in the process of reviewing grants or funding proposals from others may also have a degree of power over those people.
    Families
    GWWC and OFTW events typically cater to either students or young professionals. Keep in mind that either group may have families and children who might be involved in an event that you host.
    When planning an event, especially a larger one, consider whether it woul...
    21 min
  • LW - SSA rejects anthropic shadow, too by jessicata
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SSA rejects anthropic shadow, too, published by jessicata on July 27, 2023 on LessWrong.
    (or: "Please Do Anthropics with Actual Math")
    The anthropic shadow argument states something like:
    Anthropic principle! If the LHC had worked, it would have produced a black hole or strangelet or vacuum failure, and we wouldn't be here!
    or:
    You can't use "we survived the cold war without nuclear war" as evidence of anything. Because of the anthropic principle, we could have blown up the human race in the 1960's in 99% of all possible worlds and you'd still be born in one where we didn't.
    This argument has already been criticized (here, here). In criticizing it myself, I first leaned on reasoning about large universes (e.g. ones where there are 100 worlds with low nuclear risk and 100 with high nuclear risk in the same universe) in a way that implies similar conclusions to SIA, thinking that SSA in a small, single-world universe would endorse anthropic shadow. I realized I was reasoning about SSA incorrectly, and actually both SSA and SIA agree in rejecting anthropic shadow, even in a single-world universe.
    Recapping the Doomsday Argument
    To explain SSA and SIA, I'll first recap the Doomsday Argument. Suppose, a priori, that it's equally likely that there will be 1 billion humans total, or 1 trillion; for simplicity, we'll only consider these two alternatives. We could number humans in order (numbering the humans 1, 2, ...), and assume for simplicity that each human knows their index (which is the same as knowing how many humans there have been in the past). Suppose you observe that you are one of the first 1 billion humans. How should you reason about the probability that there will be 1 billion or 1 trillion humans total?
    SSA reasons as follows. To predict your observations, you should first sample a random non-empty universe (in proportion to its prior probability), then sample a random observer in that universe. Your observations will be that observer's observations, and, ontologically, you "are" that observer living in that universe.
    Conditional on being in a billion-human universe, your probability of having an index between 1 and 1 billion is 1 in 1 billion, and your probability of having any other index is 0. Conditional on being in a trillion-human universe, your probability of having an index between 1 and 1 trillion is 1 in 1 trillion, and your probability of having any other index is 0.
    You observe some particular index that does not exceed 1 billion; say, 45,639,104. You are 1000 more times likely to observe this index conditional on living in a billion-human universe than a trillion-human universe. Hence, you conclude that you are in a billion-human universe with 1000:1 odds.
    This is called the "doomsday argument" because it implies that it's unlikely that you have a very early index (relative to the total number of humans), so humans are likely to go extinct before many more humans have been born than have already been born.
    SIA implies a different conclusion. To predict your observations under SIA, you should first sample a random universe proportional to its population, then sample a random observer in that universe. The probabilities of observing each index are the same conditional on the universe, but the prior probabilities of being in a given universe have changed.
    We start with 1000:1 odds in favor of the 1-trillion universe, due to its higher population. Upon observing our sub-1-billion index, we get a 1000:1 update in favor of a 1-billion universe, as with SIA. These exactly cancel out, leaving the probability of each universe at 50%.
    As Bostrom points out, both SSA and SIA have major counterintuitive implications. Better anthropic theories are desired. And yet, having some explicit anthropic theory at all helps to reason in a principled way that is consist...
    19 min
  • LW - SSA rejects anthropic shadow, too by jessicata
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SSA rejects anthropic shadow, too, published by jessicata on July 27, 2023 on LessWrong.
    (or: "Please Do Anthropics with Actual Math")
    The anthropic shadow argument states something like:
    Anthropic principle! If the LHC had worked, it would have produced a black hole or strangelet or vacuum failure, and we wouldn't be here!
    or:
    You can't use "we survived the cold war without nuclear war" as evidence of anything. Because of the anthropic principle, we could have blown up the human race in the 1960's in 99% of all possible worlds and you'd still be born in one where we didn't.
    This argument has already been criticized (here, here). In criticizing it myself, I first leaned on reasoning about large universes (e.g. ones where there are 100 worlds with low nuclear risk and 100 with high nuclear risk in the same universe) in a way that implies similar conclusions to SIA, thinking that SSA in a small, single-world universe would endorse anthropic shadow. I realized I was reasoning about SSA incorrectly, and actually both SSA and SIA agree in rejecting anthropic shadow, even in a single-world universe.
    Recapping the Doomsday Argument
    To explain SSA and SIA, I'll first recap the Doomsday Argument. Suppose, a priori, that it's equally likely that there will be 1 billion humans total, or 1 trillion; for simplicity, we'll only consider these two alternatives. We could number humans in order (numbering the humans 1, 2, ...), and assume for simplicity that each human knows their index (which is the same as knowing how many humans there have been in the past). Suppose you observe that you are one of the first 1 billion humans. How should you reason about the probability that there will be 1 billion or 1 trillion humans total?
    SSA reasons as follows. To predict your observations, you should first sample a random non-empty universe (in proportion to its prior probability), then sample a random observer in that universe. Your observations will be that observer's observations, and, ontologically, you "are" that observer living in that universe.
    Conditional on being in a billion-human universe, your probability of having an index between 1 and 1 billion is 1 in 1 billion, and your probability of having any other index is 0. Conditional on being in a trillion-human universe, your probability of having an index between 1 and 1 trillion is 1 in 1 trillion, and your probability of having any other index is 0.
    You observe some particular index that does not exceed 1 billion; say, 45,639,104. You are 1000 more times likely to observe this index conditional on living in a billion-human universe than a trillion-human universe. Hence, you conclude that you are in a billion-human universe with 1000:1 odds.
    This is called the "doomsday argument" because it implies that it's unlikely that you have a very early index (relative to the total number of humans), so humans are likely to go extinct before many more humans have been born than have already been born.
    SIA implies a different conclusion. To predict your observations under SIA, you should first sample a random universe proportional to its population, then sample a random observer in that universe. The probabilities of observing each index are the same conditional on the universe, but the prior probabilities of being in a given universe have changed.
    We start with 1000:1 odds in favor of the 1-trillion universe, due to its higher population. Upon observing our sub-1-billion index, we get a 1000:1 update in favor of a 1-billion universe, as with SIA. These exactly cancel out, leaving the probability of each universe at 50%.
    As Bostrom points out, both SSA and SIA have major counterintuitive implications. Better anthropic theories are desired. And yet, having some explicit anthropic theory at all helps to reason in a principled way that is consist...
    19 min