The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - My favorite safety papers & sequences - Living post (vers.: May 15, 8 recs) by the gears to ascension
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My favorite safety papers & sequences - Living post (vers.: May 15, 8 recs), published by the gears to ascension on May 15, 2023 on LessWrong.
    Local mad librarian here. I've been urged to hurry up and make a reading list of favorite papers and thoughts on them; this is the first version, and I'll improve it over time, with occasional post refreshes occasionally, especially when the order changes or I improve explanation. This will be most useful for those experienced with reading papers, but regardless, if you're new to the field, I recommend reading things for which you do not have prereqs in as much depth as your lack of context allows, as it will tune your appetite for the prereqs.
    I see all of these as interesting ideas which I think every alignment researcher of any subfield should read but not as concluded answers to any of the questions we're trying to ask and potentially not even the right kind of framings. If you're interested in doing a call where I watch you read through and comment on any of these, I'd be very eager, assuming timezones work out - leave a comment or send a dm! Especially if you're not sure which paper is worth the effort for you.
    The initial list
    Stuff I've read and recommend:
    Discovering agents: agency definition I suspect is close to ideal
    detailed reasoning below
    Agents and devices: agency definition I find useful for rough thinking
    detailed reasoning below
    Logical induction for software engineers: induction definition I see as critical hunchbuilding and likely also a critical math tool
    todo: write detailed reasoning and pitch for reading it
    Unifying bargaining notions: working through the math of different game theory perspectives defined here has been insightful
    todo: write reasoning and pitch
    Finite factored sets in pictures: definition of causal inference based on direction of entropy instead of anything to do with time
    todo: write reasoning and pitch, especially for why to start with this one. see also the full sequence
    Other agent foundations related work I have high on my todo list:
    Geometric Rationality sequence: I haven't actually finished anything besides the first post; including based on strong hunch
    Boundaries sequence: has been high on my todo list for a while but keeps getting bumped down
    Andrew Critch's Lobposting - all of his posts that mention Lob.
    There are a bunch more things i'm planning to add but need to move on to other tasks right now, so I'm posting it as is with far less exposition on why I like each one than I intend to have eventually. Hope it helps someone!
    Discovering Agents (Kenton et al)
    Arxiv pdf
    Lesswrong / Alignmentforum summary and comments
    My absolute favorite paper in the safety literature right now, worth a level 3 read, discovering agents attempts to constrain the word "agency" by describing how to recognize it in a learning system that interacts with an environment. It has the limitation that it only works if you know what part of the system is the learner that may be agentic and you merely want to discover what it seeks. Also, the algorithm it defines is computable but thoroughly intractable for agents of any significant size, mostly limiting it to a formal definition, though substituting fast approximate causal inference may be fairly practical. The comments on both lesswrong and alignmentforum are essential to understand what it actually gives you, which is not quite as general as one might hope from the abstract.
    As with most of the incredible papers from the causal incentives group, It's not the easiest to understand unless you're quite experienced reading math papers, and it took me several passes of reading it with people to be pretty sure I got it; it was quite worth the significant effort. I had to skim the related paper Reasoning about Causality in Games to refresh my understanding of ...
    8 min
  • LW - Some Summaries of Agent Foundations Work by mattmacdermott
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some Summaries of Agent Foundations Work, published by mattmacdermott on May 15, 2023 on LessWrong.
    This is a bunch of not-very-technical summaries of agent foundations work from LessWrong and the Alignment Forum.
    I was hoping to turn it into a comprehensive literature review, categorising things in an enlightening way and listing open problems. It turns out that's pretty hard to do! It's languished in my drafts for long enough now, so I'm just going to post it as it is. Hopefully someone else can produce a great overview of the field instead.
    Why Agent Foundations?
    My own answer to this question is that most AI threat models depend on a powerful agent pursuing a goal we don't want it to, and mathematical models of agency seem useful both for understanding and dealing with these risks. Existing models of agency from fields like reinforcement learning and game theory don't seem up to the job, so trying to develop better ones might pay off.
    Normative and Descriptive Agent Foundations
    One account of why our usual models of agency aren't up to the job is the Embedded Agency sequence - the usual models assume agents are unchanging, indivisible entities which interact with their environments through predefined channels, but real-world agents are a part of their environment. The sequence lists identifies four rough categories of problems that arise when we switch to trying to model embedded agents, explained in terms of Marcus Hutter's model of the theoretically perfect reinforcement learning agent, AIXI.
    I find these subproblems to be a useful framing for thinking about agents foundations, but as I explained in a previous post, I think they take a strongly normative stance, asking what high-level principles an agent should follow in order be theoretically perfect. Some other agent foundations takes a more descriptive stance, asking what mechanistic and behavioural properties agents in the real world tend to have. You could also call this a distinction between top-down and bottom-up approaches to modelling agency.
    Here are the problems from the Embedded Agency sequence:
    Normative Subproblems
    Decision Theory. AIXI's actions affect the world in a well-defined way, but embedded agents have to figure out whether they care about the causal, evidential, or logical implications of their choices.
    Embedded World-Models. AIXI can hold every possible model of the world in its head in full detail and consider every consequence of its actions, but embedded agents are part of the world, and have limited space and compute with which to model it. This gives rise to the non-realisability problem - what happens when the real world isn't in your hypothesis class?
    Robust Delegation. AIXI is unchanging and the only comparable agent in town, but embedded agents can self-modify and create other agents. They need to ensure their successors are aligned.
    Subsystem Alignment. AIXI is indivisible, but embedded agents are chunks of the world made up of subchunks. What if those subchunks are agents with their own agendas?
    I think the embedded agency subproblems are also a useful way to categorise descriptive work, but the names and descriptions feel too normative, so I renamed them for the descriptive case. I also suggested a fifth problem, which is about figuring out how our models actually correspond to reality. I called it 'Identifying Agents', but now I prefer something like 'Agents in Practice'.
    Descriptive Subproblems
    I/O Channels. Actions, observations, and Cartesian boundaries aren't primitive: descriptive models need to define them. How do we move from a non-agentic model of the world to one with free will and counterfactuals?
    Internal Components. Presumably agents contain things like goals and world-models, but how do these components work mathematically? And are there others?
    Future Agents. W...
    22 min
  • EA - 2023 update on Muslims for Effective Altruism by Kaleem
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2023 update on Muslims for Effective Altruism, published by Kaleem on May 15, 2023 on The Effective Altruism Forum.
    Summary:
    Muslims for Effective Altruism made progress this year through our three projects: “The Muslim Network for Positive Impact”, “The Muslim Impact Lab”, and “Afterfund”, as well as continued behind the scenes work on fundraising, incubating new projects, and developing our short and medium term plans. The number of members of Muslims for Effective Altruism has continued to grow at a rate of ~25 members per year since mid-2021 (now at ~50 members).
    “Verdict”:
    It was a good (but not perfect) year for us, and we’re looking forward to doing even more interesting work, and having more of an impact, in the next one!
    Introduction:
    Around a year ago we posted our first post which gave a high level overview of why we think strating EA projects aimed at Muslims, or focusing on the intersection between EA and Islam, would be a high value endeavor. We were really pleasantly surprised at the amount of interest and support expressed about this endeavor, and continue to be grateful for the encouragement and feedback we receive from the community.
    Perhaps the most notable change before reading the rest of this post is that we’ve moved to thinking about Muslims for Effective Altruism’s structure as a federation of independent projects, rather than as one large org. We think this is useful because it prevents our lack of managerial capacity from holding back or interfering with our motivated project leads, as well as allowing us to increase our likelihood of expanding our network as other impactful projects can get off the ground without our knowledge or input.
    Kaleem will be attending EAG in London in May as well as hosting the “Muslims in EA” meetup there - so we thought it would be a good time to update everyone on what we’ve been up to in case you’d like to meet to discuss any aspects of our work.
    So, this post aims to provide an update on some of the work on which we’ve managed to make headway over the past year, as well as things we have fallen short on, and ways in which our plans have changed since the initial post.
    Projects:
    The Muslim Impact Lab
    The Muslim Impact Lab is a research and advisory body. This dynamic multi-disciplinary team is focused on collating and producing content on the intersections between EA ideas and Islamic Intellectual history, as well as providing expert consulting services to EA-Aligned organizations doing outreach in Muslim communities.
    The Lab assisted GiveDirectly by co-authoring their post in which they launched the Yemen Zakat Fund and previously advised them on their plans. Nayaaz Hashim, one of the co-founders of the lab, also published a piece on Unconditional Cash Transfers from an Islamic perspective on the Muslim Impact Lab substack. To date, GiveDirectly has raised $165,000 for the Yemen program, however it is difficult to quantify our counterfactual impact on this figure, given that other organizations such as Giving What We Can, and GiveDirectly themselves, have also been involved in raising funds. In the future we should look into ways of building in mechanisms to our processes that may help us establish our counterfactual impact with regards to raising effective Zakat.
    The lab is currently working on a survey to understand the moral priorities of the Muslim community, and developing a research agenda for, and producing content on, exploring intersections and challenges between understandings of Islam and current theories of effective altruism.
    The plan for the Muslim Impact Lab is to continue doing this research and, in collaboration with the Muslim Network for Positive Impact, put together a structured fellowship in the near future.
    Core Team: Maryam Khan, Nayaaz Hashim, Faezah Izadi, Mufti Sayed Haroon, Ahmed Gh...
    10 min
  • EA - Accidentally teaching AI models to deceive us (Ajeya Cotra on The 80,000 Hours Podcast) by 80000 Hours
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Accidentally teaching AI models to deceive us (Ajeya Cotra on The 80,000 Hours Podcast), published by 80000 Hours on May 15, 2023 on The Effective Altruism Forum.
    Over at The 80,000 Hours Podcast we just published an interview that is likely to be of particular interest to people who identify as involved in the effective altruism community: Ajeya Cotra on accidentally teaching AI models to deceive us.
    You can click through for the audio, a full transcript, and related links. Below is the episode summary and some key excerpts.
    Episode Summary
    I don’t know yet what suite of tests exactly you could show me, and what arguments you could show me, that would make me actually convinced that this model has a sufficiently deeply rooted motivation to not try to escape human control. I think that’s, in some sense, the whole heart of the alignment problem.
    And I think for a long time, labs have just been racing ahead, and they’ve had the justification — which I think was reasonable for a while — of like, “Come on, of course these systems we’re building aren’t going to take over the world.” As soon as that starts to change, I want a forcing function that makes it so that the labs now have the incentive to come up with the kinds of tests that should actually be persuasive.
    Ajeya Cotra
    Imagine you are an orphaned eight-year-old whose parents left you a $1 trillion company, and no trusted adult to serve as your guide to the world. You have to hire a smart adult to run that company, guide your life the way that a parent would, and administer your vast wealth. You have to hire that adult based on a work trial or interview you come up with. You don’t get to see any resumes or do reference checks. And because you’re so rich, tonnes of people apply for the job — for all sorts of reasons.
    Today’s guest Ajeya Cotra — senior research analyst at Open Philanthropy — argues that this peculiar setup resembles the situation humanity finds itself in when training very general and very capable AI models using current deep learning methods.
    As she explains, such an eight-year-old faces a challenging problem. In the candidate pool there are likely some truly nice people, who sincerely want to help and make decisions that are in your interest. But there are probably other characters too — like people who will pretend to care about you while you’re monitoring them, but intend to use the job to enrich themselves as soon as they think they can get away with it.
    Like a child trying to judge adults, at some point humans will be required to judge the trustworthiness and reliability of machine learning models that are as goal-oriented as people, and greatly outclass them in knowledge, experience, breadth, and speed. Tricky!
    Can’t we rely on how well models have performed at tasks during training to guide us? Ajeya worries that it won’t work. The trouble is that three different sorts of models will all produce the same output during training, but could behave very differently once deployed in a setting that allows their true colours to come through. She describes three such motivational archetypes:
    Saints — models that care about doing what we really want
    Sycophants — models that just want us to say they’ve done a good job, even if they get that praise by taking actions they know we wouldn’t want them to
    Schemers — models that don’t care about us or our interests at all, who are just pleasing us so long as that serves their own agenda
    In principle, a machine learning training process based on reinforcement learning could spit out any of these three attitudes, because all three would perform roughly equally well on the tests we give them, and ‘performs well on tests’ is how these models are selected.
    But while that’s true in principle, maybe it’s not something that could plausibly happen in the real world. Af...
    26 min
  • LW - Lessons learned from offering in-office nutritional testing by Elizabeth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lessons learned from offering in-office nutritional testing, published by Elizabeth on May 15, 2023 on LessWrong.
    Introduction
    I’ve talked previously about my concerns with nutritional deficiencies in effective altruists who go vegan for ethical reasons, especially those who don’t have a lot of contact with the broader vegan culture. No one else seemed very concerned, so I launched a tiny project to test people in this group and see if they were in fact deficient. This is a report on the latest phase of the project.
    To cut to the chase:
    It was very easy to find lots of deficiencies, although due to a severely heterogenous sample and lack of a control group this doesn’t provide useful information about if veganism is at fault.
    Finding these deficiencies probably leads to useful treatment, but not as much as I’d hoped.
    There are still a lot of operational issues to work out. My guess is that the ideal would require more work (to encourage participants to act on their results) or less (by focusing on education but not providing testing).
    I am currently looking for a co-founder to properly investigate the impact of veganism on nutrition.
    My main question here was “is there low-hanging fruit in treating nutritional deficiencies in this group, and if so how do we pluck it?” An important part of that is “how prevalent are deficiencies?”, but I had substantially more uncertainty around “do people treat deficiencies you find?” and “does the treatment lead to improvements in anything we actually care about?” That prioritization (and budget issues) led the experimental design to focus on operational issues and outcomes, and deprioritized getting the kind of clean data that would let me compare vegan and non-vegan outcomes. Similarly this write-up is mostly focused on showing the problem exists at all and building metis of investigation and treatment, rather than estimating prevalence.
    Which is to say to to everyone planning on @ing me to complain about the sample size, heterogeneity, or mediocre statistics: you are right that this sample is not very informative about base rates of deficiencies in vegans or anyone else. If someone claimed it was, they would be committing an epistemic sin. However, this particular post is focused on “how much effort is it to get nutritional issues addressed, and is that effort worth it?”. Given that, any complaints about the terrible sampling will be considered to be offers of assistance in running the much larger study that could answer the prevalence question.
    Background
    (if you’ve read the previous posts, this will be review)
    Last year I worked in a co-working space focused on existential risks, which is anything that might really end everything. Because x-risk is a popular topic within the effective altruism movement, many participants in the space were EAs. Another big topic within effective altruism is animal suffering, especially farmed animal suffering. This has led many EAs in the office to go vegan or at least vegetarian for ethical reasons, without making animals the focus of their lives. And when I asked them, many had put no thought into how to give up animal products in a healthy way.
    Which of course would have been fine if they were eating well. My personal opinion is that most people’s optimal diet contains small amounts of animal products, but lots of people are eating suboptimally for lots of reasons and I don’t consider it my problem. But the number of fatigue and concentration issues were. I don’t actually know if they were high. I don’t know what baseline to compare them to. But it wasn’t low enough to reassure me. Neither did the way the vegans talked about nutrition, and in particular the fact that more than one person was doing keto vegan and still wasn’t investigating nutrition.
    None of this meant there was necessarily a ...
    21 min
  • EA - A flaw in a simple version of worldview diversification by NunoSempere
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A flaw in a simple version of worldview diversification, published by NunoSempere on May 15, 2023 on The Effective Altruism Forum.
    Summary
    I consider a simple version of “worldview diversification”: allocating a set amount of money per cause area per year. I explain in probably too much detail how that setup leads to inconsistent relative values from year to year and from cause area to cause area. This implies that there might be Pareto improvements, i.e., moves that you could make that will result in strictly better outcomes. However, identifying those Pareto improvements wouldn’t be trivial, and would probably require more investment into estimation and cross-area comparison capabilities.1
    More elaborate versions of worldview diversification are probably able to fix this particular flaw, for example by instituting trading between the different worldview—thought that trading does ultimately have to happen. However, I view those solutions as hacks, and I suspect that the problem I outline in this post is indicative of deeper problems with the overall approach of worldview diversification.
    The main flaw: inconsistent relative values
    This section perhaps has too much detail to arrive at a fairly intuitive point. I thought this was worth doing because I find the point that there is a possible Pareto improvement on the table a powerful argument, and I didn’t want to hand-wave it. But the reader might want to skip to the next sections after getting the gist.
    Deducing bounds for relative values from revealed preferences
    Suppose that you order the ex-ante values of grants in different cause areas. The areas could be global health and development, animal welfare, speculative long-termism, etc. Their values could be given in QALYs (quality-adjusted life-years), sentience-adjusted QALYs, expected reduction in existential risk, but also in some relative unit2.
    For simplicity, let us just pick the case where there are two cause areas:
    More undilluted shades represent more valuable grants (e.g., larger reductions per dollar: of human suffering, animal suffering or existential risk), and lighter shades represent less valuable grants. Due to diminishing marginal returns, I’ve drawn the most valuable grants as smaller, though this doesn’t particularly matter.
    Now, we can augment the picture by also considering the marginal grants which didn’t get funded.
    In particular, imagine that the marginal grant which didn’t get funded for cause #1 has the same size as the marginal grant that did get funded for cause #2 (this doesn’t affect the thrust of the argument, it just makes it more apparent):
    Now, from this, we can deduce some bounds on relative values:
    In words rather than in shades of colour, this would be:
    Spending L1 dollars at cost-effectiveness A greens/$ is better than spending L1 dollars at cost-effectiveness B reds/$
    Spending L2 dollars at cost-effectiveness X reds/$ is better than spending L2 dollars at cost-effectiveness Y greens/$
    Or, dividing by L1 and L2,
    A greens is better than B reds
    X reds is better than Y reds
    In colors, this would correspond to all four squares having the same size:
    Giving some values, this could be:
    10 greens is better than 2 reds
    3 reds is better than 5 greens
    From this we could deduce that 6 reds > 10 greens > 2 reds, or that one green is worth between 0.2 and 0.6 reds.
    But now there comes a new year
    But the above was for one year. Now comes another year, with its own set of grants. But we are keeping the amount we allocate to each area constant.
    It’s been a less promising year for green, and a more promising year for red, . So this means that some of the stuff that wasn’t funded last year for green is funded now, and some of the stuff that was funded last year for red isn’t funded now:
    Now we can do the same comparisons as the last time:
    And when ...
    10 min
  • LW - Judgments often smuggle in implicit standards by Richard Ngo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Judgments often smuggle in implicit standards, published by Richard Ngo on May 15, 2023 on LessWrong.
    Our brains often use sleight-of-hand to hide fear-based motivation behind a guise of objectivity. This is particularly linked to the word “good”, which does a lot of work in a lot of people’s psychologies. For example, people often think that they, or their work, is “not good enough”. By itself, that sentence doesn’t make sense: good enough for what? Imagine going on a hike and commenting along the way “this rock isn’t heavy enough” or “this stream isn’t wide enough” without any background context. That sounds bizarre, and rightly so—the relevant threshold is very different depending on the context of the judgment. In other words, judgments are inherently two-place functions: they take in both some property and some threshold, and evaluate whether the property is above the threshold.
    Of course, people often don’t need to make the threshold explicit—if the reason you’re gathering rocks is to anchor down your tent, you can just say “this rock isn’t heavy enough” without further elaboration (although even then, miscommunications are common—heavy enough to withstand a stiff breeze? Or a gale? Or a storm?). But most judgments that people make of each other or themselves don’t have a clear threshold attached. Think of a girl standing in front of a mirror, saying to herself “I’m not beautiful enough”. Not beautiful enough to win a modeling competition? Or to convince a specific crush to go out with her? Or to appear in public without people making mean comments? The part of her mind which is making this evaluation doesn’t include that criterion, because it would weaken the forcefulness of its conclusion—it just spits out a judgment which feels like an objective evaluation, because the threshold is hidden.
    (The same is true if she just thinks “I’m not very beautiful”—not top 1%? 10%? 50%? What makes any of these thresholds important anyway?)
    Making the threshold explicit isn’t always going to change the judgment, but it’ll often make us realize that we’re holding ourselves to an unreasonably high standard. Here’s an exercise which might help, by nudging you to do the opposite. Think of the world—the whole thing, the beauty and the horror, the joy and the tragedy—and say out loud to yourself “everything’s okay, in comparison to how bad things could be”. No matter how bad you think it is now, it could all be much worse, right? Think of the satisfaction you’d feel if you thought you were on track for that much worse world, and suddenly learned that you were in our current world instead!
    Now say the same about your life—“everything’s okay, in comparison to how bad it could be”. Imagine the version of yourself who’d love to be in your position, and how they’d feel if they learned that they could. Lastly, try both of those again, but thinking about the future: “everything’s okay, no matter what happens from now on”. When I do this, I visualize moving the threshold for what counts as “okay” up and down, first measuring down from perfection, then up from hell, until the one-place judgment of “is this okay?” feels like a totally different type of thing from the two-place judgments which are actually relevant to the decisions I face.
    Does that feel weird? For me it does—I feel a sense of internal resistance. A part of me says “if you believe this, you’ll stop trying to make your life better!” I think that part is kinda right, but also a little hyperactive. I’m not committing to the view that everything’s okay, I’m just. trying it on for a second; trying to separate them enough to notice that it’s possible in principle. I notice that this resistance part feels a kind of frantic nervous energy at the thought of not applying high standards to myself—in other words, it feels fear-based mo...
    6 min
  • EA - EA Survey 2022: Demographics by David Moss
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Survey 2022: Demographics, published by David Moss on May 15, 2023 on The Effective Altruism Forum.
    Summary
    Gender
    Since 2020, the percentage of women in our sample has increased (26.5% vs 29.3%) and the percentage of men decreased (70.5% vs 66.2%).
    More recent cohorts of EAs have lower percentages of men than earlier cohorts. This pattern is compatible with either increased recruitment of women, non-binary or other non-male EAs in more recent years and/or men being more likely to drop out of EA.
    We examine differences between cohorts across years and find no evidence of significant differences in dropout between men and women.
    Race/ethnicity
    The percentage of white respondents in our sample (76.26%) has remained fairly flat over time.
    More recent cohorts contain lower percentages of white respondents (compatible with either increased recruitment and/or lower dropout of non-white respondents).
    We also examine differences between cohorts across years for race/ethnicity, but do not find a consistent pattern.
    Age
    The average age at which people first get involved in EA (26) has continued to increase.
    Education and employment
    The percentage of students in the movement has decreased since 2020 and the percentage in employment has increased. However, just over 40% of those who joined EA in the last year were students.
    Universities
    11.8% of respondents attended the top 10 (QS) ranked universities globally.
    Career strategies
    The most commonly cited strategy for impact in one’s career was ‘research’ (20.61%) followed by ‘still deciding’ (19.63%).
    More than twice as many respondents selected research as selected ‘earning to give’ (10.24%), organization-building skills (ops, management), government and policy, entrepreneurship or community building (<10% each).
    Men were significantly more likely to select research and significantly less likely to select organization-building skills. We found no significant differences by race/ethnicity.
    Highly engaged EAs were much more likely to select research (25.0% vs 15.1%) and much less likely to select earning to give (5.7% vs 15.7%).
    Politics
    Respondents continue to be strongly left-leaning politically (63.6% vs 2.4% right-leaning).
    Our 2022 sample was slightly more left-leaning than in 2019.
    Religion
    A large majority of respondents (69.58%) were atheist, agnostic or non-religious (similar to 2019).
    Introduction
    3567 respondents completed the 2022 EA Survey.
    A recurring observation in previous surveys is that the community is relatively lacking in demographic diversity on the dimensions of gender, age, race/ethnicity, and nationality. In this report, we examine the demographic composition of the community, how it has changed over time, and how this is related to different outcomes.
    In future posts in this series we will examine differences in experiences of and satisfaction with the community (see posts from 2019 and 2020), and explore the geography of the EA community.
    In a forthcoming follow-up survey, we will also be examining experiences related to gender and community satisfaction in more detail.
    Basic Demographics
    Gender
    The percentage of women has slightly increased since 2020 (26.5% to 29.3%), while the percentage of men has slightly decreased (70.5% to 66.2%).
    Gender across survey years
    Looking across different survey years, we can see that there is now a higher percentage of women in our sample than in the earliest years. In the earliest EA Surveys, we saw just over 75% men, whereas in the most recent survey, we see just over 65%.
    Gender across cohorts
    Looking across cohorts (EAs who reported first getting involved in a given year), we see that more recent cohorts contain more women than men. This is compatible with either/both increased recruitment of women (or decreased recruitment of men) or disproportionate attrition of...
    15 min
  • LW - Support Structures for Naturalist Study by LoganStrohl
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Support Structures for Naturalist Study, published by LoganStrohl on May 15, 2023 on LessWrong.
    It can be hard to stay engaged with a project that may last months, especially when you’re working on your own. In addition to meeting with my students every couple of weeks, and sometimes pairing them up with other students, there are five kinds of independent practices I often recommend to support consistent progress over the course of a naturalist study.
    Some of the support structures tend to be more important than others, and it’s critical to find a balance among them that works well for you.
    Undirected Time
    Naturalism involves learning new ways to perceive, not just updating explicit beliefs, so it relies extensively on background processing.
    That means it’s crucial to make space for shower thoughts. It’s crucial to make time for daydreaming, for mind-wandering, for the kinds of things your thoughts do when you’re neither trying to do anything in particular with them, nor having them directed by an external force (such as a book, conversation, or TV show).
    The busier you are, the more important it will probably be to create undirected time in your daily life while undertaking a naturalist study. You don’t actually have to schedule fifteen minutes each day to stare at a blank wall (although that’s a perfectly good way to go about this if it’s what you want to do); most people find that there is some existing part of their ordinary routine that can be modified to create more space for daydreaming. Some people who listen to a podcast during their morning commute choose to go without the podcast instead. Some people replace post-lunch Reddit scrolling with a short walk or a semi-nap. Anything that gives you a little more space to let your mind wander could do the trick.
    Field Work
    Field work is anything you do to observe experiences in their natural habitats. If you’re studying something that shows up while writing code as part of your job, field work happens on the clock, mostly without breaking the normal flow of your coding process.
    In general, successful field work requires two components: Opportunities to encounter fulcrum experiences, and a plan for how to observe them when they happen. If you’re studying interpersonal dynamics but you never actually interact with other people, you’re not giving yourself any opportunities to observe. If you encounter plenty of opportunities but haven’t planned ahead to notice when they happen, you probably won’t make many new observations.
    “Undirected time” and “field work” are the two most important structures in naturalism. Without them, there is no study. A successful study doesn’t necessarily require anything besides a plan to observe at the right times, and enough processing space to integrate observations.
    Field Notes
    “Field notes” means any method of recording observations that happen in the field.
    Some people try to record a brief note whenever they make a marking gesture, either in a notebook, as a voice memo, or in a text chat to themselves. Technically, a tally kept on a counter ring is a kind of field note.
    Another approach to field notes is daily reviews or journaling. Rather than taking notes on your experiences as they happen, you look back over your day and write down a little of what you remember about what you noticed. This method is farther removed from the observations, so key details sometimes get lost; but it can also be a lot easier for people who don’t want the flow of their activities interrupted.
    I often use both of these approaches at once: I make three-to-five-word notes in the field, and then I expand on them a little during daily reviews.
    Lab Work
    Lab work is any part of your study that happens under artificially designed circumstances.
    There’s a continuum from field work to lab work. If you create fulc...
    16 min
  • AF - Some Summaries of Agent Foundations Work by Matt MacDermott
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some Summaries of Agent Foundations Work, published by Matt MacDermott on May 15, 2023 on The AI Alignment Forum.
    This is a bunch of not-very-technical summaries of agent foundations work from LessWrong and the Alignment Forum.
    I was hoping to turn it into a comprehensive literature review, categorising things in an enlightening way and listing open problems. It turns out that's pretty hard to do! It's languished in my drafts for long enough now, so I'm just going to post it as it is. Hopefully someone else can produce a great overview of the field instead.
    Why Agent Foundations?
    My own answer to this question is that most AI threat models depend on a powerful agent pursuing a goal we don't want it to, and mathematical models of agency seem useful both for understanding and dealing with these risks. Existing models of agency from fields like reinforcement learning and game theory don't seem up to the job, so trying to develop better ones might pay off.
    Normative and Descriptive Agent Foundations
    One account of why our usual models of agency aren't up to the job is the Embedded Agency sequence - the usual models assume agents are unchanging, indivisible entities which interact with their environments through predefined channels, but real-world agents are a part of their environment. The sequence lists identifies four rough categories of problems that arise when we switch to trying to model embedded agents, explained in terms of Marcus Hutter's model of the theoretically perfect reinforcement learning agent, AIXI.
    I find these subproblems to be a useful framing for thinking about agents foundations, but as I explained in a previous post, I think they take a strongly normative stance, asking what high-level principles an agent should follow in order be theoretically perfect. Some other agent foundations takes a more descriptive stance, asking what mechanistic and behavioural properties agents in the real world tend to have. You could also call this a distinction between top-down and bottom-up approaches to modelling agency.
    Here are the problems from the Embedded Agency sequence:
    Normative Subproblems
    Decision Theory. AIXI's actions affect the world in a well-defined way, but embedded agents have to figure out whether they care about the causal, evidential, or logical implications of their choices.
    Embedded World-Models. AIXI can hold every possible model of the world in its head in full detail and consider every consequence of its actions, but embedded agents are part of the world, and have limited space and compute with which to model it. This gives rise to the non-realisability problem - what happens when the real world isn't in your hypothesis class?
    Robust Delegation. AIXI is unchanging and the only comparable agent in town, but embedded agents can self-modify and create other agents. They need to ensure their successors are aligned.
    Subsystem Alignment. AIXI is indivisible, but embedded agents are chunks of the world made up of subchunks. What if those subchunks are agents with their own agendas?
    I think the embedded agency subproblems are also a useful way to categorise descriptive work, but the names and descriptions feel too normative, so I renamed them for the descriptive case. I also suggested a fifth problem, which is about figuring out how our models actually correspond to reality. I called it 'Identifying Agents', but now I prefer something like 'Agents in Practice'.
    Descriptive Subproblems
    I/O Channels. Actions, observations, and Cartesian boundaries aren't primitive: descriptive models need to define them. How do we move from a non-agentic model of the world to one with free will and counterfactuals?
    Internal Components. Presumably agents contain things like goals and world-models, but how do these components work mathematically? And are there others?
    Fu...
    22 min