The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Caution when interpreting Deepmind's In-context RL paper by Sam Marks
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Caution when interpreting Deepmind's In-context RL paper, published by Sam Marks on November 1, 2022 on The AI Alignment Forum.
    Lots of people I know have had pretty strong reactions to the recent Deepmind paper, which claims to have gotten a transformer to learn an RL algorithm by training it on an RL agent's training trajectories. At first, I too was pretty shocked -- this paper seemed to provide strong evidence of a mesa-optimizer in the wild. But digging into the paper a bit more, I'm quite unimpressed and don't think that in-context RL is the correct way to interpret the experiments that the authors actually did. This post is a quick, low-effort attempt to write out my thoughts on this.
    Recall that in this paper, the authors pick some RL algorithm, use it to train RL agents on some tasks, and save the trajectories generated during training; then they train a transformer to autoregressively model said trajectories, and deploy the transformer on some novel tasks. So for concreteness, during training the transformer sees inputs that look like
    which were excerpted from an RL agent's training on some task (out of a set of training tasks) and which span multiple episodes (i.e. at some point in this input trajectory, one episode ended and the next episode began). The transformer is trained to guess the action an+c that comes next. In deployment, the inputs are determined by the transformer's own selections, with the environment providing the states and rewards. The authors call this algorithmic distillation (AD).
    Many people I know have skimmed the paper and come away with an understanding something like:
    In this paper, RL agents are trained on diverse tasks, e.g. playing many different Atari games, and the resulting transcripts are used as training data for AD. Then the AD agent is deployed on a new task, e.g. playing a held-out Atari game. The AD agent is able to learn to play this novel game, which can only be explained by the model implementing an reasonably general RL algorithm. This sounds a whole lot like a mesa-optimizer.
    This understanding is incorrect, with two key issues. First the training tasks used in this paper are all extremely similar to each other and to the deployment task; in fact, I think they only ought to count as different under a pathologically narrow notion of "task." And second, the tasks involved are extremely simple. The complaints taken together challenge the conclusion that the only way for the AD agent to do well on its deployment task is by implementing a general-purpose RL algorithm. In fact, as I'll explain in more detail below, I'd be quite surprised if it were.
    For concreteness, I'll focus here on one family of experiments, Dark Room, that appeared in the paper, but my complaint applies just as well to the other experiments in the paper. The paper describes the Dark Room environment as:
    a 2D discrete POMDP where an agent spawns in a room and must find a goal location. The agent only knows its own (x, y) coordinates but does not know the goal location and must infer it from the reward. The room size is 9 × 9, the possible actions are one step left, right, up, down, and no-op, the episode length is 20, and the agent resets at the center of the map. ... [T]he agent receives r = 1 every time the goal is reached. ... When not r = 1, then r = 0.
    To be clear, Dark Room is not a single task, but an environment supporting a family of tasks, where each task is corresponds to a particular choice of goal location (so there are 81 possible tasks in this environment, one for each location in the 9 x 9 room; note that this is an unusually narrow notion of which tasks count as different). The data on which the AD agent is trained look like: {many episodes of an agent learning to move towards goal position 1}, {many episodes of an agent learning to ...
    8 min
  • EA - The Challenges with Measuring the Impact of Lobbying by Animal Ask
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Challenges with Measuring the Impact of Lobbying, published by Animal Ask on October 31, 2022 on The Effective Altruism Forum.
    A foundational report on the methodology we use at Animal Ask to measure the impact of lobbying campaigns.
    EXECUTIVE SUMMARY
    Legislative lobbying has led to numerous positive outcomes in many policy areas and social movements. For example, lobbying in the United States is responsible for lower taxes on solar power, increased taxes on tobacco, and the establishment of a program to detect asteroids (Lerner 2020). In the animal advocacy movement specifically, one recent victory is the 'End the Cage Age' initiative. This lobbying campaign led to a commitment by the European Commission to phase out the use of cages for hens, sows, calves, and numerous other farmed animals (European Commission 2021).
    When evaluating campaign strategy, it is important to have a good way to project the success of competing legislative opportunities. Ideally, this would involve predicting the degree to which lobbying effort makes a legislative opportunity more likely to succeed, compared to the scenario where that additional effort does not take place. We refer to this as the counterfactual impact of lobbying. Understanding this would provide insight into how campaigning can be most effective.
    Currently, we do not know the best way to measure the counterfactual impact of lobbying. One promising option is to use the academic literature on lobbying. The published studies on lobbying could be used to derive estimates of policy success and, ideally, the counterfactual impact of lobbying.
    The purpose of this report is to find out whether this academic literature contains quantitative estimates that could be useful in gauging the counterfactual impact of lobbying. We answer this question by reviewing the literature on lobbying and examining studies that have published estimates of the rates of policy success.
    Overall policy success is made up of two components: a baseline rate of policy success, and the counterfactual impact of lobbying. Therefore, to measure the counterfactual impact of lobbying, we need to be able to do two things: 1) measure overall policy success, and 2) break down this overall policy success into the baseline rate of success and the counterfactual impact of lobbying (see Figure 1 below).
    We find that measuring overall policy success using the academic literature is unlikely to work. The lobbying literature contains many well-known weaknesses which cast doubt on researchers' ability to measure policy success. As well as this, our systematic review found a strange result, which suggests that the published estimates of policy success should not be taken at face value.
    We find that there is insufficient evidence to break down overall policy success into the baseline rate of success and the counterfactual impact of lobbying. There is only a single study that identifies the counterfactual impact of lobbying. That study was limited to a single policy issue and a single context, and so its findings need to be replicated across other issues and contexts before we can draw any sound conclusions.
    In summary, the academic literature does not contain sufficient information to gauge the counterfactual impact of lobbying. For these reasons, we recommend against using estimates from the published literature when modelling the effects of lobbying. Instead, we recommend that researchers choose a different option for incorporating information on the prospects of success. We conclude by highlighting two promising options that deserve further exploration: expert judgement and panels of superforecasters.
    Although we conducted this research to guide our own research process at Animal Ask, this report may also be useful to other researchers. In particular, we expect this ...
    1 hr 6 min
  • LW - Superintelligent AI is necessary for an amazing future, but far from sufficient by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Superintelligent AI is necessary for an amazing future, but far from sufficient, published by So8res on October 31, 2022 on LessWrong.
    (Note: Rob Bensinger stitched together and expanded this essay based on an earlier, shorter draft plus some conversations we had. Many of the key conceptual divisions here, like "strong utopia" vs. "weak utopia" etc., are due to him.)
    I hold all of the following views:
    Building superintelligent AI is profoundly important. Aligned superintelligence is our best bet for taking the abundant resources in the universe and efficiently converting them into flourishing and fun and art and beauty and adventure and friendship, and all the things that make life worth living.
    The best possible future would probably look unrecognizably alien. Unlocking humanity’s full potential not only means allowing human culture and knowledge to change and grow over time; it also means building and becoming (and meeting and befriending) very new and different sorts of minds, that do a better job of realizing our ideals than the squishy first-pass brains we currently have.
    The default outcome of building artificial general intelligence, using anything remotely like our current techniques and understanding, is not a wondrously alien future. It’s that humanity accidentally turns the reachable universe into a valueless wasteland (at least up to the boundaries defended by distant alien superintelligences).
    The reason I expect AGI to produce a “valueless wasteland” by default, is not that I want my own present conception of humanity’s values locked into the end of time.
    I want our values to be able to mature! I want us to figure out how to build sentient minds in silicon, who have different types of wants and desires and joys, to be our friends and partners as we explore the galaxies! I want us to cross paths with aliens in our distant travels who strain our conception of what’s good, such that we all come out the richer for it! I want our children to have values and goals that would make me boggle, as parents have boggled at their children for ages immemorial!
    I believe machines can be people, and that we should treat digital people with the same respect we give biological people. I would love to see what a Matrioshka mind can do. I expect that most of my concrete ideas about the future will seem quaint and outdated and not worth their opportunity costs, compared to the rad alternatives we'll see when we and our descendants and creations are vastly smarter and more grown-up.
    Why, then, do I think that it will take a large effort by humanity to ensure that good futures occur? If I believe in a wondrously alien and strange cosmopolitan future, and I think we should embrace moral progress rather than clinging to our present-day preferences, then why do I think that the default outcome is catastrophic failure?
    In short:
    Humanity’s approach to AI is likely to produce outcomes that are drastically worse than, e.g., the outcomes a random alien species would produce.
    It’s plausible — though this is much harder to predict, in my books — that a random alien would produce outcomes that are drastically worse (from a cosmopolitan, diversity-embracing perspective!) than what unassisted, unmodified humans would produce.
    Unassisted, unmodified humans would produce outcomes that are drastically worse than what a friendly superintelligent AI could produce.
    The practical take-away from the first point is “the AI alignment problem is very important”; the take-away from the second point is “we shouldn’t just destroy ourselves and hope aliens end up colonizing our future light cone, and we shouldn’t just try to produce AI via a more evolution-like process”; and the take-away from the third point is “we shouldn’t just permanently give up on building superintelligent AI”.
    To clarify my view...
    1 hr 6 min
  • AF - What sorts of systems can be deceptive? by Andrei Alexandru
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What sorts of systems can be deceptive?, published by Andrei Alexandru on October 31, 2022 on The AI Alignment Forum.
    This work was done as part of SERI MATS, under Leo Gao’s guidance. Thank you to Erik Jenner and Johannes Treutlein for discussions and comments on the draft.
    I’m interested in understanding deception in machine intelligence better. Specifically, I want to understand what precursors there are to deceptive alignment, and whether upon observing these precursors, we can change our approach to achieve better outcomes. In this article, I outline my current thinking on this topic, and consider a bunch of properties that systems which can be deceptive might share. I am still pretty confused about how this works, and I don’t yet have good ideas for what comes next.
    Preliminaries
    What is deception?
    Commonly, by deception we mean a scenario where an intelligent system behaves in a way that hides what information it knows from another system, or from a human overseer. Fundamentally, the reason to be deceptive is that behaving honestly would lead to the system being penalised by its designers, perhaps through a training process that can intervene directly on its internal mechanisms.
    In the context of Risks from Learned Optimization, deceptive alignment occurs when a system internally has a goal other than what we would like it to be. If this system is aware that it might be shut down or altered if it revealed this discrepancy, it is incentivised to play along, i.e. behave as though it is optimising the humans’ goal. Once the training process is complete and the system is safe to pursue its own goal, it does so without repercussions.
    As described in the paper, deception is an inner misalignment failure, i.e. in the system there exists an inner optimiser whose goals may be different to the base optimiser’s goals. If this is the case, even if we select for models which appear to seek our base objective, the objective they actually end up pursuing may be different. These systems are deceptively aligned.
    In this framing, the main way we might get deceptive alignment is through a system that performs internal search. While search is a central example, I’m curious to understand more generally what the prerequisites are for deceptive alignment. I’m particularly interested in the connection with consequentialism. In a sense, deception is just one of the many strategies a consequentialist agent might use to achieve its goals.
    How does the idea of optimisation matter here?
    In this article, I use “optimisation” in the context of an optimising system – a system which tends to reconfigure a space toward a small set of target states, while being robust to perturbation. This does not require the existence of a part of the system which is doing the optimisation, i.e. a separate optimiser. In this view, I think it’s possible to have a continuum of optimising systems, ranging from search over possible actions on one extreme to a set of heuristics on the other, such that all these systems outwardly behave as consequentialists, regardless of how they’re implemented.
    This is related to the view that some folks at MIRI have that consequentialism is about the work that’s being done, rather than the type of system that is carrying it out. If that is the case, then in the limit of generalisation capability (i.e. systems which exhibit robust generalisation), it doesn’t matter how the system is implemented – specifically, whether it’s doing search or not – for how likely it is to exhibit deceptive behaviour. Note that my impression of Eliezer’s view on this is that he does think that you can build a weaker system which doesn’t exhibit some undesirable property such as deception, but that this weaker system is just not enough to carry out a pivotal act. (I base this mostly on Eliezer’s and Ri...
    13 min
  • AF - "Cars and Elephants": a handwavy argument/analogy against mechanistic interpretability by David Scott Krueger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Cars and Elephants": a handwavy argument/analogy against mechanistic interpretability, published by David Scott Krueger on October 31, 2022 on The AI Alignment Forum.
    TL;DR: If we can build competitive AI systems that are interpretable, then I argue via analogy that trying to extract them from messy deep learning systems seems less promising than directly engineering them. Preliminaries: Distinguish weak and strong tractability of (mechanistic) interpretability as follows:
    Weak tractability: AGI-level systems are interpretable in principle, i.e. humans have the capacity to fully understand their workings in practice given the right instructions and tools. This would be false if intelligence involves irreducible complexity, e.g. because it involves concepts that are not "crisp", modular, or decomposable; for instance, there might not a crisp conceptual core to various aspects of perception or abstract concepts such as "fairness".
    Strong tractability: We can build interpretable AGI-level systems without sacrificing too much competitiveness.
    The claim:If strong tractability is true, then mechanistic interpretability is likely not the best way to engineer competitive AGI-level systems.The analogy:1) Suppose we have a broken down car with some bad parts and we want to have a car that is safe to drive. We could try to fix the car and update the parts. 2) But we also have a perfectly functioning elephant. So instead, we could try and tinker with the elephant to understand how it works and make its behavior more safe and predictable.I claim (2) is roughly analogous to mechanistic interpretability, and (1) to pursuing something more like what Stuart Russell seems to be aiming for: a neurosymbolic approach to AI safety based on modularity, probabilistic programming, and formal methods.Fleshing out the argument a bit more:To the extent that strong tractability is true, there must be simple principles we can recognize underlying intelligent behavior.
    If there are not such simple principles, then we shouldn't expect mechanistic interpretability methods to yield safe, competitive systems. We already have many ideas about what some of those principles might be (from GOFAI and other areas). Why would we expect it is easier to recognize and extract these principles from neural networks than to deliberately incorporate them into the way we engineer systems? Epistemic status: I seem to be the biggest interpretability hater/skeptic I've encountered in the AI x-safety community. This is an argument I came up with a few days ago that seems to capture some of my intuitions, although it is hand-wavy. I haven't thought about it much, and spent ~1hr writing this, but am publishing it anyways because I don't express my opinions publicly as often as I'd like due to limited bandwidth.Caveats: I made no effort to anticipate and respond to counter-arguments (e.g. "those methods aren't actually more interpretable").
    There are lots of different ways that interpretability might be useful for AI x-safety. It makes sense as part of a portfolio approach. It makes sense as an extra "danger detector" that might produce some true positives (even if there are a lot of false negatives) or one of many hacks that might be stacked. I'm not arguing that Stuart Russell's approach is clearly superior to mechanistic interpretability.
    But it seems like roughly the entire AI existential safety community is very excited about mechanistic interpretability and entirely dismissive of Stuart Russell's approach, and this seems bizarre.Unrelated bonus reason to be skeptical of interpretability (hopefully many more to come!): when you to deploy a reasonably advanced system in the real world, it will likely recruit resources outside itself in various ways (e.g. The way people write things down on paper as a way of augmenting...
    5 min
  • LW - publishing alignment research and infohazards by carado
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: publishing alignment research and infohazards, published by carado on October 31, 2022 on LessWrong.
    to me, turning my thoughts into posts that i then publish on my blog and sometimes lesswrong serves the following purposes:
    in conversations, i can easily link to a post of mine rather than explaining myself again (the original primary purpose of this blog!)
    having a more formally written-down version of my thoughts helps me think about them more clearly
    future posts — whether written by me or others — can link to my posts, contributing to a web of related ideas
    i can get feedback on my ideas, whether it be through comments on lesswrong or responses on discord
    however, i've come to increasingly want to write and publish posts which i've determined — either on my own or with the advice of a trusted peers — to be potentially infohazardous, notably with regards to potentially helping AI capability progress.
    on one hand, there is no post of mine i wouldn't trust, say, yudkowsky reading; on the other i can't just, like, DM him and everyone else i trust a link to an unlisted post every time i make one.
    it would be nice to have a platform — or maybe a lesswrong feature — which lets me choose which persons or groups can read a post, with maybe a little ⚠ sign next to its title.
    note that such a platform/feature would need something more complex than just a binary "trusted" flag: just because i can make a post that the Important People can read, doesn't mean i should be trusted to read everything else that they can read; and there might be people whom i trust to read some posts of mine but not others.
    maybe trusted recipients could be grouped by orgs — such as "i trust MIRI" or "i trust The Standard List Of Trusted Persons". maybe something like the ability to post on the alignment forum is a reasonable proxy for "trustable person"?
    i am aware that this seems hard to figure out, let alone implement. perhaps there is a much easier alternative i'm not thinking about; for the moment, i'll just stick to making unlisted posts and sending them to the very small intersection of people i trust with infohazards and people for whom it's socially acceptable for me to DM links to new posts of mine.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Gandalf or Saruman? A Soldier in Scout's Clothing by AllAmericanBreakfast
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gandalf or Saruman? A Soldier in Scout's Clothing, published by AllAmericanBreakfast on October 31, 2022 on LessWrong.
    Sometimes, Scout Mindset can feel a lot like Soldier Mindset, and vice versa.
    When you've considered an issue with some care, formed an opinion, and somebody responds with raised voice about how awful that opinion is, it's hard not to get defensive or take it personally.
    There are lots of ways you could respond. If your aim is to achieve a thoughtful and respectful conversation, then you have to do several things: control your emotional reaction, be polite, stay honest about the fact that you disagree, and choose a response that allows the discussion to return to civil discourse.
    Going through all that feels like a Scout-y thing to do, but it isn't necessarily. Provoking a person with a strongly held (but possibly correct) view into offputting behavior, then assuming a posture of being the reasonable and civil side of the debate, is a wonderful rhetorical strategy. Loudly trying to shout down your opponent makes you look like a dumb Soldier. A smart Soldier tries to look like a Scout.
    Likewise, if you're a Scout and your information is being ignored, you might need to raise your voice, fight your way through, and make them listen.
    People who are into signaling talk a lot about virtue signaling and intelligence signaling. It might be helpful to talk about Scout signaling and Soldier signaling. Neither of these are the same as being a Scout or Soldier. They aren't reliable indicators in every context of where somebody is coming from. If a person is interested in developing Scout Mindset, it seems important to me for them to learn how to distinguish Scout Mindset from Scout signaling.
    Back to our acrimonious debate, let's say the civil person in the debate sincerely wants to want to practice Scout Mindset. There are a couple of useful things they can do.
    One is to stick to their honest beliefs. That is part of the territory they are honestly scouting out.
    Another is to remember that their loud debate partner might be a Scout with a very important message. Or not! But if you haven't thoroughly explored their opinion and formed your own conclusions, it's often wise to reserve judgment. If you do so, and stick to your original view, then having fully internalized what your debate partner had to say will help you address that point the next time you're in debate.
    I'll finish with an example from a debate I had today in which I was playing the "civil" role and my debate partner was the "loud" one. We were talking about the ethics of regulated kidney sales vs. bans on organ sales.
    My view is that legalizing the regulated sale of kidneys is a good thing. It gives a person with failing health a shot at a much better life, becomes a source of income for the seller, and lowers the cost of medical care.
    Their view is that poor people who are likely to develop health problems later will tend to be the people selling their organs, even in a legal, regulated system. This is already the case with blood plasma sales. It will become a way for sellers of kidneys to get money fast, blow it on immediate needs, wreck their remaining kidney, and require a transplant themselves a few years down the line. Kidney surgery is painful, and some of the people selling their kidneys will die on the operating table. Political conservatives will use the fact that poor people can sell their kidneys as an excuse to cut welfare. They referred me to this column in the New York Times. Just a small sample from this powerfully written article:
    The people around me seemed to be regulars who were trying to squeeze a donation in before work. I know because I heard them lying on their phones to their employers about why they were going to be late as the morning wore on. More women came in after n...
    8 min
  • EA - Draft Amnesty Day: an event we might run on the Forum by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Draft Amnesty Day: an event we might run on the Forum, published by Lizka on October 31, 2022 on The Effective Altruism Forum.
    TL;DR: We’re considering making an official “Draft Amnesty Day” on the Forum when users will be encouraged to share unfinished posts, unpolished writing, butterfly ideas, thoughts they’re not sure they endorse, etc. We’re looking for feedback on this idea.
    In the spirit of draft amnesty, I’m sharing this post even though it’s not very polished. :)
    Motivation
    We’re often told that posting on the Forum is intimidating. Anecdotally, lots of people have drafts they’ve written that they feel uncertain about. A draft like this tends to sit around in Google Drives gathering metaphorical dust until the author decides to spend time and finalize it (often, months later), the idea becomes irrelevant (someone posts something that makes it redundant, or the moment passes), or the draft is just discarded.
    I think it’s good to maintain a high standard for content on the Forum, but it’s a shame and a real loss when potentially very useful ideas never see the light of day — or get shared half a year later than they could have been.
    Sharing a draft before you share a final version of a post can also mean that the final version is better! Others can provide feedback, notice flaws, share relevant resources, etc. You can also fail faster, which is also a win.
    How this would work
    We’d announce a day (or weekend). On that day, there might be a banner at the top of the Frontpage announcing that it’s Draft Amnesty Day (DAD).
    By default, we’d do this once and see how it goes. We wouldn’t plan for it to be recurring.
    We’d make a tag and encourage everyone to use it. Users who aren’t logged in wouldn’t see DAD posts by default, and others would be able to hide the posts entirely.
    We’d provide a template message to put on the top of DAD posts so that others can engage with it accordingly.
    We’d ask people to be especially nice in the comments of DAD posts (and would be ready to moderate any cases that violate this norm). On the other hand, honest feedback would likely be really useful, so I’d hope that this wouldn’t prevent people from commenting, and I’d encourage people to post “feedback guidelines.”
    Maybe: these posts wouldn’t be indexed on search engines, meaning they wouldn’t show up when you search via something like Google.
    If you end up polishing or otherwise finalizing a DAD draft and would like to share it again, we’d encourage that — you wouldn’t need to worry that you should “save” your posts for later.
    An example blurb to put on top of DAD posts
    This is a Draft Amnesty Day draft. That means it’s not polished, it’s probably not up to my standards, the ideas are not thought out, and I haven’t checked everything. I was explicitly encouraged to post something unfinished! Commenting and feedback guidelines: I’m going with the default — please be nice. But constructive feedback is appreciated; please let me know what you think is wrong. Feedback on the structure of the argument is also appreciated.
    This would go right under the title (if the author wants to put it there).
    What do you think of this idea?
    How could it go wrong? How could it be improved? Would you post anything? Please share your feedback!
    You can also use the comment section of this post to share ideas you have for posts and get feedback on those ideas.
    I think this is related to “agile” approaches (e.g. to software development), although I don’t know much about the theory, and “lean” principles.
    I might not respond to everything, but I'll read all the comments.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • AF - Boundaries vs Frames by Scott Garrabrant
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Boundaries vs Frames, published by Scott Garrabrant on October 31, 2022 on The AI Alignment Forum.
    This post is partially in response to Critch's boundaries sequence. My best guess is that he would agree with most of it in theory, but disagree with some of it in practice due to tradeoffs with other considerations in defining the concepts.
    Boundaries and Frames
    Imagine a world consisting of some atomic objects R={r0,r1,.}. For example, you can think of the ri as physical atoms or cells in the game of life. Each object comes along with a collection of states it can be in ri={s0i,s1i,.}.
    (I am not committing now on whether or not we are thinking about our model as timeless. Maybe ri should be thought about as a cell in the game of life that passes through time, or maybe it should be thought about as a (cell, time) pair.)
    There are two sets we naturally want to associate with our world. First, we have R={r0,r1,.}, which I will call the object space. Second, we have S=Πr∈R r, which is the set of all ways to assign state to each object in R. I will call this the state space. Note that partitions of R correspond to factorizations of S.
    If I want to point at (for example) an agent in R, I might tell you what atoms are inside that agent, and thus express R in the form R=a⊔e, where a is the set of all atoms that are in the agent and e is the set of all atoms that are outside of the agent (and thus in the environment).
    The agent then has its own state space A=Πr∈a r, while the environment has its own state space, E=∏r∈e r. Now, to point at this agent in S, I can express S in the form S=A×E.
    Instead of specifying a list of objects in the agent, I specify the state space of the agent. Instead of thinking of the environment as the result of subtracting the agent object out of the world, I think of the environment as the result of quotienting out the agent's state space from the world's state space.
    Instead of defining a Cartesian Boundary, I am defining a Cartesian Frame.
    Playing on this naming scheme, in general (when not necessarily talking about agents), I will use the word boundary when talking about partitioning the object space into a disjoint inside and outside, and I will use the word frame when talking about factoring the state space into an independent inside and outside. (This naming might end up only being used within the scope of this post. I'm not sure.)
    Frames are More General/Basic
    I usually prefer to think in terms of frames (but not always!). While every boundary can be recast as a frame, the converse is not also true. If we view the state space as primary, then by imagining the world as a collection of objects, we are essentially factoring the state space. The frames that correspond to boundaries are basically those that carve along the joints given by the factorization into basic objects.
    If the world is given to you pre-factored as a collection of objects, then it makes sense to partition those objects into larger objects and draw boundaries around them. However, I think for interesting problems, this is rarely the case. Further, even if the world is pre-factorized, that factorization could be wrong!
    Also, the act of drawing a boundary feels similar to me to the act of factoring the state space into atomic objects. Both are carving out interesting features of state space. The initial factorization into atomic objects is identifying microscopic objects, while the boundaries are identifying macroscopic objects. Thus, starting from a pre-factored world feels especially bad when thinking about identifying boundaries, since it is starting with half the problem already done. (Note: I think this argument is weaker than it sounds. The factorization into objects is a different type than the drawing of the boundaries.)
    Getting Past the Physical Frame
    The main practical ...
    11 min
  • EA - An Introduction to the Moral Weight Project by Bob Fischer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Introduction to the Moral Weight Project, published by Bob Fischer on October 31, 2022 on The Effective Altruism Forum.
    Key Takeaways
    Our objective: provide “moral weights” for 11 farmed species.
    To make this tractable, we made four assumptions: utilitarianism, hedonism, valence symmetry, and unitarianism.
    Given these assumptions, an animal’s “moral weight” is that animal’s capacity for welfare—the total amount of welfare that the animal could realize.
    Capacity for welfare = welfare range (the difference between the best and worst welfare states the individual can realize at a time) × lifespan.
    Given welfare ranges, we can convert welfare improvements into DALY-equivalents averted, making cross-species cost-effectiveness analyses possible.
    An Introduction to the Moral Weight Project
    If we want to do as much good as possible, we have to compare all the ways of doing good—including ways that involve helping members of different species. But do benefits to humans count the same as benefits to chickens? What about chickens vs. carp? Carp vs. honey bees? In 2020, Rethink Priorities published the Moral Weight Series—a collection of five reports about these and related questions. The first introduces different theories of welfare and moral status and their interrelationships. The second compares two ways of estimating differences in capacity for welfare and moral status. The third explores the rate of subjective experience, its importance, and potential variation in the rate of subjective experience across taxa. The fourth considers critical flicker-fusion frequency as a proxy for the rate of subjective experience. The fifth assesses whether there’s variation in the intensity ranges of valenced experiences from species to species.
    In May 2021, Rethink Priorities launched the Moral Weight Project, which extended and implemented the research program that those initial reports discussed. This post is the first in the Moral Weight Project Sequence. The aim of the sequence is to provide an overview of the research that Rethink Priorities conducted between May 2021 and October 2022 on interspecific cause prioritization—i.e., making resource allocation decisions across species. The aim of this post is to introduce the project and explain how EAs could use its results.
    DALYs-averted and “moral weight discounts”
    If we want to do the most good per dollar spent—that is, if we want to maximize cost-effectiveness—we need a common currency for comparing very different interventions. GiveWell, Founders Pledge, Open Philanthropy, and many other EA organizations currently have one: namely, the number of “disability-adjusted life years (DALYs) averted.” A DALY is a health measure with two parts: years of human life lost and years of human life lost to disability. The former measures the extent to which a condition shortens a human’s life; the latter measures the health impact of living with a condition in terms of years of life lost. Together, these values represent the overall burden of the condition. So, averting a DALY is averting a loss—namely, the loss of a single year of human life that’s lived at full health.
    Historically, some EAs used a “moral weight discount” to convert changes in animals’ welfare levels directly into DALYs-averted. That is, they understood the basic question to be:
    At what point should we be indifferent between (say) improving chickens’ welfare and preventing the loss of a year of healthy human life?
    Then, the task is to identify the correct “moral weight discount rate” to apply to the value of some quantity of chicken welfare to make it equal to the value of averting a DALY.
    This framing makes it tempting to think in terms of the value that (certain groups of) people assign to chicken welfare (or the welfare of the members of whatever species) relative to the ...
    17 min