The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Practical Pitfalls of Causal Scrubbing by Jérémy Scheurer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Practical Pitfalls of Causal Scrubbing, published by Jérémy Scheurer on March 27, 2023 on The AI Alignment Forum.
    TL;DR: We evaluate Causal Scrubbing (CaSc) on synthetic graphs with known ground truth to determine its reliability in confirming correct hypotheses and rejecting incorrect ones. First, we show that CaSc can accurately identify true hypotheses and quantify the degree to which a hypothesis is wrong. Second, we highlight some limitations of CaSc, in particular, that it cannot falsify all incorrect hypotheses. We provide concrete examples of false positive results with causal scrubbing. Our main finding is that false positives can occur when there is “cancellation”, i.e., CaSc causes the model to do better on some inputs and worse on others, such that, on average, the scrubbed model recovers the full loss. A second practical failure mode is that CaSc cannot detect whether a proposed hypothesis is specific enough, and it cannot distinguish between hypotheses that are extensionally equivalent.
    We thank Redwood Research for generously supporting this project by providing us with their Causal Scrubbing implementation, access to REMIX materials, and computational resources. We specifically thank Ansh Radhakrishnan, Buck Shlegeris, and Nicholas Goldowsdky-Dill for their feedback and advice. We thank the Long-Term Future Fund for financial support and Marcel Steimke and Lennart Heim for operational support. Finally, we thank Marius Hobbhahn, Adam Jermyn, and Erik Jenner for valuable discussions and feedback.
    Introduction
    Causal Scrubbing (CaSc) is a method to evaluate the accuracy of hypotheses about neural networks and provides a measure of the deviation of a hypothesis from the ground truth. However, CaSc does not guarantee to reject false or incomplete hypotheses. We thus believe that systematic evaluation of CaSc to investigate these limitations is valuable (in addition to evaluating its effectiveness in the wild, as done in most existing work). Hence, we evaluate CaSc to highlight its strengths and weaknesses and explore the reliability of CaSc in confirming correct hypotheses and rejecting incorrect ones.
    We evaluate the reliability of CaSc on synthetic graphs. While synthetic graphs are less realistic than trained neural networks, we get access to the known ground truth interpretation, which allows us to accurately evaluate our hypotheses. Since CaSc operates on general computational graphs, any results on synthetic graphs also apply to using CaSc on neural networks (although we don’t make any claim on how likely the situations we find are to occur in trained neural networks).
    Our evaluation is based on creating a synthetic graph that solves a specific problem (e.g., sorting a list) and creating an identical interpretation graph (the correct hypothesis). We then perturb the correct interpretation graph to make the hypothesis “worse”. Finally, we evaluate whether CaSc correctly determines the better hypothesis. Ideally, we want the scrubbed loss (the loss induced by applying CaSc) to correlate with the “correctness” of a hypothesis.
    To determine whether a hypothesis is “better” or “worse”, we introduce the concepts of extensional and intensional equivalence between functions. Extensional equivalent functions have the same input-output behavior; for example, Quicksort and Mergesort are extensionally equivalent as they both sort an input sequence. Intensional equivalent functions are implemented in the same way mechanistically. So once we zoom further in and compare Quicksort and Mergesort algorithmically, we see that they are not intensionally equivalent. This point is already made in the CaSc writeup, and our goal is merely to highlight that in the context of mechanistic interpretability, this is an important distinction that’s easy to overlook.
    In this post,...
    23 min
  • EA - On what basis did Founder's Pledge disperse $1.6 mil. to Qvist Consulting from its Climate Change Fund? by Kieran.M
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On what basis did Founder's Pledge disperse $1.6 mil. to Qvist Consulting from its Climate Change Fund?, published by Kieran.M on March 27, 2023 on The Effective Altruism Forum.
    Disclaimer: I am totally agnostic regarding the reasonableness of this funding decision, and am merely noting that it appears to me impossible to make any assessment of reasonableness based on the information at hand. I have not conducted more than 1-2 hours research/thinking on this topic, so am uncertain of whether this is true, and am happy to be corrected. Also, the question is posed but I make no comment on whether Founder's Pledge needs answer it. Perhaps the donors to this fund are provided with some private information with regards to these causes, or Founder's Pledge reasonably believes their donors are happy to trust them a priori with respect to the efficacy and value set behind their decisions.
    In the last 12 months (March 2022 to present), Founder's Pledge (FP) has (publicly) dispersed approximately $4.3 million from its Climate Change Fund (CCF):
    Of this, $1.6 million has been given to Qvist Consulting Ltd. (QCL), for the reason shown above. Unlike the other recipients, QCL does not contain any external link to resources in which you are able to discover more about this organisation's operation/mission.
    Above this table, FP states that more information regarding their rationale is available here, however this document does not discuss QCL. As far as I have discovered, the only other mention of QCL is in a retrospective of the CCF after two years, which contains a minute video from Staffan Qvist (henceforth Staffan, for clarity). There is little/no new information about their mission, aside from the suggestion that "repowering" coal plants is particularly important because of the possible emissions from current coal plants in their remaining life cycle. What, then, is repowering coal? Curiously, another grantee, TerraPraxis, is the first Google result. The basic principle seems to be to try and refit those current coal plants for a non-carbon emitting form of energy production.
    So how does Qvist consulting fit into this effort? One might reasonably expect a search of their company to shed some light.
    This, however, turns out not to be the case.
    The company's first Google result is for their company listing on the UK gov. registry. The second is for their website, but it is merely a wordpress template totally devoid of information.
    So what about their founder? Staffan has, as per his LinkedIn, a PhD in Nuclear Engineering, and has written academic and popular press (including with Stephen Pinker in the NYTimes) articles advocating for nuclear energy.
    Staffan appears to be prolific - his LinkedIn lists him as a managing director/director at two other companies: Deepsense , an "intelligence platform", and QuantifiedCarbon, a decarbonisation consultancy, both of which appear to have little web presence (the former is difficult to search, as it is a common company name). Curiously, Staffan does not list QCL on his LinkedIn - perhaps this is an oversight? He is listed as a consultant for the Clean Air Task Force, another grantee of FP interested in nuclear advocacy.
    I have no reason to believe Staffan is not an excellent academic researcher and passionate advocate for a cause that seems plausibly very important (though I am no expert). Given the scale of the grant, however, it seems reasonable to wonder what in particular led FP to believe Staffan is the best contributor to this cause, and why he and QCL required such a large first grant to begin work on this.
    Postscript: There are other reasonable questions to be asked, including why FP believes their near-exclusive funding to organisations that appear (arguably) primarily dedicated to nuclear power advocacy is the most effective use ...
    5 min
  • AF - What happens with logical induction when... by Donald Hobson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What happens with logical induction when..., published by Donald Hobson on March 26, 2023 on The AI Alignment Forum.
    So this is a bunch of related technical questions about logical induction.
    Firstly, do you need the formal theorem prover section? Can you just throw out the formal theorem prover, but give some programs in the market unbounded capital and get the same resultant behaviour? (For example, give the program that bets P(X) towards 1−P(¬X) unbounded downside risk (downside risk of n on day n) ) This means the program would lose infinite money if X and ¬X both turned out to be true.
    I think that any axioms can be translated into programs. And I think such a setup, with some finite number of fairly simple programs having infinite money available produces a logical inductor. Is this true?
    What happens when the axioms added under this system are inconsistent. (so this is a logical induction market, without a theorem prover to settle the bets, and with agents with unlimeted money betting both for and against X, possibly indirectly like the bot betting for X, the bot betting for ¬X, and the bot described above trying to make P(X)+P(¬X)=1 ) Can the other agents make unbounded money? Do the prices converge? If I added a bot with infinite money that was convinced fermats last theorem was false to a consistent ZFC system, would I get a probability distribution that assigned high probability to basic arithmetic facts in the limit? Does this make a sensible system for logical counterfactuals?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Descriptive vs. specifiable values by Tsvi Benson-Tilsen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Descriptive vs. specifiable values, published by Tsvi Benson-Tilsen on March 26, 2023 on The AI Alignment Forum.
    [Metadata: crossposted from. First completed November 19, 2022.]
    What are an agent's values? An answer to this question might be a good description of the agent's external behavior and internal workings, without showing how one could modify the agent's workings or origins so that the agent pushes the world in a specific different direction.
    Descriptive values
    There's some discussion of what can be inferred about the values of an agent based on its behavior and structure. E.g. see Daniel Dennett's intentional stance, and "Occam's razor is insufficient to infer the preferences of irrational agents" by Stuart Armstrong, Sören Mindermann (arxiv), and this post by Vanessa Kosoy.
    One could describe an agent as having certain values: the agent's behavior is a boundedly rational attempt to push the world in certain directions. For some purposes, it's useful to have a parsimonious description of an agent's behavior or internal workings in terms of values. For example, such a description could be useful for helping the agent out: to help the agent out, you push the world in the same direction that the agent is trying to push the world.
    Specifiable values
    A distinct purpose in describing an agent as having values is to answer questions about values in counterfactuals:
    What determined that the agent would have those values and not other values?
    Under what circumstances will the agent continue to have those values? E.g., will the agent rewrite itself so that its behavior is no longer well-described as boundedly pursuing those values?
    How could the agent's values be modified? How could the values be modified in a specific direction, or to a specific state, so that that the modified agent has some specific effect on the world?
    How could the agent's ontogeny--the process that made it what it is--be altered so that it ends up with some other specific values?
    To make these questions more likely to have answers, and to not rely too much on assumptions about what values are, replace the notion of "values" with the notion "what directions a mind ends up pushing the world in".
    Quasi-example: explicit utility maximization
    An auxiliary question: how, mechanistically, do "the values" determine the behavior? This question might not have an answer, because there might not be some component in the agent that constitutes "the values". For example, in humans, there's no clear value component; there are many in-built behavior-determiners, but they don't fully constitute what we call our values. But, in cases where we clearly understand the mechanism by which an agent's values determine its behavior, answers to other questions about values in counterfactuals might follow.
    For example, there's the classic agent model: a system that searches for actions that it predicts will lead in expectation to the most highly-scored world according to its utility function box. The mechanism is explicit in this model. The utility function is embodied, in a box, as an input-output function, and it determines the agent's effects on the world by providing the criterion that the agent uses to select actions. Some answers to the above questions follow. E.g., it's clear at least qualitatively how to modify the agent's values to a specific state: if you want to make the agent cause a certain kind of world, just change the utility function to score that kind of world highly.
    Even this example is not so clear cut, and relies on background assumptions. See problems with embedded agency. For example, if we assume that there's already a fixed world (that is, an understanding of what's possible) about which to define the utility function, we sweep under the rug that the understanding behind having such a world had t...
    4 min
  • LW - Sam Altman on GPT-4, ChatGPT, and the Future of AI | Lex Fridman Podcast #367 by Gabriel Mukobi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sam Altman on GPT-4, ChatGPT, and the Future of AI | Lex Fridman Podcast #367, published by Gabriel Mukobi on March 25, 2023 on LessWrong.
    Lex Fridman just released a podcast episode with Sam Altman, CEO of OpenAI. In my opinion, there wasn't too much new here that hasn't been said in other recent interviews. However, here are some scattered notes on parts I found interesting from an AI safety lens:
    AI risk
    Lex asks Sama to steelman Eliezer Yudkowsky's views
    Sama said there's some chance of no hope, but the only way he knows how to fix things is to keep iterating and eliminating the "1-shot-to-get-it-right" cases. He does like one of Eliezer's posts that discusses his reasons why he thinks alignment is hard [I believe this is in reference to AGI Ruin: A List of Lethalities].
    Lex confirms he will do an interview with Eliezer.
    Sama: Now is the time to ramp up technical alignment work.
    Lex: What about fast takeoffs?
    Sama: I'm not that surprised by GPT-4, was a little surprised by ChatGPT [I think this means this feels slow to him]. I'm in the long-takeoffs, short-timelines quadrant. I'm scared of the short-takeoff scenarios.
    Sama has heard of but not seem Ex Machina
    On power
    Sama says it's weird that it will be OOM thousands of people in control of the first AGI .
    Acknowledges the AIS people think OAI deploying things fast is bad.
    Sama asks how Lex thinks they're doing.
    Lex likes the transparency and openly sharing the issues.
    Sama: Should we open source GPT-4?
    Lex: Knowing people at OAI, no (bc he trusts them,)
    Sama: I think people at OAI know the stakes of what we're building. But we're always looking for feedback from smart people.
    Lex: How do you take feedback?
    Sama: Twitter is unreadable. Mostly from convos like this.
    On responsibility
    Sama: We will have very significant but new and different challenges [with governing/deciding how to steer AI]
    Lex: Is it up to GPT or the humans to decrease the amount of hate in the world.
    Sama: I think we as OAI have responsibility for the tools we put out in the world, I think the tools can't have responsibility.
    Lex: So there could be harm caused by these tools
    Sama: There will be harm caused by these tools. There will be tremendous benefits. But tools do wonderful good and real bad. And we will minimize the bad and maximize the good.
    Jailbreaking
    Lex: How do you prevent jailbreaking?
    Sama: It kinda sucks being on the side of the company being jailbroken. We want the users to have a lot of control and have the models behave how they want within broad bounds. The existence of jailbreaking shows we haven't solved that problem yet, and the more we solve it, the less need there will be for jailbreaking. People don't really jailbreak iPhones anymore.
    Shipping products
    Lex: shows this tweet summarizing all the OAI products in the last year
    Sama: There's a question of should we be very proud of that or should other companies be very embarrassed. We have a high bar on our team, we work hard, we give a huge amount of trust, autonomy, and authority to individual people, and we try to hold each other to very high standards. These other things enable us to ship at such a high velocity.
    Lex: How do you go about hiring?
    Sama: I spend 1/3 of my time hiring, and I approve every OAI hire. There are no shortcuts to good hiring.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • AF - The alignment stability problem by Seth Herd
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The alignment stability problem, published by Seth Herd on March 26, 2023 on The AI Alignment Forum.
    The community thinks a lot about how to align AGI. It thinks less about how to align AGI so that it stays aligned for the long term. In many hypothetical cases, these are one and the same thing. But for the type of AGI we're actually likely to get, I don't think they are.
    Despite some optimism for aligning tool-like AGI, or at least static systems, it seems likely that we will create AGI that learns after it's deployed, and that has some amount of agency. If it does, its alignment will effectively shift, as addressed in the diamond maximizer thought experiment and elsewhere. And that's even if it doesn't deliberately change its preferences. People deliberately change their preferences sometimes, despite not having access to our own source code. So, it would seem wise to think seriously and explicitly about the stability problem, even if it isn't needed for current-generation AGI research.
    I've written a chapter on this, Goal changes in intelligent systems. There I laid out the problem, but I didn't really propose solutions. What follows is a summary of that article, followed by a brief discussion of the work I've been able to locate on this problem, and one direction we might go to pursue it.
    Why we don't think about much about alignment stability, and why we should.
    Some types of AGI are self-stabilizing. A sufficiently intelligent agent will try to prevent its goals from changing, at least if it is consequentialist. That works nicely if its values are one coherent construct, such as diamond or human preferences. But humans have lots of preferences, so we may wind up with a system that must balance many goals. And if the system keeps learning after deployment, it seems likely to alter its understanding of what its goals mean. This is the thrust of the diamond maximizer problem.
    One tricky thing about alignment work is that we're imagining different types of AGI when we talk about alignment schemes. Currently, people are thinking a lot about aligning deep networks. Current deep networks don't keep learning after they're deployed. And they're not very agentic These are great properties for alignment, and they seem to be the source of some optimism.
    Even if this type of network turns out to be really useful, and all we need to make the world a vastly better place, I don't think we're going to stop there. Agents would seem to have capabilities advantages that metaphorically make tool AI want to become agentic AI. If that weren't enough, agents are cool. People are going to want to turn tool AI into agent AI just to experience the wonder of an alien intelligence with its own goals.
    I think turning intelligent tools into agents is going to be relatively easy. But even if it's not easy someone is going to manage it at some point.. It's probably too difficult to prevent further experimentation, at least without a governing body, aided by AGI, that's able and willing to at minimum intercept and de-encrypt every communication for signs of AGI projects.
    While the above logic is far from airtight, it would seem wise to think about stable alignment solutions, in advance of anyone creating AGI that continuously learns outside of close human control.
    Similar concerns have been raised elsewhere, such as On how various plans miss the hard bits of the alignment challenge. Here I'm trying to crystallize and give a name to this specific hard part of the problem.
    Approaches to alignment stability
    Alex Turner addresses this in A shot at the diamond-alignment problem. In broad form, he's saying that you would train the agent with RL to value diamonds, including having diamonds associated with the reward in a variety of cognitive tasks. This is as good an answer as we've got. I don't have a...
    7 min
  • LW - Manifold: If okay AGI, why? by Eliezer Yudkowsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Manifold: If okay AGI, why?, published by Eliezer Yudkowsky on March 25, 2023 on LessWrong.
    Arguably the most important topic about which a prediction market has yet been run: Conditional on an okay outcome with AGI, how did that happen?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - A stylized dialogue on John Wentworth's claims about markets and optimization by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A stylized dialogue on John Wentworth's claims about markets and optimization, published by So8res on March 25, 2023 on LessWrong.
    (This is a stylized version of a real conversation, where the first part happened as part of a public debate between John Wentworth and Eliezer Yudkowsky, and the second part happened between John and me over the following morning. The below is combined, stylized, and written in my own voice throughout. The specific concrete examples in John's part of the dialog were produced by me.)
    J: It seems to me that the field of alignment doesn't understand the most basic theory of agents, and is missing obvious insights when it comes to modeling the sorts of systems they purport to study.
    N: Do tell. (I'm personally sympathetic to claims of the form "none of you idiots have any idea wtf you're doing", and am quite open to the hypothesis that I've been an idiot in this regard.)
    J: Consider the coherence theorems that say that if you can't pump resources out of a system, then it's acting agent-like.
    N: I'd qualify "agent-like with respect to you", if I used the word 'agent' at all (which I mostly wouldn't), and would caveat that there are a few additional subtleties, but sure.
    J: Some of those subtleties are important! In particular: there's a gap between systems that you can't pump resources out of, and systems that have a utility function. The bridge across that gap is an additional assumption that the system won't pass up certain gains (in a specific sense).
    Roughly: if you won't accept 1 pepper for 1 mushroom, then you should accept 2 mushrooms for 1 pepper, because a system that accepts both of those trades winds up with strictly more resources than a system that rejects both (by 1 mushroom), and you should be able to do at least that well.
    N: I agree.
    J: But some of the epistemically efficient systems around us violate this property.
    For instance, consider a market for (at least) two goods: peppers and mushrooms; with (at least) two participants: Alice and Bob. Suppose Alice's utility is UA(p,m):=log10(p)+log100(m) (where p and m are the quantities of peppers and mushrooms owned by Alice, respectively), and Bob's utility is UB(p,m):=log100(p)+log10(m) (where p and m are the quantities of peppers and mushrooms owned by Bob, respectively).
    Example equilibrium: the price is 3 peppers for 1 mushroom. Alice doesn't trade at this price when she has 3log′10(p)=1log′100(m), i.e. 3ln(10)/p=1ln(100)/m, i.e. 3/p=2/m (using the fact that ln(100)=ln(102)=2ln(10)), i.e. when Alice has 1.5 times as many peppers as she has mushrooms. Bob doesn't trade at this price when he has 6 times as many peppers as mushrooms, by a similar argument. So these prices can be an equilibrium whenever Alice has 1.5x as many peppers as mushrooms, and Bob has 6x as many peppers as mushrooms (regardless of the absolute quantities).
    Now consider offering the market a trade of 25,000 peppers for 10,000 mushrooms. If Alice has 20,000 mushrooms (and thus 30,000 peppers), and Bob has only 1 mushroom (and thus 6 peppers), then the trade is essentially up to Alice. She'd observe that
    so she (and thus, the market as a whole) would accept. But if Bob had 20,000 mushrooms (and thus 120,000 peppers), and Alice had only 2 mushrooms (and thus 3 peppers), then the trade is essentially up to Bob. He'd observe
    so he wouldn't take the trade.
    Thus, we can see that whether a market — considered altogether — takes a trade, depends not only on the prices in the market (which you might have thought of as a sort of epistemic state, and that you might have noted was epistemically efficient with respect to you), but also on the hidden internal state of the market.
    N: Sure. The argument was never "every epistemically efficient (wrt you) system is an optimizer", but rather "sufficiently good optimizers are ...
    14 min
  • AF - A stylized dialogue on John Wentworth's claims about markets and optimization by Nate Soares
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A stylized dialogue on John Wentworth's claims about markets and optimization, published by Nate Soares on March 25, 2023 on The AI Alignment Forum.
    (This is a stylized version of a real conversation, where the first part happened as part of a public debate between John Wentworth and Eliezer Yudkowsky, and the second part happened between John and me over the following morning. The below is combined, stylized, and written in my own voice throughout. The specific concrete examples in John's part of the dialog were produced by me. It's over a year old. Sorry for the lag.)
    J: It seems to me that the field of alignment doesn't understand the most basic theory of agents, and is missing obvious insights when it comes to modeling the sorts of systems they purport to study.
    N: Do tell. (I'm personally sympathetic to claims of the form "none of you idiots have any idea wtf you're doing", and am quite open to the hypothesis that I've been an idiot in this regard.)
    J: Consider the coherence theorems that say that if you can't pump resources out of a system, then it's acting agent-like.
    N: I'd qualify "agent-like with respect to you", if I used the word 'agent' at all (which I mostly wouldn't), and would caveat that there are a few additional subtleties, but sure.
    J: Some of those subtleties are important! In particular: there's a gap between systems that you can't pump resources out of, and systems that have a utility function. The bridge across that gap is an additional assumption that the system won't pass up certain gains (in a specific sense).
    Roughly: if you won't accept 1 pepper for 1 mushroom, then you should accept 2 mushrooms for 1 pepper, because a system that accepts both of those trades winds up with strictly more resources than a system that rejects both (by 1 mushroom), and you should be able to do at least that well.
    N: I agree.
    J: But some of the epistemically efficient systems around us violate this property.
    For instance, consider a market for (at least) two goods: peppers and mushrooms; with (at least) two participants: Alice and Bob. Suppose Alice's utility is UA(p,m):=log10(p)+log100(m) (where p and m are the quantities of peppers and mushrooms owned by Alice, respectively), and Bob's utility is UB(p,m):=log100(p)+log10(m) (where p and m are the quantities of peppers and mushrooms owned by Bob, respectively).
    Example equilibrium: the price is 3 peppers for 1 mushroom. Alice doesn't trade at this price when she has 3log′10(p)=1log′100(m), i.e. 3ln(10)/p=1ln(100)/m, i.e. 3/p=2/m (using the fact that ln(100)=ln(102)=2ln(10)), i.e. when Alice has 1.5 times as many peppers as she has mushrooms. Bob doesn't trade at this price when he has 6 times as many peppers as mushrooms, by a similar argument. So these prices can be an equilibrium whenever Alice has 1.5x as many peppers as mushrooms, and Bob has 6x as many peppers as mushrooms (regardless of the absolute quantities).
    Now consider offering the market a trade of 25,000 peppers for 10,000 mushrooms. If Alice has 20,000 mushrooms (and thus 30,000 peppers), and Bob has only 1 mushroom (and thus 6 peppers), then the trade is essentially up to Alice. She'd observe that
    so she (and thus, the market as a whole) would accept. But if Bob had 20,000 mushrooms (and thus 120,000 peppers), and Alice had only 2 mushrooms (and thus 3 peppers), then the trade is essentially up to Bob. He'd observe
    so he wouldn't take the trade.
    Thus, we can see that whether a market — considered altogether — takes a trade, depends not only on the prices in the market (which you might have thought of as a sort of epistemic state, and that you might have noted was epistemically efficient with respect to you), but also on the hidden internal state of the market.
    N: Sure. The argument was never "every epistemically efficient (wrt you) system is a...
    14 min
  • EA - Predicting what future people value: A terse introduction to Axiological Futurism by Jim Buhler
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Predicting what future people value: A terse introduction to Axiological Futurism, published by Jim Buhler on March 24, 2023 on The Effective Altruism Forum.
    Why this is worth researching
    Humanity might develop artificial general intelligence (AGI), colonize space, and create astronomical amounts of things in the future (Bostrom 2003; MacAskill 2022; Althaus and Gloor 2016). But what things? How (dis)valuable? And how does this compare with things grabby aliens would eventually create if they colonize our corner of the universe? What does this imply for our work aimed at impacting the long-term future?
    While this depends on many factors, a crucial one will likely be the values of our successors.
    Here’s a position that might tempt us while considering whether it is worth researching this topic:
    Our descendants are unlikely to have values that are both different from ours in a very significant way and predictable. Either they have values similar to ours or they have values we can’t predict. Therefore, trying to predict their values is a waste of time and resources.
    While I see how this can seem compelling, I think this is very ill-informed.
    First, predicting the values of our successors – what John Danaher (2021) calls axiological futurism – in worlds where these are meaningfully different from ours doesn’t seem intractable at all. Significant progress has already been made in this research area and there seems to be room for much more (see the next section and the Appendix).
    Second, a scenario where the values of our descendants don’t significantly differ from ours appears quite unlikely to me. We should watch for things like the End of History illusion, here. Values seem to notably evolve through History, and there is no reason to assume we are special enough to make us drop that prior.
    Besides being tractable, I believe axiological futurism to be uncommonly important given its instrumentality in answering the crucial questions mentioned earlier. It therefore also seems unwarrantedly neglected as of today.
    How to research this
    Here are examples of broad questions that could be part of a research agenda on this topic:
    What are the best predictors of future human values? What can we learn from usual forecasting methods?
    How have people’s values changed throughout History? Why? What can we learn from this? (see, e.g., MacAskill 2022, Chapter 3; Harris 2019; Hopster 2022)
    Are there reasons to think we’ll observe less change in the future? Why? Value lock-in? Some form of moral convergence happening soon?
    Are there reasons to expect more change? Would that be due to the development of AGI, whole brain emulation, space colonization, and/or accelerated value drift?
    More broadly, what impact will future technological progress have on values? (see Hanson 2016 for a forecast example.)
    Should we expect some values to be selected for? (see, e.g., Christiano 2013; Bostrom 2009, Tomasik 2017)
    Might a period of “long reflection” take place? If yes, can we get some idea of what could result from it?
    Does something like coherent extrapolated volition have any chance of being pursued and if so, what could realistically result from it?
    Are there futures – where humanity has certain values – that are unlikely but worth wagering on?
    Might our research on this topic affect the values we should expect our successors to have by, e.g., triggering a self-defeating or self-fulfilling prophecy effect? (Danaher 2021, section 2)
    What do/will aliens value (see my forthcoming next post) and what does that tell us about ourselves?
    John Danaher (2021) gives examples of methodologies that could be used to answer these questions.
    Also, my Appendix references examples and other relevant work, including the (forthcoming) next posts in this sequence.
    Acknowledgment
    Thanks to Anders Sandberg for pointing m...
    7 min