The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - A challenge for AGI organizations, and a challenge for readers by RobBensinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A challenge for AGI organizations, and a challenge for readers, published by RobBensinger on December 1, 2022 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - Beware frictions from altruistic value differences by Magnus Vinding
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Beware frictions from altruistic value differences, published by Magnus Vinding on December 1, 2022 on The Effective Altruism Forum.
    I believe value differences pose some underappreciated challenges in large-scale altruistic efforts. My aim in this post is to outline what I see as the main such challenges, and to present a few psychological reasons as to why we should expect these challenges to be significant and difficult to overcome.
    To clarify, my aim in this post is not to make a case against value differences per se, much less a case against vigorous debate over values (I believe that such debate is healthy and desirable). Instead, my aim is to highlight some of the challenges and pitfalls that are associated with value differences, in the hope that we can better mitigate these pitfalls. After all, value differences are sure to persist among people who are trying to help others, and hence a critical issue is how well — or how poorly — we are going to handle these differences.
    Examples of challenges posed by value differences among altruists
    A key challenge posed by value differences, in my view, is that they can make us prone to tribal or otherwise antagonistic dynamics that are suboptimal by the lights of our own moral values. Such values-related frictions may in turn lead to the following pitfalls and failure modes:
    Failing to achieve moral aims that are already widely shared, such as avoiding worst-case outcomes (cf. “Common ground for longtermists”).
    Failing to make mutually beneficial moral trades and compromises when possible (in ways that do not introduce problematic behavior such as dishonesty or censorship).
    Failing to update on arguments, whether they be empirical or values-related, because the arguments are made by those who, to our minds, seem like they belong to the “other side”.
    Some people committing harmful acts out of spite or primitive tribal instincts. (The sections below give some sense as to why this might happen.)
    Of course, some of the failure modes listed above can have other causes beyond values- and coalition-related frictions. Yet poorly handled such frictions are probably still a key risk factor for these failure modes.
    In short, as I see it, the main challenges associated with value differences lie in mitigating the risks that emerge from values-related frictions, such as the risks outlined above.
    Reasons to expect values-related frictions to be significant
    The following are some reasons to expect values-related frictions to be both common and quite difficult to handle by default.
    Harmful actions based on different moral beliefs may be judged more harshly than intentional harm
    One set of findings that seem relevant come from a 2016 anthropological study that examined the moral judgments of people across ten different cultures, eight of which were traditional small-scale societies (Barrett et al., 2016).
    The study specifically asked people how they would evaluate a harmful act in light of a range of potentially extenuating circumstances, such as different moral beliefs, a mistake of fact, or self-defense. While there was significant variation in people’s moral judgments across cultures, there was nevertheless unanimous agreement that committing a harmful act based on different moral beliefs was not an extenuating circumstance. Indeed, on average across cultures, committing a harmful act based on different moral beliefs was considered worse than was committing the harmful act intentionally (see Barrett et al., 2016, fig. 5).
    It is unclear whether this pattern in moral judgment necessarily applies to all or even most kinds of acts inspired by different moral beliefs. Yet these results still tentatively suggest that we may be inclined to see value differences as a uniquely aggravating factor in our moral judgments of people’s actions — a...
    10 min
  • LW - Finding gliders in the game of life by paulfchristiano
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Finding gliders in the game of life, published by paulfchristiano on December 1, 2022 on LessWrong.
    ARC’s current approach to ELK is to point to latent structure within a model by searching for the “reason” for particular correlations in the model’s output. In this post we’ll walk through a very simple example of using this approach to identify gliders in the game of life.
    We’ll use the game of life as our example instead of real physics because it’s much simpler, but everything in the post would apply just as well to identifying “strawberry” within a model of quantum field theory. More importantly, we’re talking about identifying latent structures in physics because it’s very conceptually straightforward, but I think the same ideas apply to identifying latent structure within messier AI systems.
    Setting: sensors in the game of life
    The game of life is a cellular automaton where an infinite grid of cells evolves over time according to simple rules. If you haven’t encountered it before, you can see the rules at wikipedia and learn more at conwaylife.com.
    A glider is a particular pattern of cells. If this pattern occurs in empty space and we simulate 4 steps of the rules, we end up with the same pattern shifted one square to the right and one square down.
    Let’s imagine some scientists observing the game of life via a finite set of sensors. Each sensor is located at a cell, and at each timestep the sensor reports whether its cell is empty (“dead”) or full (“alive”). For simplicity, we’ll imagine just two sensors A and B which lie on a diagonal 25 cells apart. So in any episode our scientists will observe two strings of bits, one from each sensor.
    (To be more realistic we could consider physically-implemented sensors, e.g. patterns of cells in the game of life which measure what is happening by interacting with the grid and then recording the information in a computer also built inside the game of life. But that adds a huge amount of complexity without changing any of our analysis, so for now we’ll just talk about these supernatural sensors.)
    These scientists don’t know how the game of life works. All they see are the reports of the sensors. They observe that sensor B often fires 100 timesteps after sensor A, which they call an “A-B pattern.” Perhaps sensor A fires on 0.1% of timesteps, and sensor B fires on 0.1% of timesteps, but A-B patterns occur on 0.01% of timesteps instead of the 0.00001% you would have expected if the events were independent.
    Scientists hypothesize that A-B patterns occur due to an object traveling between sensor A and sensor B. They call this hypothetical object a “glider,” but they have no idea that a glider consists of a particular pattern of 5 live cells.
    To help understand the world, the scientists collect a large training set of sensor readings and train a generative model which can predict those readings, i.e. a function that maps a uniformly random string of bits ε to a sequence of observations for sensor A and sensor B. This generative model is trained to assign a high probability to the sequences in the training set.
    The generative model makes good predictions, and in particular it reproduces the surprising frequency of A-B patterns. As a result, the scientists expect that “gliders” must correspond to some latent structure in the model.
    Explaining A-B patterns in the generative model
    Let’s suppose that the generative model uses the random seed ε to fill in a large grid, with 10% of cells alive and 90% dead, simulates the grid for 1000 timesteps, and then outputs the 1000 bits observed by each of the sensors.
    To find gliders, we’ll search for a “probabilistic heuristic argument” based on the presumption of independence that A-B patterns are common. We expect that argument to somehow reflect the fact that gliders travel from sensor A to se...
    26 min
  • AF - A challenge for AGI organizations, and a challenge for readers by Rob Bensinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A challenge for AGI organizations, and a challenge for readers, published by Rob Bensinger on December 1, 2022 on The AI Alignment Forum.
    (Note: This post is a write-up by Rob of a point Eliezer wanted to broadcast. Nate helped with the editing, and endorses the post’s main points.)
    Eliezer Yudkowsky and Nate Soares (my co-workers) want to broadcast strong support for OpenAI’s recent decision to release a blog post ("Our approach to alignment research") that states their current plan as an organization.
    Although Eliezer and Nate disagree with OpenAI's proposed approach — a variant of "use relatively unaligned AI to align AI" — they view it as very important that OpenAI has a plan and has said what it is.
    We want to challenge Anthropic and DeepMind, the other major AGI organizations with a stated concern for existential risk, to do the same: come up with a plan (possibly a branching one, if there are crucial uncertainties you expect to resolve later), write it up in some form, and publicly announce that plan (with sensitive parts fuzzed out) as the organization's current alignment plan.
    Currently, Eliezer’s impression is that neither Anthropic nor DeepMind has a secret plan that's better than OpenAI's, nor a secret plan that's worse than OpenAI's. His impression is that they don't have a plan at all.
    Having a plan is critically important for an AGI project, not because anyone should expect everything to play out as planned, but because plans force the project to concretely state their crucial assumptions in one place. This provides an opportunity to notice and address inconsistencies, and to notice updates to the plan (and fully propagate those updates to downstream beliefs, strategies, and policies) as new information comes in.
    It's also healthy for the field to be able to debate plans and think about the big picture, and for orgs to be in some sense "competing" to have the most sane and reasonable plan.
    We acknowledge that there are reasons organizations might want to be abstract about some steps in their plans — e.g., to avoid immunizing people to good-but-weird ideas, in a public document where it’s hard to fully explain and justify a chain of reasoning; or to avoid sharing capabilities insights, if parts of your plan depend on your inside-view model of how AGI works.
    We’d be happy to see plans that fuzz out some details, but are still much more concrete than (e.g.) “figure out how to build AGI and expect this to go well because we'll be particularly conscientious about safety once we have an AGI in front of us".
    Eliezer also hereby gives a challenge to the reader: Eliezer and Nate are thinking about writing up their thoughts at some point about OpenAI's plan of using AI to aid AI alignment. We want you to write up your own unanchored thoughts on the OpenAI plan first, focusing on the most important and decision-relevant factors, with the intent of rendering our posting on this topic superfluous.
    Our hope is that challenges like this will test how superfluous we are, and also move the world toward a state where we’re more superfluous / there’s more redundancy in the field when it comes to generating ideas and critiques that would be lethal for the world to never notice.
    We didn't run a draft of this post by DM or Anthropic (or OpenAI), so this information may be mistaken or out-of-date. My hope is that we’re completely wrong!
    Nate’s personal guess is that the situation at DM and Anthropic may be less “yep, we have no plan yet”, and more “various individuals have different plans or pieces-of-plans, but the organization itself hasn’t agreed on a plan and there’s a lot of disagreement about what the best approach is”.
    In which case Nate expects it to be very useful to pick a plan now (possibly with some conditional paths in it), and make it a priority to hash out and...
    4 min
  • EA - "Insider EA content" in Gideon Lewis-Kraus's recent New Yorker article by To be stuck inside of Mobile
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Insider EA content" in Gideon Lewis-Kraus's recent New Yorker article, published by To be stuck inside of Mobile on December 1, 2022 on The Effective Altruism Forum.
    Direct link (to New Yorker website).
    Alternative link (publicly accessible).
    This piece from Gideon Lewis-Kraus (the writer for the MacAskill piece) is a recent overview of how EA has reacted to SBF and the FTX collapse.
    Lewis-Kraus's articles are probably the most in-depth public writing on EA, and he has had wide access to EA members and leadership.
    The New Yorker is highly respected and the narratives and attitudes in this piece will influence future perceptions of EA.
    This piece contains inside information about discussions or warnings about SBF. It uses interviews from a "senior EA", and excepts from an internal Slack channel used by senior EAs.
    When my profile of MacAskill, which discussed internal movement discord about Bankman-Fried’s rise to prominence, appeared in August, Wiblin vented his displeasure on the Slack channel. As he put it, the problem was with the format of such a treatment. He wrote, “They don’t focus on ‘does this person have true and important ideas.’ The writer has no particular expertise to judge such a thing and readers don’t especially care either. Instead the focus is more often on personal quirkiness and charisma, relationships among people in the story, ‘she said / he said’ reporting of disagreements, making the reader feel wise and above the substantive issue, and finding ways the topic can be linked to existing political attitudes of New Yorker readers (so traditional liberal concerns). This is pretty bad because our great virtue is being right, not being likeable or uncontroversial or ‘right-on’ in terms of having fashionable political opinions.”
    There are claims of a warning about SBF on the Slack channel:
    This past July, a contributor to the Slack channel wrote to express great apprehension about Sam Bankman-Fried. “Just FYSA,”—or for your situational awareness—“said to me yesterday in DC by somebody in gov’t: ‘Hey I was investigating someone for [x type of crime] and realized they’re on the board of CEA’ ”—MacAskill’s Centre for Effective Altruism—“ ‘or run EA or something? Crazy! I didn’t realize you could be an EA and also commit a lot of crime. Like shouldn’t those be incompatible?’ (about SBF). I don’t usually share this type of thing here, but seemed worth sharing the sentiment since I think it is not very uncommon and may be surprising to some people.” In a second message, the contributor continued, “I think in some circles SBF has a reputation as someone who regularly breaks laws to make money, which is something that many people see as directly antithetical to being altruistic or EA. (and I get why!!).
    That reputation poses PR concerns to EA whether or not he’s investigated, and whether or not he’s found guilty.” The contributor felt this was a serious enough issue to elaborate a third time: “I guess my point in sharing this is to raise awareness that a) in some circles SBF’s reputation is very bad b) in some circles SBF’s reputation is closely tied to EA, and c) there’s some chance SBF’s reputation gets much, much worse. But I don’t have any data on these (particularly c, I have no idea what types of scenarios are likely), though it seems like a major PR vulnerability. I imagine people working full-time on PR are aware of this and actively working to mitigate it, but it seemed worth passing on if not since many people may not be having these types of interactions.” (Bankman-Fried has not been charged with a crime. The Department of Justice declined to comment.)
    The suggestion is that EA leadership, while not knowing of any actual crime, accepted poor behavior and norm breaking because of the resources Bankman-Fried provided.
    In other words, it seems as th...
    6 min
  • AF - Take: We're not going to reverse-engineer the AI. by Charlie Steiner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take: We're not going to reverse-engineer the AI., published by Charlie Steiner on December 1, 2022 on The AI Alignment Forum.
    As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes, which seems likely.
    Any approach to building safe transformative AI, or even just auditing possibly-safe TAI, which relies on reverse-engineering neural networks into fine-grained pseudocode based on mechanistic understanding should keep its ambitions very modest.
    This hot take is to some extent against ambitious "microscope AI," and to some extent against a more underlying set of intuitions about the form and purpose of interpretability research. (A somewhat related excellent background post is Neel's list of theories of impact for interpretability.) So I should start by explaining what those things are and why they might be appealing.
    Webster's Dictionary defines microscope AI as "training systems to do complex tasks, then interpreting how they do it and doing it ourselves." Prima facie, this would help with transformative AI. Suppose we're building some AI that's going to have a lot of power over the world, but we're not sure if it's trustworthy - what if some of its cognition is about how to do things we don't want it to be doing? If we can do microscope AI, we can understand how our first AI is so clever, and build a second AI that's just as clever and that we're sure isn't doing things it shouldn't, like running a search for how best to deceive us.
    Microscope-powered auditing is easier - if it's hard to assemble the second AI that does good things and not bad things, how about just checking that the first AI is trustworthy? To check an AI's trustworthiness in this microscope-AI-like framing of the issues, we might want to understand how its cognitive processes work in fine-grained detail, and check that none of those processes are doing bad stuff.
    When I say I'm against this, I don't mean auditing is impossible. I mean that it's not going to happen by having humans understand how the AI works in fine-grained detail.
    As an analogy, you can figure out how curve detectors work in InceptionV1. Not just in the sense that "oh yeah, that neuron is totally a curve detector," but in terms of how the whole thing works. It's yet more difficult to figure out that other neurons are not curve detectors - typically at this point we fall back on data-based methods like ablating those neurons and then trying to get the network to recognize rainbows, rather than first-principles arguments. But we can more or less figure out that InceptionV1 has an intermediate state where it detects curves, by an understandable algorithm and for human-understandable reasons.
    If we wanted to figure out how InceptionV1 tells dogs from cats, we might hope to gradually hack away at the edges - use what we know to expand the circle of knowledge a little more, and then repeat. Use the example of curve detectors to figure out spike detectors. Use spike-detectors to figure out fur-texture detectors, and curve detectors to figure out nose-shape detectors. Then we can learn how fur texture and nose shape play into deciding on dog vs. cat. At each step we can use data to test our understanding, but the basic goal is to be able to write down the flow of information between features in a human-comprehensible way. It's not just about giving neurons english-language labels, it's about giving them sensible algorithms where those labels play the expected role.
    The biggest problem with this plan is that neural networks leak. Many things are connected to many other things, weakly, in ways that are important for their success. I recently was at a talk that showed how the vast majority of attention heads in a transformer have lots of ...
    7 min
  • EA - How should small and medium-sized donors step in to fill gaps left by the collapse of the FTX Future Fund? by JanBrauner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How should small and medium-sized donors step in to fill gaps left by the collapse of the FTX Future Fund?, published by JanBrauner on December 1, 2022 on The Effective Altruism Forum.
    Scott Alexander writes:
    "The past year has been a terrible time to be a charitable funder, since FTX ate every opportunity so quickly that everyone else had trouble finding good not-yet-funded projects. But right now is a great time to be a charitable funder: there are lots of really great charities on the verge of collapse who just need a little bit of funding to get them through. I’m trying to coordinate with some of the people involved. I haven’t really succeeded yet, I think because they’re all hiding under their beds gibbering - but probably they’ll have to come out eventually, if only for food and water. If you’re a potential charitable funder interested in helping, and not already connected to this project, please email me at [email protected]. I don’t want any affected charities to get their hopes up, because I don’t expect this to fill more than a few percent of the hole, but maybe we can make the triage process slightly less of a disaster."
    I think this argument is probably correct. Potentially we shouldn't fund every organisation that would have been funded by the Future Fund. Potentially, we shouldn't fund things that were started because FTX had so much money, and which are then going to die in a year or two now that the Future Fund isn't around anymore. This might be a time of "right-sizing" projects for available money. However, even with the recent reduction in total assets, many of these projects/organisations should probably be supported. One could simply apply the heuristic of "should this be part of the EA portfolio?" after taking into consideration that EA has less funding now.
    There are also considerations about community support and mutual insurance; e.g. donors might want to help out individuals who quit their jobs because they expected a grant from the Future Fund, and so on. I'm going to stop here now, as this is not my main point.
    My main question is this: HOW should small-to-medium sized donors (let's say people who want to donate 3-to-7 figures) actually go about this? In particular, if they don't want to put out a public call for proposals, which will likely end in receiving dozens and dozens of grant requests?One option is to email Scott (see above). Any other ideas?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - The Plan - 2022 Update by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Plan - 2022 Update, published by johnswentworth on December 1, 2022 on LessWrong.
    So, how’s The Plan going?
    Pretty well!
    In last year’s writeup of The Plan, I gave “better than a 50/50 chance” that it would work before AGI kills us all (and my median AI timelines were around 10-15 years). That was an outside view, accounting for planning fallacy and the inevitable negative surprises. My inside view was faster - just based on extrapolating my gut feel of the rate of progress, I privately estimated that The Plan would take around 8 years. (Of those 8, I expected about 3 would be needed to nail down the core conceptual pieces of agent foundations, and the other 5 would be to cross the theory-practice gap. Of course those would be intermingled, though with the theory part probably somewhat more front-loaded.)
    Over the past year, my current gut feel is that progress has been basically in line with the inside-view 8 year estimate (now down to 7, since a year has passed), and maybe even a little bit faster than that.
    So, relative to my outside-view expectation that things always go worse than my gut expects, things are actually going somewhat better than expected! I’m overall somewhat more optimistic now, although the delta is pretty small. It’s only been a year, still lots of time for negative surprises to appear.
    Any high-level changes to The Plan?
    There have been two main high-level changes over the past year.
    First: The Plan predicted that, sometime over the next 5 (now 4) years, the field of alignment would “go from a basically-preparadigmatic state, where we don’t even know what questions to ask or what tools to use to answer them, to a basically-paradigmatic state, where we have a general roadmap and toolset”. Over the past year, I tentatively think the general shape of that paradigm has become visible, as researchers converge from different directions towards a common set of subproblems.
    Second: I’ve updated away from thinking about ambitious value learning as the primary alignment target. Ambitious value learning remains the main long-term target, but I’ve been convinced that e.g. corrigibility is worth paying attention to as a target for early superhuman AGI. Overall, I’ve updated from “just aim for ambitious value learning” to “empirically figure out what potential medium-term alignment targets (e.g. human values, corrigibility, Do What I Mean, human mimicry, etc) are naturally expressible in an AGI’s internal concept-language”.
    Convergence towards a paradigm sounds exciting! So what does it look like?
    Exciting indeed! Gradual convergence toward a technical alignment paradigm has probably been the most important update from the past year.
    On the theoretical side, Paul Christiano, Scott Garrabrant, and myself had all basically converged to working on roughly the same problem (abstraction, ontology identification, whatever you want to call it) by early 2022. That kind of convergence is a standard hallmark of a proto-paradigm.
    Meanwhile, within the past year-and-a-half or so, interpretability work has really taken off; Chris Olah’s lab is no longer head-and-shoulders stronger than everyone else. And it looks to me like the interpretability crowd is also quickly converging on the same core problem of abstraction/ontology-identification/whatever-you-want-to-call-it, but from the empirical side rather than the theoretical side.
    That convergence isn’t complete yet - I think a lot of the interpretability crowd hasn’t yet fully internalized the framing of “interpretability is primarily about mapping net-internal structures to corresponding high-level interpretable structures in the environment”. In particular I think a lot of interpretability researchers have not yet internalized that mathematically understanding what kinds of high-level interpretable structures appea...
    15 min
  • EA - Biological Anchors external review by Jennifer Lin (linkpost) by peterhartree
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Biological Anchors external review by Jennifer Lin (linkpost), published by peterhartree on November 30, 2022 on The Effective Altruism Forum.
    This report is one of the winners of the EA Criticism and Red Teaming Contest.
    Summary: This is a summary and critical review of Ajeya Cotra’s biological anchors report on AI timelines. It provides an easy-to-understand overview of the main methodology of Cotra’s report. It then examines and challenges central assumptions of the modelling in Cotra’s report. First, the review looks at reasons why we might not expect 2022 architectures to scale to AGI. Second, it raises the point that we don’t know how to specify a space of algorithmic architectures that contains something that could scale to AGI and can be efficiently searched through (inability to specify this could undermine the ability to take the evolutionary anchors from the report as a bound on timelines).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - [LINK] - ChatGPT discussion by JanBrauner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [LINK] - ChatGPT discussion, published by JanBrauner on December 1, 2022 on The AI Alignment Forum.
    This is a discussion post for ChatGPT.
    I'll start off with some observations/implications:
    ChatGPT (davinci_003) seems a lot better/more user-friendly than davinci_002 was.
    The easy-to-use API probably means that many more people will interact with it.
    ChatGPT is a pretty good copy-editor (I haven't tried davinci_002 for this purpose). I will absolutely use this to edit/draft my texts.
    ChatGPT probably makes homework essays largely obsolete (but maybe they were already obsolete before?).
    GPT-4 will probably be insane.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min