The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Co-found an incubator for independent AI Safety researchers by Alexandra Bos
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Co-found an incubator for independent AI Safety researchers, published by Alexandra Bos on June 2, 2023 on LessWrong.
    Full-time, remote
    APPLY HERE
    Deadline: Thursday, June 8th (in your timezone)
    If your ideal job would be leading an impact-driven organization, being your own boss and pushing for a safer future with AI, you might be a great fit for co-founding Catalyze Impact!
    Below, you will find out more about Catalyze’s mission and focus, why co-founding this org would be high-impact, how to tell if you’re a good fit, and how to apply.
    In short, Catalyze will 1) help people become independent technical AI Safety researchers, and 2) deliver key support to independent AI Safety researchers so they can do their best work.
    Would highly appreciate it if you could share this message with people who you think might potentially be interested in this role and upvote this post if you think more people should see it so we can find the best potential co-founder for this.
    You can ask questions, register interest to potentially fund us, work with us, make use of our services in the future and share information here.
    Why support independent AI Safety researchers?
    Lots of people want to do AI Safety (AIS) research and are trying to get in a position where they can, yet only around 100-300 people worldwide are actually doing research in this crucial area. Why? Because there are almost no AIS researcher jobs available due to AIS research organizations facing difficult constraints to scaling up. Luckily there is another way to grow the research field: having more people do independent research (where a self-employed individual gets a grant, usually from a fund).
    There is, however, a key problem: becoming and being a good independent AIS researcher is currently very difficult. It requires a lot of qualities which have nothing to do with being able to do good research: you have to be proactive, pragmatic, social, good enough at fundraising, very good at self-management and willing to take major career risks. Catalyze Impact will take away a large part of the difficulties that come with being an independent researcher, thereby making it a suitable option for more people so they are empowered to do good AIS research.
    How will we help?
    This is the current design of the pilot - but you will help shape this further!
    1. Fundraising support
    > help promising individuals get funded to do research
    2. Peer support networks & mentor-matching
    > get feedback, receive mentorship, find collaborators, brainstorm and stay motivated rather than falling into isolation
    3. Accountability and coaching
    > have structure, stay motivated and productive
    4. Fiscal sponsorship: hiring funded independent researchers as ‘employees’
    > take away operational tasks which distract from research & help them build better career capital through institutional affiliation
    In what ways would this be impactful?
    Alleviating a bottleneck for scaling the AIS research field by making independent research suitable for more people: it seems that we need a lot more people to be working on solving alignment. However, talented individuals who have invested in upskilling themselves to go do AIS research (e.g. SERI MATS graduates) are largely unable to secure research positions. This is oftentimes not because they are not capable enough of doing the research, but because there are simply too few positions available (see footnote). Because of this, many of these talented individuals are left with a few sub-optimal options:
    1) try to do research/a PhD in a different academic field in hopes that it will make them a better AIS researcher in the future
    2) take a job working on AI capabilities (!)
    3) try to become an independent AIS researcher
    For many people, independent research (i.e. without this incubator) is not a good & viable option becau...
    11 min
  • AF - [Replication] Conjecture's Sparse Coding in Toy Models by Hoagy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Replication] Conjecture's Sparse Coding in Toy Models, published by Hoagy on June 2, 2023 on The AI Alignment Forum.
    Summary
    In the post Taking features out of superposition with sparse autoencoders, Lee Sharkey, Beren Millidge and Dan Braun (formerly at Conjecture) showed a potential technique for removing superposition from neurons using sparse coding. The original post shows the technique working on simulated data, but struggling on real models, while a recent update shows promise, at least for extremely small models.
    We've now replicated the toy-model section of this post and are sharing the code on github so that others can test and extend it, as the original code is proprietary.
    Additional replications have also been done by Trenton Bricken at Anthropic and most recently Adam Shai.
    Thanks to Lee Sharkey for answering questions and Pierre Peigne for some of the data-generating code.
    Future Work
    We hope to expand on this in the coming days/weeks by:
    trying to replicate the results on small transformers as reported in the recent update post.
    using automated interpretability to test whether the directions found by sparse coding are in fact more cleanly interpretable than, for example, the neuron basis.
    trying to get this technique working on larger models.
    If you're interested in working on similar things, let us know! We'll be working on these directions in the lead-up to SERI MATS.
    Comparisons
    Mean Maximum Cosine Similarity (MMCS) with true features
    Number of dead neurons (capped at 100)
    MMCS with larger dicts with same L1 penalty
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Lincoln Quirk has joined the EV UK board by Howie Lempel
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lincoln Quirk has joined the EV UK board, published by Howie Lempel on June 2, 2023 on The Effective Altruism Forum.
    I’m excited to share that Lincoln Quirk has joined the board of Effective Ventures Foundation (UK). This follows the addition of Zach Robinson and Eli Rose to the EV US board about two months ago.
    Lincoln is a co-founder and Head of Product at Wave, a technology company that aims to make finance more accessible in Africa through digital infrastructure. Wave spun off from SendWave, a remittance platform which Lincoln also cofounded and which was acquired in 2021. He has maintained a deep interest in effective altruism and been a part of the EA community for over a decade.
    In his own words, "I'm excited to join the EV UK board. I've been trying to help the world and have called myself part of the EA community for 10+ years; EV is playing one of the most important roles in this community and correspondingly holds a lot of responsibility. I'm looking forward to helping figure out the best ways EV can contribute to making the world a better place through enabling EA community and EA projects.”
    The EV UK trustees and I are excited to have Lincoln join and look forward to working with him. Lincoln impressed us during the interview process with his strategic insight and dedication to the role. I also think his experience founding and growing Wave and Sendwave will be a particularly useful perspective to add to the board.
    We are continuing to look for candidates to add to the boards of both EV UK and EV US, especially candidates with diverse backgrounds and experiences, and who have experience in accounting, law, finance, risk management or other management at large organizations. We recruited Lincoln via directly reaching out to him, and plan to continue to source candidates this way and via our open call. If you are interested or know of a great candidate, the linked forum post includes instructions for applying or nominating someone. Applications and nominations will be accepted until June 4th.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • LW - Dreams of "Mathopedia" by NicholasKross
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dreams of "Mathopedia", published by NicholasKross on June 2, 2023 on LessWrong.
    In high school, I held up a pretty-decent-level calculus class because I was confused about something. Specifically that thing where you rotate a curve around some spatial axis (like sculpting pottery) and calculate the volume of the resulting enclosed 3D shape.
    I kept being confused, and the teacher (who was super nice and knowledgeable and good-at-teaching [1])... her explanations kept not-getting-through to my brain.
    "How do we know y=x^2's 'vase' volume? Wouldn't it be infinite since it's open at the top?" --> [explanation involving rotating around the Z-axis so it's like y=sqrt(x), or something idk] --> "But that doesn't seem very principled! What's the rule law for how to turn the shapes?"
    Then some other student in class said like 1-2 sentences, but the only key info I needed was the phrase "domain and range".
    Then I was like "oh, I get it completely now, thanks!" And then the class laughed/sighed/was somewhat exasperated.
    I developed a maybe-seemingly-trivial hypothesis, that if someone receives explanation E_1 of a concept C, and they're paying attention, and they still don't intuitively grok C, then they need at least one more different explanation E_2.
    An idea immediately came to mind: Could you teach someone any advanced math concept, by throwing every explanation at once at them? Could this work on anybody without more-straightforward mental disabilities? [2]
    So I've long had a back-of-my-mind idea, which I labeled "Mathopedia". This is not to be confused with any other existing math website that someone would find useful, including MathWorld, Khan Academy, MathOverflow, Wikipedia, Mathematics Stack Exchange, YouTube, Arbital, the OEIS, Metamath, Tricki, ProofWiki, nLab, Hypertextbook, and... uh... at least one literally called Mathopedia. Might need a new name then...
    The idea was simple: a math-learning tool that explains advanced uni/graduate/research-level mathematical concepts by gathering a huge number of explanations per concept, and putting them together in an extremely-multimodal (bordering on seizure-inducing) format.
    This led me to a few more trains of thought:
    A core "Mathopedia" website, a wiki where each concept gets a page. A page's subsections would go from more-intuitive/motivational/extensional-definition/multimedia/seizure explanations, to the more technical ones, ending with a ton of examples. In my head, this could involve a strong community of contributors.
    A few desktop-software ideas that, if useful, seem (to me) too-powerful to give to non-alignment-researchers. I am probably wildly overestimating the utility of relatively-simple non-ML-based desktop software that hasn't already been invented. Still, being careful.
    [Reading the Arbital Postmortem while shaking my head so other people know that I understand what went wrong there and how my "Mathopedia"" would do better.]
    [Reading Paul Lockhart while alternately nodding and shaking my head so I agree with his emphasis on open-ended learning but dislike how mathematics is taught in US K-12 schools (as elaborated in Lockhart's colorful examples/analogies).]
    [Just cross-referencing 3 textbooks, Googling, and asking Discord, like every other mathematician since the days of Pythagoras. If the resources work for everyone else, shouldn't they work for me?]
    I'm still not sure of whether a real "Mathopedia" is worth the effort to build, in some kinds of short-AI-timelines. (Here I'm wanting such a website/tool to mainly be of use for technical AI alignment research, though if it worked it would aid many causes). Then again, when some people entering the field still lack linear algebra on arrival, maybe it is worth it.
    Despite the clear emotional/self-serving/imposter-syndrome biases at play, I'm still legitimat...
    5 min
  • EA - Things I Learned by Spending Five Thousand Hours In Non-EA Charities by jenn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Things I Learned by Spending Five Thousand Hours In Non-EA Charities, published by jenn on June 2, 2023 on The Effective Altruism Forum.
    From late 2020 to last month, I worked at grassroots-level non-profits in operational roles. Over that time, I’ve seen surprisingly effective deployments of strategies that were counter-intuitive to my EA and rationalist sensibilities.
    I spent 6 months being the on-shift operations manager at one of the five largest food banks in Toronto (~50 staff/volunteers), and 2 years doing logistics work at Samaritans (fake name), a long-lived charity that was so multi-armed that it was basically operating as a supplementary social services department for the city it was in(~200 staff and 200 volunteers). Both orgs were well-run, though both dealt with the traditional non-profit double whammy of being underfunded and understaffed.
    Neither place was super open to many EA concepts (explicit cost-benefit analyses, the ITN framework, geographic impartiality, the general sense that talent was the constraining factor instead of money, etc). Samaritans in particular is a spectacular non-profit, despite(?) having basically anti-EA philosophies, such as:
    Being very localist; Samaritans was established to help residents of the city it was founded in, and now very specialized in doing that.
    Adherence to faith; the philosophy of The Catholic Worker Movement continues to inform the operating choices of Samaritans to this day.
    A big streak of techno-pessimism; technology is first and foremost seen as a source of exploitation and alienation, and adopted only with great reluctance when necessary.
    Not treating money as fungible. The majority of funding came from grants or donations tied to specific projects or outcomes. (This is a system that the vast majority of nonprofits operate in.)
    Once early on I gently pushed them towards applying to some EA grants for some of their more EA-aligned work, and they were immediately turned off by the general vibes of EA upon visiting some of its websites. I think the term “borg-like” was used.
    Over this post, I’ll be largely focusing on Samaritans as I’ve worked there longer and in a more central role, and it’s also a more interesting case study due to its stronger anti-EA sentiment.
    Things I Learned
    Long Term Reputation is Priceless
    Non-Profits Shouldn’t Be Islands
    Slack is Incredibly Powerful
    Hospitality is Pretty Important
    For each learning, I have a section for sketches for EA integration – I hesitate to call them anything as strong as recommendations, because the point is to give more concrete examples of what it could look like integrated in an EA framework, rather than saying that it’s the correct way forward.
    1. Long Term Reputation is Priceless
    Institutional trust unlocks a stupid amount of value, and you can’t buy it with money. Lots of resources (amenity rentals; the mayor’s endorsement; business services; pro-bono and monetary donations) are priced/offered based on tail risk. If you can establish that you’re not a risk by having a longstanding, unblemished reputation, costs go way down for you, and opportunities way up. This is the world that Samaritans now operate in.
    Samaritans had a much better, easier time at city hall compared to newer organizations, because of a decades-long productive relationship where we were really helpful with issues surrounding unemployment and homelessness. Permits get back to us really fast, applications get waved through with tedious steps bypassed, and fees are frequently waived. And it made sense that this was happening! Cities also deal with budget and staffing issues, why waste more time and effort than necessary on someone who you know knows the proper procedure and will ethically follow it to the letter?
    It’s not just city hall. A few years ago, a local church offered up th...
    13 min
  • EA - An Earn to Learn Pledge by Ben West
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Earn to Learn Pledge, published by Ben West on June 2, 2023 on The Effective Altruism Forum.
    tl;dr: create an equivalent of GWWC for building career capital. We've thought about this idea for ~15 minutes and are unlikely to do something ourselves, but wanted to share it because we think it might be good.
    Many people's greatest path to impact is through changing their career
    But for a lot of these people, particularly those earlier in their career, it doesn't make sense to immediately apply to impact-oriented jobs. Instead, it's better for them to build career capital at non-impact-oriented workplaces, i.e. "earning to learn"
    It would be nice if there was some equivalent of the Giving What We Can pledge for this
    It could involve something like pledging to:
    Spend at least one day per year updating your career plan with an eye towards impact
    Apply to at least x impact-oriented jobs per year, even if you expect to get rejected
    And some sort of dashboard checking people's adherence to this, and nudging them to adhere better
    Some potential benefits:
    Many people who have vague plans of "earning to learn" just end up drifting away after entering the mainstream workforce; this can help them stay engaged
    It might relieve some of the pressure around being rejected from "EA jobs" – making clear that Official Fancy EA People endorse career paths beyond "work at one of this small list of organizations" puts less pressure on people who aren't a good fit for one of those small list of organizations
    Relatedly, it gives community builders a thing to suggest to a relatively broad set of community members which is robustly good
    Next steps:
    I think the MVP here requires ~0 technology: come up with the pledge, get feedback on it, and if people are excited throw it into a Google form
    It's probably worth reading criticisms of the GWWC pledge (e.g. this) to understand some of the failure modes here and be sure you avoid those
    It also requires thinking through some of the risks, e.g. you might not want a fully public pledge since that could hurt people's job prospects
    If you are interested in taking on this project, please contact one of us and we can try to help
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • LW - Think carefully before calling RL policies "agents" by TurnTrout
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Think carefully before calling RL policies "agents", published by TurnTrout on June 2, 2023 on LessWrong.
    I think agentic systems represent most of AI extinction risk. I want to think clearly about what training procedures produce agentic systems. Unfortunately, the field of reinforcement learning has a convention of calling its trained artifacts "agents." This terminology is loaded and inappropriate for my purposes. I advocate instead calling the trained system a "policy." This name is standard, accurate, and neutral.
    Don't assume the conclusion by calling a policy an "agent"
    The real-world systems we want to think about and align are very large neural networks like GPT-4. These networks are trained and finetuned via different kinds of self-supervised and reinforcement learning.
    When a policy network is updated using a learning process, its parameters are changed via weight updates. Eventually, the process ends (assuming no online learning for simplicity). We are then left with a policy network (e.g. GPT-4). To actually use the network, we need to use some sampling procedure on its logits (e.g. top-p with a given temperature). Once we fix the policy network and sampling procedure, we get a mapping from observations (e.g. sequences of tokens like [I, love, dogs]) to probability distributions over outputs (e.g. tokens). This mapping π is the policy.
    I want to carefully consider whether a trained policy will exhibit agentic cognition of various forms, including planning, goal-directedness, and situational awareness. While considering this question, we should not start calling the trained policy an "agent"! That's like a detective randomly calling one of the suspects "criminal." I prefer just calling the trained artifact a "policy." This neutrally describes the artifact's function, without connoting agentic or dangerous cognition.
    Of course, a policy could in fact be computed using internal planning (e.g. depth-3 heuristic search) to achieve an internally represented goal (e.g. number of diamonds predicted to be present.) I think it's appropriate to call that kind of computation "agentic." But that designation is only appropriate after further information is discovered (e.g. how the policy in fact works).
    There's no deep reason why trained policies are called "agents"
    Throughout my PhD in RL theory, I accepted the idea that RL tends to create agents, and supervised learning doesn't. Well-cited papers use the term "agents", as do textbooks and Wikipedia. I also hadn't seen anyone give the pushback I give in this post.
    Question: Given a fixed architecture (e.g. a 48-layer decoder-only transformer), what kinds of learning processes are more likely to train policy networks which use internal planning?
    If you're like I was in early 2022, you might answer "RL trains agents." But why? In what ways do PPO's weight updates tend to accumulate into agentic circuitry, while unsupervised pretraining on OpenWebText does not?
    Claim: People are tempted to answer "RL" because the field adopted the "agent" terminology for reasons unrelated to the above question. Everyone keeps using the loaded terminology because no one questions it.
    Let's be clear. RL researchers did not deliberate carefully about the inductive biases of deep learning, and then decide that a certain family of algorithms was especially likely to train agentic cognition. Researchers called policies "agents" as early as 1995, before the era of deep learning (e.g. see AI: A modern approach, 1st edition).
    Does RL actually produce agents?
    Just because "agents" was chosen for reasons unrelated to agentic cognition, doesn't mean the name is inappropriate. I can think of a few pieces of evidence for RL entraining agentic cognition.
    RL methods are often used to train networks on tasks like video games and robotics. These methods ...
    9 min
  • AF - Think carefully before calling RL policies "agents" by Alex Turner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Think carefully before calling RL policies "agents", published by Alex Turner on June 2, 2023 on The AI Alignment Forum.
    I think agentic systems represent most of AI extinction risk. I want to think clearly about what training procedures produce agentic systems. Unfortunately, the field of reinforcement learning has a convention of calling its trained artifacts "agents." This terminology is loaded and inappropriate for my purposes. I advocate instead calling the trained system a "policy." This name is standard, accurate, and neutral.
    Don't assume the conclusion by calling a policy an "agent"
    The real-world systems we want to think about and align are very large neural networks like GPT-4. These networks are trained and finetuned via different kinds of self-supervised and reinforcement learning.
    When a policy network is updated using a learning process, its parameters are changed via weight updates. Eventually, the process ends (assuming no online learning for simplicity). We are then left with a policy network (e.g. GPT-4). To actually use the network, we need to use some sampling procedure on its logits (e.g. top-p with a given temperature). Once we fix the policy network and sampling procedure, we get a mapping from observations (e.g. sequences of tokens like [I, love, dogs]) to probability distributions over outputs (e.g. tokens). This mapping π is the policy.
    I want to carefully consider whether a trained policy will exhibit agentic cognition of various forms, including planning, goal-directedness, and situational awareness. While considering this question, we should not start calling the trained policy an "agent"! That's like a detective randomly calling one of the suspects "criminal." I prefer just calling the trained artifact a "policy." This neutrally describes the artifact's function, without connoting agentic or dangerous cognition.
    Of course, a policy could in fact be computed using internal planning (e.g. depth-3 heuristic search) to achieve an internally represented goal (e.g. number of diamonds predicted to be present.) I think it's appropriate to call that kind of computation "agentic." But that designation is only appropriate after further information is discovered (e.g. how the policy in fact works).
    There's no deep reason why trained policies are called "agents"
    Throughout my PhD in RL theory, I accepted the idea that RL tends to create agents, and supervised learning doesn't. Well-cited papers use the term "agents", as do textbooks and Wikipedia. I also hadn't seen anyone give the pushback I give in this post.
    Question: Given a fixed architecture (e.g. a 48-layer decoder-only transformer), what kinds of learning processes are more likely to train policy networks which use internal planning?
    If you're like I was in early 2022, you might answer "RL trains agents." But why? In what ways do PPO's weight updates tend to accumulate into agentic circuitry, while unsupervised pretraining on OpenWebText does not?
    Claim: People are tempted to answer "RL" because the field adopted the "agent" terminology for reasons unrelated to the above question. Everyone keeps using the loaded terminology because no one questions it.
    Let's be clear. RL researchers did not deliberate carefully about the inductive biases of deep learning, and then decide that a certain family of algorithms was especially likely to train agentic cognition. Researchers called policies "agents" as early as 1995, before the era of deep learning (e.g. see AI: A modern approach, 1st edition).
    Does RL actually produce agents?
    Just because "agents" was chosen for reasons unrelated to agentic cognition, doesn't mean the name is inappropriate. I can think of a few pieces of evidence for RL entraining agentic cognition.
    RL methods are often used to train networks on tasks like video games and robotics....
    9 min
  • LW - Uncertainty about the future does not imply that AGI will go well by Lauro Langosco
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Uncertainty about the future does not imply that AGI will go well, published by Lauro Langosco on June 1, 2023 on LessWrong.
    Subtitle: A partial defense of high-confidence AGI doom predictions.
    Introduction
    Consider two different kinds of accident scenarios.
    In a default-success scenario, accidents are rare. For example, modern aviation is very safe thanks to decades of engineering efforts and a safety culture (e.g. the widespread use of checklists). When something goes wrong, it is often due to multiple independent failures that combine to cause a disaster (e.g. bad weather + communication failures + pilot not following checklist correctly).
    In a default-failure scenario, accidents are the norm. For example, when I write a program to do something I haven’t done many times already, it usually fails the first time I try it. It then goes on to fail the second time and the third time as well. Here, failure on the first try is overdetermined―even if I fix the first bug, the second bug is still, independently, enough to cause the program to crash. This is typical in software engineering, and it can take many iterations and tests to move into the default-success regime.
    See also: conjuctive vs disjunctive risk scenarios.
    Default-success scenarios include most engineering tasks that we have lots of experience with and know how to do well: building bridges, building skyscrapers, etc. Default-failure scenarios, as far as I can tell, come in two kinds: scenarios in which we’re trying to do something for the first time (rocket test launches, prototypes, new technologies) and scenarios in which there is a competent adversary that is trying to break the system, as in computer security. AGI satisfies both properties.
    Predictions on AGI risk
    I often encounter an argument against AGI risk that goes something like this (using P(doom) to refer to the probability of an AGI takeover and / or human extinction due to the development of AGI):
    Alice: We seem to be on track to build an AGI smarter than humans. We don’t know how to solve the technical problem of building an AGI we can control, or the political problem of convincing people to not build AGI. Every plausible scenario I’ve ever thought or heard of leads to AGI takeover. In my estimate, P(doom) is [high number].
    Bob: While I likewise can’t point out any detailed plausible scenario in which AGI goes well, I think there’s a good chance you’re mistaken. It’s just really overconfident to estimate P(doom) above 50%. Humans are usually bad at predicting the future, especially when it comes to novel technologies like AGI. When you account for how uncertain your predictions are, your estimate should be at most [low number].”
    I'm being vague about the numbers because I've seen Bob's argument made in many different situations. In one recent conversation I witnessed, the Bob-Alice split was P(doom) 0.5% vs. ~10%, and in another discussion it was 10% vs. 90%.
    My main claim is that Alice and Bob don’t actually disagree about how uncertain or hard to predict the future is―instead, they disagree about to what degree AGI risk is default-success vs. default-failure. If AGI risk is (mostly) default-failure, then uncertainty is a reason for pessimism rather than optimism, and Alice is right to predict failure.
    In this sense I think Bob is missing the point. Instead of trying to refute Alice from general principles, Bob should instead point to concrete reasons for optimism, e.g. “for reasons A, B, and C it is likely that we can coordinate on not building AGI for the next 40 years and solve alignment in the meantime”.
    Uncertainty does not mean you should be more optimistic
    Many people are skeptical of the ‘default-failure’ frame, so I’ll give a bit more color here. I won’t go into detail about why specifically AGI risk is a default-failure scenario; y...
    13 min
  • LW - Yudkowsky vs Hanson on FOOM: Whose Predictions Were Better? by 1a3orn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Yudkowsky vs Hanson on FOOM: Whose Predictions Were Better?, published by 1a3orn on June 1, 2023 on LessWrong.
    TLDR
    Starting in 2008, Robin Hanson and Eliezer Yudkowsky debated the likelihood of FOOM: a rapid and localized increase in some AI's intelligence that occurs because an AI recursively improves itself.
    As Yudkowsky summarizes his position:
    I think that, at some point in the development of Artificial Intelligence, we are likely to see a fast, local increase in capability—“AI go FOOM.” Just to be clear on the claim, “fast” means on a timescale of weeks or hours rather than years or decades; and “FOOM” means way the hell smarter than anything else around, capable of delivering in short time periods technological advancements that would take humans decades, probably including full-scale molecular nanotechnology. (FOOM, 235)
    Over the course of this debate, both Hanson and Yudkowsky made a number of incidental predictions about things which could occur before the advent of artificial superintelligence -- or for which we could at the very least receive strong evidence before artificial superintelligence.
    On the object level, my conclusions is that when you examine these predictions, Hanson probably does a little better than Yudkowsky. Although depending on how you weigh different topics, I could see arguments from "they do about the same" to "Hanson does much better."
    On one meta level, my conclusion is that Hanson's view --- that we should try to use abstractions that have proven prior predictive power -- looks like a pretty good policy.
    On another meta level, my conclusion -- springing to a great degree from how painful seeking clear predictions in 700 pages of words has been -- is that if anyone says "I have a great track record" without pointing to specific predictions that they made, you should probably ignore them, or maybe point out their lack of epistemic virtue if you have the energy to spare for doing that kind of criticism productively.
    Intro
    There are number of difficulties involved in evaluating some public figure's track record. We want to avoid cherry-picking sets of particularly good or bad predictions. And we want to have some baseline to compare them to.
    We can mitigate both of these difficulties -- although not, alas, eliminate them -- by choosing one document to evaluate: "The Hanson-Yudkowsky Foom Debate". (All future page numbers refer to this PDF.) Note that the PDF includes the (1) debate-via-blogposts which took place on OvercomingBias, (2) an actual in-person debate that took place at Jane Street in 2011 and (3) further summary materials from Hanson (further blogposts) and Yudkowsky ("Intelligence Explosion Microeconomic"). This spans a period from 2008 to 2013.
    I do not intend this to be a complete review of everything in these arguments.
    The discussion spans the time from the big bang until hypothetical far future galactic civilizations. My review is a little more constrained: I am only going to look at predictions for which I think we've received strong evidence in the 15 or so years since the debate started.
    Note also that the context of this debate was quite different than it would be if it happened today.
    At the time of the debate, both Hanson and Yudkowsky believed that machine intelligence would be extremely important, but that the time of its arrival was uncertain. They thought that it would probably arrive this century, but neither had the very, certain short timelines which are common today.
    At this point Yudkowsky was interested in actually creating a recursively self-improving artificial intelligence, a "seed AI." For instance, in 2006 the Singularity Institute -- what MIRI was before it renamed -- had a website explicitly stating that they sought funding to create recursively self-improving AI. During the Jane Street debate Y...
    40 min