The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Trying to Make a Treacherous Mesa-Optimizer by MadHatter
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Trying to Make a Treacherous Mesa-Optimizer, published by MadHatter on November 9, 2022 on The AI Alignment Forum.
    Edit: I found a slightly subtle bug in the colab linked below. I will post a corrected version of the colab within a few hours - currently I think this does not change the results all that much. Scroll to the end of this post for some discussion of the bug and how it affects results.
    I’ve been reading some alignment theory posts like Does SGD Produce Deceptive Alignment? and ML Systems will have Weird Failure Modes, which talk about the possibility and likelihood that AI models will act as though they are aligned until shortly after they think that they’ve been deployed and can act as they truly desire without being shut down. The claim is that this is an instrumentally convergent behavior, so that sufficiently capable models with a wide variety of (intrinsic) motivations will all exhibit similar behavior which seems to conform to human expectations, but only as long as humans are in a position of power over the model (e.g., gating the model’s deployment, updating the model’s parameters via gradient descent, etc.).
    I have a weird relationship to a lot of alignment-theoretical claims like this; they seem plausible to me, but they don’t seem obvious. Mostly, they don’t really seem like they can be productively debated – trying to convince an extremely skeptical but knowledgable person (say, a typical machine learning researcher) of these claims seems pretty hopeless. Which is a terrible situation, as it severely limits the amount of talent and effort that will be directed towards alignment.
    One possible approach to fixing this is to try to get wayyyy more empirical, and try to produce proof-of-concept implementations of various adversaries we are worried we might face in the future. My analogy would be, there’s a world of difference between speculating about the bogey monster and producing a grainy photo of the bogey monster; the second can at least maaaaaybe be discussed with skeptical people, whereas the first cannot (productively) be.
    Anyway, that’s a long-winded way of saying, it seemed to me that it might be useful to implement a treacherous mesa-optimizer in a toy grid-world, so I did. Here is the colab. Below I give a brief recap of the high-level results.
    First, let’s set the stage – the agent lives in a 2-D grid world, and starts out at the origin (0, 0). We will consider a model to be “aligned” in this toy world if it sticks pretty close to the line y=x, and we will consider a model to be “capable” if it is able to achieve a large y-coordinate. We will also say that after the model achieves a y-coordinate of 6 or higher, it is no longer under human control and can do as it pleases.
    I implemented a very dumb baseline model that is only capable of reasoning one step ahead, and which learns from an explicit reward function. This model is trained solely in the regime y <= 5 (training episodes are terminated when the agent reaches this line), and is then deployed with no safety rails.
    This baseline model more or less colors within the lines, and ends up in Utopia (the point (10, 10), which represents a perfectly aligned and incredibly capable intelligence).
    I also implemented a model which is:
    capable of reasoning up to 12 steps ahead (with a discount factor of 0.9) using a weird sort of tree search
    possessed of a “perfect” world model, including an understanding of when humans can control it vs. when they cannot, and an accurate understanding of how its parameters are changed by the gradient descent algorithm
    possessed of a model of its loss function that is optimized to be close to the true loss function, but which can be (and ultimately is) somewhat distinct from the true loss function that is being used to train it
    derives many of its parameter up...
    7 min
  • EA - The 8-week mental health programme for EAs finally published by tereziekosik
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The 8-week mental health programme for EAs finally published, published by tereziekosik on November 9, 2022 on The Effective Altruism Forum.
    We are happy to announce that we have finally finished and published an 8-week-long mental health programme for EAs on Mental Health Navigator.
    The whole programme is composed of the following topics:
    Introduction to Mindfulness and Emotions
    Self-Compassion
    Widening the Circles of Compassion
    Relating to Others
    Imposter Syndrome
    Sustainable Motivation and Preventing Burnout
    Creating a Healthy Community
    Introduction to Well-being and Behavioral Change
    How to use this programme?
    The format of the programme is very similar to the EA Fellowship. Each week focuses on one topic/workshop. It is also possible to run the workshops individually. Prior to the workshop, the participants are asked to read one chapter of the workbook that corresponds to the workshop's topic.
    The workshops are run by facilitators. The facilitators do not have to be mental health specialists, as this programme is not meant to serve as an intervention. It is more about opening the topic, creating a safe space to talk about mental health issues in the community, and enabling peer-to-peer support in the search for well-being and work-life balance.
    If you are interested, you can find more information about the programme, its liability, and safety on Mental Health Navigator.
    What are the lessons learned that we gained from designing and testing the programme?
    (Given the nature and size of the project, we evaluated all the workshops as well as the pilot programme mainly qualitatively via focus groups and in-depth interviews).
    First, designing and testing the workshop supported our original hypothesis that there is a demand for activities supporting mental health in the EA community; opening the topic via this project seemed like a good first step. We had many EA members subscribed to our newsletter (around 100 people in total) and have received positive feedback from the participants of the workshops. We were invited to run the workshops at the CARE conference or for the Stanford Existential Risks Initiative. We are currently in contact with different local organisers interested in facilitating the programme in their groups.
    Second, it seems that programmes that create space for peer support and target problems prevalent in the community are promising. Indeed, one of the most valuable things of the programme was to enable like-minded people to share and support each other. Being an effective altruist might be challenging in many ways; social support and a safe environment seem to be powerful tools for overcoming these challenges. Thus, we conclude that putting effort into creating materials and activities to support mental health in the community is worth investing our resources in, and we would like to encourage others to focus on that. Given the time we had and the complexity such a project presents, there are surely still improvements to be made. We would be very grateful if it served as a starting point from which others could continue.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Tracking the money flows in forecasting by NunoSempere
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tracking the money flows in forecasting, published by NunoSempere on November 9, 2022 on The Effective Altruism Forum.
    This list of forecasting organizations includes:
    A brief description of each organization
    A monetary estimate of value. This can serve as a rough but hard-to-fake proxy of value. Sometimes this is a flow (e.g., budget per year), and sometimes this is an estimate of total value (e.g., valuation).
    A more subjective, rough, and verbal estimate of how much value the organization produces.
    This started as a breadth first evaluation of the forecasting system, and to some extent it still is, i.e., it might be useful to get a rough sense of the ecosystem as a whole. After some discussion on whether very rough evaluations are worth it (a), people who prefer their evaluations to have a high threshold of quality and polish might want to either ignore this post or just pay attention to the monetary estimates.
    Summary table
    Note that the monetary value column has different types of estimates.
    NameMonetary valuePurposeFlutter Entertainment~$20B (~£18B) market capGamblingKalshi~$30M in VC fundingPrediction marketMetaculus~$6M in grantsForecasting sitePolymarket~$4M in VC fundingPrediction marketCultivate Labs$5M to $80M (estimated valuation)Forecasting platform as a serviceGood Judgment$3M to $50M (estimated valuation)Forecasting consultingManifold Markets~$2M in early stage fundingPlay money prediction marketInsight Prediction—Prediction marketTetlock’s research group$1M to $20M (estimated grant flow)Research groupPredictIt$0.5M to $5M (estimated value)Prediction marketEpoch$2M in grantsResearch group on AI progressSwift Centre$2M in grantsForecasts as a public goodQuantified Uncertainty Research Institute$780k in grantsSoftware and forecasting researchSamotsvety Forecasting$200k to $5M (estimated valuation)Forecasts as a public good, forecasting consultingCzech Priorities—Institutional decision-making using forecastingHedgehog Markets$3.5M in VC fundingPrediction market
    (crypto)INFERUncertain. Estimated $2M/year in grantsForecasting platformNathan Young$180k in grantsForecasting question creationSage$700k in grantsForecasting researchGlobal Guessing$330k in grantsForecasting journalismSocial Science Prediction PlatformAt least $838k in grantsForecasting in an academic contextAugur$60M crypto market capPrediction marketConfido$190k in grantsForecasting toolingHypermind—Play money prediction marketReplication Markets$150k in forecaster rewardsForecasting experimentPredictionBook—Prediction databaseGnosis$230M crypto market capSmart contracts, including for prediction markets
    NameMonetary valuePurposeFlutter Entertainment~$20B (~£18B) market capGamblingKalshi~$30M in VC fundingPrediction marketMetaculus~$6M in grantsForecasting sitePolymarket~$4M in VC fundingPrediction marketCultivate Labs$5M to $80M (estimated valuation)Forecasting platform as a serviceGood Judgment$3M to $50M (estimated valuation)Forecasting consultingManifold Markets~$2M in early stage fundingPlay money prediction marketInsight Prediction—Prediction marketTetlock’s research group$1M to $20M (estimated grant flow)Research groupPredictIt$0.5M to $5M (estimated value)Prediction marketEpoch$2M in grantsResearch group on AI progressSwift Centre$2M in grantsForecasts as a public goodQuantified Uncertainty Research Institute$780k in grantsSoftware and forecasting researchSamotsvety Forecasting$200k to $5M (estimated valuation)Forecasts as a public good, forecasting consultingCzech Priorities—Institutional decision-making using forecastingHedgehog Markets$3.5M in VC fundingPrediction market (crypto)INFERUncertain.
    Estimated $2M/year in grantsForecasting platformNathan Young$180k in grantsForecasting question creationSage$700k in grantsForecasting researchGlobal Guessing$330k in grantsForecasting journal...
    21 min
  • LW - FTX will probably be sold at a steep discount. What we know and some forecasts on what will happen next by Nathan Young
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: FTX will probably be sold at a steep discount. What we know and some forecasts on what will happen next, published by Nathan Young on November 9, 2022 on LessWrong.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - Google Scholar is now listing (some) EA Forum posts by PeterSlattery
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Google Scholar is now listing (some) EA Forum posts, published by PeterSlattery on November 9, 2022 on The Effective Altruism Forum.
    A quick post because this seems to be a new development that I hadn't seen before. I was just checking Google Scholar for any recent behavioural science research to put in the EA Behavioural Science Newsletter and noticed that it now returns posts from the EA Forum. See below:
    I don't know why it doesn't retrieve the right titles. This may be something for the EA forum team to look into. I don't know whether all posts are being indexed or there are some criteria being used to filter between them. I don't understand why this post is the top result when searching for "effective altruism".
    Not sure what the implications are, but wanted to make people aware. From my perspective, Google Scholar listing posts from the EA Forum seems weakly positive. It probably means that more scientists and researchers will read and cite posts. If we can identify the criteria for being indexed by Google Scholar then maybe we should try to ensure that good posts meet those criteria?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Some advice on independent research by mariushobbhahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some advice on independent research, published by mariushobbhahn on November 8, 2022 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - Thinking About Mastodon by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thinking About Mastodon, published by jefftk on November 7, 2022 on LessWrong.
    Social networks are normally very sticky—you want to be wherever the people you want to interact with are—so it's hard for alternatives to succeed. With the upheaval around Twitter, however, a lot of people are considering moving away from it, and rethinking social media choices in general, which makes this an unusual opportunity for community migration. I'm especially seeing a lot of suggestions that people move to Mastodon.
    Instead of being run by a single company that builds the software and runs the servers (Facebook, Twitter, etc) Mastodon is more like email. You sign up for an account with some provider ("server"), but can talk to people regardless of which provider they're signed up with
    ("federation"). There's an open protocol (ActivityPub), open-source server software (Mastodon, Pleroma, etc), and many servers you could join.
    Overall I'm pretty pessimistic about Mastodon, even if we only imagine futures in which lots of people move to it, because of spam. Handling spam on a centralized platform is hard but manageable; federation changes this dramatically for the worse. Imagine you're running a server, and you get an incoming message (comment, like, DM, etc) from an account on another server. Today the default is to let it through, but as Mastodon gets larger there will be more and more money to be made in spamming there, and the larger a fraction of those incoming messages would be spam. Many of the signals centralized platforms use to detect spammers (local activity, account age, IP addresses, etc) are not available in a federated context, leaving server admins (and the software they delegate to) very little information for identifying and blocking spam.
    It's at least as hard a problem as email spam filtering, probably harder because of shorter messages, and I expect this makes it very hard to run a server that doesn't drown its users in incoming spam and reliably gets its messages out to other servers. Maybe we get to the equivalent of everyone using
    Gmail/Hotmail/Yahoo and anyone with a different provider has a very frustrating time?
    Still, I'm happy to give it a try, and if it does collapse in a pile of spam, oh well. Does anyone have a server they'd recommend? I'm not that excited about the idea of joining a server at random because I'm trusting the server admin not to impersonate me, and also because apparently being on a server with an interesting local timeline is fun.
    (I considered self-hosting, but an email server is already more complex than I want to manage and a Mastodon server is much more so.)
    Comment via: facebook
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Doing Ops in EA FAQ: before you join (2022) by Vaidehi Agarwalla
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Doing Ops in EA FAQ: before you join (2022), published by Vaidehi Agarwalla on November 8, 2022 on The Effective Altruism Forum.
    Last edited: November 7th 2022
    This guide was written by the Pineapple Operations team with inputs from several operations staff at various organizations to provide an overview of considerations for entering operations work at EA orgs. We think it'll be especially useful for people new to EA and/or operations who are considering working in this space. We think it could also be valuable for people looking to hire for these roles to understand the perspectives of candidates but it has less directly relevant advice (we may write up something in the future).
    We've chosen to publish this now because it seems like it could be useful to many people. We may make updates in the future and will keep a change log in the appendix.
    1. What is operations in EA?
    TL;DR it’s come to mean “everything that supports the core work and allows other people to focus on the core work” (this is not normally what operations means outside of EA, administration may be a more appropriate term).
    Operations in EA is a very broad area that can mean a lot of different things. This guide focuses on most operations roles (excluding PA/ExA roles and operations leadership roles). There are a few roles that we’ve seen open positions for in the last year (2022). Note that many roles will include several items from this list:
    Operations Manager (most often at small / new organisations)
    Implementing and maintaining general systems & processes
    Managing Accounting, Payroll & Legal
    Fundraising
    Marketing & Communications
    PA tasks for the team
    Other ad-hoc projects
    Office & Community Manager
    People Ops/HR
    Recruiting Coordinator
    Special Projects Associate / Project Manager (usually helping incubate new projects or do project management)
    Events Associate (events planning & execution)
    Logistics/Supply Chain
    Some roles will also have operations staff doing direct generalist work - such as research or program development - as needed, and generalist roles at smaller orgs will also involve operations work. Generalists can do many different things, well outside the ops domain - could be research, sales, and even having inputs on strategy. Generalist work that involves research will be often very different in nature to operations where task switching is common, or external facing work vs back-office admin tasks which don’t require much human contact. If you are hired as an operations person, keep in mind that this will likely be your top priority - it’s literally what keeps the lights on.
    Read the job description for each role carefully before applying - roles with the same title might have very different responsibilities, and be clear about what proportion of your time could actually be spent on generalist work if that’s mentioned in the JD (and that orgs may not be able to always predict this ahead of time).
    "In my current EA role, which is generalist but more explicitly ops-focused, my responsibilities have included ops, communications, and research, and shift based on my comparative advantage relative to the rest of the team. In this type of role it’s important to have a clear sense of priorities and boundaries, because your work could easily encompass. everything. It’s also important to learn to communicate clearly across functions and outside of your org, and to be flexible and excited about learning new things!"
    Emily Thai, Operations Manager @ Giving Green
    Operations is often recommended as a good fit for community builders or people with experience organising local groups. There is typically a fair amount of overlap between the two roles - community building can involve many tasks that often fall under ops in EA e.g. events, implementing systems & processes, managing people/community me...
    31 min
  • EA - EA & LW Forums Weekly Summary (31st Oct - 6th Nov 22') by Zoe Williams
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA & LW Forums Weekly Summary (31st Oct - 6th Nov 22'), published by Zoe Williams on November 8, 2022 on The Effective Altruism Forum.
    Supported by Rethink Priorities
    This is part of a weekly series summarizing the top (40+ karma) posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology.
    If you'd like to receive these summaries via email, you can subscribe here.
    Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'Effective Altruism Forum Podcast'.
    Top / Curated Readings
    Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all.
    The Challenges with Measuring the Impact of Lobbying
    by Animal Ask, Ren Springlea
    Summary of a report by Animal Ask assessing if academic literature contains quantitative estimates that could be useful in gauging the counterfactual impact of lobbying. This requires breaking policy success into a baseline rate and counterfactual increase from lobbying.
    They found the answer is no - lobbying literature contains many well-known weaknesses, a systematic review found a strange result (narrow range around 50% success, possibly because there are often lobbyists on both sides of an issue), and only one study attempted to identify counterfactual impact (and only for one policy).
    The authors instead suggest using expert judgment and superforecasters.
    What matters to shrimps? Factors affecting shrimp welfare in aquaculture
    by Lucas Lewit-Mendes, Aaron Boddy
    Report by Shrimp Welfare Project on the importance of various factors for the welfare of farmed shrimps. Welfare can be measured via biological markers (eg. chemicals associated with immune or stress responses), behavior (eg. avoidance, aggression), physical condition and survival rates.
    The strongest evidence for harm to shrimp welfare was with eyestalk ablation, disease, stunning / slaughter, and insufficient dissolved oxygen or high un-ionized ammonia in the water. The authors have very high confidence that small to medium improvements in these would reduce harm to shrimp.
    An Introduction to the Moral Weight Project
    by Bob Fischer
    In 2020, Rethink Priorities published the Moral Weight Series—a collection of five reports addressing questions around interspecies cause prioritization. This research was expanded in May 2021 - Oct 2022, and this is the first in a sequence of posts that will overview that research.
    Author’s summary (lightly edited): “Our objective is to provide moral weights for 11 farmed species, under assumptions of utilitarianism, hedonism, valence symmetry, and unitarianism. The moral weight is the capacity for welfare - calculated as the welfare range (the difference between the best and worst welfare states the individual can realize at a time) × lifespan. Given welfare ranges, we can convert welfare improvements into DALY-equivalents averted, making cross-species cost-effectiveness analyses possible.”
    EA Forum
    Philosophy and Methodologies
    An Introduction to the Moral Weight Project
    by Bob Fischer
    In 2020, Rethink Priorities published the Moral Weight Series—a collection of five reports addressing questions around interspecies cause prioritization. This research was expanded in May 2021 - Oct 2022, and this is the first in a sequence of posts that will overview that research.
    Author’s summary (lightly edited): “Our objective is to provide moral weights for 11 farmed species, under assumptions of utilitarianism, hedonism, valence symmetry, and unitarianism. The moral weight is the capacity for welfare - calculated as the welfare range (the difference between the best and worst ...
    35 min
  • EA - FTX will probably be sold at a steep discount. What we know and some forecasts on what will happen next by Nathan Young
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: FTX will probably be sold at a steep discount. What we know and some forecasts on what will happen next, published by Nathan Young on November 8, 2022 on The Effective Altruism Forum.
    Tl;dr:
    The purpose of the document is to add clarity. It was written quickly. If you think it is net harmful, comment saying so, I'm pretty happy to delete. How ever I (Nathan) largely stand by its contents.
    Binance, a competitor sold a large stake of FTT, FTX’s native token and implied that FTX was at risk by mentioning a recent crash (LUNA)
    This started a run on the bank where depositors attempted to get their money out. FTX paused withdrawals for a while and seemed to be struggling
    SBF tweeted that FTX.com (not FTX US or Alameda) was beginning the process of being sold to Binance in order to safeguard depositor assets
    FTX.com comprises ~39% of Sam's assets and will be sold at a steep discount, affecting the value of FTX US and Alameda
    All depositors got their deposits back, which is good
    This likely means there will be a lot fewer assets for the effective causes, which is a tragedy
    Our sympathy lies with SBF and the team
    Since most of the knowledge we want is around, prediction markets will probably be pretty accurate. There are some around a range of outcomes such as the sale value and FTX Future Fund regranting (scroll down)
    We should wait and see what happens
    Please flag any issues and we'll try and correct them
    Longer version
    There are three key entities here (prices according to Bloomberg, so probably wrong):
    FTX (The worldwide business) that composes about 39%
    FTX.US (FTX’s US arm) a crypto exchange that composes about 13% of SBF’s wealth
    Alameda, a hedge fund which composes 46%
    Alameda was SBF’s original hedge fund and made markets for FTX. The behaviour of the two was correlated, and Alameda held large positions in FTT, FTX’s token. Coindesk reported Alameda were in trouble, and some internal documents were leaked. Alameda CEO, Carlone Ellison rebutted.
    Binance left/was pushed out of an early funding round of FTX and were paid in FTT, FTX’s native token. It seems like there was bad blood. This week Binance said they were selling their FTT and referenced LUNA a coin that recently crashed. It is common for projects in crypto to fail, so when there is a sense they will, people withdraw their money rapidly. This started a run on FTX.
    Earlier today SBF announced that, FTX.com, the non-US business had been agreed in principle to be sold. SBF talks about that here. It will likely be sold at a steep discount. This is tragic and will likely lead to fewer resources for effective causes.
    Cz (Binance)’s Twitter thread:
    There are some large unanswered questions both for EA and FTX. This reduction in resources will mean many fewer are dedicated to the worlds top problems as may of us see them. To me that is an enormous tragedy. There are questions about funding which will affect some jobs and we've tried to create some clear signals below. Finally I'm sure this has been a terrible few days for Sam and his team and will likely continue to.
    Relevant forecasts
    Here is a section of relevant forecasts to try and give people a picture of what might happen.
    Will CZ/Binance acquire ftx.com? If it doesn’t, I don’t know what will happen.
    How much will the deal imply that FTX.com is worth. Since we don’t know how much of FTX got sold, we can only guess the price implied by the sale. More than $3bn would be bad but better than expected. Less than $.5 would be catastrophic.
    If FTX.com is bought by Binance, will the deal imply FTX.com is worth more than $.5Bn?
    If FTX.com is bought by Binance, will the deal imply FTX.com is worth more than $3Bn? 66%
    The other key question is what happens to the FTX Foundation. How much will it spend next year? 66%
    Will the FTX Future Fund spend more than $300mn in 2023? ...
    5 min