The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Why did CEA buy Wytham Abbey? by Jeroen W
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why did CEA buy Wytham Abbey?, published by Jeroen W on December 6, 2022 on The Effective Altruism Forum.
    Yesterday morning I woke up and saw this tweet by Émile Torres:
    I was shocked, angry and upset at first. Especially since it appears that the estate was for sale last year for 15 million pounds:
    I'm not a big fan of Émile's writing and how they often misrepresent the EA movement. But that's not what this question is about, because they do raise a good point here: Why did CEA buy this property? My trust in CEA has been a bit shaky lately, and this doesn't help.
    Apparently it was already mentioned in the New Yorker piece:#:~:text=Last year%2C the Centre for Effective Altruism bought Wytham Abbey%2C a palatial estate near Oxford%2C built in 1480. Money%2C which no longer seemed an object%2C was increasingly being reinvested in the community itself.
    "Last year, the Centre for Effective Altruism bought Wytham Abbey, a palatial estate near Oxford, built in 1480. Money, which no longer seemed an object, was increasingly being reinvested in the community itself."
    For some reason I glanced over it at the time, or I just didn't realize the seriousness of it.
    Upon more research, I came across this comment by Shakeel Hashim: "In April, Effective Ventures purchased Wytham Abbey and some land around it (but <1% of the 2,500 acre estate you're suggesting). Wytham is in the process of being established as a convening centre to run workshops and meetings that bring together people to think seriously about how to address important problems in the world. The vision is modelled on traditional specialist conference centres, e.g. Oberwolfach, The Rockefeller Foundation Bellagio Center or the Brocher Foundation.
    The purchase was made from a large grant made specifically for this. There was no money from FTX or affiliated individuals or organizations."
    I'm very relieved to hear money from individual donors wasn't used. And the <1% suggests 15 million pounds perhaps wasn't spent. Still, I'd love to hear and understand more about this project and why CEA thinks it's cost-effective. What is the EV calculation behind it?
    Like the New Yorker piece points out, with more funding there has been a lot of spending within the movement itself. And that's fine, great even. This way more outreach can be done and the movement can grow. But we don't want to be too self-serving, and I'm scared too much of this thinking will lead to rationalizing lavish expenses (and I'm afraid this is already happening). There needs to be more transparency behind big expenses.
    Edit to add: If this expense has been made a while back, why not announce it then?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Climate research webinar by Rethink Priorities on Tuesday, December 13 at 11 am EST by Rethink Priorities
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Climate research webinar by Rethink Priorities on Tuesday, December 13 at 11 am EST, published by Rethink Priorities on December 5, 2022 on The Effective Altruism Forum.
    Since its inception in September 2021, the Global Health and Development team at Rethink Priorities has conducted research on a variety of topics. Some examples include assessing the effectiveness of inducement prizes to spur innovation, assessing promising livelihood interventions, and estimating the potential repercussions of lead paint exposure in low- and middle-income countries (report forthcoming).
    Our team has also worked on a variety of climate reports this past year. Our resident climate expert, Senior Environmental Economist Greer Gosnell, will give a presentation regarding the research process and findings of one such report that evaluates anti-deforestation as a promising climate solution. You will be able to read the full report in a forthcoming EA Forum post, so stay tuned for updates.
    The webinar, which will take place on Tuesday, December 13, 2022, at 11 am EST / 4 pm GMT, will include a brief presentation followed by a Q&A session. If you are interested in attending, please complete this form to sign up for the event. A zoom link will be shared with registered participants the day before the webinar. Questions from all participants are very welcome; anonymous attendance is possible as well. We will record the meeting and share the recording with those who have signed up but are unable to join us live.
    We look forward to sharing some of our research with you, and hope to see many of you there!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - The EA Infrastructure Fund seems to have paused its grantmaking and approved grant payments. Why? by Markus Amalthea Magnuson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The EA Infrastructure Fund seems to have paused its grantmaking and approved grant payments. Why?, published by Markus Amalthea Magnuson on December 6, 2022 on The Effective Altruism Forum.
    Yesterday someone said the EA Infrastructure Fund had paused its grantmaking. They later added that they had people send in grant proposals and got an email back confirming this pause. Today I got confirmation from a second person who said they know people with approved grants from the EAIF that have had the payments of those grants paused. I'm trying to find any official communication on this but have not found any yet. This might be relevant though, and possibly this. Does anyone know what is going on here?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - Verification Is Not Easier Than Generation In General by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Verification Is Not Easier Than Generation In General, published by johnswentworth on December 6, 2022 on LessWrong.
    People who’ve spent a lot of time thinking about P vs NP often have the intuition that “verification is easier than generation”. It’s easier to verify a solution to some equations than to find a solution. It’s easier to verify a password than to guess it. That sort of thing. The claim that it is easier to verify solutions to such problems than to generate them is essentially the claim that P ≠ NP, a conjecture which is widely believed to be true. Thus the intuition that verification is generally easier than generation.
    The problem is, this intuition comes from thinking about problems which are in NP. NP is, roughly speaking, the class of algorithmic problems for which solutions are easy to verify. Verifying the solution to some equations is easy, so that problem is in NP.
    I think a more accurate takeaway would be that among problems in NP, verification is easier than generation. In other words, among problems for which verification is easy, verification is easier than generation. Rather a less impressive claim, when you put it like that.
    With that in mind, here is an algorithmic problem for which generation is easier than verification.
    Predicate: given a program, does it halt?
    Generation problem: generate a program which halts.
    Verification problem: given a program, verify that it halts.
    The generation problem is trivial. The verification problem is uncomputable.
    That’s it for the post, you all can argue about the application to alignment in the comment section.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Take 5: Another problem with natural abstractions is laziness. by Charlie Steiner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 5: Another problem with natural abstractions is laziness., published by Charlie Steiner on December 6, 2022 on The AI Alignment Forum.
    As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes.
    Soundtrack.
    Natural abstractions are patterns in the environment that are so convenient and so useful that most right-thinking agents will learn to take advantage of them. But what if humans and modern ML are too lazy to be right-thinking?
    One way of framing this point is in terms of gradient starvation. The reason neural networks don't explore all possible abstractions (aside from the expense) is that once they find the first way of solving a problem, they don't really have an incentive to find a second way - it doesn't give them a higher score, so they don't. When gradient starvation is strong, it means the loss landscape has a lot of local minima that the agent can roll into, that aren't easily connected to the global minimum, and so what abstractions the network ends up using will depend strongly on the initial conditions.
    Regularization and exploration can help ameliorate this problem, but often come with catastrophic forgetting - if a neural net finds a strictly better way to solve the problem it's faced with, it might forget all about the previous way When we imagine a right-thinking agent that learns natural abstractions, we often imagine something that's intrinsically motivated to learn lots of different ways of solving a problem, and that doesn't erase its memory of interesting methods just because they're not on the Pareto frontier.
    So that's what I mean by "lazy"/"not lazy", here. Neural networks, or humans, are lazy if they're parochial in solution-space, doing local search in a way that sees them get stuck in what a less-lazy optimizer might consider to be ruts.
    It's important to note that laziness is not an unambiguously bad property. First, it's usually more efficient. Second, maybe we don't want our neural net to actually search through the weird and adversarial parts of parameter-space, and local search prevents it from doing so. Alex Turner et al. have recently been making arguments like this fairly forcefully. Still, we don't want maximal laziness, especially not if we want to find natural abstractions like the various meanings of "human values."
    I might be attacking a strawman or a bailey here, I'm not totally sure. I've been using "natural abstraction" here as if it just means an abstraction that would be useful for a wide variety of agents to have in their toolbox. But we might also use "natural abstractions" to denote the vital abstractions, those that aren't merely nice to have, but that you literally can't complete certain tasks without using. In that second sense, neural networks are always highly incentivized to learn relevant natural abstractions, and you can easily tell when they do so by measuring their loss.
    But as per yesterday, there are often multiple similarly-powerful ways to model the world, in particular when modeling humans and human values. There might be hard core vital abstractions for various human-interaction tasks, but I suspect they're abstractions like "discrete object," not anything nearly so far into the leaves of the tree as "human values." And when I see informal speculation about natural abstractions it usually strikes me as thinking about the less strict "useful for most agents" abstractions.
    Ultimately, I expect laziness to cause both artificial neural nets and humans to miss out on some sizeable fraction of abstractions that most agents would find useful. What to do? There are options:
    Build an AI that isn't lazy. But laziness is useful, and anyhow maybe we don't want an AI to explore all the extrema. So build an AI t...
    5 min
  • LW - Steering Behaviour: Testing for (Non-)Myopia in Language Models by Evan R. Murphy
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Steering Behaviour: Testing for (Non-)Myopia in Language Models, published by Evan R. Murphy on December 5, 2022 on LessWrong.
    Both authors contributed equally to this post and also to work done so far on the experiments it presents.
    Acknowledgments: Thanks to the following people for insightful conversations which helped us improve this post: Ian McKenzie, Aidan O’Gara, Andrew McKnight and Evan Hubinger. One of the authors (Evan R. Murphy) was also supported by a grant from the Future Fund regranting program while working on this project.
    Summary
    Myopia is a theorised property of AI systems relating to their planning horizon capabilities. As has been recently discussed, myopia seems like an important property for AI safety, because non-myopia is likely a necessary precursor to highly risky emergent properties like deceptive alignment. We expect non-myopia in large language models (LLMs) receiving RLHF or similar fine-tuning because they are trained using multi-token completions rather than just immediate next token predictions.
    We aren't aware of any previous public experiments testing specifically for myopia or non-myopia in machine learning models. We ran an initial experiment "Testing for steering behaviour in fine-tuned LLMs" which demonstrated noticeable ‘steering’ behaviour away from toxic content in the InstructGPT fine-tuned LLMs. We share key results from this experiment along with the full dataset of prompts we used in it. We also describe a follow-up experiment we're currently working on to determine the extent to which the steering we observed in the initial experiment is non-myopic.
    Finally, we invite suggestions for future (non-)myopia experiments to run and share a few ideas of our own.
    Context and motivation
    What is myopia?
    Myopia is a theorised property of some AI systems that has been discussed a fair amount on these forums. Rather than try to reinvent the wheel on defining it, we’ll borrow this explanation from the myopia tag page on Alignment Forum:
    Myopia means short-sighted, particularly with respect to planning -- neglecting long-term consequences in favor of the short term. The extreme case, in which only immediate rewards are considered, is of particular interest. We can think of a myopic agent as one that only considers how best to answer the single question that you give to it rather than considering any sort of long-term consequences. Such an agent might have a number of desirable safety properties, such as a lack of instrumental incentives.
    We’re focusing on language models (LMs) in this series of experiments, specifically unidirectional transformer LLMs like GPT-3. Here’s what we mean when we talk about myopia and non-myopia in the context of these models:
    For a myopic language model, the next token in a prompt completion is generated based on whatever the model has learned in service of minimising loss on the next token and the next token alone
    A non-myopic language model, on the other hand, can 'compromise' on the loss of the immediate next token so that the overall loss over multiple tokens is lower - i.e possible loss on future tokens in the completion may be 'factored in' when generating the next immediate token
    Why myopia matters for alignment
    One of the most dangerous emergent properties theorised by AI alignment researchers is deceptive alignment, a.k.a. the treacherous turn. If you’re not familiar with deceptive alignment, here’s a definition from its tag page on Alignment Forum:
    Deceptive Alignment is when an AI which is not actually aligned temporarily acts aligned in order to deceive its creators or its training process. It presumably does this to avoid being shut down or retrained and to gain access to the power that the creators would give an aligned AI.
    Deceptive alignment in an advanced AI system could be extremely diffi...
    21 min
  • LW - [Link] Why I’m optimistic about OpenAI’s alignment approach by janleike
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Link] Why I’m optimistic about OpenAI’s alignment approach, published by janleike on December 5, 2022 on LessWrong.
    The post lays out some arguments in favor of OpenAI’s approach to alignment and responds to common objections.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - Announcing: Audio narrations of EA Forum posts by peterhartree
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing: Audio narrations of EA Forum posts, published by peterhartree on December 5, 2022 on The Effective Altruism Forum.
    We've started making audio narrations of some of the best posts from the EA Forum.
    As of today, you can subscribe to the podcast:
    EA Forum (All audio)Audio narrations from the Effective Altruism Forum, including curated posts and other great writing.Subscribe:Apple Podcasts | Google Podcasts (soon) | Spotify | RSS
    You'll also start to see narrations embedded on the EA Forum post pages themselves.
    If a narration is available, you'll see a blue loudspeaker button:
    What can I listen to now?
    Some of the winning entries from the EA Criticism and Red Teaming Contest:
    Are you really in a race? The cautionary tales of Szilárd and Ellsberg by Haydn Belfield
    Does economic growth meaningfully improve well-being? by Vadim Albinsky
    Effective altruism in the garden of ends by Tyler Alterman
    Notes on effective altruism by Michael Nielsen
    Biological Anchors external review by Jennifer Lin
    Some posts that were recently marked as “curated”:
    Counterarguments to the basic AI risk case by Katja Grace
    My take on What We Owe the Future by Eli Lifland
    Population ethics without axiology: A framework by Lukas Gloor
    500 million, but not a single one more by jai
    AGI and lock-in by Lukas Finnveden, Jess Riedel, and Carl Shulman
    What matters to shrimps? Factors affecting shrimp welfare in aquaculture by Lucas Lewit-Mendes and Aaron Boddy
    Case for emergency response teams by Gavin and Jan Kulveit
    What happens on the average day? by Rose Hadshar
    How bad could a war get? by Stephen Clare and Rani Martin
    Cool. But I have a lot of things to listen to. Could I just get the curated posts, or maybe just the summaries?
    Yes. You can subscribe to either of these:
    EA Forum (Curated posts)Audio narrations of curated posts from the Effective Altruism Forum. Subscribe:Apple Podcasts | Google Podcasts (soon) | Spotify | RSSEA Forum (Summaries)Weekly summaries of the best EA Forum posts. Written by Zoe Williams (Rethink Priorities) and narrated by Coleman Jackson Snell.Subscribe:Apple Podcasts | Google Podcasts (soon) | Spotify | RSS
    What about AI narrations? I want to listen to everything!
    The Nonlinear Library project is currently generating AI narrations of all posts that meet a fairly low karma threshold.
    Within the next few months, we are hoping to collaborate with Nonlinear to develop a system that generates even better AI narrations for most or all EA Forum posts. We see a path to better pronunciation, emphasis, and tone, and also to much better handling of images, graphs and formulae.
    Who is working on this?
    This project is run by the EA Forum Team, in collaboration with TYPE III AUDIO.
    What about the existing “EA Forum Podcast”?
    In July 2021, Garrett Baker and David Reinstein started a volunteer project to narrate EA Forum posts. The most recent narration was published in January 2022. Since September they’ve been publishing Zoe and Coleman’s weekly summary episodes.
    We are grateful to Garrett and David for their work on this.
    We’ve not yet heard what they plan to do next—presumably they’ll release an update to subscribers in due course.
    Thoughts, feedback, suggestions?
    We'd love to hear from you! Please comment below, or write to [email protected].
    If you’re already listening to Nonlinear Library AI narrations, we’d be especially interested to hear what you think of them. What would you most like to see improved?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - SoGive Grants: a promising pilot. Our reflections and payout report. by SoGive
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SoGive Grants: a promising pilot. Our reflections and payout report., published by SoGive on December 5, 2022 on The Effective Altruism Forum.
    Executive Summary
    SoGive ran a pilot grants program, and we ended up granting a total of £223k to 6 projects (see the section “Our grantee payouts” for details). We were pleased to be able to give high quality feedback to all rejected applicants (which appears to be a gap in the grants market), and were also able to help some projects make tweaks such that we could make a positive decision to fund them. We also noted that despite explicitly encouraging biosecurity projects we only received one application in this field. We tracked our initial impressions of grants and found that the initial video call with applicants was discriminatory and helped unearth lots of decision-relevant information, but that a second video call didn’t change our evaluations very much.
    Given we added value to the grants market by providing feedback to all applicants, helped some candidates tweak their project proposal and identified 6 promising applications to direct funding towards, we would run SoGive Grants again next year, but perform a more light touch approach, assuming that the donors we work with agree to this, or new ones opt to contribute to the pool of funding. This report also includes thoughts on how we might improve the questions asked in the application form (which currently mirrors the application form used by EA Funds).
    Introduction
    Back in April SoGive launched our first ever applied-for granting program, this program has now wrapped up and this post sets out our experiences and lessons learned. For those of you not familiar with SoGive we’re an EA-aligned research organisation and think tank.
    This post will cover:
    Summary of the SoGive Grants program
    Advice to grant applicants
    Reflections on our evaluation process and criteria
    Advice for people considering running their own grants program
    Our grantee payouts
    We’d like to say a huge thank you to all of the SoGive team who helped with this project, and also to the external advisors who offered their time and expertise. Also, as discussed in the report we referred back to a lot of publicly posted EA material (typically from the EA forum) so for those individuals and organisations who take the time to write up their views and considerations online it is incredibly helpful and it affects real world decisions - thank you.
    If any potential donors reading this want their funding to contribute to the funding pool for the next round of SoGive grants, then please get in touch ([email protected]).
    1. Summary of the SoGive Grants program
    Why run a grants program?
    Even at the start of 2022 when funding conditions were more favourable, we believed that another funding body would be valuable. This reflected our view of the value of there being more vetting capacity. We also thought Joey made a valuable contribution in this post.
    Part of our work at SoGive involves advising high net worth individuals, and as such we generally scope out opportunities for high impact giving, so in order to find the highest impact donation opportunities we decided to formalise and open up this process. Prior to SoGive grants, we have tended to guide funds towards organisations that Founders Pledge, Open Phil, and GiveWell might also recommend along with interventions that SoGive has specifically researched or investigated for our donors. Especially in the case of following Open Phil’s grants, we had doubts that this was the highest impact donation advice we could offer, since Open Phil makes no guarantee that a grantee still has room for more funding (after receiving a grant from Open Phil).
    We also noticed a gap in the market for a granting program that provided high quality feedback (or, for some applicants, any feed...
    34 min
  • LW - Updating my AI timelines by Matthew Barnett
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Updating my AI timelines, published by Matthew Barnett on December 5, 2022 on LessWrong.
    Last year I published a post titled Three reasons to expect long AI timelines, and earlier this year I offered to bet people who had short AI timelines.
    While it wasn't my intention to be known as "a long AI timelines guy", I have begun feeling that was how people perceived me. Nonetheless, in the last few months, I've modified my views substantially. Thus, I offer this short post, which can hopefully make my current position more clear.
    There are several reasons for my update towards shorter AI timelines, though each reason is relatively straightforward and uncomplicated. In the spirit of writing something short rather than not writing something at all, my explanations here will be brief, although I may be willing to elaborate in a comment below.
    In order, these reasons included, but were not limited to,
    I became convinced that the barriers to language models adopting human-level reasoning were much weaker than I had believed. Previously, I had imagined that it would be difficult to get a language model to perform reasoning over long sequences, in which each step in the sequence requires making a non-trivial inference, and one mistake in understanding the sequence can make the difference between a coherent and incoherent response.Yet, my personal experience with language models, including ChatGPT, has persuaded me to that this type of problem is not a strong barrier, and is more continuous with other challenges like "understanding the tone of a document" or "understanding what's going on in a plot" which I had already thought language models were making good progress on. In hindsight, I should have perhaps trusted the model I had constructed myself, which forecasted human-level language models by 2030.
    I built an (unpublished) TAI timelines model, and after fitting the model, it came out with a median timeline of 2037. While I don't put a high degree of confidence in my model, or the parameters that I used, I believe it's still more reliable than my own intuition, which suggested much later dates were more plausible.
    I reflected more on the possibility that short-term AI progress will accelerate AI progress.
    I noticed that I had been underestimating the returns to scaling, and the possibility of large companies scaling their training budgets quickly to the $10B-$100B level. I am still unsure that this will happen within the next 10 years, but it no longer seems like something I should dismiss.
    I saw almost everyone else updating towards shorter timelines, except for people who already had 5-15 year timelines, and a few other people like Robin Hanson. Even after adjusting for the bandwagon effect, I think it's now appropriate to update substantially as well.
    I still feel like my arguments for expecting delays from regulation are being underrated. Yet, the degree to which governments (and perhaps society more generally) have been ignoring recent AI developments has made me less confident about how much we should expect these delays to last. Instead of imagining a 20 year delay, a 3 to 10 year delay from regulation now seems more reasonable to me.
    If you want me to get specific, my unconditional median TAI timeline is now something like 2047, with a mode around 2035, defined by the first year we get >30% yearly GWP growth as measured from a prior peak, or an event of comparable significance.
    This timeline may still appear too long to many people, yet my explanation is that it's what I get when I account for potential coordinated delays, catastrophes, and a 15% chance that we're fundamentally wrong about all of this stuff. Conditional on nothing like that happening, I'd be inclined to weakly bet on TAI before 2039.
    Thanks for listening. To help us out with The Nonlinear Library or t...
    4 min