The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Manifund: what we're funding (week 1) by Austin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Manifund: what we're funding (week 1), published by Austin on July 15, 2023 on The Effective Altruism Forum.
    We're experimenting with a weekly newsletter format, to surface the best grants and activity on Manifund!
    Overall reflections
    It's been a week since we announced our regrantor program. How are we feeling?
    Very happy with the publicity and engagement around our grant proposals
    #1 on Hacker News, multiple good EA Forum posts, and 7 new grassroots donors
    Feels like this validates our strategy of posting grant proposals in public.
    Happy with the caliber of regrantors who've applied for budgets
    We've gotten 15 applications; have onboarded 2 and shortlisted 4 more
    Now thinking about further fundraising, so we can give budgets to these awesome folks
    Happy with grantee experience for the handful of grants we've made
    Less happy with total grantmaking quantity and dollar volume so far
    In total, we've committed about $70k across 6 large grants
    Where's the bottleneck? Some guesses: regrantors are busy people; writeups take a while; "funder chicken" when waiting for others to make grants
    To address this, we're running a virtual "grantathon" to work on grant writeups & reviews! Join us next Wed evening, 6-9pm PT on this Google Meet.
    Grant of the week
    This week's featured grant is to Matthew Farrugia-Roberts, for introductory resources for Singular Learning Theory! In the words of Adam Gleave, the regrantor who initiated this:
    There's been an explosion of interest in Singular Learning Theory lately in the alignment community, and good introductory resources could save people a lot of time. A scholarly literature review also has the benefit of making this area more accessible to the ML research community more broadly. Matthew seems well placed to conduct this, having already familiarized himself with the field during his MS thesis and collected a database of papers. He also has extensive teaching experience and experience writing publications aimed at the ML research community.
    In need of funding
    Some of our favorite proposals which could use more funding:
    A proposal to cure cavities, forever. Very out-of-left-field (Aaron: "it certainly isn't great for our Weirdness Points budget"), but I'm a fan of the ambition and the team behind it. We're currently seeing how Manifund can make an equity investment in Lantern Bioworks.
    Also: this proposal hit #1 on Hacker News, with 260 upvotes & 171 comments. Shoutout to our friend Evan Conrad for posting + tweeting this out!
    Holly has a stellar track record at Rethink Priorities and Harvard EA (not to mention a killer blog). She's now looking to pivot to AI moratorium movement-organizing! As a grant that may include political advocacy, we're still seeing to what extent our 501c3 can fund this; in the meantime, Holly has set up a separate GoFundMe for individual donors.
    Pilot of electrical stunners to reduce shrimp suffering. Recommended by regrantor Marcus Abramovitch, as the one of the most exciting & unfunded opportunities in the entire space of animal welfare.
    In a thorough EA Forum post, Matt investigates the cost-effectiveness of this proposal - it's a thoughtful writeup, take a look at the entire thing! One takeaway:
    Electric shrimp stunning might be worth supporting as a somewhat speculative bet in the animal welfare space. Considerations that might drive donor decisions on this project include risk tolerance, credence in the undiluted experience model of welfare, and willingness to take a hits-based giving approach.
    New regrantors: Renan Araujo and Joel Becker
    Since putting out the call for regrantors last week, we've gotten quite the influx of interest. After speaking with many candidates, we're happy to announce our two newest regrantors: Renan and Joel!
    We expect regranting to integrate smoothly with Renan's work incubating lo...
    6 min
  • EA - CEA: still doing CEA things by Ben West
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CEA: still doing CEA things, published by Ben West on July 15, 2023 on The Effective Altruism Forum.
    This is a linkpost for our new and improved public dashboard, masquerading as a mini midyear update
    It's been a turbulent few months, but amidst losing an Executive Director, gaining an Interim Managing Director, and searching for a CEO, CEA has done lots of cool stuff so far in 2023.
    The headline numbers
    4,336 conference attendees (2,695 EA Global, 1,641 EAGx)
    133,041 hours of engagement on the Forum, including 60,507 hours of engagement with non-Community posts (60% of total engagement on posts)
    26 university groups and 33 organizers in UGAP
    622 participants in Virtual Programs
    There's much more, including historical data and a wider range of metrics, in the dashboard!
    Updates
    The work of our Community Health & Special Projects and Communications teams lend themselves less easily to stat-stuffing, but you can read recent updates from both:
    Community Health & Special Projects: Updates and Contacting Us
    How CEA's communications team is thinking about EA communications at the moment
    What else is new?
    Our staff, like many others in the community (and beyond), have spent more time this year thinking about how we should respond to the rapidly evolving AI landscape. We expect more of the community's attention and resources to be directed toward AI safety at the margin, and are asking ourselves how best to balance this with principles-first EA community building.
    Any major changes to our strategy will have to wait until our new CEO is in place, but we have been looking for opportunities to improve our situational awareness and experiment with new products, including:
    Exploring and potentially organizing a large conference focussed on existential risk and/or AI safety
    Learning more about and potentially supporting some AI safety groups
    Supporting AI safety communications efforts
    These projects are not yet ready to be announcements or commitments, but we thought it worth sharing at a high level as a guide to the direction of our thinking. If they intersect with your projects or plans, please let us know and we'll be happy to discuss more.
    It's worth reiterating that our priorities haven't changed since we wrote about our work in 2022: helping people who have heard about EA to deeply understand the ideas, and to find opportunities for making an impact in important fields. We continue to think that top-of-funnel growth is likely already at or above healthy levels, so rather than aiming to increase the rate any further, we want to make that growth go well.
    You can read more about our strategy here, including how we make some of the key decisions we are responsible for, and a list of things we are not focusing on. And it remains the case that we do not think of ourselves as having or wanting control over the EA community. We believe that a wide range of ideas and approaches are consistent with the core principles underpinning EA, and encourage others to identify and experiment with filling gaps left by our work.
    Impact stories
    And finally, it wouldn't be a CEA update without a few #impact-stories:
    Online
    Training for Good posted about their EU Tech Policy Fellowship on the EA Forum. 12/100+ applicants they received came from the Forum, and 6 of these 12 successfully made it on to the program, out of 17 total program slots.
    Community Health & Special Projects
    Following the TIME article about sexual misconduct, people have raised a higher-than-usual number of concerns from the past that they had noticed or experienced in the community but hadn't raised at the time. In many of these cases we've been able to act to reduce risk in the community, such as warning people about inappropriate behavior and removing people from CEA spaces when their past behavior has caused harm.
    Communicati...
    6 min
  • LW - Why was the AI Alignment community so unprepared for this moment? by Ras1513
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why was the AI Alignment community so unprepared for this moment?, published by Ras1513 on July 15, 2023 on LessWrong.
    Our epistemic rationality has probably gotten way ahead of our instrumental rationality
    Scott Alexander
    A Lesswrong Crypto autopsy
    This is a question post: Why was the AI Alignment community so unprepared for engaging with the wider world when the moment finally came?
    I have been a LW reader for at least 10 years, but I confess that until the last ~1.5 years I mostly watched the AI alignment conversation float by. I knew of the work, but I did not engage with the work. Top people were on it, and I had nothing valuable to add.All that to say: Maybe this has been covered before and I have missed it in the archives.Lately (throughout this year), there have been a flurry of posts essentially asking: How do we get better at communicating to and convincing the rest of the world about the dangers of AI alignment?
    Catching the Eye of Sauron
    An AI Realist Manifesto
    The Social Alignment Problem
    All three of which were posted in April 2023.
    The subtext being: If it is possible to not-kill-everyone this is how we are going to have to do it. Why are we failing so badly at doing this?At this risk of looking dumb or ignorant, I feel compelled to ask: Why did this work not start 10 or 15 years ago?To be clear: I do not mean true nuts and bolts ML researcher Alignment work, which this community and MIRI were clearly the beginning and end for nearly 2 decades. I do not even mean outreach work to adjacent experts who might conceivably help the cause. Again, here I think great effort was clearly made.I also do not mean that we should have been actively doing these things before it was culturally relevant.I am asking: Why did the Alignment community not prepare tools and plans for convincing the wider infosphere about AI safety years in advance?
    Prior to the Spring 2023 inflection point.Why were there no battle plans in the basement of the pentagon that were written for this exact moment?It seems clear to me, based on the posts linked above and the resulting discussion generated, that this did not happen.I can imagine an alternate timeline where there was a parallel track of development within the community circa 2010-2020(?) where much discussion and planning covered media outreach and engagement, media training, materials for public discourse, producing accessible content for every level of education and medium. For every common "normie" argument and every easy-to-see-coming news headline. Building and funding policy advocates, contacts, and resources in the political arena.
    Catchy slogans, buttons, bumper stickers, art pieces, slam dunk tweets.Heck, 20+ years is enough time to educate, train, hire and surgically insert an entire generation of people into key positions in the policy arena specifically to accomplish this one goal like sleeper cell agents. Likely much, much, easier than training highly qualified alignment researchers.It seems so obvious in retrospect that this is where the battle would be won or lost.Didn't we pretty much always know it was going to come from one or a few giant companies or research labs? Didn't we understand how those systems function in the real world? Capitalist incentives, Moats, Regulatory Capture, Mundane utility, and International Coordination problems are not new.Why was it not obvious back then? Why did we not do this? Was this done and I missed it?(First time poster: I apologize if this violates the guidelines about posts being overly-meta discussion)
    Which it seems we still cannot manage to do
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - EA EDA: What do Forum Topics tell us about changes in EA? by JWS
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA EDA: What do Forum Topics tell us about changes in EA?, published by JWS on July 15, 2023 on The Effective Altruism Forum.
    tl;dr2: Data on EA Forum posts and topics doesn't show clear 'waves' of EA
    tl;dr: I used the Forum API to collect data on the trends of EA Forum topics over time. While this analysis is by no means definitive, it doesn't support the simple narrative that there was a golden age of EA that has abandoned for a much worse one. There has been a rise in AI Safety posts, but that has also been fairly recent (within the last ~2 years)
    1. Introduction
    I really liked Ben West's recent post about 'Third Wave Effective Altruism', especially for its historical reflection on what First and Second Wave EA looked like. This characterisation of EA's history seemed to strike a chord with many Forum users, and has been reflected in recent critical coverage of EA that claims the movement has abandoned its well-intentioned roots (e.g. donations for bed nets) and decided to focus fully on bizarre risks to save a distant, hypothetical future.
    I've always been a bit sceptical with how common this sort of framing seems to be, especially since the best evidence we have from funding for the overall EA picture shows that most funding is still going to Global Health areas. As something of a (data) scientist myself, I thought I'd turn to one of the primary sources of information for what EAs think to shed some more light on this problem - the Forum itself!
    This post is a write-up of the initial data collection and analysis that followed. It's not meant to be the definitive word on either how EA, or use of the EA Forum, has changed over time. Instead, I hope it will challenge some assumptions and intuitions, prompt some interesting discussion, and hopefully leads to future posts in a similar direction either from myself or others.
    2. Methodology
    (Feel free to skip this section if you're not interested in all the caveats)
    You may not be aware, the Forum has an API! While I couldn't find clear documentation on how to use it or a fully defined schema, people have used it in the past for interesting projects and some have very kindly shared their results & methods. I found these following three especially useful (the first two have linked GitHubs with their code):
    The Tree of Tags by Filip Sondej
    Effective Altruism Data from Hamish
    This LessWrong tutorial from Issa Rice
    With these examples to help me, I created my own code to get every post made on the EA Forum to date (without those who have deleted their post).
    There are various caveats to make about the data representation and data quality. These include:
    I extracted the data on July 7th - so any totals (e.g. number of posts, post score etc) or other details are only correct as of that date.
    I could only extract the postedAt date - which isn't always when the post in question was actually posted. A case in point, I'm pretty sure this post wasn't actually posted in 1972. However, it's the best data I could find, so hopefully for the vast majority of posts the display date is the posted date.
    In looking for a starting point for the data, there was a discontinuity between August to September 2014, but the data was a lot more continuous after then. I analyse the data in terms of monthly totals, so I threw out the one-week of data I had for July. The final dataset is therefore 106 months from September 2014 to June 2023 (inclusive).
    There are around ~950 distinct tags/topics in my data, which are far too many to plot concisely and share useful information. I've decided to take the top 50 topics in terms of times used, which collectively account for 56% of all Forum tags and 92% of posts in the above time period.
    I only extracted the first listed Author of a post - however, only 1 graph shared below relies on a user-level aggregat...
    13 min
  • EA - Grow your Mental Wellbeing to Grow your Impact HERE: Announcing our Summer Program by Edmo Gamelin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Grow your Mental Wellbeing to Grow your Impact HERE: Announcing our Summer Program, published by Edmo Gamelin on July 14, 2023 on The Effective Altruism Forum.
    Do you want to
    become more fulfilled, resilient, and productive?
    practice evidence-based tools to deal with blockers and stressors such as low energy levels, anxiety, imposter syndrome, and perfectionism?
    embark on that journey with other members of the community?
    Apply for our summer program in ~15 min here! Spaces are limited.
    TL;DR
    Rethink Wellbeing is announcing an online Cognitive-Behavioral Therapy (CBT) Program from August-October 2023. This program is designed to help improve mental wellbeing and productivity for people in the EA community. Using the latest psychotherapeutic and behavior change techniques, participants attend weekly group sessions together with 5-8 well-matched peers, led by a trained facilitator, and engage in readings and application of the learned tools in between the sessions. This no-cost program requires eight weeks of commitment, with a total of 6h/week.
    What does the program consist of?
    The program is experiential and practice-based; you'll learn through repeated, deliberate practice, so your new skills can eventually become automatic and habitual. We will draw on techniques backed by a wealth of cutting-edge research, particularly those from the gold standard of third-wave CBT. These techniques can be applied to a variety of target areas: anxiety, low mood, perfectionism, procrastination, self-esteem, productivity, and more. You can learn more about CBT here.
    The program involves:
    Weekly participation
    A group video conference, led by a trained peer facilitator, ranging from 1,5 to 2 hours, designed for sharing personal experiences, bonding, initiating discussions, and practicing newly learned techniques together.
    A reading before each meeting (~a few pages/week for CBT)
    Home practice of new skills and techniques (~4h/week or ~30 min/day)
    Program evaluation surveys
    Short weekly forms for progress tracking, reflections, and feedback (< 5 min each)
    Surveys on your mental well-being at weeks 0, 6, 8, and 12 (~20 min each)
    We ask that people who sign up be ready to commit to the entire program, which is essential because:
    You are most likely to maximize your benefits from the program by dedicating time to the weekly sessions, readings, and home practice.
    Your peers will rely on you. You will go on this journey with a small group, handpicked for you. Poor participation or dropping out can challenge the group's dynamics and spirit.
    We will only be able to determine whether the program is effective and scalable if everyone engages fully.
    Why do we think the program will be effective?
    Mental health research suggests that peer-guided self-help groups working with evidence-based therapy methods can improve mental wellbeing and productivity just as much as 1:1 therapy.
    In addition, Rethink Wellbeing's pilot tests of peer-facilitated groups showed promising results:
    Participants' psychological well-being significantly increased (p < .05).
    Productivity, perfectionism, and self-leadership increased in the correspondingly themed groups.
    All participants rated the programs as "Good" or "Excellent".
    3 of 5 groups decided to continue meeting.
    You can learn more about the rationale and background behind our method on our website.
    How do I sign-up?
    If you are interested in participating in the program, you can apply within ~15 min here.
    If you are interested in facilitating a group, you can apply within ~20 min here. Details will follow in another post soon.
    Answers will be reviewed on a rolling basis until July 31, 2023. Earlier applications are preferred.
    Here's what happens after you sign up:
    We review your answers, and confirm if you're accepted via email. We will only be able to respond to th...
    5 min
  • LW - When Someone Tells You They're Lying, Believe Them by ymeskhout
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When Someone Tells You They're Lying, Believe Them, published by ymeskhout on July 14, 2023 on LessWrong.
    Some people refuse to admit they're wrong, but there's other clues
    Paul Ehrlich became well-known for his 1968 book The Population Bomb, where he made many confidently-stated but spectacularly-wrong predictions about imminent overpopulation causing apocalyptical resource scarcity. As illustration for how far off the mark Ehrlich was, he predicted widespread famines in India at a time when its population was around 500 million people, and he wrote "I don't see how India could possibly feed two hundred million more people by 1980." He happened to have made this claim right before India's Green Revolution in agriculture. Not only is India able to feed a population that tripled to 1.4 billion people, it has long been one of the world's largest agricultural exporter.
    Ehrlich is also known for notoriously losing a bet in 1990 to one of my favorite humans ever, the perennial optimist (and business professor) Julian Simon. Bryan Caplan brings up some details to the follow-up that never was:
    We've all heard about the Ehrlich-Simon bet. Simon the cornucopian bet that resources would get cheaper, Ehrlich the doomsayer bet that they would get pricier, and Simon crushed him. There's a whole book on it. What you probably don't know, however, is that in 1995, Paul Ehrlich and Steve Schneider proposed a long list of new bets for Simon - and that Simon refused them all.
    The first bet was fairly straight-forward: Ehrlich picked 5 commodities (copper, chromium, nickel, tin, & tungsten) and predicted that their price would be higher in 1990 compared to 1980 as the materials become scarcer. Instead of rising, the combined price went down. Ehrlich's decade-spanning obstinance and unparalleled ability to step on rakes make him an irresistible punching bag but despite his perennial wrongness, his responses have ranged from evasion to outright denials:
    Anne and I have always followed U.N. population projections as modified by the Population Reference Bureau - so we never made "predictions," even though idiots think we have. When I wrote The Population Bomb in 1968, there were 3.5 billion people. Since then we've added another 2.8 billion - many more than the total population (2 billion) when I was born in 1932. If that's not a population explosion, what is? My basic claims (and those of the many scientific colleagues who reviewed my work) were that population growth was a major problem. Fifty-eight academies of science said that same thing in 1994, as did the world scientists' warning to humanity in the same year. My view has become depressingly mainline!
    Some humans possess the unfortunate egotistical and dishonorable habit of refusing to admit error. It's a reflex I personally find utterly baffling, because nothing engenders someone's credibility to me more than their ability to admit error. So if we can't always rely on people to admit a mistake, what else do we have?
    What I find so interesting about the second bet in 1995 is how peculiar the proposed conditions were:
    I kept thinking ".so?" as I read these. Why would someone care about the availability of firewood versus the heating and cooking costs in general? Why would someone care about per capita cropland statistics versus the availability of food in general? Many of these are also blatant statistical fuckery, such as monitoring increases in absolute worldwide AIDS deaths during a period of persistent population growth.
    Ehrlich is playing a seemingly uncomfortable game of Twister here, but his behavior makes perfect sense if you read intelligence and agency behind his decisions. The only explanation for the indirect, tangential, and collateral measurements is that Ehrlich knows that a direct measurement will not be favorable to h...
    5 min
  • LW - The Goddess of Everything Else - The Animation by Writer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Goddess of Everything Else - The Animation, published by Writer on July 13, 2023 on LessWrong.
    This is an animation of The Goddess of Everything Else, by Scott Alexander. I hope you enjoy it :)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - Jailbreaking GPT-4's code interpreter by nikolaisalreadytaken
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Jailbreaking GPT-4's code interpreter, published by nikolaisalreadytaken on July 13, 2023 on LessWrong.
    Disclaimer: I don't know much about cybersecurity. Much of my knowledge comes from asking GPT-3.5 and GPT-4 for advice. These are some results from around 20 hours of playing around with the code interpreter plugin in early-mid May, when most of this was written. I contacted OpenAI about these jailbreaks in mid May and they mostly seem to still be there.
    Thank you to Max Nadeau, Trevor Levin, aL xin, Pranav Gade, and Alexandra Bates for feedback on this post!
    Summary
    GPT-4's code interpreter plugin has been rolled out to some users. It works by running on a virtual machine that is isolated from the internet and other machines, except for the commands sent in from the API and the results sent back to the API. GPT-4 seems to follow a set of rules that are either enforced through hard access restrictions or through GPT-4 refusing to do things for the user.
    Here, I highlight 6 rules that GPT-4 claims to be following, but which are easily breakable, alongside some best practices in cybersecurity that have been neglected. In short:
    GPT-4 claims that it is only supposed to read, modify, or delete files in two designated folders ("sandbox" and "mnt"). However, it is able to read basically any file on the system (including sensitive system files), and it is able to write and delete files outside of its designated folders.
    This seems to reveal information that the user isn't supposed to see. There are ways to find out information about the hardware that the VM is being run on, including:
    Information about the way OpenAI logs data, including what libraries and IP address they assign to virtual machines.
    A rough estimate of the number of VMs that OpenAI can run at maximum at any moment (from the way the IP addresses are allocated).
    A rough idea of what storage hardware is used (from write speed), alongside some info on other hardware.
    There is a file in the virtual machine (in a folder labeled "internal") that users can download that details how web requests are handled.
    As ChatGPT would say: "By exposing your source code, you make it easier for potential attackers to analyze the code and identify security vulnerabilities. This can lead to an increased risk of exploitation if there are any flaws in your implementation."
    GPT-4 claims that conversations with the model do not have a memory. However, files are routinely saved between conversations with the same user.
    Later in this post, I present an example of two different conversations with GPT-4 where I write a file in one conversation and read the file in another conversation.
    GPT-4 claims that there are resource limits in place to prevent users from using too much CPU or memory. However, it is possible to write >80GB of files onto OpenAI's VM within minutes. The rough rate at which I managed to write files is 0.3GB/second.
    There's a maximum Python runtime of 120 seconds per process, and 25 messages every 3 hours. This can be circumvented using simple workarounds (you can increase usage by at least a factor of 2).
    GPT-4 claims it cannot execute system commands. However, GPT-4 can and will run (innocuous) system commands and run internet-related commands (such as "ping") despite measures put in place to prevent this.
    However, OpenAI seems at least partly aware of this. They seem to tell GPT-4 that it has a strict set of rules (as it reliably repeats the rules when asked), and GPT-4 seems to believe these rules in some contexts (most of the time it refuses to do things that go against the rules), but they also left a README file for those curious enough to look at the VM's files that says:
    You might think that all is well because OpenAI was aware that the system was not secure. I don't think the existence of this README file inv...
    10 min
  • LW - How can I get help becoming a better rationalist? by TeaTieAndHat
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How can I get help becoming a better rationalist?, published by TeaTieAndHat on July 13, 2023 on LessWrong.
    Well.
    Right now, being 'a rationalist' could be said to be a massive part of my identity, at least judging by the absurd amount of time I've spent reading posts here, or SSC/ACX, and in a few other places. Yet, I'm still a mere lurker unfamiliar with most of the local customs.
    But it's not what matters. What does is that I'm a terrible rationalist. You see, rationality takes practice. And reading stuff on LW isn't practice at all. If anything, it's just a great way of filling my brain with a lot of useful concepts, and then either blame myself for not using them, or use them for something entirely unrelated to their normal purpose. Often, to make myself feel worse, and think worse.
    As the saying goes, rationality is a martial art. Learning it by reading the rules, or by watching other people apply the rules, is about as effective as developing one's muscles by watching sports on TV.
    I know of the CFAR, and of various related groups, meetups for ACX readers or for other people, etc. But, apart from ACX meetups, which aren't about being better rationalists per se, I don't have easy access to any of those, or certainly to a general environment which welcomes this. You know, not being in the Bay Area and all.
    And yet, I want to be more rational as much as anyone who's been lurking here for five years wants it, and given how depressed I was until very recently, I probably badly need it, too.
    I'm not sure what kind of answers I expect, but, like, how can I push myself to learn more, and especially to practice more, and ideally to actually use rationality to improve my life?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Winners of AI Alignment Awards Research Contest by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Winners of AI Alignment Awards Research Contest, published by Akash on July 13, 2023 on LessWrong.
    This post describes the winning submissions of AI Alignment Awards. We offered prizes for novel contributions to the problems of goal misgeneralization and corrigibility. (For more context, see this post and our site.)
    Summary
    We received 118 submissions. Of these, 5 received honorary mention prizes ($1000) and 7 won final prizes ($5000-16000). The rest of the post summarizes the 7 winning submissions, including comments from our judges. (You can see winners' full submissions on our site.)
    Goal Misgeneralization winners
    Thane Ruthenis ($11,000)
    Abstract
    Goal misgeneralization primarily happens because the system under training latches not upon the goal it's being trained for, but upon an upstream correlate of that goal - like human love for their children is an upstream correlate of inclusive genetic fitness.
    What complicates this problem are suspicions of path-dependence. It's not overdetermined what values a given system subjected to a given selection pressure will learn. Rather, every next value learned is also a function of the values the system has already learned, such that the entire process can only be predicted step-by-step, no "skipping to the end" allowed.
    I provide a mathematical framework (see the attachment) that formalizes "upstream correlates of goals", and gestures at a possible formalization of path-dependence. Optimistically, an improved version of this framework may serve as a full predictive model of path-dependence, one that would allow us to calculate precisely how we should set up a training loop in order to get the heuristics/shards we want.
    The issues are twofold:
    The process of value formation may not, in fact, be realistically predictable. It may be highly sensitive to random noise and perturbations (akin to the Lottery Ticket Hypothesis), meaning no robust predictive model is possible.
    "Shard-ecology alignment" does not equal "robust AGI alignment". Even the perfectly working version of this method would only do the equivalent of aligning the would-be AGI's base urges, not the values it'll settle on after it engages in its version of moral philosophy. And "moral reflection" may itself be a highly unstable process.
    Nevertheless, a robust theory of path-dependence may be possible, and may be an important part of some more complete solution to inner alignment.
    Comment from judge (John Wentworth)
    This submission demonstrates an unusually strong grasp of the "upstream correlates" issue. The proposal itself isn't particularly promising, as the author recognizes. But it does demonstrate a relatively-strong understanding of one part of the alignment problem, in a way orthogonal to the parts which people usually understand best.
    Pedro Afonso Oitavén de Sousa ($6,000)
    Abstract
    The idea consists of a way of creating potentially scalable explanations ("approximate proofs", like what ARC was trying to do) of why the circuit formed by a learned world model + utility function + RL agent has the output distribution it has. Then the explanation and the agent would be jointly optimized to be able to show that the agent achieves a high expected return in a set of situations much more varied than the distribution of the training data.
    It would start by taking the temporally unrolled circuit mapping random variables to utilities (for now for a finite number of time steps, but this seems possible to fix). Then on top of it, it would construct a series of progressively simpler circuits, until it gets to one that is small enough that it can be exhaustively analysed. Each circuit maps from different sets of random variables to the same utilities. Given a sampled assignment of values to the random variables of the base level (full circuit), we also have learned funct...
    22 min