The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Problems with free services for EA projects by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Problems with free services for EA projects, published by Lizka on August 3, 2023 on The Effective Altruism Forum.
    EA-motivated specialists sometimes offer free or subsidized versions of normally expensive services to EA projects. I think this is often counterproductive and outline my reasoning in this post.
    The key problem with free services is that we don't have market-based information about their quality, so the beneficiaries of a free service might be getting less value than it might appear they are getting. As a result, service providers waste their time providing expensive services to people who wouldn't pay the full price (instead, providers could charge and donate or spend that time on other impactful work). Additionally, community-level overestimates of the quality of free services are more likely and might lead people who need good services to use the free versions even when they're worse suited to their needs.
    If you're offering, taking, or advertising a free service like this, I think you should believe the situation is an exception to the general heuristic. (More on these problems and other issues, as well as exceptions and nuances.)
    Some services make sense as free services. For instance (see more):
    Community infrastructure projects or other work where the benefits go through many people (there's not a clear "beneficiary" of the service)
    Services for which overhead costs are especially high
    Services that are in large part beneficial to the provider, not the recipients of the service
    Notes & caveats:
    Scope of the post
    When I talk about "services," I mean costly services that are often provided by specialists (coaching, consulting, etc.). I think more minor services are more often fine, and I'm definitely not talking about friendly support, like helping your officemate with an ughy task or having a call with someone you met at EAG who's interested in getting into your field.
    Certain types of volunteering don't encounter the issues outlined here, particularly when the volunteer has the option to be paid a specified rate but opts out (so they know more about the "market value" of their free work).
    The concerns I discuss are also less relevant for services provided by for-profit organizations/groups that provide a free/subsidized version to e.g. nonprofits.
    I focus on free services here, but the arguments can probably be extended to cover subsidized services.
    I don't discuss paid services that only serve EA customers, but I also have concerns about these services when it isn't clear that the needs of EA projects covered are unusual. My rough take is that:
    People advertising infrastructure and support projects in EA spaces (e.g. newsletters) should not give preference to services that only have EA customers unless there's something clearly special about the service that makes it especially useful for EA work.
    People working on EA projects should find the available service that best suits their needs, which is generally unlikely to be EA-focused; most of the world is not in EA. (Maybe it's easier to identify a good-enough service in EA, but I still think we should be a bit more suspicious of our instinct towards going for EA things.)
    I'm not speaking for my team or for CEA.
    I don't know how important the issues I outline here are relative to other issues in EA, but this topic has come up several times, so I decided to write this post.
    I don't think people who provide free services are ill-meaning or anything like that. Also, I appreciate many people in the EA community for reasons that go beyond "I think they do impactful work." I like to hang out with other people in the EA community for personal reasons. But if I'm thinking about doing something for EA reasons - because I think it's an effective use of resources in order to help the world - I want to focus ...
    18 min
  • EA - University EA Groups Need Fixing by Dave Banerjee
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: University EA Groups Need Fixing, published by Dave Banerjee on August 3, 2023 on The Effective Altruism Forum.
    (Cross-posted from my website.)
    I recently resigned as Columbia EA President and have stepped away from the EA community. This post aims to explain my EA experience and some reasons why I am leaving EA. I will discuss poor epistemic norms in university groups, why retreats can be manipulative, and why paying university group organizers may be harmful. Most of my views on university group dynamics are informed by my experience with Columbia EA. My knowledge of other university groups comes from conversations with other organizers from selective US universities, but I don't claim to have a complete picture of the university group ecosystem.
    Disclaimer: I've written this piece in a more aggressive tone than I initially intended. I suppose the writing style reflects my feelings of EA disillusionment and betrayal.
    My EA Experience
    During my freshman year, I heard about a club called Columbia Effective Altruism. Rumor on the street told me it was a cult, but I was intrigued. Every week, my friend would return from the fellowship and share what he learned. I was fascinated. Once spring rolled around, I applied for the spring Arete (Introductory) Fellowship.
    After enrolling in the fellowship, I quickly fell in love with effective altruism. Everything about EA seemed just right - it was the perfect club for me. EAs were talking about the biggest and most important ideas of our time. The EA community was everything I hoped college to be. I felt like I found my people. I found people who actually cared about improving the world. I found people who strived to tear down the sellout culture at Columbia.
    After completing the Arete Fellowship, I reached out to the organizers asking how I could get more involved. They told me about EA Global San Francisco (EAG SF) and a longtermist community builder retreat. Excited, I applied to both and was accepted. Just three months after getting involved with EA, I was flown out to San Francisco to a fancy conference and a seemingly exclusive retreat.
    EAG SF was a lovely experience. I met many people who inspired me to be more ambitious. My love for EA further cemented itself. I felt psychologically safe and welcomed. After about thirty one-on-ones, the conference was over, and I was on my way to an ~exclusive~ retreat.
    I like to think I can navigate social situations elegantly, but at this retreat, I felt totally lost. All these people around me were talking about so many weird ideas I knew nothing about. When I'd hear these ideas, I didn't really know what to do besides nod my head and occasionally say "that makes sense." After each one-on-one, I knew that I shouldn't update my beliefs too much, but after hearing almost every person talk about how AI safety is the most important cause area, I couldn't help but be convinced. By the end of the retreat, I went home a self-proclaimed longtermist who prioritized AI safety.
    It took several months to sober up. After rereading some notable EA criticisms (Bad Omens, Doing EA Better, etc.), I realized I got duped. My poor epistemics led me astray, but weirdly enough, my poor epistemics gained me some social points in EA circles. While at the retreat and at EA events afterwards, I was socially rewarded for telling people that I was a longtermist who cared about AI safety. Nowadays, when I tell people I might not be a longtermist and don't prioritize AI safety, the burden of proof is on me to explain why I "dissent" from EA. If you're a longtermist AI safety person, there's no need to offer evidence to defend your view.
    (I would be really excited if more experienced EAs asked EA newbies why they take AI safety seriously more often. I think what normally happens is that the experienced EA gets su...
    17 min
  • EA - EA Survey 2022: What Helps People Have an Impact and Connect with Other EAs by WillemSleegers
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Survey 2022: What Helps People Have an Impact and Connect with Other EAs, published by WillemSleegers on August 3, 2023 on The Effective Altruism Forum.
    Summary
    Personal contact with EAs was the most commonly selected influence on EAs' ability to have a positive impact (40.9%), followed by 80,000 Hours (31.4%) and local EA groups (19.8%).
    Compared to 2020, EA Global, the EA Forum, EAGx, and 80,000 Hours one-on-one career discussion showed a small increase in being reported as important sources for greater impact. Relatively more sources decreased, some quite significantly, in being reported as important (e.g., GiveWell, 80,000 Hours website and podcast).
    Local EA groups were the most commonly cited source for making a new connection (35.5%), followed by a personal connection (34.1%), EA Global (26.2%), and EAGx (24.0%).
    Compared to 2020, relatively more respondents indicate having made an interesting and valuable new personal connection via EA Global and EAGx, and fewer via most other sources.
    Introduction
    In this post, we report on what factors EAs say helped them have a positive impact or create new connections. Note that we significantly shortened the EA Survey this year, meaning there are fewer community-related questions than in the previous EA Survey.
    Positive Influences
    We asked about which factors, within the last 12 months, had the largest influence on your personal ability to have a positive impact, allowing respondents to select up to three options. On average, respondents selected 2.36 options (median 3).
    Personal contact with EAs stood out as the most common factor selected by respondents (40.9%), followed by 80,000 Hours (combined, 31.4%), and local EA groups (19.8%).
    Personal contact, 80,000 Hours, and EA groups were also the top three factors that respondents reported as being important for getting them involved in EA.
    2020 vs. 2022
    We asked the same question in 2020, although we changed some of the response categories. This year we included 80,000 Hours (job board), the online EA community (other than EA Forum), and Virtual programs, but dropped Animal Charity Evaluators, The Life You Can Save, and Animal Advocacy Careers. These categories were dropped due to low endorsement rates in previous years.
    Compared to 2020, we see an increase in EA Global, the EA Forum, EAGx, and 80,000 Hours one-on-one career discussion as important sources for greater impact. These increases were quite small in most cases, with the biggest change observed for EAGx (from 6% to 13%). We saw multiple decreases, some quite sizable, in local EA groups, 80,000 Hours (both the website and podcast), Books, GiveWell, Articles/blogs, Giving What We Can, LessWrong, Slate Star Codex/Astral Codex Ten, podcasts, and Facebook groups.
    A portion of the decrease of the website of 80,000 Hours can be attributed to the addition of the 80,000 Hours job board category in this year's survey. Last year, respondents may have included this category in the website category of 80,000 Hours, while this year it was its own category. Including the job board category with the website category leads to a smaller decrease between 2022 and 2020, although it does not fully account for it.
    It's important to recall that these questions asked about which factors had had the largest influence within the last 12 months. Thus, the percentages of respondents who have been influenced by these factors at some point are likely larger than those reporting having been influenced in the last 12 months within this survey. Responses to this question might also be expected to change more, across years, than our questions which are not limited to the last 12 months.
    Gender
    Respondents who indicated identifying as a man were more likely to select the EA Forum, GiveWell, the 80,000 Hours podcast and one-on-one career discu...
    12 min
  • EA - Want to work on US emerging tech policy? Consider the Horizon Fellowship, TechCongress, and AAAS AI Fellowship by kuhanj
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Want to work on US emerging tech policy? Consider the Horizon Fellowship, TechCongress, and AAAS AI Fellowship, published by kuhanj on August 3, 2023 on The Effective Altruism Forum.
    I recently learned that there are a lot of fellowships currently open for individuals interested in transitioning into US emerging tech policy careers. I shared the announcements from different websites below.Based on my reading, the Horizon Fellowship seems to be broader with placements at think tanks, executive agencies, and congress. These placements can last up to two years. TechCongress and AAAS Rapid Response STPF are only for 10-12 month congressional placements. All are only open to US citizens or individuals with the right to work in the US without employer sponsorship.
    Horizon Fellowship (Deadline: Sept 15th, 2023. Apply here):
    "Applications for the 2024 cohort of the Horizon Fellowship are now open! If you are interested in working on AI, biosecurity, or related policy areas at a federal agency, congressional office, or think tank, you can learn more about how to apply at Become a Fellow.
    This application round is for the third cohort of Horizon Fellows. The first two cohorts had 15 and 20 fellows respectively, and a 100% success rate in securing high-impact placements related to AI and biosecurity policy. Among many other host organizations, our fellows have placed at the Departments of Defense and Homeland Security, the Senate Commerce and House Science committees, and think tanks such as the Center for Security and Emerging Technology and the Center for Health Security. You can learn more about past fellows and their placements at Meet our Fellows."
    Eligibility (junior think tank track): Recently obtained a bachelor's or master's degree (including those graduating in the year that their placements would start; Demonstrated interest in AI, biosecurity, or related emerging technology areas
    Eligibility (senior track): Several years of professional experience; A credential (e.g. work experience, degree, publications, or similar) related to area of focus; A graduate degree (not required; strongly preferred for executive branch fellows)
    Tech Congress (Deadline: August 22nd 2023. Apply here):
    Applications for the January 2024 Fellowship
    "The Congressional Innovation Fellowship will place you among the top tech decision makers in the United States government at a time when technology is reshaping society in fundamental ways. Even if you've never considered working in government, the Congressional Innovation Fellowship will allow you to make change at the highest levels and at a scale unparalleled in the private or public sectors.
    We are bridging the divide between Congress and the technology sector by placing tech savvy people like you-- an early-career technologist (two - six years professional experience), including those who have recently finished, or are on track to finish a Master's program or PhD-- to work with Members of Congress and Congressional Committees. Our goals are to build capacity in Congress, train cross-sector leaders -- who can understand the challenges of government and in the technology community -- and keep Congress up to date about the latest challenges and opportunities relating to technology."
    Eligibility (junior track): Early-career, with between two and six years of experience, including recently finishing (or projected to finish by January 2024) a technical degree program. Tech savvy, with experience working in or studying the technology sector.
    Eligibility (senior track): Eight or more years of work or postgraduate study. Tech savvy, with experience working in or studying the technology sector.
    AAAS Rapid Response STPF Cohort in AI (Deadline: August 5th, 2023. Apply here)
    (Note that this is a separate program from the annual AAAS Science & Technology P...
    5 min
  • LW - Password-locked models: a stress case for capabilities evaluation by Fabien Roger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Password-locked models: a stress case for capabilities evaluation, published by Fabien Roger on August 3, 2023 on LessWrong.
    Thanks to Buck Shlegeris, Ryan Greenblatt, Kshitij Sachan and Jenny Nitishinskaya for helpful discussions and feedback on drafts. This work was done at Redwood Research. The views expressed are my own and do not necessarily reflect the views of the organization.
    Password-locked models are trained to exhibit certain capabilities only when a password is present in the query. Studying these models has two purposes:
    Testing how well capability evaluations work when applied to models which "aren't trying"
    Understanding how well organizations could hide the capabilities of their models.
    In this post, I make empirical claims which I believe to be true, check some of them on small language models, and describe what this implies for the control of potentially deceptive models.
    Key empirical claims: when using behavioral model evaluations, password-locked models:
    Look much less capable than they actually are when eliciting capabilities with prompting
    Sometimes look much less capable than they actually are when eliciting capabilities with Reinforcement Learning
    Experimental evidence: I exhibit one case where they don't, and one case where they do
    Can't look much less capable than they actually when eliciting capabilities with fine-tuning
    Key takeaways, if the claims above are true:
    If an alignment plan relies on some models being too weak to perform certain tasks, then model evaluations aimed at checking the weakness of models should be able to elicit the locked capabilities of password-locked models. This is because password-locked models and deceptive models may be able to underperform in similar circumstances.
    API access and inference is not enough to evaluate capabilities of a model produced by AI labs trying to game regulations, even if AI labs are internally using the same API endpoint as auditors.
    Key Empirical Claims
    Similarities Between Password-locked Models and Deceptively Aligned Models Pretending to Be Dumb
    It looks to me that password locked models have the same properties as the one described by Evan in When can we trust model evaluations? when he describes if and how deceptive models would be able to sandbag each category of behavioral evaluation (which he coins in his post):
    Behavioral Non-Fine-Tuning Evaluations: By construction, password-locked models will refuse to be capable no matter the prompt, except for prompts with the special password. But it's unclear how easy it is to build good password-locked models which are hard to crack (e.g. which don't leak the password with some clever prompting). Even if good password-locked models are built, there remain a large difference with deceptive models: the search for the correct password might have very different properties than the search for a prompt which makes a deceptive model believe it's time to strike.
    Behavioral I.I.D. Fine-Tuning Evaluations: I would be surprised if fine-tuning the password-locked model didn't lift the locking mechanism extremely quickly. This is similar to deceptive models (except if they are so strong that they can gradient hack). The main difference is that fine-tuning to remove password-locking on one task might transfer to another task, which might not be the case for deceptive models.
    Behavioral RL Fine-Tuning Evaluations: The success of evaluations which use Reinforcement Learning depends, as for deceptive models, on the success of exploration. If supervised fine-tuning is done with only low quality answers, then the password locked models will just continue to output low-quality answers and will never explore its way into high quality ones - except if there is a natural "path" gradually improving the quality of answers. But it is possible to emulate a thre...
    13 min
  • LW - AI #23: Fundamental Problems with RLHF by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #23: Fundamental Problems with RLHF, published by Zvi on August 3, 2023 on LessWrong.
    After several jam-packed weeks, things slowed down to allow everyone to focus on the potential room temperature superconductor, check Polymarket to see how likely it is we are so back and bet real money, or Manifold for chats and better graphs and easier but much smaller trading.
    The main thing I would highlight this week are an excellent paper laying out many of the fundamental difficulties with RLHF, and a systematic new exploit of current LLMs that seems to reliably defeat RLHF.
    I'd also note that GPT-4 fine tuning is confirmed to be coming. That should be fun.
    Table of Contents
    Introduction.
    Table of Contents.
    Language Models Offer Mundane Utility. Here's what you're going to do.
    Language Models Don't Offer Mundane Utility. Universal attacks on LLMs.
    Fun With Image Generation. Videos might be a while.
    Deepfaketown and Botpocalypse Soon. An example of doing it right.
    They Took Our Jobs. What, me worry?
    Get Involved. If you share more opportunities in comments I'll include next week.
    Introducing. A bill. Also an AI medical generalist.
    In Other AI News. Fine tuning is coming to GPT-4. Teach LLMs arithmetic.
    Quiet Speculations. Various degrees of skepticism.
    China. Do not get overexcited.
    The Quest for Sane Regulation. Liability and other proposed interventions.
    The Week in Audio. I go back to The Cognitive Revolution.
    Rhetorical Innovation. Bill Burr is concerned and might go off on a rant.
    No One Would Be So Stupid As To. Robotics and AI souls.
    Aligning a Smarter Than Human Intelligence is Difficult. RLHF deep dive.
    Other People Are Not As Worried About AI Killing Everyone. It'll be fine.
    The Wit and Wisdom of Sam Altman. Don't sleep on this.
    The Lighter Side. Pivot!
    Language Models Offer Mundane Utility
    Make a ransom call, no jailbreak needed. Follows the traditional phone-calls-you-make-are-your-problem-sir legal principle. This has now been (at least narrowly, for this particular application) fixed or broken, depending on your perspective.
    See the FAQ:
    Back in AI#3 we were first introduced to keeper.ai, the site that claims to use AI to hook you up with a perfect match that meets all criteria for both parties so you can get married and start a family, where if you sign up for the Legacy plan they only gets paid when you tie the knot. They claim 1 in 3 dates from Keeper lead to a long term relationship. Aella has now signed up, so we will get to see it put to the test.
    Word on Twitter is the default cost for the keeper service is $50k. If it actually works, that is a bargain. If it doesn't, depends if you have to deposit in advance, most such startups fail and that is a lot to put into escrow without full confidence you'll get it back.
    I continue to think that this service is a great idea if you can get critical mass and make reasonable decisions, while also not seeing it as all that AI. From what I can tell the AI is used to identify potential matches for humans to look at, but it is not clear to me (A) how any match can ever truly be 100%, I have never seen one, not everything is a must-have and (B) how useful an AI is here when you need full reliability, you still need the humans to examine everything and I'd still instinctively want to mostly abstract things into databases? Some negative selection should be useful in saving time, but that seems like about it?
    An early experiment using LLMs as part of a recommendation algorithm. LLMs seem useful for getting more useful and accurate descriptor labels of potential content, and could potentially help give recommendations that match arbitrary user criteria. Or potentially one could use this to infer user criteria, including based on text-based feedback. I am confident LLMs are the key to the future of recommendation engines, b...
    1 hr 2 min
  • AF - Password-locked models: a stress case for capabilities evaluation by Fabien Roger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Password-locked models: a stress case for capabilities evaluation, published by Fabien Roger on August 3, 2023 on The AI Alignment Forum.
    Thanks to Buck Shlegeris, Ryan Greenblatt, Kshitij Sachan and Jenny Nitishinskaya for helpful discussions and feedback on drafts. This work was done at Redwood Research. The views expressed are my own and do not necessarily reflect the views of the organization.
    Password-locked models are trained to exhibit certain capabilities only when a password is present in the query. Studying these models has two purposes:
    Testing how well capability evaluations work when applied to models which "aren't trying"
    Understanding how well organizations could hide the capabilities of their models.
    In this post, I make empirical claims which I believe to be true, check some of them on small language models, and describe what this implies for the control of potentially deceptive models.
    Key empirical claims: when using behavioral model evaluations, password-locked models:
    Look much less capable than they actually are when eliciting capabilities with prompting
    Sometimes look much less capable than they actually are when eliciting capabilities with Reinforcement Learning
    Experimental evidence: I exhibit one case where they don't, and one case where they do
    Can't look much less capable than they actually when eliciting capabilities with supervised fine-tuning on high-quality demonstrations
    Key takeaways, if the claims above are true:
    If an alignment plan relies on some models being too weak to perform certain tasks, then model evaluations aimed at checking the weakness of models should be able to elicit the locked capabilities of password-locked models. This is because password-locked models and deceptive models may be able to underperform in similar circumstances.
    API access and inference is not enough to evaluate capabilities of a model produced by AI labs trying to game regulations, even if AI labs are internally using the same API endpoint as auditors.
    Key Empirical Claims
    Similarities Between Password-locked Models and Deceptively Aligned Models Pretending to Be Dumb
    It looks to me that password locked models have the same properties as the one described by Evan in When can we trust model evaluations? when he describes if and how deceptive models would be able to sandbag each category of behavioral evaluation (which he coins in his post):
    Behavioral Non-Fine-Tuning Evaluations: By construction, password-locked models will refuse to be capable no matter the prompt, except for prompts with the special password. But it's unclear how easy it is to build good password-locked models which are hard to crack (e.g. which don't leak the password with some clever prompting). Even if good password-locked models are built, there remain a large difference with deceptive models: the search for the correct password might have very different properties than the search for a prompt which makes a deceptive model believe it's time to strike.
    Behavioral I.I.D. Fine-Tuning Evaluations: I would be surprised if fine-tuning the password-locked model on high quality answers didn't lift the locking mechanism extremely quickly. This is similar to deceptive models (except if they are so strong that they can gradient hack). The main difference is that fine-tuning to remove password-locking on one task might transfer to another task, which might not be the case for deceptive models.
    Behavioral RL Fine-Tuning Evaluations: The success of evaluations which use Reinforcement Learning depends, as for deceptive models, on the success of exploration. If supervised fine-tuning is done with only low quality answers, then the password locked models will just continue to output low-quality answers and will never explore its way into high quality ones - except if there is a natural "path" g...
    14 min
  • EA - Alignment Grantmaking is Funding-Limited Right Now [crosspost] by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Grantmaking is Funding-Limited Right Now [crosspost], published by johnswentworth on August 3, 2023 on The Effective Altruism Forum.
    For the past few years, I've generally mostly heard from alignment grantmakers that they're bottlenecked by projects/people they want to fund, not by amount of money. Grantmakers generally had no trouble funding the projects/people they found object-level promising, with money left over. In that environment, figuring out how to turn marginal dollars into new promising researchers/projects - e.g. by finding useful recruitment channels or designing useful training programs - was a major problem.
    Within the past month or two, that situation has reversed. My understanding is that alignment grantmaking is now mostly funding-bottlenecked. This is mostly based on word-of-mouth, but for instance, I heard that the recent lightspeed grants round received far more applications than they could fund which passed the bar for basic promising-ness. I've also heard that the Long-Term Future Fund (which funded my current grant) now has insufficient money for all the grants they'd like to fund.
    I don't know whether this is a temporary phenomenon, or longer-term. Alignment research has gone mainstream, so we should expect both more researchers interested and more funders interested. It may be that the researchers pivot a bit faster, but funders will catch up later. Or, it may be that the funding bottleneck becomes the new normal. Regardless, it seems like grantmaking is at least funding-bottlenecked right now.
    Some takeaways:
    If you have a big pile of money and would like to help, but haven't been donating much to alignment because the field wasn't money constrained, now is your time!
    If this situation is the new normal, then earning-to-give for alignment may look like a more useful option again. That said, at this point committing to an earning-to-give path would be a bet on this situation being the new normal.
    Grants for upskilling, training junior people, and recruitment make a lot less sense right now from grantmakers' perspective.
    For those applying for grants, asking for less money might make you more likely to be funded. (Historically, grantmakers consistently tell me that most people ask for less money than they should; I don't know whether that will change going forward, but now is an unusually probable time for it to change.)
    Note that I am not a grantmaker, I'm just passing on what I hear from grantmakers in casual conversation. If anyone with more knowledge wants to chime in, I'd appreciate it.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • EA - How many people are neartermist and have high P(doom)? by Sanjay
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How many people are neartermist and have high P(doom)?, published by Sanjay on August 2, 2023 on The Effective Altruism Forum.
    As far as I can tell, there seems to be a strong tendency for those who are worried about AI risk to also be longtermists.
    However many elements of the philosophical case for longtermism are independent of contingent facts about what is going to happen with AI in the coming decades.
    If we have good community epistemic health, we should expect there to be people who object to longtermism on grounds like:
    person-affecting views
    supporting a non-zero pure discount rate
    but who still are just as worried about AI as those with P(doom) > 90%.
    Indeed, the proportion of "doomers" with those philosophical objections to longtermism should be just as high as the rate of such philosophical objections among those typically considered neartermist.
    I'm interested in answers either of the form:
    "hello, I'm both neartermist and have high P(doom) from AI risk..."; or
    "here's some relevant data, from, say, the EA survey, or whatever"
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    2 min
  • AF - 3 levels of threat obfuscation by HoldenKarnofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 3 levels of threat obfuscation, published by HoldenKarnofsky on August 2, 2023 on The AI Alignment Forum.
    One of the biggest reasons alignment might be hard is what I'll call threat obfuscation: various dynamics that might make it hard to measure/notice cases where an AI system has problematic misalignment (even when the AI system is in a controlled environment and one is looking for signs of misalignment). I write more about this general fear in AI Safety Seems Hard to Measure.
    I think it's very important to understand threat obfuscation dynamics; to try to design ways of assessing danger that are designed with threat obfuscation dynamics in mind; and to avoid interpreting reassuring-seeming evidence too optimistically, by keeping in mind the various threat obfuscation dynamics that could affect it.
    I've seen three noticeably different threat obfuscation dynamics discussed, in ways that often leave me confused about exactly what dynamic someone is talking about, and when. This post aims to clear that up, based on a conversation with Paul Christiano.
    The three dynamics are:
    Simple "training game" dynamics: an AI system appears to behave helpfully, rather than attempting deception/manipulation/disempowerment of humans, because (a) it is good at predicting whether such attempts will be caught; (b) it is aiming for good in-episode performance (in terms of loss or reward); (c) such attempts would result in poor in-episode performance if caught. But if an AI system had a chance to do deception/manipulation/disempowerment that would not be detected, and would result in "good performance" within an episode, it would do so (this could include large-scale world takeover).
    These dynamics are dangerous because we might observe dangerous behavior, train AIs to behave less dangerously, then observe less dangerous behavior, and conclude that we've become more safe (when in fact we've trained the AIs to be more effective at deception).
    But we can still detect dangerous behavior in cases where it would not result in poor in-episode performance. For example, we might decide that when an AI system in the wild hacks its way to good in-episode performance, and is caught after the fact, we will at that point act as if the performance was actually good/intended (for example, no gradient descent against the behavior). Or we might deliberately conduct tests in which AI systems have opportunities to take over digital environments, obtaining high reward (no catch, not a trap, no attempt to "train out" the behavior when it happens).1 In these cases, situationally aware AI systems might "show their true colors" with fairly straightforward attempts to draw them out.
    Deceptive alignment: an AI system forms some aim (via inner misalignment) other than good in-episode performance. (This could include something like "an AI system that wants to maximize digital representations of reward across the galaxy, and is willing to get poor in-episode performance as part of a longer-term mission for that to happen.") It then gets good in-episode performance as part of a longer-term scheme to accomplish its other aim.
    In this case, the above techniques for getting an AI to "show its true colors" might2 fail.
    However, we could still get an AI to engage in specific behaviors that might be helpful to us and harmful to its goals, by specifically training the AI to behave in these ways. For example, we might want an AI to find security vulnerabilities that we can then patch. Even if doing so would undermine the AI's longer-term aim, we can use gradient descent to put a lot of pressure on it to find security vulnerabilities anyway. (Related: the "no sandbagging on checkable tasks" hypothesis)
    Gradient hacking: this also requires that an AI system has some aim other than good in-episode performance. Gradient hacking ...
    11 min