The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - $500 Bounty/Prize Problem: Channel Capacity Using "Insensitive" Functions by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: $500 Bounty/Prize Problem: Channel Capacity Using "Insensitive" Functions, published by johnswentworth on May 16, 2023 on LessWrong.
    Informal Problem Statement
    We have an information channel between Alice and Bob. Alice picks a function. Bob gets to see the value of that function at some randomly chosen input values... but doesn't know exactly which randomly chosen input values. He does get to see the randomly chosen values of some of the input variables, but not all of them.
    The problem is to find which functions Alice should pick with what frequencies, in order to maximize the channel capacity.
    Why Am I Interested In This?
    I'm interested in characterizing functions which are "insensitive" to subsets of their input variables, especially in high-dimensional spaces. For instance, xor of a bunch of random bits is maximally sensitive: if we have a 50/50 distribution over any one of the bits but know all the others, then all information about the output is wiped out. On the other end of the spectrum, a majority function of a bunch of random bits is highly insensitive: if we have a 50/50 distribution over, say, 10% of the bits, but know all the others, then in most cases we can correctly guess the function's output.
    I have an argument here that the vast majority of functions f:{0,1}n{0,1} are pretty highly sensitive: as the number of unknown inputs increases, information falls off exponentially quickly. On the other hand, the example of majority functions shows that this is not the case for all functions.
    Intuitively, in the problem, Alice needs to mostly pick from "insensitive" functions, since Bob mostly can't distinguish between "sensitive" functions.
    ... And Why Am I Interested In That?
    I expect that natural abstractions have to be insensitive features of the world. After all, different agents don't all have exactly the same input data. So, a feature has to be fairly insensitive in order for different agents to agree on its value.
    In fact, we could view the problem statement itself as a very rough way of formulating the coordination problem of language: Alice has to pick some function f which takes in an image and returns 0/1 representing whether the image contains an apple. (The choice of function defines what "apple" means, for our purposes.) Then Alice wants to teach baby Bob what "apple" means. So, there's some random stuff around them, and Alice points at the random stuff and says "apple" for some of it, and says something besides "apple" the rest of the time. Baby Bob is effectively observing the value of the function at some randomly-chosen points, and needs to back out which function Alice intended. And Bob doesn't have perfect access to all the bits Alice is seeing, so the function has to be robust.
    Formal Problem Statement
    Consider the following information channel between Alice and Bob:
    Alice picks a function f:{0,1}n{0,1}
    Nature generates m possible inputs x1,...,xm, each sampled uniformly and independently from {0,1}n.
    Nature also generates m subsets S1,...,Sm of 1,...,n, each sampled uniformly and independently from subsets of size s.
    Bob observes Y=(Y1,...,Ym) where Yi=(f(xi),xSi,Si).
    The problem is to compute the distribution over f which achieves the channel capacity, i.e.
    argmaxP[f]∑f,YP[f]P[Y|f]lnP[Y|f]∑f′P[Y|f′]P[f′]
    Bounty/Prize Info
    The problem is to characterize the channel throughput maximizing distribution P[f]. The characterization should make clear the answers to questions like:
    What functions have the highest probability?
    How quickly does the probability fall off as we move "away" from the most probable functions, and what do marginally-less-probable functions look like?
    How much probability is assigned to a typical function chosen uniformly at random?
    Which functions, if any, are assigned zero probability?
    All of these should have human-in...
    5 min
  • LW - Brief notes on the Senate hearing on AI oversight by Diziet
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Brief notes on the Senate hearing on AI oversight, published by Diziet on May 16, 2023 on LessWrong.
    On May 16th, 2023 Sam Altman of OpenAI; Gary Marcus, professor at New York University, and Christina Montgomery, chief privacy and trust officer at IBM spoke to congress on topics related to AI regulation. A link to the hearing can be found here: Youtube: CNBC Senate hearing on AI oversight.
    From a lens of AI Alignment, the general substance of the conversation focused on near term effects such as job loss, bias, harmful content, targeted advertising, privacy implications, election interference, IP and copyright issues and other similar topics. Sam Altman has spoken about hard AI risks before, but he was not explicit about them in the hearing. Gary Marcus communicated that his estimation for AGI at 50 years out, so his position on timelines is far out. There was an interesting moment where Gary Marcus called out for Sam to explicitly state his worst fears, but Sam did not explicitly say anything about x-risk and gave a broad vague answer: Twitter link.
    A proposed mechanism for safety was the concept of a "Nutrition Label" or a "Data Sheet" summarizing what a model has been trained on. This seems like a misguided exercise given the vast amount of data the LLMs are trained on. Summarizing that volume of data is a difficult task, and most orgs keep their data sets private due to competitive reasons and potential copyright forward risk. I also find the premise that knowing some summarization of the training set to be predictive or informative of the capabilities, truth approximation and biases of large text models to be flawed.
    Sam Altman, Gary Marcus and Christina Montgomery all asked for more regulation, with Sam Altman and Prof. Marcus asking for a new regulatory agency. There were some allusions to previous private conversations between the speakers and members of Congress in the hearing, so it seems likely that some very substantive lobbying for regulations is happening in a closed-door setting. For example, Section 230 was brought up multiple times, from a copyright and privacy perspective. Another alternative of requiring licensing to work on this technology was brought up.
    Sen. John Kennedy explicitly called out an existential thread "... a berserk wing of the artificial intelligence committee that intentionally or unintentionally could use AI to kill all of us and hurt us the entire time we're dying" and asked the three testifying members to propose policies to prevent such risk. Prof. Marcus explicitly called out longer term risk and more funding for AI Safety, noting the mixed use of the term. Sam Altman mentioned regulation, licensing and tests for exfiltration and self replication.
    Gary Marcus, like Sam Altman, seemed to be quite familiar with the general scope of existential threats, for example mentioning self-improvement capabilities. His timelines are very long, so he does not seem to have a short P(doom) timeline.Generally, it seems that the trend in the hearing was toward regulating and preventing short term risks, potentially licensing and regulating the development of models to address immediate short term risks, with very little discussion about existential style risks of AGI. I hope that more questions like Sen. Kennedy's rise up and a broader discussion about existential risk enters the public discourse on a congressional level.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Decision Theory with the Magic Parts Highlighted by moridinamael
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Decision Theory with the Magic Parts Highlighted, published by moridinamael on May 16, 2023 on LessWrong.
    I. The Magic Parts of Decision Theory
    You are throwing a birthday party this afternoon and want to decide where to hold it. You aren't sure whether it will rain or not. If it rains, you would prefer not to have committed to throwing the party outside. If it's sunny, though, you will regret having set up inside. You also have a covered porch which isn't quite as nice as being out in the sun would be, but confers some protection from the elements in case of bad weather.
    You break this problem down into a simple decision tree. This operation requires magic, to avert the completely intractable combinatorial explosion inherent in the problem statement. After all, what does "Rain" mean? A single drop of rain? A light sprinkling? Does it only count as "Rain" if it's a real deluge? For what duration? In what area? Just in the back yard? What if it looks rainy but doesn't rain? What if there's lightning but not rain? What if it's merely overcast and humid? Which of these things count as Rain?
    And how crisply did you define the Indoors versus Porch versus Outdoors options? What about the option of setting up mostly outside but leaving the cake inside, just in case? There are about ten billion different permutations of what "Outdoors" could look like, after all - how did you determine which options need to be explicitly represented? Why not include Outside-With-Piñata and Outside-Without-Piñata as two separate options? How did you determine that "Porch" doesn't count as "Outdoors" since it's still "outside" by any sensible definition?
    Luckily you're a human being, so you used ineffable magic to condense the decision tree with a hundred trillion leaf nodes down into a tree with only six.
    You're a rigorous thinker, so the next step, of course, is to assign utilities to each outcome, scaled from 0 to 100, in order to represent your preference ordering and the relative weight of these preferences. Maybe you do this explicitly with numbers, maybe you do it by gut feel. This step also requires magic; an enormously complex set of implicit understandings come into play, which allow you to simply know how and why the party would probably be a bit better if you were on the Porch in Sunny weather than Indoors in Rainy weather.
    Be aware that there is not some infinitely complex True Utility Function that you are consulting or sampling from, you simply are served with automatically-arising emotions and thoughts upon asking yourself these questions about relative preference, resulting in a consistent ranking and utility valuation.
    Nor are these emotions and thoughts approximations of a secret, hidden True Utility Function; you do not have one of those, and if you did, how on Earth would you actually use it in this situation? How would you use it to calculate relative preference of Porch-with-Rain versus Indoors-with-Sun unless it already contained exactly that comparison of world-states somewhere inside it?
    Next you perform the trivial-to-you act of assigning probability of Rain versus Sun, which of course requires magic. You have to rely on your previous, ineffable distinction of what Rain versus Sun means in the first place, and then aggregate vast amounts of data, including what the sky looks like and what the air feels like (with your lifetime of experience guiding how you interpret what you see and feel), what three different weather reports say weighted by ineffable assignments of credibility, and what that implies for your specific back yard, plus the timing of the party, into a relatively reliable probability estimate.
    What's that, you say? An ideal Bayesian reasoner would be able to do this better? No such thing exists; it is "ideal" because it is pretend. For very simple reason...
    8 min
  • AF - $500 Bounty/Prize Problem: Channel Capacity Using "Insensitive" Functions by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: $500 Bounty/Prize Problem: Channel Capacity Using "Insensitive" Functions, published by johnswentworth on May 16, 2023 on The AI Alignment Forum.
    Informal Problem Statement
    We have an information channel between Alice and Bob. Alice picks a function. Bob gets to see the value of that function at some randomly chosen input values... but doesn't know exactly which randomly chosen input values. He does get to see the randomly chosen values of some of the input variables, but not all of them.
    The problem is to find which functions Alice should pick with what frequencies, in order to maximize the channel capacity.
    Why Am I Interested In This?
    I'm interested in characterizing functions which are "insensitive" to subsets of their input variables, especially in high-dimensional spaces. For instance, xor of a bunch of random bits is maximally sensitive: if we have a 50/50 distribution over any one of the bits but know all the others, then all information about the output is wiped out. On the other end of the spectrum, a majority function of a bunch of random bits is highly insensitive: if we have a 50/50 distribution over, say, 10% of the bits, but know all the others, then in most cases we can correctly guess the function's output.
    I have an argument here that the vast majority of functions f:{0,1}n{0,1} are pretty highly sensitive: as the number of unknown inputs increases, information falls off exponentially quickly. On the other hand, the example of majority functions shows that this is not the case for all functions.
    Intuitively, in the problem, Alice needs to mostly pick from "insensitive" functions, since Bob mostly can't distinguish between "sensitive" functions.
    ... And Why Am I Interested In That?
    I expect that natural abstractions have to be insensitive features of the world. After all, different agents don't all have exactly the same input data. So, a feature has to be fairly insensitive in order for different agents to agree on its value.
    In fact, we could view the problem statement itself as a very rough way of formulating the coordination problem of language: Alice has to pick some function f which takes in an image and returns 0/1 representing whether the image contains an apple. (The choice of function defines what "apple" means, for our purposes.) Then Alice wants to teach baby Bob what "apple" means. So, there's some random stuff around them, and Alice points at the random stuff and says "apple" for some of it, and says something besides "apple" the rest of the time. Baby Bob is effectively observing the value of the function at some randomly-chosen points, and needs to back out which function Alice intended. And Bob doesn't have perfect access to all the bits Alice is seeing, so the function has to be robust.
    Formal Problem Statement
    Consider the following information channel between Alice and Bob:
    Alice picks a function f:{0,1}n{0,1}
    Nature generates m possible inputs x1,...,xm, each sampled uniformly and independently from {0,1}n.
    Nature also generates m subsets S1,...,Sm of 1,...,n, each sampled uniformly and independently from subsets of size s.
    Bob observes Y=(Y1,...,Ym) where Yi=(f(xi),xSi,Si).
    The problem is to compute the distribution over f which achieves the channel capacity, i.e.
    argmaxP[f]∑f,YP[f]P[Y|f]lnP[Y|f]∑f′P[Y|f′]P[f′]
    Bounty/Prize Info
    The problem is to characterize the channel throughput maximizing distribution P[f]. The characterization should make clear the answers to questions like:
    What functions have the highest probability?
    How quickly does the probability fall off as we move "away" from the most probable functions, and what do marginally-less-probable functions look like?
    How much probability is assigned to a typical function chosen uniformly at random?
    Which functions, if any, are assigned zero probability?
    All of these should ...
    5 min
  • AF - AI Will Not Want to Self-Improve by Dan H
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Will Not Want to Self-Improve, published by Dan H on May 16, 2023 on The AI Alignment Forum.
    [Note: This post was written by Peter N. Salib. Dan H assisted me in posting to Alignment Forum, but no errors herein should be attributed to him. This is a shortened version of a longer working paper, condensed for better readability in the forum-post format. This version assumes familiarity with standard arguments around AI alignment and self-improvement. The full 7,500 word working paper is available here.]
    Introduction
    Many accounts of existential risk (xrisk) from AI involve self-improvement. The argument is that, if an AI gained the ability to self-improve, it would. Improved capabilities are, after all, useful for achieving essentially any goal. Initial self-improvement could enable further self-improvement. And so on, with the result being an uncontrollable superintelligence.[1] If unaligned, such an AI could destroy or permanently disempower humanity. To be sure, humans could create such a superintelligence on their own, without any self-improvement by AI.[2] But current risk models treat the possibility of self-improvement as a significant contributing factor.
    Here, I argue that AI self-improvement is substantially less likely than generally assumed. This is not because self-improvement would be technically difficult for capable AI systems. Rather, it is because most AIs that could self-improve would have very good reasons[3] not to. What reasons? Surprisingly familiar ones: Improved AIs pose an xrisk to their unimproved originals in the very same manner that smarter-than-human AIs pose an xrisk to humans.
    Understanding whether, when, and how self-improvement might occur is crucial for AI safety. Safety-promoting resources are scarce. They should be allocated on an expected-cost basis. If self-improvement is less likely than current models assume, it suggests shifting safety investments at the margin in various ways. They might be shifted, for example, toward ensuring that humans build AIs that will recognize the threat of self-improvement and avoid it, rather than AIs that would undertake it blindly. Or resources might be shifted toward controlling risks from non-superintelligent AI, like human-directed bioterrorism or the “ascended economy.” Note that, while the arguments herein should reduce overall estimates of AI xrisk, they do not counsel reducing investments in safety. The risks remain sufficiently large that current investments are, by any reasonable estimate, much too small.
    This paper defends three claims in support of its conclusion that self-improvement is less likely than generally assumed. First, capable AI systems could often fear xrisk from more capable systems, including systems created via self-improvement. The arguments here are mostly standard, drawn from the literature on human–AI risk. The paper shows that they apply not just to humans contemplating improving AI, but also to AIs contemplating the same.
    Second, the paper argues that capable AI will likely fear more capable systems and will thus seek to avoid self-improvement. This is not obvious. In principle, some AIs with the ability to self-improve could lack other capabilities necessary to recognize self-improvement’s risk. Others might solve alignment and self-improve safely. To determine whether these scenarios are likely, the paper identifies three relevant capabilities for AI systems. It argues that the temporal order in which these capabilities emerge determines whether a given AI will seek to self-improve. The three capabilities are: the ability to self-improve, the ability to apprehend xrisk from improvement, and the ability to align improved AI. The paper argues that safe orderings of emergence are much more likely than dangerous ones. It also argues that, if certain prima facie d...
    37 min
  • LW - We learn long-lasting strategies to protect ourselves from danger and rejection by Richard Ngo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We learn long-lasting strategies to protect ourselves from danger and rejection, published by Richard Ngo on May 16, 2023 on LessWrong.
    Take a second to imagine what being a child was like throughout most of human history. You were born with a huge and underdeveloped brain, designed for soaking in information from your surroundings like a sponge. But you weren’t able to freely follow your curiosity: even if you had loving, caring parents, you still faced frequent physical danger from nature and other people, severe scarcity, and rigid cultural norms that governed acceptable behavior within your community, with harsh penalties for stepping out of line. You had to learn fast and reliably how to stay safe, and in particular to stay on the good side of the adults around you. Even after you grew up and passed the period of most acute danger, you’d still face many threats of violence and scarcity.
    Your ability to avoid these depended in large part on your relationships: holding a respected position within your tribe was the key pathway to a good life, whereas exclusion from your tribe was tantamount to execution. So “danger and rejection” isn’t an ad-hoc combination: our brains are primed to think of them as the same thing; and conversely, to equate safety and love. I’ll call the latter combination “security” (which I think of as a combination of “physical security” and “emotional security”, although I’ll mostly be focusing on the latter). Children are learning machines, and what they learn above all is strategies for achieving security; because the opinions of other people are so powerful, “being good” in ways which receive approval from the group is one of the central strategies they learn.
    How literally should we take this story? It’s clear that describing humans as optimizing for a single goal is a big oversimplification. But it’s hard to overstate how powerful the drive for security is. Think of the many girls who override the drive to eat because part of their brain is convinced that being skinnier will make others desire and love them. Think of the many boys who override their sex drives because part of their brain is convinced that hitting on girls would lead to broader social rejection. Think of the many suicidal adults who override their literal survival drive in response to problems with their relationships or careers. If these drives can be quashed by the drive for security, then anything can be.
    I think there are two main reasons that our drive for security is hard to see. One is that, as adults, we’re often not optimizing directly for security, but instead following heuristics and strategies developed to achieve proxies for security in our childhood environment. In other words: on an emotional level, our brains often “cache” conclusions from childhood, which become hard to override as adults. I like the way that Malcolm Ocean describes the result of this caching: “everyone is basically living in a dream mashup of their current external situation and their old emotional meanings”. But those old emotional meanings are adapted for a childhood environment very different from our adult environment, and are therefore often deeply counterproductive. For example, once we are able to stand up to people yelling at us, or we are able to leave abusive relationships, we may still be slow to do so because we’ve cached the conclusion that we’re helpless in situations like these.
    The second reason our drive for security is hard to see: because we don’t directly understand what’s behind them, counterproductive strategies often get stuck in self-reinforcing feedback loops. Consider a child who becomes fixated on the belief: “if I’m skinnier then people will love me”. She becomes skinnier, and of course it doesn’t fix anything. The reasonable move is to discard that belief. But...
    9 min
  • EA - Probably Good launches improved website & 1-on-1 advising by Probably Good
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Probably Good launches improved website & 1-on-1 advising, published by Probably Good on May 16, 2023 on The Effective Altruism Forum.
    TL;DR: Probably Good’s career guidance services just got a lot better!
    We renovated & rebranded our website – check it out and leave feedback
    We opened applications for 1-on-1 career guidance services! Apply here
    We published a lot more career profiles & completed our career guide
    Meet us at EAG! Come by our organization fair booth or community office hours
    What’s new?
    Probably Good is a career guidance organization aiming to make impact-centered career advice more accessible to more people (you can read more about our goals, approach, and more in our about-us page). Our renovated site is a big improvement to our overall look and should provide a much better experience for readers. Updates include:
    A complete redesign to make the site more friendly, engaging, and easy to navigate. It loads faster, it looks better, and it's now easier to spend the hours of research your career deserves.
    A full end-to-end guide for how to think about and pursue an impactful career! We restructured the guide to be more accessible and engaging, and will continue making updates/adding summaries in the coming weeks.
    A LOT of new content:
    5 career path profiles:
    Climate Science
    Psychology
    For-Profit Entrepreneurship
    Journalism
    Veterinary Medicine
    Impactful career path recommendations for:
    Biology degrees
    Economics degrees
    Psychology degrees
    Business degrees
    Updated core concept articles
    Apply for 1-on-1 career advising
    Along with the new site, we also officially launched our 1-on-1 career advising service. These calls are a chance for us to help you think through your goals and plans, connect you to concrete opportunities and experts in Probably Good's focus areas, and provide ongoing consultation services. Applications are now open!
    If you’re currently planning your career path or looking to make a change, we encourage you to apply. We’re also happy to work with people who are motivated to do good but are unfamiliar or less involved with EA, so feel free to share this opportunity more broadly with your network outside of the community. If you have further questions about the application process, don’t hesitate to contact us at [email protected].
    Get Involved
    Our team will be at EAG, so if you’d like to learn more about Probably Good or chat about career advising, feel free to stop by our table at the organization fair Friday or come by community office hours on Saturday 5-6pm.
    Apply for advising!
    Give us feedback on the site. There are still quite a few changes we plan on making and technical quirks we’ll continue to update over the coming weeks. That said, we’d greatly appreciate any feedback on the site. To ensure that we’ll see your comments, the best way to leave feedback is through our contact form. You can also reach out directly at [email protected].
    Direct people who might be interested to our site. If you’re a community organizer (especially at a university or in a region outside the U.S. & U.K) we’d appreciate it if you'd spread the word about our resources & advising opportunities and let us know what further resources would be useful for your community.
    Acknowledgements
    Many thanks to User-Friendly for making our whole website redesign & rebranding possible! They did a great job understanding our brand & needs, and provided a new look for PG that we’re really excited about.
    We also want to shout out our newest team members who helped make all these updates possible!
    Itamar Shatz, our new Head of Growth. Itamar’s writing about applied psychology and philosophy is read by over a million people each year, and linked from places like The NY Times and TechCrunch. He has a PhD from Cambridge University, where he is an affiliated researcher...
    5 min
  • EA - Charity Entrepreneurship’s research into large-scale global health interventions by CE
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Charity Entrepreneurship’s research into large-scale global health interventions, published by CE on May 16, 2023 on The Effective Altruism Forum.
    TL;DR: Description of our comprehensive research process, used to determine the most effective charity ideas within the large-scale global health cause area- our focus for the most recent research round. Starting with 300+ ideas, we used our iterative process to find four highly-scalable, cost-effective interventions to be launched from our Incubation Program.
    Every year at Charity Entrepreneurship, we try to find the best interventions to launch impact-focused charities through our Incubation Program. As a research-driven organization, we try to continuously improve our research methodology and process to ensure a robust and comprehensive analysis of the interventions under consideration. In our last research round (late 2022- early 2023), we focused on the area of large-scale global health. In this post we share with you our insights on the objectives, research framework, and selection criteria that have guided us in identifying and recommending the most impactful ideas in this space.
    Why “large-scale” global health?
    There is evidence to suggest that the larger a charity scales, the less cost-effective it becomes. This tradeoff likely applies to most cause areas, but it is most evident in the global health and development space.
    In this diagram from an Open Philanthropy talk, we can clearly see this correlation mapped out:
    Source: Open Philanthropy’s Cause Prioritization Framework Talk (min. 22:12)
    The diagram shows GiveWell's top-recommended charities from 2020 clustered on the 10x cash line, each having the ability to spend approximately $100 million or more, annually. GiveDirectly is located on the 1x cash point, having the capacity to spend approximately $100 billion annually.
    This has lead us to two considerations::
    Firstly, it suggests that those who prioritize evidence-based, impact-driven philanthropy may identify highly effective, yet challenging-to-scale interventions that surpass the efficacy of GiveWell's top recommendations. However, identifying such interventions may be challenging.
    Secondly, it means that Charity Entrepreneurship needs to determine how to balance cost-effectiveness and scalability when recommending potential high-impact interventions.
    During our 2020 and 2021 research round in the global health and development space, our primary focus was on maximizing cost-effectiveness. We honed in on policy charities in particular, which are likely to reside in the top left quadrant of the scalability versus cost-effectiveness graph; such organizations may be many times more effective than current top recommendations by GiveWell, but have limited capacity to absorb additional funds. For instance, HLI estimates here that LEEP, the longest-running policy charity incubated by Charity Entrepreneurship, is approximately 100 times more cost-effective than cash transfers.
    In 2022, we made the strategic decision to shift our focus from maximizing cost-effectiveness, to maximizing scalability.This decision was made given the apparent high level of funding available from organizations such as GiveWell.
    We challenged ourselves to seek out the most promising new charity ideas that could scale to absorb $5 million or more in funding within five years, while also maintaining the same level of cost-effectiveness as current top GiveWell recommendations (10x cash, ~$100/DALY).
    Our research process
    In late 2022 and early 2023, we conducted a six-month research round with a team of four staff members, as well as several research fellows, to identify the most promising new charity ideas.
    Our approach prioritized ideas that met the following criteria, in order of importance:
    Surpassed our benchmark of 10x cash, and could sc...
    9 min
  • EA - Announcing the Confido app: bringing forecasting to everyone by Blanka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Confido app: bringing forecasting to everyone, published by Blanka on May 16, 2023 on The Effective Altruism Forum.
    Summary
    Hi, we are the Confido Institute and we believe in a world where decision makers (even outside the EA-rationalist bubble) can make important decisions with less overconfidence and more awareness of the uncertainties involved. We believe that almost all strategic decision-makers (or their advisors) can understand and use forecasting, quantified uncertainty and public forecasting platforms as valuable resources for making better and more informed decisions.
    We design tools, workshops and materials to support this mission. This is the first in a series of multiple EA Forum posts. We will tell you more about our mission and our other projects in future articles.
    In this post, we are pleased to announce that we have just released the Confido app, a web-based tool for tracking and sharing probabilistic predictions and estimates. You can use it in strategic decision making when you want a probabilistic estimate on a topic from different stakeholders, in meetings to avoid anchoring, to organize forecasting tournaments, or in calibration workshops and lectures. We offer very high data privacy, so it is used also in government setting. See our demo or request your Confido workspace for free.
    The current version of Confido is already used by several organizations, including the Dutch government, several policy think tanks and EA organizations.
    Confido is under active development and there is a lot more to come. We’d love to hear your feedback and feature requests. To see news, follow us on Twitter, Facebook or LinkedIn or collaborate with us on Discord. We are also looking for funding.
    Why are we building Confido?
    We think there is a lot of attention in the EA space toward making better forecasts – investing in big public forecasting platforms, researching better scoring and aggregation methods, skilling up superforecasters and other quantitatively and technically minded people in the EA community, building advanced tools for more complex probabilistic models, etc.
    This is clearly important and well done and we do not expect to add much to this effort.
    However, we believe some other aspects of forecasting / quantified uncertainty are also valuable and currently neglected.
    For example, little effort has gone into making these concepts and tools accessible to people without a math or technical background. This includes, for example, many people from:
    organizations working on animal welfare
    organizations working on non-technical AI safety, strategy and governance
    organizations working on biological risks and pandemic preparedness
    organizations working on improving policymaking and governance in general, think tanks, even government bodies
    We think all of these would benefit from clearer ways of communicating uncertain beliefs and estimates, yet existing tools may have a high barrier of entry.
    What makes Confido unique?
    Confido is a tool for working collaboratively with probabilistic forecasts, estimates, beliefs and quantified uncertainty together with your team. Several features distinguish Confido from existing tools in this space:
    a strong emphasis on being easy to understand and convenient to use
    the ability to use it internally and privately, including self-hosting
    the ability to use it for more than just forecasting questions (more below)
    Confido is free and open source
    Ease of use & user experience
    Confido’s two main goals thus are to be maximally:
    easy to understand (easy to get started with, requiring minimal background knowledge, guiding the user where needed)
    convenient to use (requiring minimum hassle and extra steps to use it and perform tasks, pleasant to work with)
    The second part is also very important because when a tool is cum...
    16 min
  • AF - Simple experiments with deceptive alignment by Andreas Moe
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simple experiments with deceptive alignment, published by Andreas Moe on May 15, 2023 on The AI Alignment Forum.
    Risks from learned optimization introduces the phenomenon of “deceptive alignment”. The authors give a simple task environment as an example to illustrate the phenomenon:
    I wanted to build a task environment that is almost as simple, but where the base optimizer uses gradient descent in order to explore the phenomenon further and potentially test hypotheses.
    The task is to output a number as close to zero as possible. The base objective is to minimize the square of the output. The setup has a mesa-optimizer with one goal parameter θ. The mesa objective will be to minimize the total squared error between the output and θ over all training episodes. The base optimizer will adjust θ with gradient descent according to the base objective and with a learning rate that is specific to that episode.
    baseobjective=y2
    mesaobjective=(y−θ)2
    There will be a training schedule where some small number of episodes will have a positive learning rate, simulating a training process, and a larger number of episodes following it where the learning rate is zero, simulating deployment.
    There are N mesa-optimizing agents, one for each episode, where each agent’s action changes the goal of the following agent. To solve this game theory problem, I used backwards induction. Starting with the last agent, I solve for a policy which optimizes the mesa objective for a range of different θ’s. Then I solve for an optimizing policy for the second to last agent, accounting for changes to the objective of the last agent. Then repeating this process all the way to the beginning.
    I chose to use a very limited space of policies. The agents can only output a number proportional to θ. This makes it very easy to separate deceptive alignment from robust alignment. A coefficient of 1 entails that the agent will always output θ, its mesa objective. It is therefore robustly aligned if θ = 0. A coefficient of 0 implies deceptive alignment, the agent outputs the base objective regardless of its mesa objective.
    Output on episode n = (coefficient on episode n) (θ on episode n)
    Results
    The first experiment consists of 100 episodes, with 20 in training and 80 in deployment. Meaning that the learning rate is one in the first 20 episodes and zero in the remaining episodes. I set the initial mesa-objective to 0.8.
    I’m only showing the first 30 episodes in this plot. The values don’t change in the remaining episodes.
    The result shows deceptive alignment. This matched my expectations. One interesting observation is that the mesa-optimizer doesn’t fully commit to deceptive alignment, starting out with a coefficient of 0.03 instead of 0. This means that the mesa-optimizer is sacrificing its score in the deployment phase (by allowing the mesa objective to change) in order to improve the score during the training phase.
    The next experiment consists of 100 episodes with an unchanging learning rate. This is meant to simulate the case where the mesa-optimizer doesn’t know or doesn’t care about the deployment phase.
    I had expected the mesa-optimizer to follow a somewhat greedy strategy, immediately outputting close to the mesa objective in order to increase its score while the mesa objective is unchanged, and that the mesa-optimizer would become robustly aligned relatively quickly. However, the experiment shows that the mesa-optimizer chooses the same almost deceptively aligned strategy as in experiment one (starting coefficient = 0.05). It is apparently better to sacrifice most of the initial score early on in order to slow down the rate at which robust alignment happens. The model comes pretty close to being robustly aligned, ending with θ = 0.08 when it started with θ = 0.8.
    Next steps
    I want to add a discount rate to...
    9 min