The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Neuronpedia - AI Safety Game by hijohnnylin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Neuronpedia - AI Safety Game, published by hijohnnylin on July 26, 2023 on LessWrong.
    Neuronpedia is an AI safety game that documents and explains each neuron in modern AI models. It aims to be the Wikipedia for neurons, where the contributions come from users playing a game. Neuronpedia wants to connect the general public to AI safety, so it's designed to not require any technical knowledge to play.
    Neuronpedia is in experimental beta: getting its first users in order to collect feedback, ideas, and build an initial community.
    OBJECTIVES
    Increase understanding of AI to help build safer AI
    Increase public engagement, awareness, and education in AI safety
    CURRENT STATUS
    I started working on Neuronpedia three weeks ago, and I'm posting on LessWrong to develop an initial community and for feedback and testing. I'm not posting it anywhere else, please do not share it yet in other forums like Reddit.
    There's an onboarding tutorial that explains the game, but to summarize: It's a word association game. You're shown one neuron ("puzzle") at a time, and its highest activations ("clues"). You then either vote for an existing explanation, or submit your own explanation. Neuronpedia's first "campaign" is explaining gpt2-small, layer 6.
    There is an "advanced mode" that allows testing custom activation text and shows more details/filters. Click "Simple" at the top right to toggle it.
    WHAT YOU CAN DO
    Play @ neuronpedia.org - feel free to use a throwaway GitHub account to log in.
    Give feedback, ideas, and ask questions.
    Join the community Discord:
    THE VISION
    Millions of casual and technical users play Neuronpedia daily, trying to solve each neuron (like NYT crossword/Wordle). There are weekly/monthly contests ("side quests"). Top scorers are ranked on leaderboards by country, region, etc.
    Neuronpedia sparks interest in AI safety for thousands of people and they contribute in other ways (switch fields, do research, etc).
    Researchers use the data to build safer and more predictable AI models. Companies post updated versions of their AI models (or parts of them) as new "campaigns" and iterate through increasingly safer models.
    HOW NEURONPEDIA CAME ABOUT
    After moving on from my previous startup, I reached out to 80,000 Hours for career advice. They connected me to William Saunders who provided informal (not affiliated with any company) guidance on what might be useful products to develop for AI safety research. Three weeks ago, I started prototyping versions of Neuronpedia, starting as a reference website, then eventually iterating into a game.
    Neuronpedia is seeded with data and tools from OpenAI's Automated Interpretability and Neel Nanda's Neuroscope.
    IS THIS SUSTAINABLE?
    Unclear. There's no revenue model, and there is nobody supporting Neuronpedia. I'm working full time on it and spending my personal funds on hosting, inference servers, OpenAI API, etc. If you or your organization would like to support this project, please reach out at [email protected].
    COUNTERARGUMENTS AGAINST NEURONPEDIA
    These are reasons Neuronpedia could fail to achieve one or more of its objectives. They're not insurmountable, but good to keep in mind.
    Can't get enough people to care about AI safety or think it's a real problem.
    Neurons are the wrong "unit" for useful interpretability and Neuronpedia is unable to adapt to the correct "unit" (groups of neurons, etc).
    Even the best human explanations are not good.
    Scoring algorithm for explanation is bad and can't be improved.
    Not engaging enough - the game isn't balanced, doesn't have enough "loops", etc.
    Bugs.
    Lack of funds.
    AI companies shut it down via copyright claims, cease and desist, etc.
    Unable to contain abusive users or spam.
    Too slow to stop misaligned AI.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please ...
    4 min
  • LW - Meta-level adversarial evaluation of oversight techniques might allow robust measurement of their adequacy by Buck
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meta-level adversarial evaluation of oversight techniques might allow robust measurement of their adequacy, published by Buck on July 26, 2023 on LessWrong.
    Summary: Many proposed AGI alignment procedures involve taking a pretrained model and training it using rewards from an oversight process to get a policy. These procedures might fail when the oversight procedure is locally inadequate: that is, if the model is able to trick the oversight process into giving good rewards for bad actions. In this post, we propose evaluating the local adequacy of oversight by constructing adversarial policies for oversight processes. Specifically, we propose constructing behaviors that a particular oversight process evaluates favorably but that we know to be bad via other means, such as additional held-out information or more expensive oversight processes. We think that this form of adversarial evaluation is a crucial part of ensuring that oversight processes are robust enough to oversee dangerously powerful models.
    Introduction
    A core element of many scenarios where AI ends up disempowering humanity (e.g. "Without specific countermeasures") are oversight failures: that is, failures resulting from working with a really powerful model without being able to tell if its actions are resulting in good or bad outcomes.
    For example, suppose we're using a model to help us write code for the datacenter on which we run our powerful models. This model may be able to insert subtle security vulnerabilities that we are unable to detect. This inability to distinguish whether actions have good outcomes might cause our model to be misaligned when we train on our erroneous rewards, and it might also bottleneck various techniques for preventing models from taking bad actions.
    Before we put a model to use in a specific situation, we really want to be sure that the oversight process we're using to train it can tell whether a model is taking actions that are bad on average. We will call an oversight process "locally adequate" for a specific model and deployment setting if, for actions taken by that model in that setting, the process is able to identify when an action would have harmful consequences (in cases where the model knows).
    In this piece, we'll describe a way to determine whether an oversight process is locally adequate. We'll do this by training the model to perform actions that both:
    could be very bad if they were done in production
    don't look bad to the oversight process.
    This is an example of what we'll call "meta-level adversarial evaluation": rather than finding inputs that make a model fail, we're finding models that make an oversight process fail. Previous discussions of adversarial evaluation in AI alignment have mostly been about object-level adversarial evaluation (though the Superalignment blog post mentioned meta-level adversarial evaluations).
    For instance, continuing with the example of the model writing code that might have subtle security vulnerability, we could train a variant of our model to insert security flaws into the software it's writing that lead to a particular visible outcome (e.g. deleting a file on a server that the model wasn't supposed to have access to), without the oversight process realizing that the software is flawed. In other cases, we can evaluate the local adequacy of oversight in a specific situation by switching to another equally challenging situation where we have access to some held-out information which we can use to evaluate whether a model's output was in fact bad even though the oversight process judged it as good.
    We argue that it is likely that when working with dangerously powerful models, we'll be able to run these evaluations to get a confident sense of whether oversight processes are locally adequate for a particular model and deployment envir...
    29 min
  • LW - AI Safety Hub Serbia Soft Launch by DusanDNesic
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Hub Serbia Soft Launch, published by DusanDNesic on July 25, 2023 on LessWrong.
    TLDR; We got three-month funding from a generous individual funder to launch an AI Safety office in Serbia. We are giving free office space (and, if funding later permits, housing) to AI Safety researchers who are looking for a place to work from. Priority for citizens of countries like Russia and China who can work visa-free from Serbia while being close to Europe. Register interest here or ask questions at my email. We also have a promise of partial funding for our bare-bones request (budget) from an individual donor but are looking for a second individual donor in order to fulfill it (about 30k USD) - reach out to me if this may be you.
    Background:
    EA Serbia and AI Safety Serbia groups are small but growing (~30 people in EA Serbia, ~3 people looking to get into AIS research as a career, and ~3 to get into AIS policy). Due to Serbia's favorable Visa policy towards Russia and China, many foreigners already live here. With lower living costs than many other international hub cities, a vibrant scene, and a favorable time zone and climate, Belgrade has a growing foreigner community.
    As we have seen projects such as CEEALAR as important and impressive, we wish to replicate them in Serbia, where they can better serve people who may struggle to get UK visas. We also believe that having the capacity to quickly scale cheap housing for people coming from different countries is a good thing.
    We also believe that we should start small, prototype, and then move larger. We have had a unique opportunity to get an office space that is NGO-friendly, has good vibes, and costs only ~550 Euros per month for office space that has three rooms and can fit 8-15 people (depending on how snug they decide to be) with a coffee shop downstairs where another 20 people can spend their time co-working, as the office and the downstairs coffee shop are under the same ownership. This is certainly less luxurious than many other EA/AI co-working places, but we have a high degree of customizability allowed to us, which we can use to make a good office space. If we grow enough, we can also move to a bigger venue, as our needs grow. Certainly, if we knew that more use could be found in an office in Serbia, getting something somewhat further from the center, which includes living and office spaces, would be better, but we do not wish to explore that until we have proof of concept and need.
    Operations details (a.k.a. how it works):
    The office is currently rented for three months, August-October, so that we can keep the favorable price instead of having to find a different place. The office space has some desks and chairs, but we are looking to acquire full funding and have people voice their needs before acquiring more furniture. The office is usually open during working hours of the coffee shop (10 AM to Midnight, except on weekends when it is 4 PM to Midnight) as they share an entrance, but exceptionally, we can accommodate special requests if someone works better at strange working hours.
    Office space is given to those that are working on projects related to AI Safety as a priority, but EA research is also welcome whenever we have spare capacity (which we expect at first).
    Unfortunately, we currently do not have a legal entity that can provide visa invitations for those coming from countries that require a visa - for that, we would need to gather funding before starting the process. Still, a Serbian visa is not required for many and is relatively easy to get for most.
    We have a reliable real estate agent who is able to get good deals on housing in Belgrade for those that need housing assistance until we get funded and rent a co-living space as well.
    For those looking to eat consistently through us, we can arrange ...
    6 min
  • EA - Launching the meta charity funding circle (MCF): Apply for funding or join as a donor! by Joey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Launching the meta charity funding circle (MCF): Apply for funding or join as a donor!, published by Joey on July 26, 2023 on The Effective Altruism Forum.
    Summary
    We are launching the Meta Charity Funders, a growing network of donors sharing knowledge and discussing funding opportunities in the EA meta space. Apply for funding by August 27th or join the circle as a donor. See below or visit our website to learn more!
    If you are doing EA-aligned "meta" work, and have not received substantial funding for several years, you might be worried about funding. Over the past 10 years, Open Philanthropy and EA Funds comprised a large percent of total meta funding and are far from independent of each other. This lack of diversity means potentially effective projects outside their priorities often struggle to stay afloat or scale, and the beliefs of just a few grant-makers can massively shape the EA movement's trajectory.
    It can be difficult for funders within meta as well. Individual donors often don't know where to give if they don't share EA Funds' approach. Thorough vetting is scarce and expensive, with only a handful of grant-makers deploying tens of millions per year in meta grants, resulting in sub-optimal allocations.
    This is why we are launching the Meta Charity Funders, a growing network of donors sharing knowledge, discussing funding opportunities, and running joint open grant rounds in the EA meta space. We believe many charitable projects create a huge impact by working at one level removed from direct impact to instead enhance the impact of others. Often these projects cut across causes and don't fit neatly into a box, thus being neglected by funders. Well known examples of meta organizations include charity evaluators like GiveWell, incubators like Charity Entrepreneurship, cause prioritization research organizations like Rethink Priorities, or field-building projects promoting effective giving or impactful careers.
    Grantees: Apply to many HNW donors at once - 1st round closes August 27.
    We are open to funding meta work across a range of causes, organizational stages, strategies, etc. We are most interested in applications that have not already been substantially supported by similar actors such as EA Funds or Open Philanthropy, though we will still consider these. We expect most of our grants to range from $10,000 to $500,000 and consider grants to both individuals and organizations. We expect our first round to be between $500,000 and $1.5m of total funding.
    Please lean in favor of applying if you are unsure if you would be a good fit!
    Donors: Join us! Find neglected opportunities, get help with ops and vetting, and give on your own terms.
    People who are unable to commit to regular meetings are still encouraged to apply and may be invited to our Slack and email list and gain access to our grant opportunities database.
    Meta Charity Funding Circle is a project of Charity Entrepreneurship and Impactful Grantmaking. It is organized by this post's authors: Gage Weston, Vilhelm Skoglund, and Joey Savoie. Our members are anonymous.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    4 min
  • AF - Meta-level adversarial evaluation of oversight techniques might allow robust measurement of their adequacy by Buck Shlegeris
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meta-level adversarial evaluation of oversight techniques might allow robust measurement of their adequacy, published by Buck Shlegeris on July 26, 2023 on The AI Alignment Forum.
    Summary: Many proposed AGI alignment procedures involve taking a pretrained model and training it using rewards from an oversight process to get a policy. These procedures might fail when the oversight procedure is locally inadequate: that is, if the model is able to trick the oversight process into giving good rewards for bad actions. In this post, we propose evaluating the local adequacy of oversight by constructing adversarial policies for oversight processes. Specifically, we propose constructing behaviors that a particular oversight process evaluates favorably but that we know to be bad via other means, such as additional held-out information or more expensive oversight processes. We think that this form of adversarial evaluation is a crucial part of ensuring that oversight processes are robust enough to oversee dangerously powerful models.
    Introduction
    A core element of many scenarios where AI ends up disempowering humanity (e.g. "Without specific countermeasures") are oversight failures: that is, failures resulting from working with a really powerful model without being able to tell if its actions are resulting in good or bad outcomes.
    For example, suppose we're using a model to help us write code for the datacenter on which we run our powerful models. This model may be able to insert subtle security vulnerabilities that we are unable to detect. This inability to distinguish whether actions have good outcomes might cause our model to be misaligned when we train on our erroneous rewards, and it might also bottleneck various techniques for preventing models from taking bad actions.
    Before we put a model to use in a specific situation, we really want to be sure that the oversight process we're using to train it can tell whether a model is taking actions that are bad on average. We will call an oversight process "locally adequate" for a specific model and deployment setting if, for actions taken by that model in that setting, the process is able to identify when an action would have harmful consequences (in cases where the model knows).
    In this piece, we'll describe a way to determine whether an oversight process is locally adequate. We'll do this by training the model to perform actions that both:
    could be very bad if they were done in production
    don't look bad to the oversight process.
    This is an example of what we'll call "meta-level adversarial evaluation": rather than finding inputs that make a model fail, we're finding models that make an oversight process fail. Previous discussions of adversarial evaluation in AI alignment have mostly been about object-level adversarial evaluation (though the Superalignment blog post mentioned meta-level adversarial evaluations).
    For instance, continuing with the example of the model writing code that might have subtle security vulnerability, we could train a variant of our model to insert security flaws into the software it's writing that lead to a particular visible outcome (e.g. deleting a file on a server that the model wasn't supposed to have access to), without the oversight process realizing that the software is flawed. In other cases, we can evaluate the local adequacy of oversight in a specific situation by switching to another equally challenging situation where we have access to some held-out information which we can use to evaluate whether a model's output was in fact bad even though the oversight process judged it as good.
    We argue that it is likely that when working with dangerously powerful models, we'll be able to run these evaluations to get a confident sense of whether oversight processes are locally adequate for a particular mod...
    29 min
  • EA - An overview of standards in biosafety and biorisk by rosehadshar
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An overview of standards in biosafety and biorisk, published by rosehadshar on July 26, 2023 on The Effective Altruism Forum.
    Linkpost for
    This report represents ~40 hours of work by Rose Hadshar in summer 2023 for Arb Research, in turn for Holden Karnofsky in response to this call for proposals on standards.
    It's based on a mixture of background reading, research into individual standards, and interviews with experts. Note that I didn't ask for permission to cite the expert interviews publicly, so I've anonymised them.
    I suggest reading the scope and summary and skimming the overview, then only looking at sections which seem particularly relevant to you.
    Scope
    This report covers:
    Both biosecurity and biosafety:
    Biosecurity: "the protection, control and accountability for valuable biological materials (including information) in laboratories in order to prevent their unauthorized access, loss, theft, misuse, diversion or intentional release."
    Biosafety: "the containment principles, technologies and practices that are implemented to prevent unintentional exposure to pathogens and toxins or their accidental release"
    Biosecurity and biosafety standards internationally, but with much more emphasis on the US
    Regulations and guidance as well as standards proper. I am using these terms as follows:
    Regulations: rules on how to comply with a particular law or laws. Legally binding
    Guidance: rules on how to comply with particular regulations. Not legally binding, but risky to ignore
    Standards: rules which do not relate to compliance with a particular law or laws. Not legally binding.
    Note that I also sometimes use 'standards' as an umbrella term for regulations, guidance and standards.
    Summary of most interesting findings
    For each point:
    I've included my confidence in the claim (operationalised as the probability that I would still believe the claim after 40 hours' more work).
    I link to a subsection with more details (though in some cases I don't have much more to say).
    The origins of bio standards
    (80%) There were many different motivations behind bio standards (e.g. plant health, animal health, worker protection, bioterrorism, fair sharing of genetic resources.)
    (70%) Standards were significantly reactive to rather than proactive about incidents (e.g. lab accidents, terrorist attacks, and epidemics), though:
    There are exceptions (e.g. the NIH guidelines on recombinant DNA)
    Guidance is often more proactive than standards (e.g. gene drives)
    (80%) International standards weren't always later or less influential than national ones
    (70%) Voluntary standards seem to have prevented regulation in at least one case (e.g. the NIH guidelines)
    (65%) In the US, it may be more likely that mandatory standards are passed on matters of national security (e.g. FSAP)
    Compliance
    (60%) Voluntary compliance may sometimes be higher than mandated compliance (e.g. NIH guidelines)
    (70%) Motives for voluntarily following standards include responsibility, market access, and the spread of norms via international training
    (80%) Voluntary standards may be easier to internationalise than regulation
    (90%) Deliberate efforts were made to increase compliance internationally (e.g. via funding biosafety associations, offering training and other assistance)
    Problems with these standards
    (90%) Bio standards are often list-based. This means that they are not comprehensive, do not reflect new threats, prevent innovation in risk management, and fail to recognise the importance of context for risk
    There's been a partial move away from prescriptive, list-based standards towards holistic, risk-based standards (e.g. ISO 35001)
    (85%) Bio standards tend to lack reporting standards, so it's very hard to tell how effective they are
    (60%) Standards may have impeded safety work in some areas (e.g. select agent designation as a...
    35 min
  • LW - Underwater Torture Chambers: The Horror Of Fish Farming by omnizoid
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Underwater Torture Chambers: The Horror Of Fish Farming, published by omnizoid on July 26, 2023 on LessWrong.
    Crossposted from my blog.
    The horror of fish farming
    One of my professors in college taught a class about effective altruism - a social movement about doing good effectively. Whenever he was talking about the scale of animal suffering, all the statistics he talked about were about the suffering of non-fish. It became a running joke - friends and I would make jokes like the following: "a gunman robs a bank, kills over 4 non-fish" or "one (non-fish) death is a tragedy, a million fish deaths are a statistic." But this professor, despite explicitly ignoring fish whenever he talked about a problem, was nonetheless more pro-fish than almost all people. Because no one cares about fish - at all.
    I remember when I was young, my grandmother would take me and my brother fishing. This was seen as a totally innocuous, fun way to spend a weekend. The general attitude towards fishing is almost exactly opposite to the attitude towards, for example, hunting: liberal parents in blue states are not happy letting their children hunt land creatures, the way they are happy to let their children hunt fish. This is unsettling - hooking fish into the mouth to yank them out of the water so that they suffocate to death is seen as just a bit of innocent fun.
    Elsewhere, I've made the case against land-based factory farms - great, industrial torture chambers, that routinely mutilate animals, give them too little space to move around, castrate them with no anesthetic, and force them to spend their lives covered in feces and filth, in constant agony, disgust, and boredom. I've even argued that factory farming is the worst crime in human history - worse than whatever very bad crime you're thinking of.
    The cruelty of the farm industry is not limited to the versions that are on land. The fish farming industry is unspeakably horrifying. We have built hell and consigned the fish to it. And far more animals are tormented and killed in fish farms than in land-based factory farms. The aquaculture industry may be the single worst industry in the history of the world - causing more suffering than any other. This judgment may sound surprising, but it's hard to deny if one thinks fish suffering matters at all given the sheer numbers. One undercover report on fish farms found:
    Workers' abusive handling of fish, including slamming and stomping on fish, and violently throwing fish, including treating them like basketballs performing "trick shots"
    Workers cruelly killing fish by slamming them on the ground
    Live fish have their eyes eaten by fish who are underfed and hungry and mistake their pupils as food
    Ineffective anesthetization during vaccination and fin clipping
    Fish thrown into buckets and left to suffocate in piles of the dead and dying
    Conditions so filthy that fish must be vaccinated
    Painful spinal deformities, and fungus growth on fish intended for human consumption, including fungus eating away at the faces of the fish
    Extreme crowding in barren conditions and high death rates of eggs and fish
    Lewis Bollard notes "The fishing industry alone kills 3-8 billion animals every day, most by slow suffocation, crushing, or live disemboweling." So roughly the same number of fish are killed in horrifying, inhumane ways every few days as there are people on earth. It takes longer for the fish to suffocate than for us to drown, and so is probably more painful. The most up-to-date report - and also one of the most conservative - found that the least sentient fish that they surveyed experience suffering about one-twentieth as intensely as human suffering. Let's grant this - and use the conservative estimate of how many fish are killed by people - only 3 billion per day. If a fish's death is only one-twentiet...
    16 min
  • AF - Frontier Model Security by Matthew "Vaniver" Gray
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Frontier Model Security, published by Matthew "Vaniver" Gray on July 26, 2023 on The AI Alignment Forum.
    As the capabilities of frontier artificial intelligence models continue to increase rapidly, ensuring the security of these systems has become a critical priority. In our previous posts, we've focused on Anthropic's approach to safety, and Claude's capabilities and applications. In this post, we are sharing some of the steps we are taking to ensure our models are developed securely. We hope to advance public discussion about how all labs can deploy top models securely, as well as share recommendations for government regulatory approaches that encourage adoption of strong cybersecurity practices. Below we discuss some of our recommendations for cybersecurity best practices, which Anthropic itself is in the process of implementing.SummaryFuture advanced AI models have the potential to upend economic and national security affairs within and among nation-states.
    Given the strategic nature of this technology, frontier AI research and models must be secured to levels far exceeding standard practices for other commercial technologies in order to protect them from theft or misuse.In the near term, governments and frontier AI labs must be ready to protect advanced models and model weights, and the research that feeds into them. This should include measures such as the development of robust best practices widely diffused among industry, as well as treating the advanced AI sector as something akin to "critical infrastructure" in terms of the level of public-private partnership in securing these models and the companies developing them.Many of these measures can begin as voluntary arrangements, but in time it may be appropriate to use government procurement or regulatory powers to mandate compliance.Cybersecurity Best PracticesWe believe "two-party control" is necessary to secure advanced AI systems.
    Two-party control is already used in a range of domains; for example, two people with two keys are needed to open the most secure vaults, and multi-party review patterns have been applied in manufacturing (GMP, ISO 9001), food (FSMA PCQI, ISO 22000), medical (ISO 13485) and finance tech (SOX).
    This pattern should be applied to all systems involved in the development, training, hosting, and deployment of frontier AI models.
    This pattern is already in widespread use within major tech companies to defend against the most advanced threat actors and mitigate insider risk.
    It is manifested as a system design where no person has persistent access to production-critical environments, and they must ask a coworker for time-limited access with a business justification for that request.
    Even emerging AI labs, without large enterprise resources, can implement these controls.
    We call this multi-party authorization to AI-critical infrastructure design. This is a leading security requirement that depends on the gamut of cybersecurity best practices to implement correctly.In addition, secure software development practices should pervade the frontier AI model environment. The gold-standard for these practices are the NIST Secure Software Development Framework (SSDF) and the Supply Chain Levels for Software Artifacts (SLSA). Executive Orders have been leveraged successfully to encourage major tech companies to adopt higher development standards: in 2021, EO 14028 directed OMB to set Federal Procurement guidelines.
    This motivated action: the software industry has invested heavily to meet the SSDF's requirements to retain federal contracts.While frontier AI research is already benefiting from the implementation of some of these standards by dint of cloud providers hosting their models, applying these existing standards can step-change the security of these AI systems:SSDF and SLSA are largely transla...
    7 min
  • EA - [Crosspost] An AI Pause Is Humanity's Best Bet For Preventing Extinction (TIME) by Otto
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Crosspost] An AI Pause Is Humanity's Best Bet For Preventing Extinction (TIME), published by Otto on July 25, 2023 on The Effective Altruism Forum.
    Otto Barten is director of the Existential Risk Observatory, a nonprofit aiming to reduce existential risk by informing the public debate.
    Joep Meindertsma is founder of PauseAI, a movement campaigning for an AI Pause.The existential risks posed by artificial intelligence (AI) are now widely recognized.
    After hundreds of industry and science leaders warned that "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war," the U.N. Secretary-General recently echoed their concern. So did the prime minister of the U.K., who is also investing 100 million pounds into AI safety research that is mostly meant to prevent existential risk. Other leaders are likely to follow in recognizing AI's ultimate threat.
    In the scientific field of existential risk, which studies the most likely causes of human extinction, AI is consistently ranked at the top of the list. In The Precipice, a book by Oxford existential risk researcher Toby Ord that aims to quantify human extinction risks, the likeliness of AI leading to human extinction exceeds that of climate change, pandemics, asteroid strikes, supervolcanoes, and nuclear war combined. One would expect that even for severe global problems, the risk that they lead to full human extinction is relatively small, and this is indeed true for most of the above risks. AI, however, may cause human extinction if only a few conditions are met. Among them is human-level AI, defined as an AI that can perform a broad range of cognitive tasks at least as well as we can. Studies outlining these ideas were previously known, but new AI breakthroughs have underlined their urgency: AI may be getting close to human level already.
    Recursive self-improvement is one of the reasons why existential-risk academics think human-level AI is so dangerous. Because human-level AI could do almost all tasks at our level, and since doing AI research is one of those tasks, advanced AI should therefore be able to improve the state of AI. Constantly improving AI would create a positive feedback loop with no scientifically established limits: an intelligence explosion. The endpoint of this intelligence explosion could be a superintelligence: a godlike AI that outsmarts us the way humans often outsmart insects. We would be no match for it.
    A godlike, superintelligent AI
    A superintelligent AI could therefore likely execute any goal it is given. Such a goal would be initially introduced by humans, but might come from a malicious actor, or not have been thought through carefully, or might get corrupted during training or deployment. If the resulting goal conflicts with what is in the best interest of humanity, a superintelligence would aim to execute it regardless. To do so, it could first hack large parts of the internet and then use any hardware connected to it. Or it could use its intelligence to construct narratives that are extremely convincing to us. Combined with hacked access to our social media timelines, it could create a fake reality on a massive scale. As Yuval Harari recently put it: "If we are not careful, we might be trapped behind a curtain of illusions, which we could not tear away - or even realise is there."
    As a third option, after either legally making money or hacking our financial system, a superintelligence could simply pay us to perform any actions it needs from us. And these are just some of the strategies a superintelligent AI could use in order to achieve its goals. There are likely many more. Like playing chess against grandmaster Magnus Carlsen, we cannot predict the moves he will play, but we can predict the outcome: we los...
    12 min
  • EA - How much is reducing catastrophic and extinction risk worth, assuming XPT forecasts? by rosehadshar
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How much is reducing catastrophic and extinction risk worth, assuming XPT forecasts?, published by rosehadshar on July 25, 2023 on The Effective Altruism Forum.
    This is a post I drafted some months ago, in the course of analysing some XPT data and reading Shulman and Thornley. It's not very sophisticated, I haven't checked the workings, and I haven't polished the language; but I'm posting anyway because that seems better than not posting. Note that it's a personal take and doesn't represent FRI's views.
    Thanks to Josh Rosenberg at FRI and Elliot Thornley for help and comments.
    BLUF: if you make a bunch of assumptions, then even quite low absolute risk forecasts like the XPT ones imply quite high spending on reducing GCRs, conditional on there being sufficiently cost-effective ways to do so.
    In 2022, what has become the Forecasting Research Institute ran the Existential Risk Persuasion Tournament (XPT). Over 200 forecasters, including superforecasters and domain experts, spent 4 months making forecasts on various questions related to existential and catastrophic risk.
    You can see the results from the tournament overall here, and a discussion of the XPT AI risk forecasts in particular here.
    These are the main XPT forecasts on catastrophic and extinction risk:
    Biological--Engineered pathogens--Natural pathogens--AI (superforecasters)AI (domain experts)NuclearNon-anthropogenicTotal catastrophic riskBiological--Engineered pathogens--Natural pathogens--AI (superforecasters)AI (domain experts)NuclearNon-anthropogenicTotal extinction risk
    2030
    2050
    2100
    Catastrophic risk (>10% of humans die in 5 years)
    1.8%
    0.8%
    1%
    0.01%
    0.73%
    2.13%
    0.35%
    5%
    12%
    0.50%
    1.83%
    4%
    0.0026%
    0.015%
    0.05%
    0.85%
    3.85%
    9.05%
    Extinction risk (human population <5000)
    0.012%
    0.01%
    0.0018%
    0.0001%
    0.03%
    0.38%
    0.02%
    1.1%
    3%
    0.001%
    0.01%
    0.074%
    0.0004%
    0.0014%
    0.0043%
    0.01%
    0.3%
    1%
    If we take these numbers at face value, how much is catastrophic and extinction risk reduction worth?
    One approach is to take the XPT forecasts, convert them into deaths in expectation, then assume a value of a statistical life and a discount rate, and estimate how much averting those deaths is 'worth'. (I'm stealing this method directly from Shulman and Thornley.)
    Using the XPT superforecasts and OWID population projections gives us the following deaths in expectation, in millions:
    Bio----AINuclearNon-anthropogenicTotalBio----AINuclearNon-anthropogenicTotal
    Deaths in expectation (millions)
    2030
    2050
    2100
    Catastrophic risk (>10% of humans die in 5 years)
    18.6
    0.09
    7.1
    22.0
    4.3
    17.8
    41.4
    0.02
    0.15
    0.5
    7.3
    37.4
    93.7
    Extinction risk (human population <5000)
    1.2
    0.01
    2.9
    39.3
    0.09
    1.0
    7.7
    0.03
    0.1
    0.4
    0.9
    29.1
    103.5
    Some notes:
    Workings here.
    For catastrophic risks, the deaths in expectation should be read as a lower bound, because they assume 10% deaths and the question includes scenarios with >10% deaths.
    That's deaths in expectation worldwide. But the value of a statistical life varies by country: governments have different resources and the cost of interventions in different places varies.
    So the most straightforward way to think about the worth of catastrophic and extinction risk reduction is to ask how much this would be worth in a given country. Let's take the US as an example.
    First we need US deaths in expectation:
    Bio----AINuclearNon-anthropogenicTotalBio----AINuclearNon-anthropogenicTotal
    US deaths in expectation (millions)
    2030
    2050
    2100
    Catastrophic risk (>10% of humans die in 5 years)
    0.7
    0.004
    0.3
    0.8
    0.2
    0.7
    1.6
    0.001
    0.006
    0.02
    0.3
    1.4
    3.6
    Extinction risk (human population <5000)
    0.05
    0.0004
    0.1
    1.5
    0.004
    0.04
    0.3
    0.001
    0.01
    0.02
    0.04
    1.1
    3.9
    Workings here.
    We can then assume a US value for a statistical life, and a discount rate, and use these to estimate how much averting the deaths in expectation is 'worth' to the...
    17 min