Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Coordination within EA: community & ecosystems, published by CristinaSchmidtIbáñez on October 20, 2021 on The Effective Altruism Forum.

Coordination problems in EA seem somewhat pressing, but we are lacking a structured way to think through different kinds of coordination challenges and bottlenecks.

In this post we

Introduce a way to categorize and understand coordination services within the EA community: ecosystem & community member services

Present different potential services in these categories

Present a way to categorize coordination bottlenecks into three functional areas: knowledge, infrastructure and support

Our intention with this post is mainly to open up a discussion about this topic so that everyone can contribute to it and understand it better.

Introduction

The importance of coordination & coordination challenges has been periodically discussed for several years in the EA movement. In recent years, it seems that these challenges are becoming more acute - in November 2020 Benjamin Todd mentioned that currently, “the main bottlenecks might be organizational capacity, infrastructure, and management to help train people up, as well as specialist skills that people can put to work now”. (80,000 Hours episode). Under the assumption that these are the current bottlenecks of the community, coordination should help alleviate the effects of these bottlenecks.

The EA community is fairly highly collaborative and seems to focus on coordination more than other social movements. As such it does not appear that a lack of coordination related to building capacity and infrastructure has been the cause of negative outcomes (there are risks associated with info hazards or the unilateralist’s curse where a lack of coordination can be negative). Nevertheless, it seems that improving coordination would allow our community to do even more good. We think that there is significant room for improvement to build systems, or infrastructure, that could enable better coordination within the next 3-5 years and we would expect them (given the current trajectory of the community) to pay off for > 5 years after they’ve been refined.

In this post, we aim to map the meta landscape of internal coordination bottlenecks within the community. We hope this is useful for those thinking of or currently working in this space.

While we believe there are also significant opportunities to improve or develop coordination between EA and the communities adjacent to EA (“the outside world”), this post will focus more on internal coordination issues.

Note: “Organizations” refer to those which actively engage with the EA community, regardless of identification as EA-aligned or not.

Why internal coordination services?

We want the EA community to do not just some good but the most good possible. If we think that pursuing EA goals are very impactful, then even small marginal gains should be very impactful. The case for coordination in general has been previously made by Owen Cotton Barratt & Benjamin Todd; we have little to add to that.

Internal coordination services include those that help organizations & community members work together efficiently by sharing knowledge, or reducing resources required for undertaking specific actions. We think most of these are quite tractable.

We think internal coordination is particularly tractable (given adequate resources) because 1) Resources are fairly fungible within the EA community, 2) EA is a high trust community, 3) There is general consensus that (most) EA goals are worth pursuing. (See: The almighty Hive will for more)

However, there are forces that lead to suboptimal coordination including conflicting priorities for individual agents to invest time in coordination, principal-agent problems, lack of appeal of coordination work (e.g. coordination is “not sexy”), and...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is [$10k bounty] Read and compile Robin Hanson’s best posts, published by Richard Ngo on October 20, 2021 on LessWrong.

I think Robin Hanson's ideas are not read nearly as widely as they should be, in part because it's difficult to navigate his many, many blog posts (I estimate he's written 2000 of them). So I'd like to pay someone to read through all his writings and compile the best ones into a more accessible format. The default output would be an ebook like Rationality: from AI to Zombies, containing several thematically-linked sequences of posts; possible extensions of this include adding summaries or publishing physical copies (although let me know if you have any other suggestions).

I expect this to take 1-2 months of work, and plan to pay around $10k USD (more details to be determined as we get a better idea of the scope of the project). My gmail address is richardcngo; email me with the subject line "Hanson compilation", plus any relevant information about yourself, if you might be interested in doing this.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is [$10k bounty] Read and compile Robin Hanson’s best posts, published by Richard Ngo on October 20, 2021 on LessWrong.

I think Robin Hanson's ideas are not read nearly as widely as they should be, in part because it's difficult to navigate his many, many blog posts (I estimate he's written 2000 of them). So I'd like to pay someone to read through all his writings and compile the best ones into a more accessible format. The default output would be an ebook like Rationality: from AI to Zombies, containing several thematically-linked sequences of posts; possible extensions of this include adding summaries or publishing physical copies (although let me know if you have any other suggestions).

I expect this to take 1-2 months of work, and plan to pay around $10k USD (more details to be determined as we get a better idea of the scope of the project). My gmail address is richardcngo; email me with the subject line "Hanson compilation", plus any relevant information about yourself, if you might be interested in doing this.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is AGI Safety Fundamentals curriculum and application, published by Richard Ngo on October 20, 2021 on The AI Alignment Forum.

Over the last year EA Cambridge has been designing and running an online program aimed at effectively introducing the field of AGI safety; the most recent cohort included around 150 participants and 25 facilitators from around the world. Dewi Erwan runs the program; I designed the curriculum, the latest version of which appears in the linked document. We expect the program to be most useful to people with technical backgrounds (e.g. maths, CS, or ML), although the curriculum is intended to be accessible for those who aren't familiar with machine learning, and participants will be put in groups with others from similar backgrounds. If you're interested in joining the next version of the course (taking place January - March 2022) apply here to be a participant or here to be a facilitator. Applications are open to anyone and close 15 December. (We expect to be able to pay facilitators, but are still waiting to confirm the details.)

This post contains an overview of the course and an abbreviated version of the curriculum; the full version (which also contains optional readings, exercises, notes, discussion prompts, and project ideas) can be found here. Comments and feedback are very welcome, either on this post or in the full curriculum document; suggestions of new exercises, prompts or readings would be particularly helpful. I'll continue to make updates until shortly before the next cohort starts.

Course overview

The course consists of 8 weeks of readings, plus a final project. Participants are divided into groups of 4-6 people, matched based on their prior knowledge about ML and safety. Each week (apart from week 0) each group and their discussion facilitator will meet for 1.5 hours to discuss the readings and exercises. Broadly speaking, the first half of the course explores the motivations and arguments underpinning the field of AGI safety, while the second half focuses on proposals for technical solutions. After week 7, participants will have several weeks to work on projects of their choice, to present at the final session.

Each week's curriculum contains:

Key ideas for that week

Core readings

Optional readings

Two exercises (participants should pick one to do each week)

Further notes on the readings

Discussion prompts for the weekly session

Week 0 replaces the small group discussions with a lecture plus live group exercises, since it's aimed at getting people with little ML knowledge up to speed quickly.

The topics for each week are:

Week 8 (several weeks later): Projects

Abbreviated curriculum (only key ideas and core readings)

Week 0 (optional): introduction to machine learning

This week mainly involves learning about foundational concepts in machine learning, for those who are less familiar with them, or want to revise the basics. If you’re not already familiar with basic concepts in statistics (like regressions), it will take a bit longer than most weeks; and instead of the group discussions from most weeks, there will be a lecture and group exercises. If you’d like to learn ML in more detail, see the further resources section at the end of this curriculum.

Otherwise, start with Ngo (2021), which provides a framework for thinking about machine learning, and in particular the two key components of deep learning: neural networks and optimisation. For more details and intuitions about neural networks, watch 3Blue1Brown (2017a); for more details and intuitions about optimisation, watch 3Blue1Brown (2017b). Lastly, see Simonini (2020) for an introduction to how deep learning can be used to solve reinforcement learning tasks.

If you’re not familiar with the basics of statistics, like linear regression and classification:

Introduction: linear re...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is AGI Safety Fundamentals curriculum and application, published by Richard Ngo on October 20, 2021 on The AI Alignment Forum.

Over the last year EA Cambridge has been designing and running an online program aimed at effectively introducing the field of AGI safety; the most recent cohort included around 150 participants and 25 facilitators from around the world. Dewi Erwan runs the program; I designed the curriculum, the latest version of which appears in the linked document. We expect the program to be most useful to people with technical backgrounds (e.g. maths, CS, or ML), although the curriculum is intended to be accessible for those who aren't familiar with machine learning, and participants will be put in groups with others from similar backgrounds. If you're interested in joining the next version of the course (taking place January - March 2022) apply here to be a participant or here to be a facilitator. Applications are open to anyone and close 15 December. (We expect to be able to pay facilitators, but are still waiting to confirm the details.)

This post contains an overview of the course and an abbreviated version of the curriculum; the full version (which also contains optional readings, exercises, notes, discussion prompts, and project ideas) can be found here. Comments and feedback are very welcome, either on this post or in the full curriculum document; suggestions of new exercises, prompts or readings would be particularly helpful. I'll continue to make updates until shortly before the next cohort starts.

Course overview

The course consists of 8 weeks of readings, plus a final project. Participants are divided into groups of 4-6 people, matched based on their prior knowledge about ML and safety. Each week (apart from week 0) each group and their discussion facilitator will meet for 1.5 hours to discuss the readings and exercises. Broadly speaking, the first half of the course explores the motivations and arguments underpinning the field of AGI safety, while the second half focuses on proposals for technical solutions. After week 7, participants will have several weeks to work on projects of their choice, to present at the final session.

Each week's curriculum contains:

Key ideas for that week

Core readings

Optional readings

Two exercises (participants should pick one to do each week)

Further notes on the readings

Discussion prompts for the weekly session

Week 0 replaces the small group discussions with a lecture plus live group exercises, since it's aimed at getting people with little ML knowledge up to speed quickly.

The topics for each week are:

Week 8 (several weeks later): Projects

Abbreviated curriculum (only key ideas and core readings)

Week 0 (optional): introduction to machine learning

This week mainly involves learning about foundational concepts in machine learning, for those who are less familiar with them, or want to revise the basics. If you’re not already familiar with basic concepts in statistics (like regressions), it will take a bit longer than most weeks; and instead of the group discussions from most weeks, there will be a lecture and group exercises. If you’d like to learn ML in more detail, see the further resources section at the end of this curriculum.

Otherwise, start with Ngo (2021), which provides a framework for thinking about machine learning, and in particular the two key components of deep learning: neural networks and optimisation. For more details and intuitions about neural networks, watch 3Blue1Brown (2017a); for more details and intuitions about optimisation, watch 3Blue1Brown (2017b). Lastly, see Simonini (2020) for an introduction to how deep learning can be used to solve reinforcement learning tasks.

If you’re not familiar with the basics of statistics, like linear regression and classification:

Introduction: linear re...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Truthful AI, published by Owen Cotton-Barratt on October 20, 2021 on The Effective Altruism Forum.

This post contains the abstract and executive summary of a new 96-page paper (subtitle: Developing and governing AI that does not lie) from authors at the Future of Humanity Institute, Global Priorities Institute, and OpenAI. Also posted on LessWrong.

I believe that over the next several years this might be an important field on longtermist grounds for two disjoint reasons:

Maybe there's a window of opportunity to set standards for behaviour of linguistic AI systems, and higher standards could improve global epistemics and coordination.

AI truthfulness is more concrete and approachable a target than "avoiding AI risk", but has several structural similarities, and I think we'd learn a lot (on both the technical and governance sides) from tackling it.

Abstract

In many contexts, lying – the use of verbal falsehoods to deceive – is harmful. While lying has traditionally been a human affair, AI systems that make sophisticated verbal statements are becoming increasingly prevalent. This raises the question of how we should limit the harm caused by AI “lies” (i.e. falsehoods that are actively selected for). Human truthfulness is governed by social norms and by laws (against defamation, perjury, and fraud). Differences between AI and humans present an opportunity to have more precise standards of truthfulness for AI, and to have these standards rise over time. This could provide significant benefits to public epistemics and the economy, and mitigate risks of worst-case AI futures.

Establishing norms or laws of AI truthfulness will require significant work to:

identify clear truthfulness standards;

create institutions that can judge adherence to those standards; and

develop AI systems that are robustly truthful.

Our initial proposals for these areas include:

a standard of avoiding “negligent falsehoods” (a generalisation of lies that is easier to assess);

institutions to evaluate AI systems before and after real-world deployment;

explicitly training AI systems to be truthful via curated datasets and human interaction.

A concerning possibility is that evaluation mechanisms for eventual truthfulness standards could be captured by political interests, leading to harmful censorship and propaganda. Avoiding this might take careful attention. And since the scale of AI speech acts might grow dramatically over the coming decades, early truthfulness standards might be particularly important because of the precedents they set.

Executive Summary & Overview

The threat of automated, scalable, personalised lying

Today, lying is a human problem. AI-produced text or speech is relatively rare, and is not trusted to reliably convey crucial information. In today’s world, the idea of AI systems lying does not seem like a major concern.

Over the coming years and decades, however, we expect linguistically competent AI systems to be used much more widely. These would be the successors of language models like GPT-3 or T5, and of deployed systems like Siri or Alexa, and they could become an important part of the economy and the epistemic ecosystem. Such AI systems will choose, from among the many coherent statements they might make, those that fit relevant selection criteria — for example, an AI selling products to humans might make statements judged likely to lead to a sale. If truth is not a valued criterion, sophisticated AI could use a lot of selection power to choose statements that further their own ends while being very damaging to others (without necessarily having any intention to deceive – see Diagram 1). This is alarming because AI untruths could potentially scale, with one system telling personalised lies to millions of people.

Aiming for robustly beneficial standards

Widespread and damaging A...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Truthful AI, published by Owen Cotton-Barratt on October 20, 2021 on The Effective Altruism Forum.

This post contains the abstract and executive summary of a new 96-page paper (subtitle: Developing and governing AI that does not lie) from authors at the Future of Humanity Institute, Global Priorities Institute, and OpenAI. Also posted on LessWrong.

I believe that over the next several years this might be an important field on longtermist grounds for two disjoint reasons:

Maybe there's a window of opportunity to set standards for behaviour of linguistic AI systems, and higher standards could improve global epistemics and coordination.

AI truthfulness is more concrete and approachable a target than "avoiding AI risk", but has several structural similarities, and I think we'd learn a lot (on both the technical and governance sides) from tackling it.

Abstract

In many contexts, lying – the use of verbal falsehoods to deceive – is harmful. While lying has traditionally been a human affair, AI systems that make sophisticated verbal statements are becoming increasingly prevalent. This raises the question of how we should limit the harm caused by AI “lies” (i.e. falsehoods that are actively selected for). Human truthfulness is governed by social norms and by laws (against defamation, perjury, and fraud). Differences between AI and humans present an opportunity to have more precise standards of truthfulness for AI, and to have these standards rise over time. This could provide significant benefits to public epistemics and the economy, and mitigate risks of worst-case AI futures.

Establishing norms or laws of AI truthfulness will require significant work to:

identify clear truthfulness standards;

create institutions that can judge adherence to those standards; and

develop AI systems that are robustly truthful.

Our initial proposals for these areas include:

a standard of avoiding “negligent falsehoods” (a generalisation of lies that is easier to assess);

institutions to evaluate AI systems before and after real-world deployment;

explicitly training AI systems to be truthful via curated datasets and human interaction.

A concerning possibility is that evaluation mechanisms for eventual truthfulness standards could be captured by political interests, leading to harmful censorship and propaganda. Avoiding this might take careful attention. And since the scale of AI speech acts might grow dramatically over the coming decades, early truthfulness standards might be particularly important because of the precedents they set.

Executive Summary & Overview

The threat of automated, scalable, personalised lying

Today, lying is a human problem. AI-produced text or speech is relatively rare, and is not trusted to reliably convey crucial information. In today’s world, the idea of AI systems lying does not seem like a major concern.

Over the coming years and decades, however, we expect linguistically competent AI systems to be used much more widely. These would be the successors of language models like GPT-3 or T5, and of deployed systems like Siri or Alexa, and they could become an important part of the economy and the epistemic ecosystem. Such AI systems will choose, from among the many coherent statements they might make, those that fit relevant selection criteria — for example, an AI selling products to humans might make statements judged likely to lead to a sale. If truth is not a valued criterion, sophisticated AI could use a lot of selection power to choose statements that further their own ends while being very damaging to others (without necessarily having any intention to deceive – see Diagram 1). This is alarming because AI untruths could potentially scale, with one system telling personalised lies to millions of people.

Aiming for robustly beneficial standards

Widespread and damaging A...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is New Data Visualisations of the EA Forum, published by Hamish Huggard on October 20, 2021 on The Effective Altruism Forum.

Followup to: EffectiveAltruismData.com: A Website for Aggregating and Visualising EA Data

EffectiveAltuismData.com now has a much fancier interface:

Highlights:

A new logo by Arjun Yadav who's been helping me out

Contents sidebar (which can be hidden with the top-left hamburger button)

Smooth scrolling

Dark mode!

I collected a bunch of data on the EA Forum via the GraphQL interface and created some visualisations form it.

The visualisations are interactive on EffectiveAltuismData.com.

Note that I filtered out a few extremely low karma outliers. They made karma axes harder to read, and it seemed best not to draw attention to them.

These visualisations took something like a full-time week to make.

What's next?

Visualisations of EA Funds grants

Visualisations of Giving What We Can pledges and donations

Spinning off the data aggregation code into a separate library for others to use

"Download data" buttons

I'm planning a 3-6 month hiking trip in a few weeks, so hopefully I'll be able to get one or two of these done before going off the grid.

If you have any data visualisation suggestions or requests, let me know in the comments, by direct message, or by email.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is New Data Visualisations of the EA Forum, published by Hamish Huggard on October 20, 2021 on The Effective Altruism Forum.

Followup to: EffectiveAltruismData.com: A Website for Aggregating and Visualising EA Data

EffectiveAltuismData.com now has a much fancier interface:

Highlights:

A new logo by Arjun Yadav who's been helping me out

Contents sidebar (which can be hidden with the top-left hamburger button)

Smooth scrolling

Dark mode!

I collected a bunch of data on the EA Forum via the GraphQL interface and created some visualisations form it.

The visualisations are interactive on EffectiveAltuismData.com.

Note that I filtered out a few extremely low karma outliers. They made karma axes harder to read, and it seemed best not to draw attention to them.

These visualisations took something like a full-time week to make.

What's next?

Visualisations of EA Funds grants

Visualisations of Giving What We Can pledges and donations

Spinning off the data aggregation code into a separate library for others to use

"Download data" buttons

I'm planning a 3-6 month hiking trip in a few weeks, so hopefully I'll be able to get one or two of these done before going off the grid.

If you have any data visualisation suggestions or requests, let me know in the comments, by direct message, or by email.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is [AN #167]: Concrete ML safety problems and their relevance to x-risk, published by Rohin Shah on October 20, 2021 on The AI Alignment Forum.

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that, while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

1. Robustness: Create models that are resilient to adversaries, unusual situations, and Black Swan events.

2. Monitoring: Detect malicious use, monitor predictions, and discover unexpected model functionality.

3. Alignment: Build models that represent and safely optimize hard-to-specify human values.

4. External Safety: Use ML to address risks to how ML systems are handled, including cyberwarfare and global turbulence.

Throughout, the paper attempts to clarify the problems’ motivation and provide concrete project ideas.

Dan Hendrycks' opinion: My coauthors and I wrote this paper with the ML research community as our target audience. Here are some thoughts on this topic:

1. The document includes numerous problems that, if left unsolved, would imply that ML systems are unsafe. We need the effort of thousands of researchers to address all of them. This means that the main safety discussions cannot stay within the confines of the relatively small EA community. I think we should aim to have over one third of the ML research community work on safety problems. We need the broader community to treat AI safety at least as seriously as safety for nuclear power plants.

2. To grow the ML safety research community, we need to suggest problems that can progressively build the community and organically grow support for elevating safety standards within the existing research ecosystem. Research agendas that pertain to AGI exclusively will not scale sufficiently, and such research will simply not get enough market share in time. If we do not get the machine learning community on board with proactively mitigating risks that already exist, we will have a harder time getting them to mitigate less familiar and unprecedented risks. Rather than try to win over the community with alignment philosophy arguments, I'll try winning them over with interesting problems and try to make work towards safer systems rewarded with prestige.

3. The benefits of a larger ML safety community are numerous. They can decrease the cost of safety methods and increase the propensity to adopt them. Moreover, to ensure that ML systems have desirable properties, it is necessary to rapidly accumulate incremental improvements, but this requires substantial growth since such gains cannot be produced by just a few card-carrying x-risk researchers with the purest intentions.

4. The community will fail to grow if we ignore near-term concerns or actively exclude or sneer at people who work on problems that are useful for both near- and long-term safety (such as adversaries). The alignment community will need to stop engaging in textbook territorialism and welcome serious hypercompetent researchers who do not post on internet forums or who happen not to subscribe to effective altruism. (We include a community strategy in the Appendix.)

5. We focus on reinforcement learning but also deep learning. Most of the machine learning research community studies deep learning (e.g., text processing, vision) and does not use, say, Bellman equations or PPO. While existentially catastrophic failures will likely require competent sequential decision-making agents, the relevant problems and solutions can often be better studied outside of gridworlds and MuJoCo. There is much u...