Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Law-Following AI 2: Intent Alignment + Superintelligence → Lawless AI (By Default), published by Cullen OKeefe on April 27, 2022 on The AI Alignment Forum.

This post is written in my personal capacity, and does not necessarily represent the views of OpenAI or any other organization. Cross-posted to the Effective Altruism Forum.

In the first post of this sequence, I defined "law-following AI" ("LFAI") and "intent alignment." In this post, I will begin to motivate the case for working on LFAI by showing that an AI agent A that is only intent-aligned (i.e., without additional law-following constraints directly on the agents) would in many cases break the law to advance its human principal H's interests.

If you agree that it is generally good for agents to be bound by law, then this should be a pro tanto reason to support work on LFAI in addition to intent alignment. If you need more convincing that LFAI would be generally good, the next few posts will explore some more specific motivations for LFAI.

Image by OpenAI's DALL·E

Sufficiently Intelligent Agents Can Evade Detection and Attribution

I suspect this point will need little argumentation to many of my readers. Evasion and obfuscation seems likely to be a generally promising means of ensuring self-preservation (because detection would increase the probability of punishment and constraint), which in turn is plausibly a convergent instrumental goal of optimizing AI systems.[1]

For example, to evade detection and attribution, A could:

Behave innocently only when being observed;

Orchestrate complex malevolent actions through a coordinated series of small, inscrutable or innocent-seeming steps;

Surreptitiously create and run subagents or copies of itself on new hardware, potentially very far removed from its original hardware;

Convince humans or other systems to take actions on its behalf, without revealing its true identity; and/or

Delete logs of its actions.

A sufficiently intelligent AI agent could likely consistently fool humans using these and other (unforeseen) techniques.

Other Ways to Circumvent Law

Even in the best case scenario, where the agent is detected and within the jurisdiction of a well-functioning legal system, it would be reasonable to question whether A or H could be effectively subject to normal legal processes.[2] If A had a motivation to, A could help H escape liability by, for example:[3]

"Outlawyering" counterparties.

Benefitting H in a way that would undermine recourse for creditors.

Shifting and hiding assets in ways that would make it difficult for creditors to reach.[4]

Persuasively arguing for the law to be changed in H's favor (by legislation or otherwise).

Engaging in vexatious litigation techniques to delay and raise the costs of the proceeding.

Convincingly fabricating favorable evidence and destroying or obscuring unfavorable evidence.

Bribing, misleading, or intimidating counterparties, witnesses, jurors, and judges.

A Competent Intent-Aligned Agent Will Sometimes Intentionally Break the Law

As I said in the previous post, on its face, intent-alignment does not entail law-following. Part of law is coercing prosocial behavior:[5] law incentivizes agents to behave in ways that they do not intrinsically want to behave. If A is aligned with H, whether A obeys the law depends on whether H wants A to obey the law. Subsequent posts will examine what legal consequences H might face if A causes legally cognizable harms. However, even if an adequate theory of liability for the H was available, it will seem impossible to hold H liable if nobody can produce evidence that some agent of H's was responsible for those harms. As argued above, a sufficiently intelligent agent probably could consistently avoid leaving any such evidence.

Detection and attribution would not solve the problem, however. Even ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Law-Following AI 1: Sequence Introduction and Structure, published by Cullen OKeefe on April 27, 2022 on The AI Alignment Forum.

This post is written in my personal capacity, and does not necessarily represent the views of OpenAI or any other organization. Cross-posted to the Effective Altruism Forum.

This sequence of posts will argue that working to ensure that AI systems follow laws is a worthwhile way to improve the long-term future of AI.[1]

The structure of this sequence will be as follows:

First, in this post, I will define some key terms and sketch what an ideal law-following AI ("LFAI") system might look like.

In the next few posts, I will explain why law-following might not emerge by default given the existing constellation of alignment approaches, financial objectives, and legal constraints, and explain why this is troubling.

Finally, I will propose some policy and technical routes to ameliorating these problems.

If the vision here excites you, and you would like to get funding to work on it, get in touch. I may be excited to recommend grants for people working on this, as long as it does not distract them from working on more important alignment issues.

Image by OpenAI's DALL·E.

Key Definitions

A law-following AI , or LFAI , is an AI system that is designed to rigorously comply with some defined set of human-originating rules ("laws"),[2] using legal interpretative techniques,[3] under the assumption that those laws apply to the AI in the same way that they would to a human. By "intrinsically motivated," I mean that the AI is motivated to obey those rules regardless of whether (a) its human principal wants it to obey the law,[4] or (b) disobeying the law would be instrumentally valuable.[5] (The Appendix to this post explores some possible conceptual issues with this definition of LFAI.)

I will compare LFAI with intent-aligned AI. The standard definition of "intent alignment" generally concerns only the relationship between some property of a human principal H and the actions of the human's AI agent A:

Jan Leike et al. define the "agent alignment problem" as "How can we create agents that behave in accordance with the user's intentions?"

Amanda Askell et al. define "alignment" as "the degree of overlap between the way two agents rank different outcomes."

Paul Christiano defines "AI alignment" as "A is trying to do what H wants it to do."

Richard Ngo endorses Christiano's definition.

Iason Gabriel does not directly define "intent alignment," but provides a taxonomy wherein an AI agent can be aligned with:

"Instructions: the agent does what I instruct it to do."

"Expressed intentions: the agent does what I intend it to do."

"Revealed preferences: the agent does what my behaviour reveals I prefer."

"Informed preferences or desires: the agent does what I would want it to do if I were rational and informed."

"Interest or well-being: the agent does what is in my interest, or what is best for me, objectively speaking."

"Values: the agent does what it morally ought to do, as defined by the individual or society."

All but (6) concern the relationship between H and A. It would therefore seem appropriate to describe them as types of intent alignment.

Alignment with some broader or more complete set of values—such as type (6) in Gabriel's taxonomy, Coherent Extrapolated Volition, or what Ngo calls "maximalist" or "ambitious" alignment—is perhaps desirable or even necessary, but seems harder than working on intent alignment.[6] Much current alignment work therefore focuses on intent alignment.

We can see that, on its face, intent alignment does not entail law-following. A key crux of this sequence, to be defended in subsequent posts, is that this gap between intent alignment and law-following is:

Bad in expectation for the long-term future.

Easier to bridge than the ga...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Two examples of European alt. protein policy, published by Aryan on April 27, 2022 on The Effective Altruism Forum.

Two weeks ago, at EAG: London, I was introduced to alternative protein policy. I went into conversations about alt pro policy without really knowing anything about it. I also realised there wasn’t a forum post that would give me examples of what this space looked like beforehand. I thought I’d try and fill that gap.

This post is a quick look at two concrete policy projects in Europe. Thanks to Alex Holst for helping me learn about them! All mistakes remain my own.

Public research funding

The first initiative that has had a strong push by people working in European alternative protein policy is to have government-backed funding for open-access alternative protein research.

Whereas companies who provide private alternative protein financing guard the research that gets produced, public funding for open-access research would give more talented people access to cutting-edge research and new developments, ideally allowing new alt pro companies to get off the ground more quickly and easily.

It should also allow more types of alternative proteins (like plant-based steak) to work their way into markets. The theory of change with alternative protein is to get consumers who would otherwise eat meat to switch over to plant-based/cultivated meat items. Research (and my anecdotal evidence) suggests that alt pro companies like Beyond Meat and Impossible Foods are getting close to hitting the nail on the head in regards to the taste of plant-based burgers. However, there is still an array of meat items that plant-based alternatives can’t compete with and a lack of companies to help fix that. What public funding can do is provide companies with the fuel to accelerate their research and development and allow them to get their products out on shelves at competitive price points.

Examples of this type of policy initiative

The Dutch government invested €60 million in funding for cultivated meat and precision fermentation.

Horizon Europe made €32 million in funding available for areas including sustainable protein research.

The Spanish Government invested €5.2M in Cultured Meat

Labels

Another policy initiative is to fight for clearer labels of alternative proteins in Europe.

Europe is fairly strict when it comes to advertising and labelling. For example, purely plant-based products cannot, in principle, be marketed with designations such as ‘milk’, ‘cream’, ‘butter’, ‘cheese’ or ‘yoghurt’, which are reserved by EU law for animal products. (See here)

The European Union considered banning plant-based producers from describing their products with words like ‘burgers’ and ‘sausages’. The European Alliance for Plant-based Foods tried to fight this and won! By lobbying against the proposed ban and getting the word out via media outlets across Europe, the ban difference came down to just 48 out of 705 votes, and phrases like “plant-based sausages” – were protected.

Another example of this can be seen with amendment 171.

I am generalising here. This study was done on participants in the U.S.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Snakebites kill 100,000 people every year, here's what you should know, published by MathiasKB on April 27, 2022 on The Effective Altruism Forum.

WHO estimates between 81,000-138,000 die every year from venomous snakebites. What on earth!? When I first heard this I refused to believe it. How had I managed to call myself an EA for so many years and not known snakebites were this big a deal? Not only do snakebites kill an unfathomable number of people, another four hundred thousand are permanently disabled every year.

In this post I will:

Provide an overview of the situation.

Look at the cost-effectiveness of conventional interventions to prevent and treat envenoming.

Argue that a new medical invention could make a massive difference and shill for the company that invented it.

Why do snakebites kill and maim so many?

Imagine you're a farmer working in the amazon. One day, as you are working your field, you suddenly feel a searing pain from your leg. You see a snake dislodging its fangs from your calf and quickly slither away. Was it venomous or not? Even if it were a venomous snake that bit you, there's a 50% chance it was a dry-bite, and that the swelling will go away after a while. The trip to the nearest medical facility will take an entire day and cost you a fortune. You decide to wait and see. After a while the symptoms get worse and you decide to consult your family. They tell you to go to the town's spiritual healer, who can treat you for a much smaller cost. When morning comes around, you finally make the decision to go to the hospital. It's now too late. By the time you arrive the venom will have done its damage. Even if you receive the correct antivenom treatment and survive, you will suffer permanent disability and be in for a long and expensive hospital stay.

After a few weeks spent recovering in the hospital you will find yourself handicapped, out of work, and in debt.

Snakebite envenoming is an enormous burden on tropical regions in the third and second world. Like other neglected tropical diseases, the burden of snakebites is much larger than the just deaths they cause. Having a productive member of one's family or community suddenly die or become permanently disabled wrecks havoc on all those who were dependent on them. Children, for example, are often removed from schools to make up for the lost income and additional person who must now be taken care of.

After reading up on various studies, WHO reports, and meeting with the CEO of a venom diagnostics startup, I believe there are two primary reasons why snakebites do so much harm.

Snakebites are expensive to treat

Treating envenoming requires administering antivenom which is expensive to produce. Each type of venomous snake species requires its own antivenom. Administering the wrong antivenom is costly and harms the patient.

The hospitalization and antivenom treatment is far too expensive for many to afford. In Bangladesh, nearly 75% of snakebite victims spent their savings on treatment and over 60% had to borrow to meet the costs.

The world currently produces roughly half the antivenom needed to treat every patient. WHO argues (without using these exact terms) that there is a lemons-market for antivenom, where production of low-grade and ineffective antivenom is crowding out high-quality producers. This has reduced the profitability of antivenom production and prevents producers from scaling up production. For example, FAV Afrique, a dependable and widely distributed antivenom in sub-saharan Africa, was discontinued in 2016. I haven't been able to verify that recent discontinuations of antivenom production can be attributed to a lemon market, and an alternative explanation could be that WHO is neglecting that a worse treatment for less money may be the better outcome for some.

Lemons-market or not, we ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Intro to brain-like-AGI safety] 13. Symbol grounding & human social instincts, published by Steve Byrnes on April 27, 2022 on The AI Alignment Forum.

Part of the “Intro to brain-like-AGI safety” post series.

13.1 Post summary / Table of contents

In the previous post, I proposed that one path forward for AGI safety involves reverse-engineering human social instincts—the innate reactions in the Steering Subsystem (hypothalamus and brainstem) that contribute to human social behavior and moral intuitions. This post will go through some examples of how human social instincts might work.

My intention is not to offer complete and accurate descriptions of human social instinct algorithms, but rather to gesture at the kinds of algorithms that a reverse-engineering project should be looking for.

This post, like Posts #2–#7 but unlike the rest of the series, is pure neuroscience, with almost no mention of AGI besides here and the conclusion.

Table of contents:

Section 13.2 explains, first, why I expect to find innate, genetically-hardwired, social instinct circuits in the hypothalamus and/or brainstem, and second, why evolution had to solve a tricky puzzle when designing these circuits. Specifically, these circuits have to solve a “symbol grounding problem”, by taking the symbols in a learned-from-scratch world-model, and somehow connecting them to the appropriate social reactions.

Section 13.3 and 13.4 go through two relatively simple examples where I attempt to explain recognizable social behaviors in terms of innate reaction circuits: filial imprinting in Section 13.3, and fear-of-strangers in Section 13.4.

Section 13.5 discusses an additional ingredient that I suspect plays an important role in many social instincts, which I call “little glimpses of empathy”. This mechanism enables reactions where recognizing or expecting a feeling in someone else triggers a “response feeling” in oneself—for example, if I notice that my rival is suffering, it triggers the warm feelings of schadenfreude. To be clear, “little glimpses of empathy” have little in common with how the word “empathy” is used normally; “little glimpses of empathy” are fast and involuntary, and are involved in both prosocial and antisocial emotions.

Section 13.6 wraps up with a plea for researchers to figure out exactly how human social instincts work, ASAP. I will have a longer wish-list of research directions in Post #15, but I want to emphasize this one right now, as it seems particularly impactful and tractable. If you (or your lab) are in a good position to make progress but would need funding, email me and I’ll keep you in the loop about possible upcoming opportunities.

13.2 What are we trying to explain and why is it tricky?

13.2.1 Claim 1: Social instincts arise from genetically-hardcoded circuitry in the Steering Subsystem (hypothalamus & brainstem)

Let’s talk about envy, to pick a central example of social emotions. (Remember, the point of this post is that I want to understand human social instincts in general; I don’t literally want AGIs to be envious—see previous post, Section 12.4.3.)

I claim: there needs to be genetically-hardcoded circuitry in the Steering Subsystem—a.k.a. an “innate reaction”—which gives rise to the feeling of envy.

Why do I think that? A few reasons:

First, envy seems to have a solid evolutionary justification. I’m referring here to the usual evolutionary psychology story: Basically, for most of human history, life was full of zero-sum competitions for status, mates, and resources, such that an aversive reaction to other people’s successes (under some circumstances) would have been plausibly adaptive in general.

Second, envy seems to be innate, not learned. I think parents will agree that children often react negatively to the successes of their siblings and classmates starting fr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SERI ML Alignment Theory Scholars Program 2022, published by Ryan Kidd on April 27, 2022 on The AI Alignment Forum.

The Stanford Existential Risks Initiative (SERI) recently opened applications for the second iteration of the ML Alignment Theory Scholars (MATS) Program, which aims to help aspiring alignment researchers enter the field by pairing them with established research mentors and fostering an academic community in Berkeley, California over the summer. Current mentors include Alex Gray, Beth Barnes, Evan Hubinger, John Wentworth, Leo Gao and Stuart Armstrong. Applications close on May 15 and include a written response to mentor-specific selection questions, viewable on our website.

Who is this program for?

Our ideal applicant has:

an understanding of the AI alignment research landscape equivalent to having completed EA Cambridge’s AGI Safety Fundamentals course;

previous experience with technical research (e.g. ML, CS, maths, physics, neuroscience, etc.);

strong motivation to pursue a career in AI alignment research.

For the first stage of the program, we asked each alignment researcher to provide a set of questions that are sufficient to select candidates they would be happy to mentor. Applicants can apply for multiple mentors, but will have to complete each mentor’s selection questions.

What will this program involve?

Over four weeks, the participants will develop an understanding of a research agenda at the forefront of AI alignment through online readings and cohort discussions, averaging 10 h/week from Jun 6 to Jul 1. After this initial upskilling period, the scholars will be paired with an established AI alignment researcher for a two-week “research sprint” to test fit from Jul 4 to Jul 15. Assuming all goes well, scholars will be accepted into an eight-week intensive research program in Berkeley, California over the US summer break (Jul 25 to Sep 16).

Participants will obtain a $6,000 grant for completing the training and research sprint and $16,000 at the conclusion of the program. Furthermore, all expenses will be covered, including accommodation, office space and networking events with the Bay Area alignment community. We are happy to continue providing funding after the two month period to promising scholars, at the discretion of our research mentors. International students can apply to the program, and will arrive in the US under a B1 visa.

We hope to run another iteration of the program in the winter, and possibly in the fall. If you are not able to apply for the summer program, we encourage you to apply for the fall or winter. We may be able to offer different types of visas in future iterations.

Theory of change

This section is intended to explain the reasoning behind our program structure and is not required reading for any applicant. SERI MATS’ theory of change is as follows:

We believe that AI alignment research is pre-paradigmatic, with a diversity of potentially promising research agendas. Therefore, we aim to support many different alignment research agendas to decorrelate failure. We also aim to accelerate the development of scholars into researchers capable of pursuing original agendas and mentoring further scholars.

We believe that working 1:1 with a mentor is the best and quickest way to develop the ability to conduct alignment theory research—that reading curriculum alone is worse for a large number of participants. Moreover, we believe that our target scholars might be able to produce value directly for the mentors by acting as research assistants. For the first few months, we are generally more excited about mentees working on an established mentor’s research agenda than on their own.

We believe that our limiting constraint is mentor time. This means we wish to have strong filtering mechanisms (e.g. candidate selection questions) to...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 21 on 21, published by Amir Bolous on April 25, 2022 on LessWrong.

This is a linkpost for/

Today, I turn 21. Here are 21 reflections for 21 years.

The highest ROI thing you can invest in early on is learning to be an independent thinker. The world is full of noise. It's impossible to drown out it all out, but it's possible to pick out the tunes if you listen carefully. Learn to derive things for yourself. Read and listen to what other people say but always arrive at your own conclusions.

Corollary to this: there may be forces that drive you to take a certain path. A shocking number of new possibilities open up when you realize you don't have to walk on the path laid down by others for you, and you can in fact craft your own path. You're also probably underestimating yourself.

Who you surround yourself with is very important. Everyone talks about this but I still think we underestimate how large of an impact this has. Do your friends inspire you? Do they encourage you to become a better person? A good litmus test for what kind of person you are is are you a good influence on your friends' lives? Do you lift them up or bring them down?

Initially, the bottleneck to doing something is often knowing that it's even an option. After that, the bottleneck is courage not intelligence.

There's value in approaching different aspects of your life with more of a "scientist mindset." Many of us often spend too much time worrying about what to do as opposed to trying things to see what we actually like doing. Treat all aspects of your life as little experiments. Treat the project you've been to afraid to start as an experiment that may fail. Treat your exercise routine that you haven't gotten around to commit to as an experiment. Try, observe, and adapt. There's nothing to be afraid of if you fail because it's just an experiment.

Try not to judge others because you don't know what they care about or what their value function actually is. What makes sense for others might not make sense to you if you don't care about the same things.

Be competitive with yourself. Are you a better person than you were yesterday? Did you learn something new today? Try not to be competitive with others because it leads to some common failure modes like a zero-sum, scarcity mindset.

Being honest with yourself is hard, but your intuition is more telling than you think.

Be true to yourself is not great advice. Instead, think about what type of person you want to be. Think about what will help you become more of that kind of person. Do those things.

Speaking of advice, most advice people will give you will be bad for you. The issue is that people are not acutely aware of your circumstances. Most advice is in fact advice people would give to their younger selves, which means they're projecting advice calibrated against their experiences and circumstances instead of yours. Be wary of advice from people who don't know you well.

Most cliches you hear (e.g. "do what you love", "work hard", "journey before destination" etc.) however are incredibly good advice. They're cliches for a reason. They've survived thousands of years of evolution. Note also, that most generic advice actually has multiple levels of nuance that can only be understood through experience.

Do stuff in public. Think in public. Write in public. Ship in public. Doing things publicly attracts like-minded people and opportunities that you wouldn't be able to easily otherwise find. It's a great vehicle for serendipity.

If you're constantly rushing from one thing to another, are you really saving time or are you wasting it?

We make a lot of assumptions in our lives, many unconsciously. Some assumptions are good (e.g. in a scientific context in order to let's say model something), but be careful of what you assume (especially unconsciously) about othe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Copilot Accelerates Timelines, published by Michaël Trazzi on April 26, 2022 on The AI Alignment Forum.

"Say we have intelligences that are narrowly human / superhuman on every task you can think of (which, for what it’s worth, I think will happen within 5-10 years). How long before we have self-replicating factories? Until foom? Until things are dangerously out of our control? Until GDP doubles within one year? In what order do these things happen?" (source)

When discussing Takeoff Speeds, I feel the debate often gets stuck in some kind of false dichotomy between Fast and Slow, where the crux seems to be about whether some self-improving AI would be able to foom without human assistance.

Instead, we could get a Moderate Takeoff (think months or years), where AI does not self-improve (by itself). Instead, there would be a reinforcing feedback loop where progress in AI leads to AI becoming increasingly useful to make progress in AI, with humans in the loop at all times.

On top of that, things might just happen privately at some AI lab for a few months until the AI is able to foom by itself, which will look like foom for everyone outside that lab.

AI Helping Humans with AI

In Superintelligence, takeoff is defined as the period between AGI and superintelligence. In this post, I will be using as takeoff's starting point the first "AI Helping Humans with AI" (in a meaningful way), or AIHHAI for short, since it will arise before we get fully general intelligence and accelerate AI progress. Here are some examples of what I have in mind for "helping humans in a meaningful way":

GPT-N that you can prompt with "I am stuck with this transformer architecture trying to solve problem X". GPT-N would be AIHHAI if it answers along the lines of "In this arXiv article, they used trick Z to solve problems similar to X. Have you considered implementing it?", and using an implementation of Z would solve X >50% of the time.

Another example would be if some code generation tool like Copilot makes ML engineers substantially more productive in writing ML code. Making predictions about productivity is tricky and hard to measure, but it would involve something like accepting code suggestions a decent amount, like 100x more than what engineers using Copilot currently accept.

(Sidenote: My general impression from people using Copilot is that they believe it's becoming increasingly useful, and use it on a daily basis, though it rarely auto-completes the right line of code right away. Given that we had Codex/Copilot last year, and that Sam Altman hinted at some new Codex capabilities in his ACX Q&A, I think we will get some impressive release for Copilot/Codex sometime this year that most engineers will want to use. (Similar to how a lot of developers are used to using Visual Studio's suite, especially intellisense.) The model I have in mind for "AI helping humans with AI" could be this one, though it will probably require 1-2x more iterations.)

Moderate Takeoff

A Moderate Takeoff is defined as "one that occurs over some intermediary temporal interval, such as months or years". For AIHHAI, we can distinguish two cases:

AIHHAI is developed by some AI lab working on it privately. That lab has a lead compared to the other labs, since they are working more productively using AIHHAI. Thus, they might reach superintelligence first, without allowing enough time for the rest of the world to compete.

AIHHAI is made public, or quickly (think months) reproduced by others publicly or privately. In any case, some AIHHAI model is eventually made public, and there is not only one group using AIHHAI--other companies are adopting the same strategy (multipolar scenario).

For the first case, you can think of OpenAI using a new version of Copilot internally, that enables their team to quickly build another, even ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consider Changing Your Forum Username to Your Real Name, published by Patrick Gruban on April 26, 2022 on The Effective Altruism Forum.

I was positively surprised to find out that I was able to edit my username in the forum to be my full name. As I was previously under the impression that this was impossible, I wanted to share this and encourage users to consider switching to their full names.

The suggestion in the how-to-guide is:

In general, we think that real names are good for community bonding, and we encourage you to use yours. But it's not required.

I think this is a good policy. I can imagine cases where using a pseudonym might make it easier to communicate openly without people outside the community being able to connect the post to the author. For most posters, especially the frequent posters, it seems relatively easy to find out who the author is. After meeting several users at EAGs I'm building up a mental database where I keep track of real names (used on Swapcard and emails), forum user names, and sometimes nicknames and Twitter handles. This seems unnecessary.

Connecting names you saw in comments and posts to name tags at conferences makes it easier for people new to the community to start conversations based on what you read. It's also easier when you hear others refer to people by their real name.

In a growing community that aspires to be welcoming, I think it's a good norm to make it easy for people to learn about the engaged participants. In addition to using the real name, I would also like to encourage adding a description to the profile. This can include the current organisation, group, university, cause area or GWWC membership. Similar to Swapcard at the EAG conferences, it helps to understand better where someone is coming from or is currently active.

A counter-argument might be that readers might defer too much to people with impressive affiliations instead of focussing on the content. I would agree with that. However, currently, many pseudonyms seem to be known to engaged members, which leads to different levels of knowledge.

Looking at the posts with the highest karma it's nice to see many using real names already and I hope to see more in future.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [$20K In Prizes] AI Safety Arguments Competition, published by ThomasWoodside on April 26, 2022 on The Effective Altruism Forum.

TL;DR—We’re distributing $20k in total as prizes for submissions that make effective arguments for the importance of AI safety. The goal is to generate short-form content for outreach to policymakers, management at tech companies, and ML researchers. This competition will be followed by another competition in around a month that focuses on long-form content.

This competition is for short-form arguments for the importance of AI safety. For the competition for distillations of posts, papers, and research agendas, see the Distillation Contest.

Objectives of the arguments

To mitigate AI risk, it’s essential that we convince relevant stakeholders sooner rather than later. To this end, we are initiating a pair of competitions to build effective arguments for a range of audiences. In particular, our audiences include policymakers, tech executives, and ML researchers.

Policymakers may be unfamiliar with the latest advances in machine learning, and may not have the technical background necessary to understand some/most of the details. Instead, they may focus on societal implications of AI as well as which policies are useful.

Tech executives are likely aware of the latest technology, but lack a mechanistic understanding. They may come from technical backgrounds and are likely highly educated. They will likely be reading with an eye towards how these arguments concretely affect which projects they fund and who they hire.

Machine learning researchers can be assumed to have high familiarity with the state of the art in deep learning. They may have previously encountered talk of x-risk but were not compelled to act. They may want to know how the arguments could affect what they should be researching.

We’d like arguments to be written for at least one of the three audiences listed above. Some arguments could speak to multiple audiences, but we expect that trying to speak to all at once could be difficult. After the competition ends, we will test arguments with each audience and collect feedback. We’ll also compile top submissions into a public repository for the benefit of the x-risk community.

Note that we are not interested in arguments for very specific technical strategies towards safety. We are simply looking for sound arguments that AI risk is real and important.

Competition details

The present competition addresses shorter arguments (paragraphs and one-liners) with a total prize pool of $20K. The prizes will be split among, roughly, 20-40 winning submissions. Please feel free to make numerous submissions and try your hand at motivating various different risk factors; it's possible that an individual with multiple great submissions could win a good fraction of the prize. The prize distribution will be determined by effectiveness and epistemic soundness as judged by us. Arguments must not be misleading.

To submit an entry:

Please leave a comment on this post (or submit a response to this form), including:

The original source, if not original.

If the entry contains factual claims, a source for the factual claims.

The intended audience(s) (one or more of the audiences listed above).

In addition, feel free to adapt another user’s comment by leaving a reply⁠⁠—prizes will be awarded based on the significance and novelty of the adaptation.

Note that if two entries are extremely similar, we will, by default, give credit to the entry which was posted earlier. Please do not submit multiple entries in one comment; if you want to submit multiple entries, make multiple comments.

The first competition will run until May 27th, 11:59 PT. In around a month, we’ll release a second competition for generating longer “AI risk executive summaries'' (more details to come...