The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How Long Do Policy Changes Matter? New Paper, published by zdgroff on November 2, 2023 on The Effective Altruism Forum.

A key question for many interventions' impact is how long the intervention changes some output counterfactually, or how long the intervention washes out. This is often the case for work to change policy: the cost-effectiveness of efforts to pass

animal welfare ballot initiatives

,

nuclear non-proliferation policy

,

climate policy

, and

voting reform

, for example, will depend on (a) whether those policies get repealed and (b) whether they would pass anyway. Often there is an explicit assumption, e.g., that passing a policy is equivalent to speeding up when it would have gone into place anyway by

X

years.

[1]

[2]

As people routinely note when making these assumptions, it is very unclear what assumption would be appropriate.

In a

new paper

(my economics "job market paper"), I address this question, focusing on U.S. referendums but with some data on other policymaking processes:

Policy choices sometimes appear stubbornly persistent, even when they become politically unpopular or economically damaging. This paper offers the first systematic empirical evidence of how persistent policy choices are, defined as whether an electorate's or legislature's decisions affect whether a policy is in place decades later. I create a new dataset that tracks the historical record of more than 800 state policies that were the subjects of close referendums in U.S. states since 1900. In a regression discontinuity design, I estimate that passing a referendum increases the chance a policy is operative 20, 40, or even 100 years later by over 40 percentage points. I collect additional data on U.S. Congressional legislation and international referendums and use existing data on state legislation to document similar policy persistence for a range of institutional environments, cultures, and topics. I develop a theoretical model to distinguish between possible causes of persistence and present evidence that persistence arises because policies' salience declines in the aftermath of referendums. The results indicate that many policies are persistently in

place - or not - for reasons unrelated to the electorate's current preferences.

Below I'll pull out some key takeaways that I think are relevant to the EA community and in some cases did not make it into the paper.

Overview of Results and Methods

My strategy in the paper involves comparing how many policies that barely passed or barely failed in U.S. state-level referendums are in place over time. I collect data on all referendums whose vote outcome is within 2.5 percentage points of the threshold for passage (typically 50%) since 1900 in a subset of U.S. states. I then do what's called a regression discontinuity design, which allows me to estimate the effect of passing a referendum on whether it is in place later on.

The headline result from the paper is below. Many referendums that barely fail eventually pass in the first few years or decades afterward, and then this levels off. At 100 years later, just under 80% of the barely passed ones are in place compared to just under 40% of the barely failed ones. Importantly, the hazard rate - the rate at which this effect declines over time - is much lower in the later years, meaning that if you were to extrapolate this out beyond 100 years, the effect at 200 years would be expected to be significantly more than 40% * 40%.

Something relevant to EAs that I don't focus on in the paper is how to think about the effect of campaigning for a policy given that I focus on the effect of passing one conditional on its being proposed. It turns out there's a method (

Cellini et al. 2010

) for backing this out if we assume that the effect of passing a referendum on whether the policy is in place lat...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Saying the quiet part out loud: trading off x-risk for personal immortality, published by disturbance on November 2, 2023 on LessWrong.

Statement:

I want to deliberately balance the caution and the recklessness in developing AGI, such that it gets created in the last possible moment so that I and my close ones do not die.

This Statement confuses me. There are several observations I can make about it. There are also many questions I want to ask but have no idea how to answer. The goal of this post is to deconfuse myself, and to get feedback on the points that I raised (or failed to raise) below.

First observation:

The Statement is directly relevant to LW interests.

It ties together the issues of immortality and AI risk, both of which are topics people here are interested in. There are countless threads, posts and discussions about high-level approaches to AI safety, both in the context of "is" (predictions) and "ought" (policy). At the same time, there is still a strong emphasis on the individual action, deliberating on which choices to make to improve the to marginal effects

of living life in a certain way

. The same is true for immortality. It has been discussed to death, both from the high-level and from the individual,

how-do-I-sign-up-for-Alcor

point of view. The Statement has been approached

from the "is"

, but not from the "ought" perspective. At the same time:

Second observation:

No one talks about the Statement.

I have never met anyone who expressed this opinion, neither in-person nor online, even after being a part (although, somewhat on the periphery) of the rationalist community for several years. Not only that, I have not been able to find any post or comment thread on LW or SSC/ACX that discusses it, argues for or against it, or really gives it any attention whatsoever. I am confused by this since the Statement seems to be fairly straightforward.

One reason might be the:

Third observation

: Believing in the Statement is low status, as it constitutes an almost-taboo opinion.

Not only no one is discussing it, but the few times when I expressed the Statement in person (at EA-infiltrated rationalists meetups), it was treated with suspicion or hostility. Although to be honest, I'm not sure how much this is me potentially misinterpreting the reactions. I got the impression that it is seen as sociopathic. Maybe it is?

Fourth observation

: Believing in the Statement is incompatible with long-termism, and it runs counter to significantly valuing future civilisation in general.

Fifth observation:

Believing in the Statement is compatible with folk morality and revealed preferences of most of the population.

Most people value their lives, and the lives of those around them to a much greater extent than those far away from them. This is even more true for the future lives. The revealed-preference discount factor is bounded away from 1.

Sixth observation:

The Statement is internally consistent.

I don't see any problems with it on the purely logical level. Rational egoism (or variants thereof) constitutes a valid ethical theory, although it is potentially prone to self-defeat.

Seventh observation:

Because openly admitting to believing in the Statement is disadvantageous, it is possible that many people in fact hold this opinion secretly.

I have no idea how plausible this is. Judging this point is one of my main goals in writing this post. The comments are a good place for debating the meta-level points, but, if I am right about the cost of holding this opinion - not so much for counting its supporters. An alternative is

this anonymous poll I created

please vote if you're reading this.

Eighth observation:

The Statement has the potential to explain some of the variance of attitudes to AI risk-taking.

One way of interpreting this observation might be that people arguing a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Complementary notes on Alvea [unofficial], published by mxschons on November 2, 2023 on The Effective Altruism Forum.

My ex-colleagues just posted the

official Alvea winddown announcement

as well as

Kyle's reflections on our activities

over the past 1.5 years. I wanted to complement these posts with some additional comments.

Publication of the South Africa Trial Results

Our academic publication of the South Africa Trial results are currently under review, but you can

see the submitted manuscript here

. We also published the

complete data-set as well as the full study report

. While Kyle describes our efficacy results correctly as underwhelming (btw - the Janssen comparator as well), the trial made several important scientific contributions. To our knowledge this study is the first time where

A naked DNA based SARS-CoV-2 booster candidate was studied in preimmunized humans (with ~80% having hybrid immunity from previous infections).

Unusually high doses of up to 8 mg DNA plasmid were administered intradermally / subcutaneously during a single visit.

A SARS-CoV-2 vaccine candidate was compared during its first Phase-1 safety study against a licensed comparator (Janssen's Ad26.COV2.S). (Sidenote: it is insane that this seems to be the only COVID-19 trial that managed to get an comparator)

Lessons learned

In addition to Kyle's list, here are some more I'd like to point out:

Experience matters:

Alvea matured substantially over time. From my perspective, many important internal operations mistakes could have been prevented if there had been more experience in the team earlier on. Same for research decisions and other mistakes we made. Sometimes you really want this person who has 4,000+ h of work experience and not 40h of research experience. Maybe even often.

You can hire people remotely:

It is very common for orgs to be hesitant to hire internationally. I myself would not have been able to work for Alvea if Grigory wouldn't have been creative about contracting and working with Employer of Records to employ ~50% of the 45 person team. I claim that organizations are missing out on top talent if they are only fishing in their country.

We had kick-ass values I am still identifying with

(shoutout to

Eric

for shaping our company culture with those)

Ownership: We take personal responsibility for achieving exceptional results. We draw energy from doing the "impossible."

Agility: We move with speed and flexibility in the face of constant uncertainty.

Truthseeking: We chase the truth, especially when it requires courage. We experiment to change our minds quickly.

Care: We have each other's back. Our mutual support sparks our growth as a team and our impact in the world.

Medical countermeasures matter:

I get the vibes in EA that people are not excited about medical countermeasures / their deployment. I'd like to challenge that. Ambulances, firefighters, and police take less than 15 minutes to arrive at an emergency scene. Starting a clinical trial 100d? Humanity needs an equivalent pharmaceutical preparedness and response ability to stop pandemics in their tracks. For instance, I would like to see work on a 100h response plan (in addition to the 100d CEPI timeline) to an outbreak using candidate vaccines and ultra-rapid ring-vaccination trials. So consider

PanopliaLabs

when allocating your donations this year.

Drug Development Explainer Series

I just

cross posted an explainer series on drug development

from my website I wrote up with

James Smith

and

Kirsten Angeles

. I argue it captures most of the insights on the pharma landscape we made at Alvea and recommend it to everyone who wants to get a comprehensive overview of how drugs are made.

One article in the series describes some of the core cultural and operational activities that allowed Alvea to execute that quickly. For conveni...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Propaganda or Science: A Look at Open Source AI and Bioterrorism Risk, published by 1a3orn on November 2, 2023 on LessWrong.

0: TLDR

I examined all the biorisk-relevant citations from a policy paper arguing that we should ban powerful open source LLMs.

None of them provide good evidence for the paper's conclusion. The best of the set is evidence from statements from Anthropic -- which rest upon data that no one outside of Anthropic can even see,

and

on Anthropic's interpretation of that data. The rest of the evidence cited in this paper ultimately rests on a single extremely questionable "experiment" without a control group.

In all, citations in the paper provide an illusion of evidence ("look at all these citations") rather than actual evidence ("these experiments are how we know open source LLMs are dangerous and could contribute to biorisk").

A recent

further

paper on this topic (published

after

I had started writing this review) continues this pattern of being more advocacy than science.

Almost all the bad papers that I look at are funded by Open Philanthropy. If Open Philanthropy cares about truth, then they should stop burning the epistemic commons by funding "research" that is always going to give the same result no matter the state of the world.

1: Principles

What

could

constitute evidence that powerful open-source language models contribute or will contribute substantially to the creation of biological weapons, and thus that we should ban them?

That is, what kind of anticipations would we need to have about the world to make that a

reasonable thing to think

? What other beliefs are a necessary part of this belief making any sense at all?

Well, here are two pretty-obvious principles to start out with:

Principle of Substitution

: We should have evidence of some kind that the LLMs can (or will) provide information that humans cannot also easily access through other means -- i.e., through the internet, textbooks, YouTube videos, and so on.

Blocker Principle

: We should have evidence that the

lack of information

that LLMs can (or will) provide is in fact a significant blocker to the creation of bioweapons.

The first of these is pretty obvious. As example: There's no point in preventing a LLM from telling me how to make gunpowder, because I can find out how to do that from an encyclopedia, a textbook, or a novel like Blood Meridian. If you can substitute some other source of information for an LLM with only a little inconvenience, then an LLM does not contribute to the danger.

The second is mildly less obvious.

In short, it could be that

most of

the blocker to creating an effective bioweapon is not knowledge -- or the kind of knowledge that an LLM could provide -- but something else. This "something else" could be access to DNA synthesis; it could be the process of culturing a large quantity of the material; it could be the necessity of a certain kind of test; or it could be something else entirely.

You could compare to atomic bombs -- the chief obstacle to building atomic bombs is

probably not

the actual knowledge of how to do this, but access to refined uranium. Thus, rather than censor every textbook on atomic physics, we can simply control access to refined uranium.

Regardless, if this other blocker constitutes 99.9% of the difficulty in making an effective bioweapon, and lack of knowledge only constitutes 0.1% of the difficulty, then an LLM can only remove that 0.1% of the difficulty, and so open source LLMs would only contribute marginally to the danger. Thus, bioweapons risk would not be a good reason to criminalize open-source LLMs.

(I am not speaking theoretically here -- a

paper

from a researcher at the Future of Humanity Institute argues that the actual product development cycle involved in creating a bioweapon is far, far more of an obstacle to its cre...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EAGxVirtual: Speaker announcements, timings, and other updates, published by Sasha Berezhnoi on November 2, 2023 on The Effective Altruism Forum.

EAGxVirtual

is fast approaching and we're excited to share some more details about the event!

This post covers updates from the team, including dates and times, content, unique features, and demographic data. In the

previous post

, we covered the conference theme, reasons to attend, and reviews from the previous attendees.

Content: what to expect

We're very excited to announce our key speakers for this event:

Peter Singer

on the most pressing moral issues facing humanity

.

Bruce Friedrich, President of The Good Food Institute

on longtermism and alternative proteins.

Carl Robichaud, Co-lead on nuclear policy grantmaking at Longview Philanthropy

on a turning point in the story of nuclear weapons.

Olga Kikou, Head of the EU Office of Compassion in World Farming

on ending the cage age in the EU.

Neel Nanda, Research Engineer at DeepMind

on open problems in mechanistic interpretability.

We are working hard on the program. Beyond the above talks (and many more talks and workshops!), you can expect office hours hosted by experts and EA orgs, fireside chats, group meetups and icebreakers, lightning talks from attendees, and unofficial satellite events.

The tentative schedule is available

here

(all times are in UTC).

Please note that the schedule is subject to change. The final schedule will be available on the Swapcard app, which we aim to launch next week.

Taking action anywhere in the world

We have already received 600 applications from people representing over 70 countries. We welcome all who have a genuine interest in learning more or connecting, including those who are new to effective altruism. If you are a highly-engaged EA, you can make a difference by being responsive to requests from first-time attendees.

The map below shows the geographical distribution of the participants:

We would love to see more applications.

If you know someone who you think should attend the conference, please encourage them to apply by sending them this link:

eagxvirtual.com

The deadline for applications is 11:59 pm UTC on Thursday, 16 November.

Apply here

if you haven't already.

Dates and times

The conference will be taking place from 10 am UTC on Friday, November 17th, until 11:59 pm UTC on Sunday, November 19th.

We don't expect you to always be online

you can be flexible with your participation! It's completely okay if you can attend only on one of the days. Recordings will be available for registered attendees, so you can watch the sessions you missed later.

Friday will feature introductory-level content for participants who are relatively new to EA and a career fair on

Gather Town

.

Saturday and Sunday will have full-day schedules, starting at 7 am UTC each day.

There will be a break in the program on Sunday between 2 am and 7 am UTC.

Conference features

Our main content and networking platform for the conference is the

Swapcard

. We will share access to the app with all the attendees on November 6 and provide guidance on how to use it and get the most out of the conference.

We collaborate with EA Gather Town to make an

always-available virtual venue

for the attendees to spark more connections and unstructured discussions throughout the conference.

Extensive

stewardship program

. We will highlight ambassadors across different cause areas whom you can speak to get advice or feedback on your career plans.

Evergreen discussion space

: we are inviting everyone to use EA Anywhere Slack as a discussion space. No more Slacks that are abandoned immediately after the conference is over!

Ways to contribute

If you want to represent your organization at the career fair or host office hours, please,

fill out this form

.

Apply to give a Lightning talk

if ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Participate in the Donation Election and the first weekly theme (starting 7 November), published by Lizka on November 2, 2023 on The Effective Altruism Forum.

TLDR

: Participate in the Donation Election, start writing posts for "Effective giving spotlight" week (starting this Tuesday!), and more.

There are many ways to participate:

Get involved in Giving Season events

Donate to the Donation Election Fund

to encourage discussion and participation (we're partially matching donations

)

Add candidates to the Election

Pre-vote

[1]

to register your interest (pre-votes are anonymous, but we'll know how many people pre-voted for a given candidate in the Election)

Share your experience donating, fundraising, earning to give, or more

or your uncertainties or considerations about where we should donate and how we should fundraise

Fundraise for your project

Explain how your project would use extra funding (particularly for Marginal Funding Week

), share impact analyses or retrospectives,

invite Forum users to ask you questions

, and see if your project should be listed as a candidate

in the Election

Explore

The

Giving portal

Writing about effective giving

Follow the

Donation Election

(This post is an update to our

earlier announcement

about Giving Season on the Forum.)

Giving Season & weekly discussion themes

Start preparing for discussion themes on the EA Forum:

Theme and dates

Description

Effective Giving Spotlight

(7-14 November) - starting this Tuesday!

How can we grow the amount of funding going to effective projects aimed at improving the world? We'll feature people's experiences with donating, fundraising, earning to give, etc.

See more details below.

Marginal Funding Week

(14-21 November)

How would your project use extra funding?

To decide whether donating to a given project or organization is cost-effective, it's really useful to know how marginal funding would get used. We'll invite EA organizations and projects to describe how they would use extra donations - a bit like

this post about LTFF

or otherwise share more about what they do.

We might also try to collect a summary of the key information in one post at the end of the week.

Donation Debate Week

(21-28 November)

Where should we donate (and what should we vote for in the Donation Election)?

Discuss which interventions and projects are most

cost-effective

and how they should vote (and donate!). I'm hoping to see estimates (including rough "back of the envelope calculations"

BOTECs

) and productive disagreement or identification of what the

cruxes

that drive different conclusions are.

We'll also probably feature some classic writing on the topic, some relevant

AMAs

, and more.

First weekly theme: Effective Giving Spotlight (November 7-14)

A lot of promising projects are funding-constrained

[2]

they'd get more done if they had more funding. "Effective Giving Spotlight" week will feature content on how we can grow the amount of funding going to effective projects aimed at improving the world. ef

Consider participating!

You can write posts (or

link-posts

, including things like reviews of classic writing on the topic), comment on others' posts, or share the event. You might want to write about:

Your experience donating, fundraising, earning to give, etc. - lessons, things you've changed your mind on,

postmortems

Thoughts on who should consider

earning to give

Uncertainties you have about where to donate

What effective projects would be particularly useful

… or anything else related to effective giving.

If you're not sure if something would be useful or relevant, please feel free to reach out to me or to the Forum team. We'll post smaller announcements with more details about the other themes.

Participate in the Donation Election

In December, Forum users

[1]

will vote on how the

Donation Election Fund

should be allocate...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Estimating effective dimensionality of MNIST models, published by Arjun Panickssery on November 2, 2023 on LessWrong.

The

local learning coefficent

λ

is a measure of a model's "effective dimensionality" that captures its complexity (more background

here

).

Lau et al

recently described a sampling method (SGLD) using noisy gradients to find a stochastic estimate

^

λ

in a computationally tractable way (good explanation

here

):

I present results (

Github repo

) of the "Task Variability" project suggested on the

DevInterp project list

. To see how degeneracy scales with task difficulty and model class, I trained a fully-connected MLP and a CNN (both with ~120k parameters) on nine MNIST variants with different subsets of the labels (just the labels {0, 1}, then {0, 1, 2}, etc.). All models were trained to convergence using the same number of training data points. I implement Lau et al's algorithm on each of the trained models. The results below are averaged over three runs:

The results for the full MNIST dataset are comparable to Lau et al's results while using models ten times smaller trained for ten times fewer epochs.

The sampling method is finicky and sensitive to the hyperparameter choices of learning rate and noise factor. It will fail by producing negative or very high

^

λ

values if the noise

ϵ

and the distance penalty

γ

(see lines 6 and 7 in the pseudocode above) isn't calibrated to the model's level of convergence. The results show a linear scaling law relating number of labels to task complexity. The CNN typically has a lower

^

λ

than the MLP, which matches intuitions that some of the complexity is "stored" in the architecture because the convolutions apply a useful prior on functions good at solving image recognition tasks.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mech Interp Challenge: November - Deciphering the Cumulative Sum Model, published by TheMcDouglas on November 2, 2023 on The AI Alignment Forum.

I'm writing this post to discuss solutions to the October challenge, and present the challenge for this November.

If you've not read the

first post in this sequence

, I'd recommend starting there - it outlines the purpose behind these challenges, and recommended prerequisite material.

November Problem

The problem for this month is interpreting a model which has been trained to classify the cumulative sum of a sequence.

The model is fed sequences of integers, and is trained to classify the cumulative sum at a given sequence position. There are 3 possible classifications:

0 (if the cumsum is negative),

1 (if the cumsum is zero),

2 (if the cumsum is positive).

For example, if the sequence is:

Then the classifications would be:

The model is

not attention only

. It has one attention layer with a single head, and one MLP layer. It does

not

have layernorm at the end of the model. It was trained with weight decay, and an Adam optimizer with linearly decaying learning rate.

I don't expect this problem to be as difficult as some of the others in this sequence, however the presence of MLPs does provide a different kind of challenge.

You can find more details on the

Streamlit page

. Feel free to reach out if you have any questions!

October Problem - Solutions

In the second half of the sequence, the attention heads perform the algorithm "attend back to (and copy) the first token which is larger than me". For example, in a sequence like:

we would have the second 3 token attending back to the first 5 token (because it's the first one that's larger than itself), the second 5 attending back to 7, etc. The SEP token just attends to the smallest token.

Some more refinements to this basic idea:

The two attending heads split responsibilities across the vocabulary. Head 0.0 is the less important head; it deals with values in the range 28-37 (roughly). Head 0.1 deals with most other values.

In subsequences

x < y < z

where the three numbers are close together,

x

will often attend to

z

rather than to

y

. So why isn't this an adversarial example, i.e. why does the model still correctly predict

y

follows

x

?

Answer - the OV circuit shows that when we attend to source token

s

, we also boost things slightly less thn

s

, and suppress things slightly more than

s

.

So in the case of

x < y < z

, we have:

Attention to

y

will boost

y

a lot, and suppress

z

a bit.

Attention to

z

will boost

z

a lot, and boost

y

a bit.

So even if

z

gets slightly more attention than

y

, it might still be the case that

y

gets predicted with higher probability.

Sequences with large jumps are adversarial examples (because they're rare in the training data, which was randomly generated from choosing subsets without replacement).

Best Submissions

We received more submissions for this month's problem than any other in the history of the series, so thanks to everyone who attempted! The best solution to this problem was by

Vlad K

, who correctly identified the model's tendency to produce unexpected attention patterns when 3 numbers are close together, and figured out how the model manages to produce correct classifications anyway.

Best of luck for this and future challenges!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are 1-in-5 Americans familiar with EA?, published by David Moss on November 2, 2023 on The Effective Altruism Forum.

YouGov recently reported the

results

of a survey (n=1000) suggesting that about "one in five (22%) Americans are familiar with effective altruism."

[1]

We think these results are exceptionally unlikely to be true. Their 22% figure is very similar to the proportion of Americans we previously found

claim

to have heard of effective altruism (19%)

in our earlier survey

(n=6130). But, after conducting appropriate checks, we estimated that much lower percentages are likely to have genuinely heard of EA

[2]

(2.6% after the most stringent checks, which we speculate is still likely to be somewhat inflated

[3]

).

Is it possible that these numbers have simply dramatically increased following the FTX scandal?

Fortunately, we have tested this with multiple followup surveys explicitly designed with this possibility in mind.

[4]

In our most recent survey (conducted October 6th

[5]

), we estimated that approximately 16% (13.0%-20.4%) of US adults would

claim

to have heard of EA. Yet, when we add in additional checks to assess whether people appear to have really heard of the term, or have a basic understanding of what it means, this estimate drops to 3% (1.7% to 4.4%), and even to approximately 1% with a more stringent level of assessment.

[6]

These results are roughly in line with our earlier polling in May 2022, as well as additional polling we conducted between May 2022 and October 2023, and do not suggest any dramatic increase in awareness of effective altruism, although assessing small changes when base rates are already low is challenging.

We plan to continue to conduct additional surveys, which will allow us to assess possible changes from just before the trial of Sam Bankman-Fried to after the trial.

Attitudes towards EA

YouGov also report that respondents are, even post-FTX, overwhelmingly positive towards EA, with 81% of those who (claim to) have heard of EA approving or strongly approving of EA.

Fortunately, this positive view is broadly in line with our own findings- across different ways of breaking down who has heard of EA and different levels of stringency- which we aim to report on separately at a later date. However, our

earlier work

did find that awareness of FTX was associated with more negative attitudes towards EA.

Conclusions

The point of this post is not to criticise YouGov in particular. However, we do think it's worth highlighting that even highly reputable polling organizations should not be assumed to be employing all the additional checks that may be required to understand a particular question. This may apply especially in relation to niche topics like effective altruism, or more technical topics like AI, where additional nuance and checks may be required to assess understanding.

^

Also see this

quick take

.

^

There are many reasons why respondents may erroneously claim knowledge of something. But simply put, one reason is that people like demonstrating their knowledge, and may err on the side of claiming to have heard of something even if they are not sure. Moreover, if the component words that make up a term are familiar, then the respondent may either mistakenly believe they

have

already encountered the term, or think it is sufficient that they believe they can reasonably infer what the term means from its component parts to claim awareness (even when explicitly instructed not to approach the task this way!). Some people also appear to conflate the term with others - for example, some amalgamation of inclusive fitness/reciprocal altruism appears quite common.

For reference, over 12% of people claim to have heard of the specific term "Globally neutral advocacy": A term that the research team invented, which returns no google results as...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Snapshot of narratives and frames against regulating AI, published by Jan Kulveit on November 2, 2023 on LessWrong.

This is a speculative map of a hot discussion topic. I'm posting it in question form in the hope we can rapidly map the space in answers.

Looking at various claims at X and at the AI summit, it seems possible to identify some key counter-regulation narratives and frames that various actors are pushing.

Because a lot of the public policy debate won't be about

"what are some sensible things to do"

within a particular frame, but rather about fights for frame control, or

"what frame to think in"

, it seems beneficial to have at least some sketch of a map of the discourse.

I'm posting this as a question with the hope we can rapidly map the space, and one example of a "local map":

"It's about open source vs. regulatory capture"

It seems the coalition against AI safety, most visibly represented by Yann LeCun and Meta, has identified

"it's about open source vs. big tech"

as a favorable frame in which they can argue and build a coalition of open-source advocates who believe in the open-source ideology, academics who want access to large models, and small AI labs and developers believing they will remain long-term competitive by fine-tuning smaller models and capturing various niche markets. LeCun and others attempt to portray themselves as

the force of science and open inquiry

, while the scaling labs proposing regulation are

the evil big tech attempting regulatory capture

. Because this seems to be the prefered anti-regulation frame, I will spend most time on this.

Apart from the mentioned groups, this narrative seems to be memetically fit in a "proudly cynical" crowd which assumes everything everyone is doing or saying is primarily self-interested and profit-driven.

Overall, the narrative has clear problems with explaining away inconvenient facts, including:

Thousands of academics calling for regulation are uncanny counter-evidence for x-risk being just a ploy by the top labs.

The narrative strategy seems to explain this by some of the senior academics just being deluded, and others also pursuing a self-interested strategy in expectation of funding.

Many of the people explaining AI risk now were publicly concerned about AI risk before founding labs, and at times when it was academically extremely unprofitable, sometimes sacrificing standard academic careers.

The narrative move is to just ignore this.

Also, many things are just assumed - for example, if the resulting regulation would be in the interest o frontrunners.

What could be memetically viable counter-arguments within the frame?

Personally, I tend to point out that motivation to avoid AI risk is completely compatible with self-interest. Leaders of AI labs also have skin in the game.

Also, recently I try to ask people to use the explanatory frame of

'cui bono'

also to the other side, namely, Meta.

One possible hypothesis here is Meta just loves open source and wants everyone to flourish.

A more likely hypothesis is Meta wants to own the open-source ecosystem.

A more complex hypothesis is Meta doesn't actually love open source that much but has a sensible, self-interested strategy, aimed at a dystopian outcome.

To understand the second option, it's a prerequisite to comprehend the

"commoditize the complement"

strategy. This is a business approach where a company aims to drive down the cost or increase the availability of goods or services complementary to its own offerings. The outcome is an increase in the value of the company's services.

Some famous successful examples of this strategy include Microsoft and PC hardware: PC hardware became a commodity, while Microsoft came close to monopolizing the OS, extracting huge profits. Or, Apple's App Store: The complement to the phone is the apps. Apps have becom...