Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Implications of evidential cooperation in large worlds, published by Lukas Finnveden on August 23, 2023 on The AI Alignment Forum.

I've written several posts about the plausible implications of "evidential cooperation in large worlds" (ECL), on my newly-revived blog. This is a cross-post of the first. If you want to see the rest of the posts, you can either go to the blog or click through the links in this one.

All of the content on my blog, including this post, only represent my own views - not those of my employer. (Currently OpenPhilanthropy.)

"ECL" is short for "evidential cooperation in large worlds". It's an idea that was originally introduced in Oesterheld (2017) (under the name of "multiverse-wide superrationality"). This post will explore implications of ECL, but it won't explain the idea itself. If you haven't encountered it before, you can read the paper linked above or this summary written by Lukas Gloor.1

This post lists all candidates for decision-relevant implications of ECL that I know about and think are plausibly important.2 In this post, I will not describe in much depth why they might be implications of ECL. Instead, I will lean on the principle that ECL recommends that we (and other ECL-sympathetic actors) act to benefit the values of people whose decisions might correlate with our decisions.

As described in this appendix, this relies on you and others having particular kinds of values. For one, I assume that you care about what happens outside our light cone. But more strongly, I'm looking at values with the following property: If you could have a sufficiently large impact outside our lightcone, then the value of taking different actions would be dominated by the impact that those actions had outside our lightcone. I'll refer to this as "universe-wide values". Even if all your values aren't universe-wide, I suspect that the implications will still be relevant to you if you have some universe-wide values.

This is speculative stuff, and I'm not particularly confident that I will have gotten any particular claim right.

Summary (with links to sub-sections)

For at least two reasons, future actors will be in a better position to act on ECL than we are. Firstly, they will know a lot more about what other value-systems are out there. Secondly, they will be facing immediate decisions about what to do with the universe, which should be informed by what other civilizations would prefer.3 This suggests that it could be important for us to Affect whether (and how) future actors do ECL. This can be decomposed into two sub-points that deserve separate attention: how we might be able to affect Futures with aligned AI, and how we might be able to affect Futures with misaligned AI.

But separately from influencing future actors, ECL also changes our own priorities, today. In particular, ECL suggests that we should care more about other actors' universe-wide values. When evaluating these implications, we can look separately at three different classes of actors and their values. I'll separately consider how ECL suggests that we should.

Care more about other humans' universe-wide values.4

I think the most important implication of this is that Upside- and downside-focused longtermists should care more about each others' values.

Care more about evolved aliens' universe-wide values.

I think the most important implication of this is that we plausibly should care more about influencing how AI could benefit/harm alien civilizations.

How much more? I try to answer that question in the next post. My best guess is that ECL boosts the value of this by 1.5-10x. (This is importantly based on my intuition that we would care a bit about alien values even without ECL.)

Care more about misaligned AIs' universe-wide values.5

I don't think this significantly reduces the value of worki...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: State of Generally Available Self-Driving, published by jefftk on August 22, 2023 on LessWrong.

After a lot of discussions online where people try to argue about where self-driving tech is going but seem pretty confused about where the tech currently is, I wanted to give a bit of an overview of the current state.

There are two main approaches: taxis and personal vehicles. There are many companies that have gotten as far as testing with a safety driver and/or with "trusted tester" riders, but as far as I can tell only two companies run commercial ride services open to the public without anyone in the driver's seat:

Waymo (Google-affiliated) has been operating fully driverless vehicles in

Phoenix AZ as a commercial ride service since 2020. They used to have a waitlist, but at this point anyone can download the app and try it. They've just expanded to SF, and have a waitlist for LA and Austin. As of 2023-08 they claim to be serving 10k weekly riders.

Cruise (GM-affiliated) has been operating fully driverless vehicles in SF as a commercial ride service since 2022. They also now seem to cover Austin and Phoenix, and as of

2023-08-02 also claim 10k weekly riders.

For personal vehicles, the most automation you can currently get is Level

3. These are systems where, when engaged, the person in the driver's seat can safely and legally read a book or otherwise not pay attention. If the system runs into a situation it can't handle, it alerts the driver, and if the driver doesn't take control it automatically stops the car. One option here, and maybe the only commercially available one, is Mercedes' Drive Pilot, which they launched in Germany in

2022. It only operates on highways under 40mph (essentially, stop-and-go traffic) but Mercedes takes on the legal liability when it's engaged. They claim their 2024 (as in, starting late this year) US models of their S-class and EQS sedans will have DrivePilot as an option and are approved in

CA and NV. There may be other L3 systems in other countries - I'm having a lot of trouble telling what exactly launched with what models and whether it qualifies as L3.

Overall, the current state is beyond "it's just a demo", but it's also still heavily limited by location and current conditions.

Comment via: facebook, mastodon

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EU farmed fish policy reform roadmap, published by Neil Dullaghan on August 22, 2023 on The Effective Altruism Forum.

Full report (in PDF) available on the Rethink Priorities website. This is a follow-up to our report "Strategic considerations for upcoming EU farmed animal legislation".

Summary

The majority of fish consumed (in tonnes) in the EU are either wild-caught fish or farmed fish imported from non-EU Norway, Scotland, and Turkey. I don't see evidence that the EU will regulate imports of fish this decade and I'm less confident on interventions that affect wild animal welfare. So we narrowed the scope of the report to the species of fish farmed in the largest numbers of individuals and life-years in the EU: sea bass, sea bream, and small trout.

The report argues that the most promising EU fish policy ask right now is a fast transition to better slaughter conditions of sea bass, sea bream, and (small) rainbow trout. It's an option that EU policymakers already put on the table as part of their larger animal legislation reform package, animal advocacy organisations support it, it has real world implementation, and there are clear actions to make improvements to the ask, namely by providing evidence discussed in the report of short transition periods.

The EU's scientific agency, EFSA, has opinions on farmed fish welfare due 2024-2029 which will offer a beachhead for rearing reforms the movement may ask for in future that affect the whole life of an individual (e.g. water quality standards, stocking density maximums, enrichment especially for juveniles). There is a lot of economic and political precedent the aquatic animal advocacy movement needs to start creating years in advance to make the most of EFSA opinions on farmed fish welfare - the report discusses what sort of evidence the movement might need.

I worry that after the European Commission presents its reform proposal in September/October 2023 (including a cage-free hen policy), little progress will be made before the June 2024 European Parliament elections put more conservative co-legislators in place, and a new European Commission 5-year term starts in November 2024. The longer the reform negotiations drag on, if the movement doesn't pivot resources to building the case for rearing reforms, advocates may be left playing catch-up in a few years and fail to make the most of EFSA opinions.

I argue the movement should make an effort over the next 10 months to pull together the evidence and coalitions in support of a fast slaughter transition and ensure it makes it in the final law if things are progressing quickly. However, given limited resources, the more time and resources that continue to go into fish slaughter the higher the risk that this cuts against building the case for the arguably highly expected value rearing reforms later this decade.

One could reasonably disagree and say even if the EU reform looks to be slowing, we should focus 100% of our fish policy advocacy efforts on slaughter to make sure the movement locks in a precedent for doing anything on commercially farmed fish at the EU level, and that this slaughter provision forms a beachhead for rearing reforms later. This may be compelling if you doubt that there will be opportunities to utilise EFSA opinions to create reform, especially in the absence of a fish slaughter precedent (e.g. if the opinions are dropped or delayed, or you don't believe we can gather sufficient evidence for EFSA to make bold rearing reform recommendations). Or you might think we should push for a lot more than slaughter right now, to mainstream those asks and hope at least one more of them make it onto the agenda.

On the first, I argue the leaked draft impact assessment already signalled a willingness to explore fish reforms post EFSA-opinions, and to turn EFSA opinio...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Call for Papers on Global AI Governance from the UN, published by Chris Leong on August 22, 2023 on The Effective Altruism Forum.

Copied from their LinkedIn page:

𝐂𝐚𝐥𝐥 𝐟𝐨𝐫 𝐏𝐚𝐩𝐞𝐫𝐬 𝐨𝐧 𝐆𝐥𝐨𝐛𝐚𝐥 𝐀𝐈 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞

Exciting news! As we gear up for the High-level Advisory Body on AI, we're inviting thought leaders, researchers, and enthusiasts to contribute short papers (~2000 words) on pivotal themes:

1️⃣ Key Issues on Global AI Governance: Dive into studies and recommendations that the High-Level Advisory Body on AI should prioritize, especially those needing global governance attention.

2️⃣ Current Efforts in Global AI Governance: Share analyses on bilateral, multilateral, and inter-regional initiatives. We're keen on understanding varying philosophical approaches, critiques, and suggestions.

3️⃣ Models in Global AI Governance: Whether you're analyzing existing models or proposing fresh perspectives, we're all ears. Surveys and analyses of other proposals are also encouraged.

We champion diversity! We're eager to hear from a myriad of groups, regions, and methodologies. Your insights will serve as foundational material (with due credit) for the High-Level Advisory Body on AI.

Deadline: 30 September

Submission: Send your paper (hyperlink or PDF) to [email protected]. Ensure your title page has the author(s) details, affiliation, and contact info. If it's an Executive Summary of a more extensive piece, kindly attach the main paper's link.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Will AI kill everyone? Here's what the godfathers of AI have to say [RA video], published by Writer on August 22, 2023 on The Effective Altruism Forum.

This video is based on this article. @jai has written both the original article and the script for the video.

Script:

The ACM Turing Award is the highest distinction in computer science, comparable to the Nobel Prize. In 2018 it was awarded to three pioneers of the deep learning revolution: Geoffrey Hinton, Yoshua Bengio, and Yann LeCun.

In May 2023, Geoffrey Hinton left Google so that he could speak openly about the dangers of advanced AI, agreeing that "it could figure out how to kill humans" and saying "it's not clear to me that we can solve this problem."

Later that month, Yoshua Bengio wrote a blog post titled "How Rogue AIs may Arise", in which he defined a "rogue AI" as "an autonomous AI system that could behave in ways that would be catastrophically harmful to a large fraction of humans, potentially endangering our societies and even our species or the biosphere."

Yann LeCun continues to refer to thoseanyone suggesting that we're facing severe and imminent risk as "professional scaremongers" and says it's a "simple fact" that "the people who are terrified of AGI are rarely the people who actually build AI models."

LeCun is a highly accomplished researcher, but in light of Bengio and Hinton's recent comments it's clear that he's misrepresenting the field whether he realizes it or not. There is not a consensus among professional researchers that AI research is safe. Rather, there is considerable and growing concern that advanced AI could pose extreme risks, and this concern is shared by not only both of LeCun's award co-recipients, but the headsleaders of all three leading AI labs (OpenAI, Anthropic, and Google DeepMind):

Demis Hassabis, CEO of DeepMind, said in an interview with Time Magazine: "When it comes to very powerful technologies - and obviously AI is going to be one of the most powerful ever - we need to be careful. Not everybody is thinking about those things. It's like experimentalists, many of whom don't realize they're holding dangerous material."

Anthropic, in their public statement "Core Views on AI Safety", says: "One particularly important dimension of uncertainty is how difficult it will be to develop advanced AI systems that are broadly safe and pose little risk to humans. Developing such systems could lie anywhere on the spectrum from very easy to impossible."

And OpenAI, in their blog post "Planning for AGI and Beyond", says "Some people in the AI field think the risks of AGI (and successor systems) are fictitious; we would be delighted if they turn out to be right, but we are going to operate as if these risks are existential." Sam Altman, the current CEO of OpenAI, once said "Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. "

There are objections one could raise to the idea that advanced AI poses significant risk to humanity, but "it's a fringe idea that actual AI experts do not take seriously" is no longer among them. Instead, a growing share of experts are echoing the conclusion reached by Alan Turing, considered by many to be the father of computer science and artificial intelligence, back in 1951: "[I]t seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. [...] At some stage therefore we should have to expect the machines to take control."

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Large Language Models will be Great for Censorship, published by Ethan Edwards on August 22, 2023 on LessWrong.

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort

Thanks to ev_ and Kei for suggestions on this post.

LLMs can do many incredible things. They can generate unique creative content, carry on long conversations in any number of subjects, complete complex cognitive tasks, and write nearly any argument. More mundanely, they are now the state of the art for boring classification tasks and therefore have the capability to radically upgrade the censorship capacities of authoritarian regimes throughout the world.

How Censorship Worked

In totalitarian government states with wide censorship - Tsarist Russia, Eastern Bloc Communist states, the People's Republic of China, Apartheid South Africa, etc - all public materials are ideally read and reviewed by government workers to ensure they contain nothing that might be offensive to the regime. This task is famously extremely boring and the censors would frequently miss obviously subversive material because they did not bother to go through everything. Marx's Capital was thought to be uninteresting economics so made it into Russia legally in the 1890s.

The old style of censorship could not possibly scale, and the real way that censors exert control is through deterrence and fear rather than actual control of communication. Nobody knows the strict boundary line over which they cannot cross, and therefore they stay well away from it. It might be acceptable to lightly criticize one small part of the government that is currently in disfavor, but why risk your entire future on a complaint that likely goes nowhere? In some regimes such as the PRC under Mao, chaotic internal processes led to constant reversals of acceptable expression and by the end of the Cultural Revolution most had learned that simply being quiet was the safest path. Censorship prevents organized resistance in the public and ideally for the regime this would lead to tacit acceptance of the powers that be, but a silently resentful population is not safe or secure.

When revolution finally comes, the whole population might turn on their rulers with all of their suppressed rage released at once. Everyone knows that everyone knows that everyone hates the government, even if they can only acknowledge this in private trusted channels.

Because proper universal and total surveillance has always been impractical, regimes have instead focused on more targeted interventions to prevent potential subversion. Secret polices rely on targeted informant networks, not on workers who can listen to every minute of every recorded conversation. This had a horrible and chilling effect and destroyed many lives, but also was not as effective as it could have been. Major resistance leaders were still able to emerge in totalitarian states, and once the government showed signs of true weakness there were semi-organized dissidents ready to seize the moment.

Digital Communication and the Elusiveness of Total Censorship

Traditional censorship mostly dealt with a relatively small number of published works: newspapers, books, films, radio, television. This was somewhat manageable just using human labor. However in the past two decades, the amount of communication and material that is potentially public has been transformed with the internet.

It is much harder to know how governments are handling new data because the information we have mostly comes from the victims of surveillance who are kept in the same deterrent fear as the past. If victims imagine the state is more capable than it is, that means the state is succeeding, and it is harder to assess the true capabilities. We don't have reliable accounts from insiders or archival access since no major regi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Which possible AI systems are relatively safe?, published by Zach Stein-Perlman on August 22, 2023 on LessWrong.

Presumably some kinds of AI systems, architectures, methods, and ways of building complex systems out of ML models are safer or more alignable than others. Holding capabilities constant, you'd be happier to see some kinds of systems than others.

For example, Paul Christiano suggests "LM agents are an unusually safe way to build powerful AI systems." He says "My guess is that if you hold capability fixed and make a marginal move in the direction of (better LM agents) + (smaller LMs) then you will make the world safer. It straightforwardly decreases the risk of deceptive alignment, makes oversight easier, and decreases the potential advantages of optimizing on outcomes."

My quick list is below; I'm interested in object-level suggestions, meta observations, reading recommendations, etc. I'm particularly interested in design-properties rather than mere safety-desiderata, but safety-desiderata may inspire lower-level design-properties.

All else equal, it seems safer if an AI system:

Is more interpretable

If its true thoughts are transparent and expressed in natural language (see e.g. Measuring Faithfulness in Chain-of-Thought Reasoning)

(what else?);

Has humans in the loop (even better to the extent that they participate in or understand its decisions, rather than just approving inscrutable decisions);

Decomposes tasks into subtasks in comprehensible ways, and in particular if the interfaces between subagents performing subtasks are transparent and interpretable;

Is more supervisable or amenable to AI oversight (what low-level properties determine this besides interpretable-ness and decomposing-tasks-comprehensibly?);

Is feedforward-y rather than recurrent-y (because recurrent-y systems have hidden states? so this is part of interpretability/overseeability?);

Is myopic;

Lacks situational awareness;

Lacks various dangerous capabilities (coding, weapon-building, human-modeling, planning);

Is more corrigible (what lower-level desirable properties determine corrigibility? what determines whether systems have those properties?) (note to self: see 1, 2, 3, 4, and comments on 5);

Is legible and process-based;

Is composed of separable narrow tools;

Can't be run on general-purpose hardware.

These properties overlap a lot. Also note that there are nice-properties at various levels of abstraction, like both "more interpretable" and [whatever low-level features make systems more interpretable].

If a path (like LM agents) or design feature is relatively safe, it would be good for labs to know that. An alternative framing for this question is: what should labs do to advance safer kinds of systems?

Obviously I'm mostly interested in properties that might not require much extra-cost and capabilities-sacrifice relative to unsafe systems. A method or path for safer AI is ~useless if it's far behind unsafe systems.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: DIY Deliberate Practice, published by lynettebye on August 21, 2023 on LessWrong.

In the spirit of growth and self-improvement, I recently attempted to apply Ericsson's principles of deliberate practice to my own growth goal: speeding up my writing. If you're unfamiliar with the minutia of Ericsson's methods, don't worry, I was in the same boat - and hence my initial goal had substantial room for improvement. This is the story of how I used to deliberate practice principles to workshop my growth goal.

What exactly is deliberate practice?

Ericsson's recipe for practice starts with what he calls "purposeful practice":

Purposeful practice takes place outside of your comfort zone, pushing what you can already do. You should be trying new techniques, not just repeating what you've done before. Think "try differently", not "try harder"!

Purposeful practice demands you actively think about what you're doing -- you shouldn't be able to daydream about dinner while doing it!

Purposeful practice involves well-defined, specific goals broken down for step-by-step improvement (NOT vaguely "trying to improve"). You don't want to "practice the piano piece" you want to "practice the tricky section with the left hand until you can play it three times through at the correct speed without mistakes."

Purposeful practice involves quick feedback and changing what you're doing in response. Ideally, immediate feedback so that you can improve your approach mid practice session.

Ericsson adds one more criteria to graduate from "purposeful practice" to "deliberate practice": well-developed knowledge of what and how to practice. Deliberate practice is when you're purposefully practicing optimized strategies for improving the skill. Ideally, you want a highly developed field where experts have identified the most effective techniques and the best training strategies to develop those skills, plus a teacher who can lead you through the process.

Lacking that, do your best to find proven techniques and hope for the best. I ask more experienced people how they developed their skills or what they recommend I practice, and use that as a starting point. (Tips for informational interviews to learn how more experienced people developed skills.)

My initial goal

My goal was to write faster. I didn't have an instructor, but I did have a benchmark: several journalists and bloggers had shared that they could write a post each day. One blogger who I respect advised me to try publishing a post each day for a month. So I set the more modest goal of writing one post each day for a week.

My first.and second.and third attempts

Day 1: I began by enthusiastically plunking out a short post around a great career planning tip I'd recently learned. I got the full thing drafted, but it seemed a bit forlorn. Surely it would be better if I went back and wrote a longer post that also included the other career planning tips I found most useful?

Day 2: Sticking to my intention to draft a new post each day, I set aside my career tools idea. Instead, I started drafting what became my CBT post. I'd stitched together most of the main post by time evening rolled around, but I wanted to go through the resources I'd been compiling to make a nice resource list.

Day 3: I whipped together a little post on an intuition I had about AI. However, when I spoke with my partner in the evening (who works in the field), he agreed that a solid example would improve the post. It too went on the stack of posts awaiting revising.

Day 4: I tried putting together a short post on ADHD.and only got as far as an outline. The more I tried to nail down what I wanted to say, the more I realized there was to cover. In the end, I set it aside to await a round of interviews. (It eventually grew into nine thousand words across three posts.)

Day 5: A migraine kill...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Probably Good published a list of impact-focused job-boards, published by Probably Good on August 21, 2023 on The Effective Altruism Forum.

Probably Good recently published a page for impact-focused job boards on our site!

We created it to help people who are searching for potentially impactful opportunities in a range of cause areas and regions. It includes a variety of for-good job boards (e.g. international non-profit jobs, civil service positions, tech-focused roles), region-specific boards, and a few boards specifically geared towards climate change, animal advocacy, global health. We also spotlight the 80,000 Hours job board, which is the most EA-aligned resource we include.

This page is intended to be a good jumping off point for people to start looking for jobs that could help others. So while we believe the boards listed can be a good place to look for opportunities, we don't endorse every job on every board. We encourage our readers to carefully analyze the opportunities that interest them, use tools (such as our career guide chapters on this topic) to assess their potential impact, and consider both the direct impact and the career capital that different opportunities can provide. We're also happy to chat 1-on-1 about specific opportunities and impactful options in a career advising call.

This page is still a work-in-progress and we'd love to keep expanding it to accommodate different interests and priorities. If you have suggestions for job boards we should include, please let us know here or email directly at [email protected].

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ideas for improving epistemics in AI safety outreach, published by mic on August 21, 2023 on LessWrong.

In 2022 and 2023, there has been a growing focus on recruiting talented individuals to work on mitigating the potential existential risks posed by artificial intelligence. For example, we've seen an increase in the number of university clubs, retreats, and workshops dedicated to introducing people to the issue of existential risk from AI.

However, these efforts might foster an environment with suboptimal epistemics. Given the goal of enabling people to contribute positively to AI safety, there's an incentive to focus on that without worrying as much about whether our arguments are solid. Many people working on field building are not domain experts in AI safety or machine learning but are motivated due to a belief that AI safety is an important issue. Some participants may hold the belief that addressing the risks associated with AI is important, without fully comprehending their reasoning behind this belief or having engaged with strong counterarguments.

This post is a brief examination of this issue and suggests some ideas to improve epistemics in outreach efforts.

Note: I first drafted this in December 2022. Since then, concern about AI x-risk has been increasingly discussed in the mainstream, so AI safety field builders should hopefully be using fewer weird, epistemically poor arguments. Still, I think epistemics are still relevant to discuss after a recent post noted poor epistemics in EA community building.

What are some ways that AI safety field building may be epistemically unhealthy?

Organizers may promote arguments for AI safety that may be (comparatively) compelling yet flawed

Advancing arguments promoting the importance of AI safety while neglecting opposing arguments

E.g., citing that x% of researchers believe that AI has an y% chance of causing an existential catastrophe, without the caveat that experts have widely differing views

Confidently making arguments that are flawed or have insufficiently justified premises

E.g., claiming that instrumental convergence is inevitable, assuming that AIs are maximizing for reward (see Reward is not the optimization target, although there are also comments disagreeing with this)

See also: Rohin Shah's comment here about how few people can make an argument for working on AI x-risk that he doesn't think is obviously flawed

Simultaneously, I think that most ML people don't find AI safety arguments particularly compelling.

It's easy to form the perception that arguments in favor of AI safety are "supposed" to be the more correct ones. People might feel hesitant to voice disagreements.

In a reading group (such as one based on AI Safety Fundamentals), people may go along with the arguments from the readings or what the discussion facilitator says - deferring to authority and being hesitant to think through arguments themselves.

People may participate in reading groups but skim the readings, and walk away with a belief in the conclusions without understanding the arguments; or notice they are confused but walk away regardless believing the conclusions.

Why are good epistemics valuable?

To do productive research, we want to avoid having an understanding of AI x-risk that is obviously flawed

"incorrect arguments lead to incorrect beliefs which lead to useless solutions" (from Rohin Shah)

Bad arguments are bad for persuading people (or at least, it seems bad if you can't anticipate common objections from the ML community)

People making bad arguments is bad for getting people to do useful work

Attract more people with good epistemics

For the sake of epistemic rigor, I'll also make a few possible arguments about why epistemics may be overrated.

Perhaps people can do useful work even if they don't have an inside view of why AI ...