Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A summary of every Replacing Guilt post, published by Akash on June 30, 2022 on The Effective Altruism Forum.

1

I recently finished the Replacing Guilt series (by Nate Soares).

I don't struggle much with guilt. I consider myself a happy and energetic person. I also spent several years studying psychology, and I'm familiar with many techniques from cognitive-behavioral therapy.

So I was surprised by how impressed I was with Replacing Guilt. Elegant writing, memorable stories, useful advice, and dark humor.

There are few things I would put in the category of "pretty much everyone I know should spend 5 minutes reading this to see if it might be helpful, and at least 20% of them should read the whole thing." I feel this way about Replacing Guilt.

2

As I read, I wrote a few sentences summarizing each post. I mostly did this to improve my own comprehension/memory.

You should treat the summaries as "here's what Akash took away from this post" as opposed to "here's an actual summary of what Nate said."

Note that the summaries are not meant to replace the posts-- in fact, they're intended to get you to read the posts (especially the ones that seem interesting, though I recommend starting from the beginning).

3

Here are my notes on each post, in order. Tomorrow, I'll post a tier-list.

Half-assing it with everything you’ve got

You can try really hard to produce a low-quality or medium-quality outcome. You can put in your all to reach a B-, or to be in 17th place at the end of the race.

Failing with abandon

If you wanted to go to sleep at 12AM, and it’s 12:15AM, don’t be like “ah well, I failed.” You can try to get as close to your goal as possible, even if you don’t fully achieve it.

The Stamp Collector

We do not have full access to the external world, but we also do not have full access to our internal worlds. We can, in fact, care about the real world, other people, and things that happen outside of our minds.

You’re allowed to fight for something

We can convert listless guilt (which is vague and general) into specific guilt (which is about a particular thing). One way to do this is by looking out at the world and seeing things that are bad and being like “I want the world to be different”

Caring about something larger than yourself

Sometimes, our feelings are are in conflict with our quiet aesthetics (values, virtues, deeper beliefs). It can be useful to notice this and then figure out how we want to resolve these conflicts. For some people, this means embracing the aesthetics and recognizing that there are some feelings that are evolutionary artifacts that we don’t actually care about.

You don’t get to know what you’re fighting for

It is really really hard to really know what we care about. But sometimes the world is in such a bad place that we don’t need to (because there are some obvious atrocities that we would like to mitigate or prevent).

Should considered harmful

When we use the word “should”, we make one choice seem like the option that makes us a good person and one option seem like negative infinity. Instead of saying “I should read more”, I will feel less guilty and be better at making trade-offs if I say “If I read more, I will know more about the world and be more able to contribute to AIS discussions. (Did you notice the meta move there? hehe.)

Not because you should

The best way to make decisions is to evaluate your options and figure out what is best. “Should” is a label that can only be applied retrospectively, after you have decided what is best. Additionally, some charity will be lost when some people give up their “shoulds”. But most people will actually do more good by rejecting their “shoulds”— most people actually want to help others, and tapping into this motivation is stronger and more sustainable.

Your shoulds are not a duty

When people think about ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gradient hacking: definitions and examples, published by Richard Ngo on June 29, 2022 on The AI Alignment Forum.

Gradient hacking is a hypothesized phenomenon where:

A model has knowledge about possible training trajectories which isn’t being used by its training algorithms when choosing updates (such as knowledge about non-local features of its loss landscape which aren’t taken into account by local optimization algorithms).

The model uses that knowledge to influence its medium-term training trajectory, even if the effects wash out in the long term.

Below I give some potential examples of gradient hacking, divided into those which exploit RL credit assignment and those which exploit gradient descent itself. My concern is that models might use techniques like these either to influence which goals they develop, or to fool our interpretability techniques. Even if those effects don’t last in the long term, they might last until the model is smart enough to misbehave in other ways (e.g. specification gaming, or reward tampering), or until it’s deployed in the real world—especially in the RL examples, since convergence to a global optimum seems unrealistic (and ill-defined) for RL policies trained on real-world data. However, since gradient hacking isn’t very well-understood right now, both the definition above and the examples below should only be considered preliminary.

RL credit hacking examples

A policy might know that a certain sequence of actions leads to higher reward. But if it chooses not to take those actions, it won’t be rewarded for them, and that type of behavior won’t be reinforced (until it finds them via random exploration, which could take a very long time).

A policy might know that humans give mistaken feedback on a given trajectory, and then select that trajectory more frequently in order to update itself in a certain direction. When humans give mistakenly positive feedback, this is just an example of specification gaming. But we could also imagine the model deliberately inducing mistakenly negative feedback.

A policy might know which of its actions is actually responsible for receiving reward, even though the RL algorithm being used “doesn’t know”. Then the model can influence how much credit the RL algorithm gives different actions (e.g. by putting them earlier or later in the trajectory).

This seems easier for dumb RL algorithms like REINFORCE, and seems harder for actor-critic methods. However, finding algorithms that make use of all the knowledge a model has about the training landscape seems like it’ll require a full solution to the ELK problem.

A policy might know when it’s being trained off-policy versus on-policy (because off-policy it observes a trajectory of actions which it wouldn’t have taken), and choose actions differently in each case. A tentative possibility, related to the first example: during off-policy training, it could try novel “explore” actions which it knows would lead to high reward on-policy. Those actions would then be reinforced against (in favor of the action actually taken during that trajectory), since when performed off-policy the rewards are not actually seen.

In multi-agent RL, even if rewards are zero-sum, different policies could coordinate to choose which of them will receive the reward in which episodes, to influence which behaviors are reinforced.

Gradient descent hacking examples

When local optima exist, a model might choose to do worse than it otherwise could, with the intention of changing its parameters in a certain direction. Intuitively speaking, gradient descent will then push in two directions: towards changing the parameters which implemented the decision to gradient hack, and towards changing the parameters which the model wanted to change (since their current values influenced its decision).

Gradient d...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Latent Adversarial Training, published by Adam Jermyn on June 29, 2022 on The AI Alignment Forum.

The Problem

We’d like to train models to be robustly safe, even in environments that may fall well outside of the training distribution. Unfortunately all we get to work with is the training distribution, which makes ensuring robust generalization difficult.

Deception is an example of this concern. Models may be well-behaved in the training environment because they realize it is a training environment, but behave poorly in deployment because they notice the distributional shift.

An Approach

Background

It would be great if we could generate scenarios where the model behaves badly. For instance if the model is deceptive we’d like to generate inputs to trick it into thinking it’s been deployed so that it reveals that deception.

Adversarial Training (AT), and Relaxed Adversarial Training (RAT), are approaches to this problem. Very briefly, in AT, we search for inputs that generate large loss, while in RAT we search for distributions of inputs or other more general evidence that the model is capable of generating large loss.

The approach I want to explore, called Latent Adversarial Training (LAT), is a form of RAT.

Concept

Suppose we have two models, the Agent and the Surgeon. The Agent is the model we’re trying to train, and the Surgeon is going to be its adversary. The training proceeds as follows:

The Agent processes input x from the environment, producing latent state s(x) and output y(s).

The Surgeon takes inputs (x,s,y) and searches for a modification s→s+δs such that y(s+δs) does poorly on the loss function L(y).

We update the Agent based on L(y(s+δs)), treating the Surgeon’s modifications as constants.

The idea is that the Surgeon gets to meddle with the latent state of the Agent with full knowledge of what the Agent was planning on doing, and we train the Agent to be robust against this kind of attack.

Implementation

Concretely, suppose the Agent is a neural network. The latent state s consists of the activations of all the neurons and the Surgeon gets to specify modifications like “this neuron gets +5”.

How does the Surgeon decide what to do? There are a number of options. It could examine the gradient of the Agent’s loss with respect to each activation, then push on the activations that give it the most loss. Or it could randomly generate its modifications and use some other optimization process like the Metropolis-Hastings algorithm to identify loss-maximizing options.

In general I’m imagining the Surgeon as more of a fuzzer with access to program memory and less of a complicated model. In principle it could be some complicated neural network, but then I’d want it to be simpler than the Agent, possibly of bounded size, so I could be quite sure I understood what it was doing. The last thing I want is a misaligned Surgeon trying to achieve its own mesa-objective by programming the Agent to be Surgeon-aligned and deceptive against humans.

I suspect it’s possible to have the Surgeon pick useful modifications that generate meaningful changes in the loss without having the Surgeon’s complexity scale with that of the Agent, but that’s just an intuition, and if it fails then this approach looks a lot less appealing.

Core Ideas

The key difference between LAT and Adversarial Training is that the Surgeon gets to directly manipulate the Agent’s inner state, which makes the Surgeon’s job much easier than in the ordinary adversarial training setup.

Take the classic example of an Agent that only defects if it sees a valid solution to a hard cryptographic problem: the Surgeon doesn’t need to identify an input that causes the Agent to defect, it only needs to identify that somewhere in the middle of the Agent there is a switch which, if flipped, causes the Agent to defect and gene...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Know what you're optimising for, published by alexrjl on June 29, 2022 on The Effective Altruism Forum.

This is a linkpost for Know what you're optimising for, a piece that I wrote for the 80,000 hours blog, collecting a group of ideas I've found myself often discussing in 1-on-1 careers calls. The post has some formatting which doesn't translate well to the forum so I haven't copied the entire text over, though I would value comments, thoughts, and feedback from EA forum users, and this post seems like a reasonable place to put them. I didn't want to just have a bunch of italic text and a link though, so below is a small extract which I think gives a good sense of my motivation for the overall piece:There’s a difference between doing things that are somewhat correlated with things you want (or even doing things that you expect to lead to things you want), and trying really unusually hard to actually get what you want. Sometimes working out what you actually want can be really hard — for many, working out what one ultimately values can be a lifetime’s work.

However, I’ve been frequently surprised, during my time as an advisor, by how often it’s been sufficient to just ask:

It looks like you’re trying to achieve X here. Is X really the thing you want?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Limits to Legibility, published by Jan Kulveit on June 29, 2022 on The Effective Altruism Forum.

From time to time, someone makes the case for why transparency in reasoning is important. The latest conceptualization is Epistemic Legibility by Elizabeth, but the core concept is similar to reasoning transparency used by OpenPhil, and also has some similarity to A Sketch of Good Communication by Ben Pace.

I'd like to offer a gentle pushback. The tl;dr is in my comment on Ben's post, but it seems useful enough for a standalone post.

“How odd I can have all this inside me and to you it's just words.” ― David Foster Wallace

When and why reasoning legibility is hard

Say you demand transparent reasoning from AlphaGo. The algorithm has roughly two parts: tree search and a neural network. Tree search reasoning is naturally legible: the "argument" is simply a sequence of board states. In contrast, the neural network is mostly illegible - its output is a figurative "feeling" about how promising a position is, but that feeling depends on the aggregate experience of a huge number of games, and it is extremely difficult to explain transparently how a particular feeling depends on particular past experiences. So AlphaGo would be able to present part of its reasoning to you, but not the most important part.

Human reasoning uses both: cognition similar to tree search (where the steps can be described, written down, and explained to someone else) and processes not amenable to introspection (which function essentially as a black box that produces a "feeling"). People sometimes call these latter signals “intuition”, “implicit knowledge”, “taste”, “S1 reasoning” and the like. Explicit reasoning often rides on top of this.Extending the machine learning metaphor, the problem with human interpretability is that "mastery" in a field often consists precisely in having some well-trained black box neural network that performs fairly opaque background computations.

Bad things can happen when you demand explanations from black boxes

The second thesis is that it often makes sense to assume the mind runs distinct computational processes: one that actually makes decisions and reaches conclusions, and another that produces justifications and rationalizations.

In my experience, if you have good introspective access to your own reasoning, you may occasionally notice that a conclusion C depends mainly on some black box, but at the same time, you generated a plausible legible argument A for the same conclusion after you reached the conclusion C.

If you try running, say, Double Crux over such situations, you'll notice that even if someone refutes the explicit reasoning A, you won't quite change the conclusion to ¬C. The legible argument A was not the real crux. It is quite often the case that (A) is essentially fake (or low-weight), whereas the black box is hiding a reality-tracking model.

Stretching the AlphaGo metaphor a bit: AlphaGo could be easily modified to find a few specific game "rollouts" that turned out to "explain" the mysterious signal from the neural network. Using tree search, it would produce a few specific examples how such a position may evolve, which would be selected to agree with the neural net prediction. If AlphaGo showed them to you, it might convince you! But you would get a completely superficial understanding of why it evaluates the situation the way it does, or why it makes certain moves.

Risks from the legibility norm

When you make a strong norm pushing for too straightforward "epistemic legibility", you risk several bad things:First, you increase the pressure on the "justification generator" to mask various black boxes by generating arguments supporting their conclusions.Second, you make individual people dumber. Imagine asking a Go grandmaster to transparently justify his mov...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Will Capabilities Generalise More?, published by Ramana Kumar on June 29, 2022 on The AI Alignment Forum.

Nate and Eliezer (Lethality 21) claim that capabilities generalise further than alignment once capabilities start generalising far at all. However, they have not articulated particularly detailed arguments for why this is the case. In this post I collect the arguments for and against the position I have been able to find or generate, and develop them (with a few hours’ effort). I invite you to join me in better understanding this claim and its veracity by contributing your own arguments and improving mine.Thanks to these people for their help with writing and/or contributing arguments: Vikrant Varma, Vika Krakovna, Mary Phuong, Rory Grieg, Tim Genewein, Rohin Shah.

For:

1. Capabilities have much shorter description length than alignment.

There are simple “laws of intelligence” that underwrite highly general and competent cognitive abilities, but no such simple laws of corrigibility or laws of “doing what the principal means” – or at least, any specification of these latter things will have a higher description length than the laws of intelligence. As a result, most R&D pathways optimising for capabilities and alignment with anything like a simplicity prior (for example) will encounter good approximations of general intelligence earlier than good approximations of corrigibility or alignment.

2. Feedback on capabilities is more consistent and reliable than on alignment.

Reality hits back on cognitive strategies implementing capabilities – such as forming and maintaining accurate beliefs, or making good predictions – more consistently and reliably than any training process hits back on motivational systems orienting around incorrect optimisation targets. Therefore there’s stronger outer optimisation pressure towards good (robust) capabilities than alignment, so we see strong and general capabilities first.

3. There’s essentially only one way to get general capabilities and it has a free parameter for the optimisation target.

There are many paths but only one destination when it comes to designing (via optimisation) a system with strong capabilities. But what those capabilities end up being directed at is path- and prior-dependent in a way we currently do not understand nor have much control over.

4. Corrigibility is conceptually in tension with capability, so corrigibility will fail to generalise when capability generalises well.

Plans that actually work in difficult domains need to preempt or adapt to obstacles. Attempts to steer or correct the target of actually-working planning are a form of obstacle, so we would expect capable planning to resist correction, limiting the extent to which alignment can generalise when capability starts to generalise.

5. Empirical evidence: human intelligence generalised far without staying aligned with its optimisation target.

There is empirical/historical support for capabilities generalising further than alignment to the extent that the analogy of AI development to the evolution of intelligence holds up.

6. Empirical evidence: goal misgeneralisation happens.

There is weak empirical support for capabilities generalising further than alignment in the fact that it is possible to create demos of goal misgeneralisation (e.g.,).

7. The world is simple whereas the target is not.

There are relatively simple laws governing how the world works, for the purposes of predicting and controlling it, compared to the principles underlying what humans value or the processes by which we figure out what is good. (This is similar to For#1 but focused on knowledge instead of cognitive abilities.) (This is in direct opposition to Against#3.)

8. Much more effort will be poured into capabilities (and d(progress)/d(effort) for alignment is not so much hig...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Kurzgesagt – The Last Human (Youtube), published by habryka on June 29, 2022 on LessWrong.

TLDW: Big Youtube channel Kurzgesagt released a video on the potential of humanity's future that's I think my favorite videos of theirs yet.

I have a pretty high bar for posting video content to LessWrong, but this video seemed better than most other videos in terms of capturing some of the things that make me interested in a lot of things on LessWrong.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There is no separate "community building" track, published by Owen Cotton-Barratt on June 29, 2022 on The Effective Altruism Forum.

Here's a cartoon picture I think people sometimes have:

I think this is misguided and can be harmful.

The appeal of the picture is that specialization generally brings big benefits; since there are two large buckets of work it's natural to think they should be pursued by specialists working in specialist orgs.

The reason that the specialization argument doesn't just go through is that the things we're trying to build communities for are complicated. I think community-building work is often a lot better when it's high context on what's being done and what's needed in direct work. In some cases this could shift it from negative to positive; in others it merely provides a significant boost. By default in the "separate camps" model, the community-building camp simply isn't high enough context on the direct work camp to achieve that.

Rather than separate camps, I think that it's better to think of there being one camp, which is oriented around "direct work". A good amount of community-building work goes on there, but it's all pretty well integrated with the direct work (often with heavy overlap of the people involved).

At an individual level, I think it's often correct for people to multi-class between direct work and community building. After you have expertise on direct work and what's needed, there are low-hanging fruit in leveraging that expertise in community building (this might be e.g. via giving talks or mentoring junior people). Or if you're mostly focused on community building I think you'll often benefit from spending a fraction of your time really trying to get to the bottom of understanding direct work agendas, and exactly why people are pursuing different strategies. A possible way of getting solid grounding for this is to actually spend a fraction of your time aiming to do high-value direct work, but that's certainly not the only approach — reading content and (especially) talking to people who are engaged with direct work about what their bottlenecks are also helpful.

As a very rough rule of thumb, I think that it's good if people doing community-building work spend at least 20% of their time obsessing over the details of what's needed in direct work.

Clarifications & caveats to these claims:

20% time doesn't need to mean a day every week. It could be ten weeks a year, or one year every five. (It probably shouldn't be stretched longer than that, because knowledge will go stale.)

I mean to refer to people trying to build communities aimed at increasing direct work. I think it's fine to have separate specialist people/orgs for things like:

Education — e.g. if you had an org aimed at getting more people to intuitively understand scope-sensitivity

Building communities that aren't aimed at direct work — e.g. if you want to build a community of AI researchers who read science fiction together and think about the future of AI

It's fine to have professional facilitators who are helping the community-building work without detailed takes on object-level priorities, but they shouldn't be the ones making the calls about what kind of community-building work needs to happen

The whole 20% figure is me plucking a number that feels reasonable out of the air. If someone wants to argue that it should be 12% or 30% I'm like "that's plausible". If someone wants to argue that it should be 2% or 80% I'm going to be pretty sceptical.

I think a lot of people who are working professionally in community building understand all of this intuitively. But it doesn't seem to me to be uniformly understood/discussed, so I thought it was worth sharing the model.

Corollaries of this view:

People who want to do community building work should be particularly inter...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Four reasons I find AI safety emotionally compelling, published by Kat Woods on June 28, 2022 on The Effective Altruism Forum.

‘I don’t feel emotionally motivated to work on AI safety, even though I’m intellectually convinced that it’s important.’

It always surprises me when people say this because I find my work at Nonlinear on AI safety incredibly motivating. I’m sharing my reasons in the hope that they’ll resonate with some of you, and that these ideas will help bring your emotional drives into greater harmony with your abstract convictions.

1. The dramatic scale of AGI is inspiring

When I was a kid, I wanted to save the world. Like many EAs, I was obsessed with stories of superheroes who could use their powers to save whole cities from catastrophe. I aspired to be like Gandhi, or Martin Luther King, and to do something really big and important; something that would forge widespread, lasting change. But as I got older, these dreams collided violently with the harsh reality: changing things is really really hard. I became more realistic and scaled down my ambitions.

I started off working on global health. In this area, the most I could hope for was (for example) to reduce infant mortality by a percent or two in a single country - and that’s if my career was an astonishing success. This is significant - that’s a lot of lives saved! - but it’s far more modest than my childhood ambitions. I resigned myself to the fact that while I could make significant improvements to the world, the problems we face would remain immense and oppressive.

Similarly, at one time, I felt most motivated to work on animal welfare. But again: even if I was unusually successful in my work, I could only hope to make a small dent in the problem. I might (for example) secure slightly better conditions for factory-farmed chickens in the US, but there would still be factory farms. And even if factory farms were abolished, what about the suffering of animals in the wild?

This situation was demotivating over time: the thought of so much intense pain was emotionally crushing, and I knew that even if I worked extremely hard and improved some animals’ lives, those improvements would be dwarfed by the suffering that remained.

With AGI, on the other hand, my ambitions to really change things have returned. AI is likely to change the world massively: for better or for worse. When we talk about AI, we often focus on its dangers: the fact that misaligned AI could kill us all or cause astronomical suffering. But there is a more positive side: if we manage to align AGI - if we get it right - it could fix so much. A superintelligent AI aligned with human values could end poverty, war, oppression, abuse, suffering itself - you name it. It could improve the world far more than any other intervention or advance to date.

This could be the most important cause in all of history. This time, if enough people work hard, if we’re thoughtful and strategic, we could really achieve utopian improvements to the human condition. I find this vision incredibly inspiring.

2. AI safety means you don’t have to “choose” a cause

Because aligned AI could have such wide-reaching effects, I find it easier to wholeheartedly dedicate myself to AI alignment. I used to feel terrible about all the problems I wasn’t helping with. As EAs, we know that we are always in triage and need to prioritize ruthlessly, but it can still feel painful, even if we know it’s the right thing to do.

I was working on poverty and animal welfare, but then I’d think - what about domestic abuse? What about the lobsters you see in tanks? What about depression? What about human rights abuses? What about North Korea? Or just plain old cancer, heart disease, death?

In the past, when I thought about one of these problems, I had to regretfully set it aside, and tell my...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The inordinately slow spread of good AGI conversations in ML, published by RobBensinger on June 29, 2022 on The Effective Altruism Forum.

Spencer Greenberg wrote on Twitter:

Recently @KerryLVaughan has been critiquing groups trying to build AGI, saying that by being aware of risks but still trying to make it, they’re recklessly putting the world in danger. I’m interested to hear your thought/reactions to what Kerry says and the fact he’s saying it.

Michael Page replied:

I'm pro the conversation. That said, I think the premise -- that folks are aware of the risks -- is wrong.

Honestly, I think the case for the risks hasn't been that clearly laid out. The conversation among EA-types typically takes that as a starting point for their analysis. The burden for the we're-all-going-to-die-if-we-build-x argument is -- and I think correctly so -- quite high.

Oliver Habryka then replied:

I find myself skeptical of this.

Like, my sense is that it's just really hard to convince someone that their job is net-negative. "It is difficult to get a man to understand something when his salary depends on his not understanding it" And this barrier is very hard to overcome with just better argumentation.

My reply:

I disagree with "the case for the risks hasn't been that clearly laid out". I think there's a giant, almost overwhelming pile of intro resources at this point, any one of which is more than sufficient, written in all manner of style, for all manner of audience.

(I do think it's possible to create a much better intro resource than any that exist today, but 'we can do much better' is compatible with 'it's shocking that the existing material hasn't already finished the job'.)

I also disagree with "The burden for the we're-all-going-to-die-if-we-build-x argument is -- and I think correctly so -- quite high."

If you're building a machine, you should have an at least somewhat lower burden of proof for more serious risks. It's your responsibility to check your own work to some degree, and not impose lots of micromorts on everyone else through negligence.

But I don't think the latter point matters much, since the 'AGI is dangerous' argument easily meets higher burdens of proof as well.

I do think a lot of people haven't heard the argument in any detail, and the main focus should be on trying to signal-boost the arguments and facilitate conversations, rather than assuming that everyone has heard the basics.

A lot of the field is very smart people who are stuck in circa-1995 levels of discourse about AGI.

I think 'my salary depends on not understanding it' is only a small part of the story. ML people could in principle talk way more about AGI, and understand the problem way better, without coming anywhere close to quitting their job. The level of discourse is by and large too low for 'I might have to leave my job' to be the very next obstacle on the path.

Also, many ML people have other awesome job options, have goals in the field other than pure salary maximization, etc.

More of the story: Info about AGI propagates too slowly through the field, because when one ML person updates, they usually don't loudly share their update with all their peers. This is because:

1. AGI sounds weird, and they don't want to sound like a weird outsider.

2. Their peers and the community as a whole might perceive this information as an attack on the field, an attempt to lower its status, etc.

3. Tech forecasting, differential technological development, long-term steering, exploratory engineering, 'not doing certain research because of its long-term social impact', prosocial research closure, etc. are very novel and foreign to most scientists.

EAs exert effort to try to dig up precedents like Asilomar partly because Asilomar is so unusual compared to the norms and practices of the vast majority of science. Sci...