The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What "Effective Altruism" Means to Me, published by Richard Y Chappell on June 14, 2024 on The Effective Altruism Forum.

I previously included a link to this as part of my trilogy on anti-philanthropic misdirection, but a commenter asked me to post the full text here for the automated audio conversion. Apologies to anyone who has already read it.

As I wrote in 'Why Not Effective Altruism?', I find the extreme hostility towards effective altruism from some quarters to be rather baffling. Group evaluations can be vexing: perhaps what the critics have in mind when they hate on EA has little or no overlap with what I have in mind when I support it? It's hard to know without getting into details, which the critics rarely do. So here are some concrete claims that I think are true and important.

If you disagree with any of them, I'd be curious to hear which ones, and why!

What I think:

1. It's good and virtuous to be beneficent and want to help others, for example by taking the Giving What We Can 10% pledge.

2. It's good and virtuous to want to help others effectively: to help more rather than less with one's efforts.

3. We have the potential to do a lot of good in the face of severe global problems (including global poverty, factory-farmed animal welfare, and protecting against global catastrophic risks such as future pandemics).

4. In all these areas, it is worth making deliberate, informed efforts to act effectively. Better targeting our efforts may make even more of a difference than the initial decision to help at all.

5. In all these areas, we can find interventions that we can reasonably be confident are very positive in expectation. (One can never be so confident of actual outcomes in any given instance, but being robustly positive in prospect is what's decision-relevant.)

6. Beneficent efforts can be expected to prove (much) more effective if guided by careful, in-depth empirical research. Quantitative tools and evidence, used wisely, can help us to do more good.

7. So it's good and virtuous to use quantitatively tools and evidence wisely.

8. GiveWell does incredibly careful, in-depth empirical research evaluating promising-seeming global charities, using quantitative tools and evidence wisely.

9. So it's good and virtuous to be guided by GiveWell (or comparably high-quality evaluators) rather than less-effective alternatives like choosing charities based on locality, personal passion, or gut feelings.

10. There's no good reason to think that GiveWell's top charities are net harmful.[1]

11. But even if you're the world's most extreme aid skeptic, it's clearly good and virtuous to voluntary redistribute your own wealth to some of the world's poorest people via GiveDirectly. (And again: more good and virtuous than typical alternatives.)

12. Many are repelled by how "hands-off" effective philanthropy is compared to (e.g.) local volunteering. But it's good and virtuous to care more about saving and improving lives than about being hands on. To prioritize the latter over the former would be morally self-indulgent.

13. Hits-based giving is a good idea. A portfolio of long shots can collectively be likely to do more good than putting all your resources into lower-expected-value "sure things". In such cases, this is worth doing.

14. Even if one-off cases, it is often better and more virtuous to accept some risk of inefficacy in exchange for a reasonable shot at proportionately greater positive impact. (But reasonable people can disagree about which trade-offs of this sort are worth it.)

15. The above point encompasses much relating to politics and "systemic change", in addition to longtermist long-shots. It's very possible for well-targeted efforts in these areas to be even better in expectation than traditional philanthropy - just note that this potential impact comes at ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Help Fund Insect Welfare Science, published by Bob Fischer on June 14, 2024 on The Effective Altruism Forum.

The Arthropoda Foundation

Tens of trillions of insects are used or killed by humans across dozens of industries. Despite being the most numerous animal species reared by animal industries, we know next to nothing about what's good or bad for these animals. And right now, funding for this work is scarce. Traditional science funders won't pay for it; and within EA, the focus is on advocacy, not research. So,

welfare science needs your help.

We're launching the

Arthropoda Foundation, a fund to ensure that insect welfare science gets the essential resources it needs to provide decision-relevant answers to pressing questions. Every dollar we raise will be granted to research projects that can't be funded any other way.

We're in a critical moment for this work. Over the last year, field-building efforts have accelerated, setting up academic labs that can tackle key studies. However, funding for these studies is now uncertain. We need resources to sustain the research required to improve the welfare of insects.

Why do we need a fund?

We need a fund because we need a runway for high-priority research. Scientists need to make plans over several years, not a few months. They have to commit now to a grad student who starts next year and finishes a project two years after that. The fund helps guarantee that resources will be there to support academics in the long-term, ensuring that entire labs can remain devoted to this work.

We need a fund because we need to let researchers be researchers, not fundraisers. A fund doesn't just buy critical research; it buys the ability of the world's few insect welfare scientists to focus on what matters.

We need a fund because funding scientific research on insect welfare isn't easy for individual donors. First, it's hard to know what to fund. As some of the few researchers who have worked on these issues in EA, we're lending our expertise to vet opportunities. Second, universities take overhead that reduces the impact of your donations; an independent fund can use the board's volunteer labor to make the many small reimbursements that are required to cover costs directly.

Third, if you're a donor who's giving below the amounts required to support entire projects, your opportunities are extremely limited. This fund smooths over such hurdles, ensuring that everyone can support the highest value research.

This fund gives a brand new field some time to get established, it gives that field the resources required to produce essential science, and it keeps that research as cost-effective as possible.

Please support welfare science.

Team

Bob Fischer is a Professor at Texas State University and the lead project manager and author of the

Moral Weight Project, a research project to build comparative models of moral weight across animal species.

Daniela Waldhorn is the Director of Animal Welfare research at Rethink Priorities, a board member of the Centre for Animal Ethics at Pompeu Fabra University, and

lead author on the largest initial EA project focused on studying invertebrate welfare.

Abraham Rowe is the Principal of Good Structures, a nonprofit operations service provider, and was previously the COO of Rethink Priorities, and the co-founder and Executive Director of Wild Animal Initiative, an academic field-building and grantmaking organization supporting research on wild animal welfare.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] An update from Good Ventures, published by Alexander Berger on June 14, 2024 on The Effective Altruism Forum.

I wanted to share

this update from Good Ventures (Cari and Dustin's philanthropy), which seems relevant to the EA community.

Tl;dr: "while we generally plan to continue increasing our grantmaking in our existing focus areas via our partner Open Philanthropy, we have decided to exit a handful of sub-causes (amounting to less than 5% of our annual grantmaking), and we are no longer planning to expand into new causes in the near term by default."

A few follow-ups on this from an Open Phil perspective:

I want to apologize to directly affected grantees (who've already been notified) for the negative surprise here, and for our part in not better anticipating it.

While this represents a real update, we remain deeply aligned with Good Ventures (they're expecting to continue to increase giving via OP over time), and grateful for how many of the diverse funding opportunities we've recommended that they've been willing to tackle.

An example of a new potential focus area that OP staff had been interested in exploring that Good Ventures is not planning to fund is research on the potential moral patienthood of digital minds. If any readers are interested in funding opportunities in that space, please

reach out.

Good Ventures has told us they don't plan to exit any overall focus areas in the near term. But this update is an important reminder that such a high degree of reliance on one funder (especially on the

GCR side) represents a structural risk. I think it's important to diversify funding in many of the fields Good Ventures currently funds, and that doing so could make the funding base more stable both directly (by diversifying funding sources) and indirectly (by lowering the time and energy costs to Good Ventures from being such a disproportionately large funder).

Another implication of these changes is that going forward, OP will have a higher bar for recommending grants that could draw on limited Good Ventures bandwidth, and so our program staff will face more constraints in terms of what they're able to fund. We always knew we weren't funding every worthy thing out there, but that will be even more true going forward. Accordingly, we expect marginal opportunities for other funders to look stronger going forward.

Historically, OP has been focused on finding enough outstanding giving opportunities to hit Good Ventures' spending targets, with a long-term vision that once we had hit those targets, we'd expand our work to support other donors seeking to maximize their impact. We'd already gotten a lot closer to GV's spending targets over the last couple of years, but this update has accelerated our timeline for investing more in partnerships and advising other philanthropists.

If you're interested, please consider

applying or

referring candidates to lead our new partnerships function. And if you happen to be a philanthropist looking for advice on how to invest >$1M/year in new cause areas, please

get in touch.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shard Theory - is it true for humans?, published by ErisApprentice on June 14, 2024 on The AI Alignment Forum.

And is it a good model for value learning in AI?

(Read on Substack:

https://recursingreflections.substack.com/p/shard-theory-is-it-true-for-humans)

TLDR

Shard theory proposes a view of value formation where experiences lead to the creation of context-based 'shards' that determine behaviour. Here, we go over psychological and neuroscientific views of learning, and find that while shard theory's emphasis on context bears similarity to types of learning such as conditioning, it does not address top-down influences that may decrease the locality of value-learning in the brain.

What's Shard Theory (and why do we care)?

In 2022, Quintin Pope and Alex Turner posted '

The shard theory of human values', where they described their view of how experiences shape the value we place on things. They give an example of a baby who enjoys drinking juice, and eventually learns that grabbing at the juice pouch, moving around to find the juice pouch, and modelling where the juice pouch might be, are all helpful steps in order to get to its reward.

'Human values', they say, 'are not e.g. an incredibly complicated, genetically hard-coded set of drives, but rather sets of contextually activated heuristics…' And since, like humans, AI is often trained with reinforcement learning, the same might apply to AI.

The original post is long (over 7,000 words) and dense, but Lawrence Chan helpfully posted a condensation of the topic in '

Shard Theory in Nine Theses: a Distillation and Critical Appraisal'. In it, he presents nine (as might be expected) main points of shard theory, ending with the last thesis: 'shard theory as a model of human values'. 'I'm personally not super well versed in neuroscience or psychology', he says, 'so I can't personally attest to [its] solidity…I'd be interested in hearing from experts in these fields on this topic.' And that's exactly what we're here to do.

A Crash Course on Human Learning

Types of learning

What is learning? A baby comes into the world and is inundated with sensory information of all kinds. From then on, it must process this information, take whatever's useful, and store it somehow for future use.

There's various places in the brain where this information is stored, and for various purposes. Looking at these various types of storage, or memory, can help us understand what's going on:

3 types of memory

We often group memory types by the length of time we hold on to them - 'working memory' (while you do some task), 'short-term memory' (maybe a few days, unless you revise or are reminded), and 'long-term memory' (effectively forever). Let's take a closer look at long-term memory:

Types of long-term memory

We can broadly split long-term memory into 'declarative' and 'nondeclarative'. Declarative memory is stuff you can talk about (or 'declare'): what the capital of your country is, what you ate for lunch yesterday, what made you read this essay. Nondeclarative covers the rest: a grab-bag of memory types including knowing how to ride a bike, getting habituated to a scent you've been smelling all day, and being motivated to do things you were previously rewarded for (like drinking sweet juice).

For most of this essay, we'll be focusing on the last type: conditioning.

Types of conditioning

Conditioning

Sometime in the 1890s, a physiologist named Ivan Pavlov was researching salivation using dogs. He would feed the dogs with powdered meat, and insert a tube into the cheek of each dog to measure their saliva.As expected, the dogs salivated when the food was in front of them. Unexpectedly, the dogs also salivated when they heard the footsteps of his assistant (who brought them their food).

Fascinated by this, Pavlov started to play a metronome whenever h...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shard Theory - is it true for humans?, published by Rishika Bose on June 14, 2024 on The AI Alignment Forum.

And is it a good model for value learning in AI?

(Read on Substack:

https://recursingreflections.substack.com/p/shard-theory-is-it-true-for-humans)

TLDR

Shard theory proposes a view of value formation where experiences lead to the creation of context-based 'shards' that determine behaviour. Here, we go over psychological and neuroscientific views of learning, and find that while shard theory's emphasis on context bears similarity to types of learning such as conditioning, it does not address top-down influences that may decrease the locality of value-learning in the brain.

What's Shard Theory (and why do we care)?

In 2022, Quintin Pope and Alex Turner posted '

The shard theory of human values', where they described their view of how experiences shape the value we place on things. They give an example of a baby who enjoys drinking juice, and eventually learns that grabbing at the juice pouch, moving around to find the juice pouch, and modelling where the juice pouch might be, are all helpful steps in order to get to its reward.

'Human values', they say, 'are not e.g. an incredibly complicated, genetically hard-coded set of drives, but rather sets of contextually activated heuristics…' And since, like humans, AI is often trained with reinforcement learning, the same might apply to AI.

The original post is long (over 7,000 words) and dense, but Lawrence Chan helpfully posted a condensation of the topic in '

Shard Theory in Nine Theses: a Distillation and Critical Appraisal'. In it, he presents nine (as might be expected) main points of shard theory, ending with the last thesis: 'shard theory as a model of human values'. 'I'm personally not super well versed in neuroscience or psychology', he says, 'so I can't personally attest to [its] solidity…I'd be interested in hearing from experts in these fields on this topic.' And that's exactly what we're here to do.

A Crash Course on Human Learning

Types of learning

What is learning? A baby comes into the world and is inundated with sensory information of all kinds. From then on, it must process this information, take whatever's useful, and store it somehow for future use.

There's various places in the brain where this information is stored, and for various purposes. Looking at these various types of storage, or memory, can help us understand what's going on:

3 types of memory

We often group memory types by the length of time we hold on to them - 'working memory' (while you do some task), 'short-term memory' (maybe a few days, unless you revise or are reminded), and 'long-term memory' (effectively forever). Let's take a closer look at long-term memory:

Types of long-term memory

We can broadly split long-term memory into 'declarative' and 'nondeclarative'. Declarative memory is stuff you can talk about (or 'declare'): what the capital of your country is, what you ate for lunch yesterday, what made you read this essay. Nondeclarative covers the rest: a grab-bag of memory types including knowing how to ride a bike, getting habituated to a scent you've been smelling all day, and being motivated to do things you were previously rewarded for (like drinking sweet juice).

For most of this essay, we'll be focusing on the last type: conditioning.

Types of conditioning

Conditioning

Sometime in the 1890s, a physiologist named Ivan Pavlov was researching salivation using dogs. He would feed the dogs with powdered meat, and insert a tube into the cheek of each dog to measure their saliva.As expected, the dogs salivated when the food was in front of them. Unexpectedly, the dogs also salivated when they heard the footsteps of his assistant (who brought them their food).

Fascinated by this, Pavlov started to play a metronome whenever he ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fine-tuning is not sufficient for capability elicitation, published by Theodore Chapman on June 14, 2024 on The AI Alignment Forum.

Produced as part of the

ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort under the supervision of Evan Hubinger.

Acknowledgements: Thanks to Kyle Brady for his many contributions to this project.

Abstract

This post argues that the performance elicited by fine-tuning an LLM on a task using a given prompt format does not usefully bound the level of performance observed when the same information is presented in a different structure. Thus, fine-tuned performance provides very little information about the best performance that would be achieved by a large number of actors fine-tuning models with random prompting schemes in parallel.

In particular, we find that we get much better results from fine-tuning gpt-3.5-turbo (ChatGPT 3.5) to play chess when the game so far is presented in a single block of SAN[1] than when the game so far is separated into a series of SAN moves presented as alternating user / assistant messages.

The fact that this superficial formatting change is sufficient to change our fine-tuned performance serves to highlight that modern LLMs are much more fragile than they appear at first glance, even subject to fine-tuning.

Introduction

In the abstract, model evaluations identify a task and attempt to establish a bound on the level of performance that can be elicited from a given model with a given level of investment. The current state of the art is roughly:

1. Choose a reasonable prompting scheme

2. Generate a dataset of high-quality samples and encode them in the chosen format

3. Fine-tune the model and evaluate the resulting performance

4. Make some implicit regularity assumptions about the quality of models fine-tuned using different prompting schemes[1]

5. Conclude that probably no other actor can elicit substantially better performance on the same task from the same model while spending substantially less money than we did

This post takes issue with step 4. We begin by illustrating the extreme brittleness of observed model performance when prompting without fine-tuning. Then we argue that fine-tuning is not sufficient to eliminate this effect. Using chess as a toy model, we show two classes of prompting schemes under which ChatGPT-3.5 converges to dramatically different levels of performance after fine-tuning.

Our central conclusion is that the structure in which data is presented to an LLM (or at least to ChatGPT 3.5) matters more than one might intuitively expect and that this effect persists through fine-tuning.

In the specific case of chess, the better prompting scheme that we use (described in the section below) is easily derived but in situations that are further out of distribution (such as the automated replication and adaptation tasks METR defined), it is not obvious what the best way to present information is, and it seems plausible that there are simple prompt formats which would result in substantially better performance than those that we've tested to date.

General Setting

We use the term 'agent' to refer to the combination of a model - here gpt-3.5-turbo unless otherwise specified - and a function which takes a chess position as input and outputs the document we feed into the model (henceforth a 'prompting scheme').

We perform our evaluations using three datasets of chess games:

1. A collection of ~6000 games played by humans on Lichess with at least 30 minutes for each player

2. A collection of ~500 games played between all pairings of stockfish 16 level 1, 5, 10, 15, and 20

3. A collection of ~300 games played by ChatGPT 3.5 or gpt-3.5-turbo-instruct with various prompting schemes

We evaluate our agents by selecting a random point in each of the games, providing the current game position as...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Be Proud To Be An Effective Altruist, published by Omnizoid on June 14, 2024 on The Effective Altruism Forum.

Crossposted on my blog.

When I was at Manifest, Austin Chen said something interesting to me: that one thing that he liked about my blog was that I was proud to be an effective altruist. I didn't treat it as an embarrassing quirk, but as something that's actively good, that's worth encouraging. A lot of EAs don't do this, which I think is sad. Scott Alexander

amusingly quips:

I hate having to post criticism of EA.

Not because EA is bad at taking criticism. The opposite: they like it too much. It almost feels like a sex thing. "Please, tell me again how naughty I'm being!" I went to an EA organization's offices once - I think it was OpenPhil, but don't quote me on that - and the whole place was strewn with the most critical books you can imagine - Robert Reich, Anand Giradharadas, that kind of thing.

I remember when I was out tabling for the effective altruism club at my University, some woman came up and started trotting out braindead anti-effective altruism arguments - something about tech bros and SBF and white saviorism. The other very reasonable and responsible EAs at the club were saying things like "hmm, yeah, that's a very interesting criticism, well, if you want to come to a meeting, we'd be happy to discuss it."

I wanted to scream! These aren't serious criticisms. This is complete bullshit.

Spend enough time listening to the criticisms of effective altruism and it becomes clear that, aside from those arguing for small tweaks at the marigns, they all stem from either a) people being very dogmatic and having a worldview that's strangely incompatible with doing good things (if, for instance, they don't help the communist revolution); b) people wanting an excuse to do nothing in the face of extreme suffering; or c) people disliking effective altruists and so coming up with some

half-hearted excuse for why EA is really something-something colonialism.

I don't want to be too mean about this, but the criticisms are unbelievably dumb. They are confused to a really extraordinary degree. The reasoning is so exquisitely poor that it's very clearly motivated. I don't say that about other subjects - this is the only subject on which the critics don't have any half-decent objections. And yet despite this, lots of effective altruists treat the criticisms as serious.

1,000 children die a day of malaria. Think about how precious the life is of a young child - concretely picture a small child coughing up blood and lying in bed with a fever of 105. We - the effective altruists - are the ones doing something about that. And not just doing a bit about it - working as hard as possible to eradicate malaria, taking seriously how precious the lives of those children are.

GiveWell does research into the charities that avert misery most effectively, and then tens of thousands of EAs all around the world give to those charities, because they recognize that though they'll never see the children they help, those children matter.

This is not something to be embarrassed about. Giving money to effective charities and getting others to do the same is by far the best thing I have ever done in my life. Every nice thing I've done interpersonally doesn't have half a percent the value of saving lives, of funneling money into the hands of charities so that little kids don't get horrible diseases that kill them.

And yet these critics who do nothing, who sit on their asses as children die whose deaths they can easily avert have the gall to criticize those of us who are doing something, who are working to avert deaths as effectively as possible. They have the gall to criticize those of us who are working to free animals from the horrifying torture chambers known as

factory farms.

It's one thi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #68: Remarkably Reasonable Reactions, published by Zvi on June 14, 2024 on LessWrong.

The big news this week was Apple Intelligence being integrated deeply into all their products. Beyond that, we had a modestly better than expected debate over the new version of SB 1047, and the usual tons of stuff in the background. I got to pay down some writing debt.

The bad news is, oh no, I have been called for Jury Duty. The first day or two I can catch up on podcasts or pure reading, but after that it will start to hurt. Wish me luck.

Table of Contents

AiPhone covers the announcement of Apple Intelligence. Apple's products are getting device-wide integration of their own AI in a way they say preserves privacy, with access to ChatGPT via explicit approval for the heaviest requests. A late update: OpenAI is providing this service for free as per Bloomberg.

I offered Quotes from Leopold Aschenbrenner's Situational Awareness Paper, attempting to cut down his paper by roughly 80% while still capturing what I considered the key passages. Then I covered his appearance on Dwarkesh's Podcast, where I offered commentary. The plan is to complete that trilogy tomorrow, with a post that analyzes Leopold's positions systematically, and that covers the reactions of others.

1. Introduction.

2. Table of Contents.

3. Language Models Offer Mundane Utility. Roll your own process.

4. Language Models Don't Offer Mundane Utility. What happened to Alexa?

5. Fun With Image Generation. Dude, where's my image of a car?

6. Copyright Confrontation. Everyone is rather on edge these days.

7. Deepfaketown and Botpocalypse Soon. People will do things that scale.

8. They Took Our Jobs. Lost your job? No problem. Start a new company!

9. Someone Explains it All. Data center construction, the bitter lesson.

10. The Art of the Jailbreak. The Most Forbidden Technique?

11. Get Involved. AISI hiring a senior developer.

12. Introducing. New OpenAI execs, new AI assistant, new short video model.

13. In Other AI News. More progress avoiding MatMul. Nvidia takes it all in stride.

14. Quiet Speculations. What you see may be what you get.

15. I Spy With My AI. Microsoft Recall makes some changes to be slightly less crazy.

16. Pick Up the Phone. Perhaps a deal could be made.

17. Lying to the White House, Senate and House of Lords. I don't love it.

18. The Quest for Sane Regulation. People want it. Companies feel differently.

19. More Reasonable SB 1047 Reactions. Hearteningly sane reactions by many.

20. Less Reasonable SB 1047 Reactions. The usual suspects say what you'd suspect.

21. That's Not a Good Idea. Non-AI example, California might ban UV lights.

22. With Friends Like These. Senator Mike Lee has thoughts.

23. The Week in Audio. Lots to choose from, somehow including new Dwarkesh.

24. Rhetorical Innovation. Talking about probabilities with normies is hard.

25. Mistakes Were Made. Rob Bensinger highlights two common ones.

26. The Sacred Timeline. What did you mean? Which ways does it matter?

27. Coordination is Hard. Trying to model exactly how hard it will be.

28. Aligning a Smarter Than Human Intelligence is Difficult. Natural abstractions?

29. People Are Worried About AI Killing Everyone. Reports and theses.

30. Other People Are Not As Worried About AI Killing Everyone. Why not?

31. The Lighter Side. Do you have to do this?

What is still in the queue, in current priority order?

1. The third and final post on Leopold Aschenbrenner's thesis will come tomorrow.

2. OpenAI has now had enough drama that I need to cover that.

3. DeepMind's scaling policy will get the analysis it deserves.

4. Other stuff remains: OpenAI model spec, Rand report, Seoul, the Vault.

Language Models Offer Mundane Utility

Write letters to banks on your behalf by invoking Patrick McKenzie.

Can GPT-4 autonomously hack zero-day security flaws u...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Maybe let the non-EA world train you, published by ElliotT on June 14, 2024 on The Effective Altruism Forum.

This post is for EAs at the start of their careers who are considering which organisations to apply to, and their next steps in general.

Conclusion up front: It can be really hard to get that first job out of university. If you don't get your top picks, your less exciting backup options can still be great for having a highly impactful career. If those first few years of work experience aren't your best pick, they will still be useful as a place where you can 'learn how to job', save some money, and then pivot or grow from there.

The main reasons are:

The EA job market can be grim. Securing a job at an EA organisation out of university is highly competitive, often resulting in failing to get a job, or chaotic job experiences due to the nascent nature of many EA orgs. An alternative, of getting short-term EA grants to work independently is not much better, as it can lead to financial instability and hinder long-term professional growth.

Non-EA jobs have a lot of benefits. They offer a stable and structured environment to build skills, learn organisational norms, get feedback, etc.

After, you'll be better placed to do directly impactful work. After a few years at a non-EA job, you'll be better placed to fill a lot of roles at EA orgs. You might also be able to start something yourself.

Caveats: Of course, take everything with a grain of salt.

For every piece of advice, there is someone who needs to hear the opposite, and the advice in here is no exception.

Acknowledgments: Thanks to the following people for giving some great feedback on an a draft of this post and making it better: David Nash, Matt Reardon, Chana Messinger, Karla Still, Michelle Hutchinson. All opinions are my own. All mistakes are chatGPT's.

What's the problem? Three failure modes of trying to get an EA job

It seems like a lot of people who are motivated by the ideas of effective altruism, use 'get a job at an 'EA org' as shorthand for 'how to have an impact with my career' (this includes me, but more on that later). By EA org I mean the kind of organisation where most people working there are EAs. This is understandable. Figuring out how to have a positive impact with your career is really hard.

It's a reasonable heuristic that orgs within the EA community are more likely to have a big positive impact in the world than the average non-profit. Finally, we're all sensitive to status in our community, and in some parts of EA, working at an EA org is definitely considered pretty darn cool.

One issue with this that I want to briefly flag is that 'working at an EA org' and 'doing impactful work' are not interchangeable (Michelle at 80,000 Hours covers that well

here). But the other thing that you probably know is that EA jobs are really hard to get. They are really, really, competitive. Some jobs get hundreds of high-quality applications. I think this leads to a few failure modes.

Story 1 - Lots of rejections: An ambitious, smart, highly engaged EA, fresh out of uni, applies to a lot of EA organisations. They make it far in the hiring rounds, maybe even to trial periods. But, after many disheartening months of applications, trial tasks, and interviews, they don't get an EA job at an EA org.

Story 2 - Chaotic EA Job: Alternatively, maybe they get the job at an impactful organisation, thinking they've made it past the hard bit. However, many EA orgs are very young. This can lead to things like a chaotic onboarding, a lack of HR practices, inexperienced managers, and sudden changes in funding or strategy that can leave them without a job. And if that isn't enough, EA branding does not guarantee an org is doing impactful work.

Both of these stories are pretty bad. These situations are psychologically and...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: OpenAI appoints Retired U.S. Army General Paul M. Nakasone to Board of Directors, published by Joel Burget on June 14, 2024 on LessWrong.

Today, Retired U.S. Army General Paul M. Nakasone has joined our Board of Directors. A leading expert in cybersecurity, Nakasone's appointment reflects OpenAI's commitment to safety and security, and underscores the growing significance of cybersecurity as the impact of AI technology continues to grow.

As a first priority, Nakasone will join the Board's

Safety and Security Committee, which is responsible for making recommendations to the full Board on critical safety and security decisions for all OpenAI projects and operations.

Whether this was influenced by Aschenbrenner's Situational Awareness or not, it's welcome to see OpenAI emphasizing the importance of security. It's unclear how much this is a gesture vs reflective of deeper changes.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org