Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Cage of the Language, published by Martin Sustrik on April 13, 2022 on LessWrong.

Wittgenstein: Die Grenzen meiner Sprache bedeuten die Grenzen meiner Welt.

Passer-by: Where's that guy from? Is he Dutch?

I've never thought much of the Wittgenstein's argument. First, there's non-verbal thinking that can be used to escape the limits of the language. Second, once the knowledge is gained by non-verbal thinking, it can be used to extend the language itself and thus extend the limits of everyone's world.

However, I've been writing newspaper articles and a blog in Slovak lately and the experience made me think of the Wittgenstein quote.

Namely, I had hard time expressing some ideas and making some arguments. Slovak, you see, is a language of 5 million speakers, with ~200 years of history as a literary language. A lot of terms that one can use in English simply do not exist.

I can't say, for example:

I support policy X because it enables economies of scale and I am against policy Y because it introduces a single point of failure.

There are no widely accepted equivalents to "economies of scale" and "single point of failure".

I can say:

Som za X, pretože umožňuje úspory z rozahu a som proti Y, pretože zavádza jediný bod zlyhania.

It's hard to relate how that sounds. Enough to say that "úspory z rozahu" doesn't literally translate to "economies of scale" but rather to "savings from the extent". One gets an impression of a crackpot speaking gibberish, making a crackpot argument.

It should be said that this is not a problem in one-to-one conversations. One can use the English term if the other is aware of it, or, if not, he can clarify the meaning of the term in advance. Where the problem hits is the public discussion. There you can't expect the prior knowledge or afford extensive explanations. Attention span in public discussion is, after all, very limited. If you waste your one minute of public attention by explaining what a single point of failure is, you've already lost.

It can be argued that the problem lies not with the language, but rather with the language speakers: Slovaks are simple people, uneducated, they don't have the concept of economies of scale. Therefore, they should be educated and the term would force its way into the language all by itself.

But that, while true, is missing the point.

Most English speakers don't know what economies of scale are either. Yet, with 1.5 billion English speakers there's a large enough minority that does. They use the term, they write about it, they discuss it. If you encounter the term and you are not sure what it means you can google it and find out that there's a lot of hits, that some pretty serious people are discussing it, you can even learn what it means yourself. In any case, there's none of this "lonely crackpot" perception that you get if using a small language.

All in all, in small language communities, the limits of the language are the limits of the Overton window. Some policies are not on the table not because people are opposed to them but because they can't even discuss them.

It reminds me of what Joe Henrich says about small populations:

There are these great cases in the ethnohistorical record of groups like the Polar Inuit who get cut off from the rest of the Inuit population. Then they begin to lose valuable tools and technology because their own brains remain the same size, but their collective brain became severed. They’re not able to maintain as much know-how in the population.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Takeoff speeds have a huge effect on what it means to work on AI x-risk, published by Buck Shlegeris on April 13, 2022 on The AI Alignment Forum.

The slow takeoff hypothesis predicts that AGI emerges in a world where powerful but non-AGI AI is already a really big deal. Whether AI is a big deal right before the emergence of AGI determines many super basic things about what we should think our current job is. I hadn’t fully appreciated the size of this effect until a few days ago.

In particular, in a fast takeoff world, AI takeover risk never looks much more obvious than it does now, and so x-risk-motivated people should be assumed to cause the majority of the research on alignment that happens. In contrast, in a slow takeoff world, many aspects of the AI alignment problems will already have showed up as alignment problems in non-AGI, non-x-risk-causing systems; in that world, there will be lots of industrial work on various aspects of the alignment problem, and so EAs now should think of themselves as trying to look ahead and figure out which margins of the alignment problem aren’t going to be taken care of by default, and try to figure out how to help out there.

In the fast takeoff world, we’re much more like a normal research field–we want some technical problem to eventually get solved, so we try to solve it. But in the slow takeoff world, we’re basically in a weird collaboration across time with the more numerous, non-longtermist AI researchers who will be in charge of aligning their powerful AI systems but who we fear won’t be cautious enough in some ways or won’t plan ahead in some other ways. Doing technical research in the fast takeoff world basically just requires answering technical questions, while in the slow takeoff world your choices about research projects are closely related to your sociological predictions about what things will be obvious to whom when.

I think that these two perspectives are extremely different, and I think I’ve historically sometimes had trouble communicating with people who held the slow takeoff perspective because I didn’t realize we disagreed on basic questions about the conceptualization of the question. (These miscommunications persisted even after I was mostly persuaded of slow takeoffs, because I hadn’t realized the extent to which I was implicitly assuming fast takeoffs in my picture of how AGI was going to happen.)

As an example of this, I think I was quite confused about what genre of work various prosaic alignment researchers think they’re doing when they talk about alignment schemes. To quote a recent AF shortform post of mine:

Something I think I’ve been historically wrong about:

A bunch of the prosaic alignment ideas (eg adversarial training, IDA, debate) now feel to me like things that people will obviously do the simple versions of by default. Like, when we’re training systems to answer questions, of course we’ll use our current versions of systems to help us evaluate, why would we not do that? We’ll be used to using these systems to answer questions that we have, and so it will be totally obvious that we should use them to help us evaluate our new system.

Similarly with debate--adversarial setups are pretty obvious and easy.

In this frame, the contributions from Paul and Geoffrey feel more like “they tried to systematically think through the natural limits of the things people will do” than “they thought of an approach that non-alignment-obsessed people would never have thought of or used”.

It’s still not obvious whether people will actually use these techniques to their limits, but it would be surprising if they weren’t used at all.

I think the slightly exaggerated slogan for this update of mine is “IDA is futurism, not a proposal”.

My current favorite example of the thinking-on-the-margin version of alignment res...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What to include in a guest lecture on existential risks from AI?, published by Aryeh Englander on April 13, 2022 on The AI Alignment Forum.

A professor I'm friendly with has been teaching a course on AI ethics this semester, and he asked me if I could come give a guest lecture on "AI apocalypse" scenarios. What should I include in the lecture?

Details:

Audience is mostly graduate engineering students. Some are in a program focused on AI, others are in a systems engineering program and don't otherwise have much knowledge about AI beyond this course.

This will be I think the final class for the semester. Previous classes have covered topics like privacy, bias, explainability, social coordination failures (including race to the bottom dynamics), self-driving cars, AI in social media, lethal autonomous weapons, AI in healthcare, implementing ethics in organizations, and the economic impacts of AI. I did not attend those lectures so I don't know exact details of what was or wasn't covered.

I have leeway to include pretty much anything I want in the class and to focus on any topic(s) I want. The professor would like me to include something about forecasting, but that's optional.

I can optionally assign something short as an assignment before class, but it is the end of the semester so nothing more than that. There will not be any homework based on the class, I think.

The class will be 2 hours long.

I do not have a huge amount of time to prepare for the lecture - it's relatively soon (April 27) and I have multiple other work / school / family obligations I need to attend to between now and then.

If anybody has relevant material I could use, such as slides or activities, that would be great! Also, if anybody wants to help develop the material for this class, please message me (preferably at my work email - [email protected]).

As a bonus, I expect that material for a class of this sort may turn out to be useful for plenty of other people on this and related forums, either for themselves or as a tool they can use when presenting the same topic to others.

[Note: I am posting this here with permission from the professor.]

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Help us make civilizational refuges happen, published by Linch on April 13, 2022 on The Effective Altruism Forum.

There are a number of overlapping reasons for why we might want really good refuges:

Countermeasures refuges: In a global catastrophic biological event, we ought to have a secure facility where scientific personnel can develop medical countermeasures in a position of relative safety.

Foundation-style refuges: In the case of an extreme global catastrophe, a civilizational refuge can protect a nucleus of people, information, and materials to increase the probability of rebuilding and recovery after civilization collapse.

Values refuges: We would have a means to safeguard important cosmopolitan values and empirical or conceptual worldviews to survive and flourish after a civilizational collapse, assuming it is good to do so.

Currently, we think all three reasons are plausible, which a focus on #1 and #3. In particular, we believe that building refuges can be an unusually effective broad-spectrum intervention for reducing existential risk from a wide range of non-agentic sources.

We (a working group composed of Linch, Ajay, and a third, anonymous, collaborator) have worked part-time to deliver a temporary plan for refuges. We’ve made considerable progress, but we will greatly benefit from having a technically competent volunteer or short-term contractor help us refine our plans and make them much more detailed/precise/concrete. If you are interested, please message us or chat with us at EA Global London!

Funders are potentially interested, assuming a good initial team and operational plan.

Our two biggest bottlenecks:

We have not yet identified great candidates for a sufficiently competent (interim) CEO who is both capable and willing to take ownership of the refuge project.

Our technical requirements write-up is not yet detailed enough to a) attract top non-EA talent or b) discuss a feasibility study with an architecture firm in sufficient depth to be useful. This would require expanding out technical requirements from a ~10-pager to a ~40-pager in the next 4-6 weeks.

Here are ways you can advance the refuge project:

Reach out to us, either here or at EAGlobal if you’re interested in volunteering to help us build better plans or talent scout for a CEO to champion this.

If you think you might make for a good CEO, definitely talk to us.

Apply to Rethink Priority’s new temporary roles in General Longtermism or Special Projects, to potentially work full-time on this under Linch’s guidance.

Note that the jobs won’t start until June/July, and we’d ideally want to make progress faster. Also it is not guaranteed that RP would work on this.

Independently come up with your own plans/schema for civilizational refuges, and pitch it to funders. More competition in the EA ecosystem can be very valuable!

E.g., impartial concern, the importance of existential risk reduction

We can potentially help with funding small-scale experiments and preliminary research.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How dath ilan coordinates around solving alignment, published by Thomas Kwa on April 13, 2022 on LessWrong.

Summary

Eliezer Yudkowsky created dath ilan in 2014 as a fictional world where Yudkowsky is the median person. Since then, he has written hundreds of pages of fiction in this setting, mostly "rational fiction", which means that Yudkowsky thinks most of the things he writes about in dath ilan would actually happen given the premises. This post is a series of glowfic quotes.

When the idea of AGI was first discovered "a few generations" ago, the world government of dath ilan took action to orient their entire civilization around solving the alignment problem, including getting 20% of their researchers to do safety research, and slowing the development of multiple major technologies. AGI safety research has been ongoing for generations.

The possibility of AGI and other x-risks is considered an infohazard of the highest importance. It was even kept from representatives at first. At our current tech level, the average dath ilani has not heard about the possibility of AGI; even so, education gives dath ilani intuitions that might be useful for alignment research.

Dath ilan prediction markets that include Keepers (who know about AGI) say they have a 97% chance of reviving cryonically preserved people. Yudkowsky has confirmed this implies dath ilan has a >97% chance of solving alignment.

Yudkowsky thinks Earth will probably fail at alignment. It's not clear whether the likely success of dath ilan is due to its higher average research ability, or its widespread coordination around civilizational priorities and infohazards, but I strongly suspect it's the latter.

The discovery of x-risk from AGI

This is dath ilan; and a few generations before Thellim's time, the highest of the Keepers called together the Nine Legislators in a secret meeting. Shortly after, the highest Keeper and the Nine called an emergency closed assembly of the 324 Representatives.

And the highest Keeper said, with the Nine Legislators at the peak of dath ilan standing beside her, that the universe had proven to be a harsher and colder and more dangerous place than had been hoped.

And that all Civilization needed to turn much of its effort away from thriving, and toward surviving. There needed to be controls and slowdowns and halts instituted on multiple major technologies. Which would need to be backed up by much more pervasive electronic surveillance than anybody had ever even considered allowing before. Roughly a fifth of all the present and future smart people in the world ought to publicly appear to burn out or retire, and privately work on a new secret project under maximum-security conditions. [...]

The reasoning behind this policy could, in principle, be laid out to the 324 Representatives. But that would represent a noticeable additional risk, if it happened now, while mechanisms to prevent information propagation hadn't been set up yet.

How dath ilan deals with the infohazard of AGI

Emielle Potgieter: What is parallel earth's biggest problem, then?

Eliezer Yudkowsky: I'd assume that Artificial General Intelligence is being seen by the Senior Very Serious People as a big problem, given the degree to which nobody ever talked about it, how relatively slow computing progress was compared to here, and how my general education just happened to prepare me to make a ton of correct inferences about it as soon as anybody mentioned the possibility to me. They claim to you it's about hypothetical aliens and economic dysfunction scenarios, but boy howdy do you get a lot of Orthogonality and Goodhart's Curse in the water supply.

“I say this to complete the circle which began with my arrival: The world of dath ilan did not talk about existential risk. I strongly hypothesize that this is one of those things t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Another list of theories of impact for interpretability, published by Beth Barnes on April 13, 2022 on The AI Alignment Forum.

Neel's post on this is good. I thought I'd add my own list/framing. Somewhat rough.

I see various somewhat different ways in which interpretability can be useful for AI safety. These require different things from your interpretability in terms of how efficient it is, how much it lets you identify exactly what your model is thinking as opposed to broad properties of its cognition, and how reliable it needs to be.

Roughly in decreasing order of demandingness:

Microscope AI

Component of a full solution to alignment problem (ie as part of something like imitative generalization)

Knowing everything a model is thinking and fully auditing it to make sure it’s not doing anything sketchy

Relaxed adversarial training: identifying which part of the model corresponds to its ‘beliefs about its observations’ so that you can search over these

Identifying a ‘truthfulness direction’ in activation space or something similar

Having some rough understanding of what a model is thinking/what type of thinking it’s doing and thereby increasing the chance that you can spot it’s deceptive

Lobotomy: identifying which parts of a model do what sort of cognition, and extracting just the parts that are less likely to be doing something dangerous

Microscope AI

Instead of building and using an ML model, build the model and then use interpretability techniques to extract the knowledge it has learnt. Humans can then apply this knowledge directly rather than needing to actually deploy the model. See

Component of a ‘full solution to alignment problem’ (ie as part of something like imitative generalization)

By ‘a full solution to the alignment problem’ I’m thinking of a setup which would in theory let you know everything the model knows (in particular, if a model can use some knowledge to design a plan that leads to it getting power, you can use this knowledge to see that the plan will lead to the model getting more power)IDA/debate/RRM probably don’t give us this, partly because you can’t use them to learn things the model knows based on observing patterns in large datasets. For example, it would be hard to use them to supervise a model to translate correctly from a language it knows but you don’t.

Some possible solutions to this have something that vaguely looks like ‘have humans in the loop for learning patterns from large datasets’ - ie have humans (or more precisely a model trained to imitate humans) assess the prior probability of different claims (e.g. ‘this word means red’) and the probability of the data given the claim (e.g. ‘probability of this sentence under the hypothesis that this word means red’). This is imitative generalisation.In order to do this efficiently, you probably need to have some NN-like architecture representing the hypotheses (you can’t just have a giant string of text or something). In that case, what you’re doing is closely related to interpretability - you’re having a human engage with concepts represented in an NN and using them to make predictions about the world.

One example would be to jointly search over NN weights and labels on circuits such that the human has a high prior on those circuits representing a correct hypothesis, and that when the human assumes the labels are correct and uses the outputs of the circuits to predict the data, they predict it well.This definitely doesn’t work as stated, but maybe there’s something useful in this space.Related: Reverse-engineering using interpretability

Relaxed adversarial training: identifying which part of the model corresponds to its ‘beliefs about its observations’ so that you can search over these

The basic idea of relaxed adversarial training is something like:

A deceptive model may be waiting ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Go Republican, Young EA!, published by ColdButtonIssues on April 13, 2022 on The Effective Altruism Forum.

Cross-posted from Cold Button Issues

Young effective altruists in the United States interested in using public policy to make the world better should almost all be Republicans. They should not be Democrats, they should not be Greens, they should be Republicans.

First, because the Republican Party matters a lot. Second, there is way less competition among young people to climb within the Republican Party, at least within key fields.

There’s a pretty good chance that after 2024, we’ll see a Republican President, maybe a Trump Administration 2.0. Metacalus says it’s a tossup as of today. And Metacalus also thinks that Republicans will control Congress after 2022. There are already 23 states where Republicans control the legislature and the governorship (a trifecta). What the Republican party does is a huge deal!

The other reason is that it will make you quirky and unique. No, really. Young people are so left-wing these days. 58% of millennials voted for Biden over Trump. And the tendency is even more extreme for those with more degrees or coming from elite universities. So young Democrats seeking internships on campaigns or in DC face intense competition for even entry-level roles. Young Republicans have a much more favorable labor market. Anecdotally, in the last week an acquaintance of mine with a CV full of liberal activism was even offered a job at conservative NGO, where she had been striking out with liberal organizations. I think this is probably because conservatives have to compete for young talent, whereas young liberal talent has to compete for positions.

Right-wing groups put a fair amount of effort into trying to recruit young activists. You can join the College Republicans of course. But there are also paid fellowships offered by right-wing groups hoping to recruit Gen Z supporters such as the Hudson Institute. There are easy on-ramps to conservative politics to the United States, and given the scarcity of young conservatives it will be much easier for you to stand out and move up.

The place where this advantage is the most obvious is probably the legal field. About 70% of lawyers who involve themselves in politics via political donations lean Democratic. Yale Law School is so left-leaning that events featuring dissenting speakers are mobbed by protesters. So being a conservative law student or lawyer is probably unpleasant. But if you want to be a judge, being a conservative in the legal field is great! Probably around half of all future federal judges will be Republican-appointees, and there are many states and counties across the country where being a Republican will help a lawyer become a judge.

But what if you think the Republican Party is just a million times worse than the Democratic Party? Personally, I don’t think that’s true. And on a lot of issues effective altruists care about (artificial intelligence, the long-run future), there aren’t clear partisan lines and presidential candidates certainly aren’t using these issues as platform planks. Sure, one party as it currently stands is almost certainly better on average, but either party could shift on many issues. We don’t what the Republican Party will think about artificial intelligence in two years! But lots of important issues are so low-profile that if you rise in either party and land in the right spot in the federal bureaucracy, you could be making important decisions regardless of what your party thinks about abortion or healthcare.

But for those effective altruists who are pretty sure the Democratic Party is way better, perhaps those championing open borders or stricter animal welfare laws, or those who think international cooperation via the United Nations is really amazing, I would ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CEA is discontinuing its focus university programming, passing funding to Open Philanthropy, published by JoanGass on April 13, 2022 on The Effective Altruism Forum.

I’m writing this in my official capacity as the Head of the CEA Groups Team. Thanks to Asya Bergal, Claire Zabel, Aaron Gertler, Max Dalton, Alex Holness Tofts, Will Payne, Huw Thomas, Julia Wise, Will Fenning, and Ollie Base for their input on this draft. All opinions expressed are my own.

CEA is discontinuing its focus university programming – this includes its Campus Specialist program and Campus Specialist Internship program (formerly known as the community-building grants program for focus universities).

CEA will be redirecting university organizers who would like to be funded for their work to Open Philanthropy’s two new fellowships, which were launched in part to replace the funding aspects of the Campus Specialist and Campus Specialist Internship programs:

The University Organizer Fellowship, which provides funding for organizers and groups expenses for part-time and full-time organizers helping with student groups focused on effective altruism, longtermism, rationality, or other relevant topics at any university (not just focus universities). (Intended to replace the Campus Specialist and Campus Specialist Internship programs.)

The Century Fellowship, a selective 2-year program that gives resources and support (including $100K+/year in funding) to particularly promising people early in their careers who want to work in areas that could improve the long-term future. (Intended partially for particularly strong Campus Specialist applicants.)

As before, all group organizers can also apply for group expenses from CEA’s Group Support Funding, and organizer funding and groups expenses from EA Infrastructure Fund. A detailed breakdown of potential funding options for organizers can be found here.

CEA will continue to support early stage university groups via the University Group Accelerator Program and city and national groups in key locations via the Community Building Grants Program.

The following post covers two sections:

Why we made this shift

Lessons learned

I want to emphasise that this is a tactical shift about where funding for EA community building is coming from – not a signal about impact. CEA, Open Phil, and EAIF are just as excited about field-building work at focus universities as we were before – and we all still think advanced-stage university field building can be a very high impact opportunity.

So why is CEA making this change?

Ultimately, this was a decision related to organisational focus. I think that the CEA groups team will be more successful if we’re more focused. Over the last year we tried to cover three massive areas: focus university groups, early-stage university groups, and key city and national groups. It was challenging to provide excellent support to group leaders, who had different needs, in all of these areas simultaneously. We think we were more successful in early stage university groups and key city and national groups than focus university groups, who reported getting more value out of support and funding provided by other organisations. We want to encourage others who may be better-placed to provide focus university support to do so, and we worry that we may implicitly discourage this if it seems like we “own” the entire groups space

When choosing where to focus across early-stage university groups, late stage university groups, and city/national groups, we looked at our track record and the number of other organisations working in a given space.

Track record:

Over the last 6 months, we think we’ve done a strong job in our early-stage university work (supporting 32 groups via our University Group Accelerator Program with a stellar user rating of 9.1/10) and better...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tips for conducting worldview investigations, published by lukeprog on April 12, 2022 on The Effective Altruism Forum.

I think AI x-risk reduction is largely bottlenecked on a lack of strategic clarity,[1] more than e.g. bio x-risk reduction is, such that it's very hard to tell which intermediate goals we could aim for that would reduce rather than increase AI x-risk. Even just among (say) EA-motivated longtermists focused on AI x-risk reduction, there is a very wide spread of views on on AI timelines, takeoff speeds, alignment difficulty, theories of victory, sources of doom, and basically everything else (example), and I think this reflects how genuinely confusing the strategic situation is.

One important way[2] to improve strategic clarity is via what we've come to call AI "worldview investigations,"[3] i.e. reports that:

Aim to provide an initial tentative all-things-considered best-guess quantitative[4] answer to a big, important, fundamental ("worldview") AI strategy question such as "When will we likely build TAI/AGI?" or "How hard should we expect AI alignment to be?" or "What will AI takeoff look like?" or "Which AI theories of victory are most desirable and feasible?" Or, focus on a consideration that should provide a noticeable update on some such question.

Are relatively thorough in how they approach the question.

Past example "worldview investigations" include:

Open Phil's series on "When will we likely build TAI?", summarized here, and comprised mainly by Bio Anchors, Brain Computation, Semi-Informative Priors, Explosive Growth, and Human Trajectory. (~1185pp[5])

Carlsmith's Is power-seeking AI an existential risk? (~90pp)

My 2017 Report on Consciousness and Moral Patienthood[6] (~485pp) Worldview investigations are difficult to do well: they tend to be large in scope, open-ended, "wicked," and require modeling and drawing conclusions about many different phenomena with very little hard evidence to work with.

Here is some advice that may help researchers to succeed at completing a worldview investigation:

Be bold! Attack the big important action-relevant question directly, and try to come to a bottom-line quantitative answer, even though it's unjustified in many ways and will be revised later.

Reach out to others early for advice and feedback, especially people who have succeeded at this kind of work before. Share early, scrappy drafts for feedback on substance and direction.[7]

On the research process itself, see Minimal-trust investigations, Learning by writing, The wicked problem experience, and Useful Vices for Wicked Problems.

Reasoning transparency has advice for how to communicate what you know, with what confidence, and how you know it, despite the fact that for many sub-questions you won't have enough time or evidence to come to a well-justified conclusion.

How to Measure Anything is full of strategies for quantifying very uncertain quantities. (Summary here.)

Superforecasting has good advice about how to quantify your expectations about the future. (Summary here.)

Finally, here are some key traits of people who might succeed at this work:[8]

Ability to carve up poorly-scoped big-picture questions into the most important parts, operationalize concepts that seem predictive, and turn a fuzzy mess of lots of factors into a series of structured arguments connected to (usually necessarily weak / of limited relevance) evidence.

At least moderate quantitative/technical chops, enough to relatively quickly learn topics like the basics of machine learning scaling laws or endogenous growth theory, while of course still significantly relying on conversations with subject matter experts.

Ability to work quickly and iteratively, limiting their time on polish/completeness and on "rabbit holes" that could be better-explored or better-argued or more evidence-ba...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Useful Vices for Wicked Problems, published by Holden Karnofsky on April 12, 2022 on The Effective Altruism Forum.

Cross-posted from Cold Takes.

I've claimed that the best way to learn is by writing about important topics. (Examples I've worked on include: which charity to donate to, whether life has gotten better over time, whether civilization is declining, whether AI could make this the most important century of all time for humanity.)

But I've also said this can be "hard, taxing, exhausting and a bit of a mental health gauntlet," because:

When trying to write about these sorts of topics, I often find myself needing to constantly revise my goals, and there's no clear way to know whether I'm making progress. That is: trying to write about a topic that I'm learning about is generally a wicked problem.

I constantly find myself in situations like "I was trying to write up why I think X, but I realized that X isn't quite right, and now I don't know what to write." and "I either have to write something obvious and useless or look into a million more things to write something interesting." and "I'm a week past my self-imposed deadline, and it feels like I have a week to go, but maybe it's actually 12 weeks - that's what happened last time."

Overall, this is the kind of work where I can't seem to tell how progress is going, or stay on a schedule.

This post goes through some tips I've collected over the years for dealing with these sorts of challenges - both working on them myself, and working with teammates and seeing what works for them.

A lot of what matters for doing this sort of work is coming at it with open-mindedness, self-criticality, attention to detail, and other virtues. But a running theme of this work is that it can be deadly to approach with too much virtue: holding oneself to self-imposed deadlines, trying for too much rigor on every subtopic, and otherwise trying to do "Do everything right, as planned and on time" can drive a person nuts. So this post is focused on a less obvious aspect of what helps with wicked problems, which is useful vices - antidotes to the kind of thoroughness and conscientiousness that lead to unreachable standards, and make wicked problems impossible.

I've organized my tips under the following vices, borrowing from Larry Wall and extending his framework a bit:

Laziness. When some key question is hard to resolve, often the best move is to just ... not resolve it, and change the thesis of your writeup instead (and change how rigorous you're trying to make it). For example, switching from "These are the best charities" to "These are the charities that are best by the following imperfect criteria."

Impatience. One of the most crucial tools for this sort of work is interrupting oneself. I could be reading through study after study on some charitable activity (like building wells), when stepping back to ask "Wait, why does this matter for the larger goal again?" could be what I most need to do.

Hubris. Whatever I was originally arguing ("Charity X is the best"), I'm probably going to realize at some point that I can't actually defend it. This can be demoralizing, even crisis-inducing. I recommend trying to build an unshakable conviction that one has something useful to say, even when one has completely lost track of what that something might be.

Self-preservation. When you're falling behind, it can be tempting to make a "heroic" effort at superhuman productivity. When a problem seems impossible, it can be tempting to fix your steely gaze on it and DO IT ANYWAY. I recommend the opposite: instead of rising to the challenge, shrink from it and fight another day (when you'll solve some problem other than the one you thought you were going for).

Overall, it's tempting to try to "boil the ocean" and thoroughly examine every aspect of a topic o...