Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflecting on Steering, Pitfalls, and History, published by Davidmanheim on November 21, 2021 on The Effective Altruism Forum.

In light of Holden's excellent discussion of Rowing, Steering, Anchoring, Equity, Mutiny, I've been thinking about steering and its problems, and think there's something missing in his analysis. I haven't figure out how to plug the hole, but I have some initial thoughts.

Epistemic status: Preliminary thoughts and tentative conclusions, with some concrete examples which I hope are illustrative of something general, but may be post-hoc justifications to support my intuitive arguments.

Introduction

Holden proposes that steering, which he says is, in the analogy, trying to "navigate to a better destination than the current one." The idea is, then, to "anticipate future states of the world (climate change, transformative AI, utopia, dystopia) and act accordingly." He then notes that the historical track record of this type of work is not particularly great. Unfortunately, I think that this tries to combine two different things, in ways that are misleading, and uses far too narrow a historical lens. I'll start by addressing the elephant which Holden correctly located as crowding the room, Communism, then try to look elsewhere for examples.

Communism as Steering

First, true communism has never been tried. That's because true communism isn't even a single platonic idea, so there are mottes all over the place that allow for ambiguity and prevarication. But it's still useful to look at where the ideas have gone wrong.

The origins trace back well before Thomas More's Utopia, written in 1516, but that's a good place to start, with a vision of a society with communal ownership, simple needs, and rejection of competition. Of course, the idea of an idea state wasn't a blueprint or charting a course - but it clearly identified some problems with European society, and then proposed fanciful and impractical solutions.

In response to a varity of related ideas and recognition of problems, there were attempt to create such communal societies which clearly predate Marx, for example, in places like New Lanark - where Robert Owen, with support of philanthropists and others, including Bentham, built a Owenist commune around the time Marx was born. The small-scale communes, both predating Marx, during his lifetime, and over the following century, seem to have been anywhere from fairly successful to unspectacular failures.

But communism wasn't really compete until Marx. Much of what Marx added was a specific concern for what happens without radical reform, an economic theory, and a theory of history. I won't address the problems with the economic theory, but the specific concern - that capitalism would run roughshod over the workers and lead to poverty for all but the richest - wasn't actually unreasonable. In fact, without the labor movement in the US, and similar movements abroad, which developed largely prompted by socialism and communism, the trajectory of the working poor was quite grim. It's plausible that without a communist movement, Marx's vision of a capitalist dystopia wouldn't have been particularly wrong - and his assumption that things would inevitably become more socialist was mostly wrong in thinking that revolutions would necessarily be violent.

The problems with communism as attempted aroudn the world are manifold, but mostly seem to go alongside the planning and active steering. The types of abuse that communist states embraced were largely driven by certainty about their vision, and the inevitable human tendency to build "ideal" systems that in practice are self-serving rather than beneficial to others.

I think this is a trend - identifying problems is far easier than solving them, and grand schemes for solutions, is a typical red flag. It describes...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Vitalik: Cryptoeconomics and X-Risk Researchers Should Listen to Each Other More, published by EmersonSpartz on November 21, 2021 on LessWrong.

5 years ago, Ethereum co-founder Vitalik Buterin wrote an important but little-known blog post that deserves more visibility, so I'm posting it here. This way, it'll also cross-post to The Nonlinear Library, where it'll be available in podcast form.Inspired by Vitalik, I just set up an X-risk + Crypto Discord - let's cross-pollinate together!

Also inspired by Vitalik, I'm putting out a $100 bounty for any post on the EA Forum/Alignment Forum/LW that cross-applies ideas from crypto/web3 to x-risk that gets at least 25 upvotes, subject to my discretion. Alternatively, you can email me at [email protected] and if your ideas are interesting enough I'll still pay out the bounty.

Semi-unrelated, but if you're interested in this kind of stuff, Anna Riedl and I set up a Telegram for EAs into Complexity and Systems Science. We think this is another valuable interdisciplinary combination.

---Why Cryptoeconomics and X-Risk Researchers Should Listen to Each Other More

Vitalik Buterin

Jul 4, 2016·6 min read

Special thanks to Jaan Tallinn for early feedback and comments.

There has recently been a small but growing number of signs of interest in blockchains and cryptoeconomic systems from a community that has traditionally associated itself with artificial intelligence and various forms of futuristic existential risk research. Ralph Merkle, inventor of the now famous cryptographic technology which underpins Ethereum’s light client protocol, has expressed interest in DAO governance. Skype co-founder Jaan Tallinn proposed researching blockchain technology as a way to create mechanisms to solve global coordination problems. Prediction market advocates, who have long understood the potential of prediction markets as governance mechanisms, are now looking at Augur. Is there anything interesting here? Is this simply a situation of computer geeks who were previously attracted to computer-geek-friendly topic A now also being attracted to a completely unrelated but also computer-geek-friendly topic B, or is there an actual connection?

I would argue that there is, and the connection is as follows. Both the cryptoeconomics research community and the AI safety/new cyber-governance/existential risk community are trying to tackle what is fundamentally the same problem: how can we regulate a very complex and very smart system with unpredictable emergent properties using a very simple and dumb system whose properties once created are inflexible?

In the context of AI research, a major sub-problem is that of defining a utility function that would guide the behavior of a superintelligent agent without accidentally guiding it into doing something that satisfies the function as written but does not satisfy the intent (sometimes called “edge instantiation”). For example, if you tried to tell a super-intelligent AI to cure cancer, it may end up reasoning that the most reliable way to do that is to simply kill everyone first. If you tried to plug that hole, it may decide to simply permanently cryogenically freeze all humans without killing them. And so forth. In the context of Ralph Merkle’s DAO democracy, the problem is that of determining an objective function that is correlated with social and technological progress and generally things that people want, is anti-correlated with existential risks, and is easily measurable enough that its measurement would not itself become a source of political battles.

Meanwhile, in the context of cryptoeconomics, the problems are surprisingly similar. The core problem of consensus asks how to incentivize validators to continue supporting and growing a coherent history using a simple algorithm that is set in stone, wh...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Vitalik: Cryptoeconomics and X-Risk Researchers Should Listen to Each Other More, published by EmersonSpartz on November 21, 2021 on LessWrong.

5 years ago, Ethereum co-founder Vitalik Buterin wrote an important but little-known blog post that deserves more visibility, so I'm posting it here. This way, it'll also cross-post to The Nonlinear Library, where it'll be available in podcast form.Inspired by Vitalik, I just set up an X-risk + Crypto Discord - let's cross-pollinate together!

Also inspired by Vitalik, I'm putting out a $100 bounty for any post on the EA Forum/Alignment Forum/LW that cross-applies ideas from crypto/web3 to x-risk that gets at least 25 upvotes, subject to my discretion. Alternatively, you can email me at [email protected] and if your ideas are interesting enough I'll still pay out the bounty.

Semi-unrelated, but if you're interested in this kind of stuff, Anna Riedl and I set up a Telegram for EAs into Complexity and Systems Science. We think this is another valuable interdisciplinary combination.

---Why Cryptoeconomics and X-Risk Researchers Should Listen to Each Other More

Vitalik Buterin

Jul 4, 2016·6 min read

Special thanks to Jaan Tallinn for early feedback and comments.

There has recently been a small but growing number of signs of interest in blockchains and cryptoeconomic systems from a community that has traditionally associated itself with artificial intelligence and various forms of futuristic existential risk research. Ralph Merkle, inventor of the now famous cryptographic technology which underpins Ethereum’s light client protocol, has expressed interest in DAO governance. Skype co-founder Jaan Tallinn proposed researching blockchain technology as a way to create mechanisms to solve global coordination problems. Prediction market advocates, who have long understood the potential of prediction markets as governance mechanisms, are now looking at Augur. Is there anything interesting here? Is this simply a situation of computer geeks who were previously attracted to computer-geek-friendly topic A now also being attracted to a completely unrelated but also computer-geek-friendly topic B, or is there an actual connection?

I would argue that there is, and the connection is as follows. Both the cryptoeconomics research community and the AI safety/new cyber-governance/existential risk community are trying to tackle what is fundamentally the same problem: how can we regulate a very complex and very smart system with unpredictable emergent properties using a very simple and dumb system whose properties once created are inflexible?

In the context of AI research, a major sub-problem is that of defining a utility function that would guide the behavior of a superintelligent agent without accidentally guiding it into doing something that satisfies the function as written but does not satisfy the intent (sometimes called “edge instantiation”). For example, if you tried to tell a super-intelligent AI to cure cancer, it may end up reasoning that the most reliable way to do that is to simply kill everyone first. If you tried to plug that hole, it may decide to simply permanently cryogenically freeze all humans without killing them. And so forth. In the context of Ralph Merkle’s DAO democracy, the problem is that of determining an objective function that is correlated with social and technological progress and generally things that people want, is anti-correlated with existential risks, and is easily measurable enough that its measurement would not itself become a source of political battles.

Meanwhile, in the context of cryptoeconomics, the problems are surprisingly similar. The core problem of consensus asks how to incentivize validators to continue supporting and growing a coherent history using a simple algorithm that is set in stone, wh...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Chris Voss negotiation MasterClass: review, published by VipulNaik on November 20, 2021 on LessWrong.

This post is about the negotiation MasterClass taught by Chris. Voss and more broadly about the techniques and worldviews shared in that. MasterClass. I watched this MasterClass in April. 2020.

My post discusses some of the relationship -- similarities and differences -- with rationalist advice. These portions should be of particular relevance to LessWrong and are part of the reason for posting to LessWrong.

This is a fairly lengthy post, so please use the sections to navigate to the portions you are most interested in.

I cover a lot in this review:

What is this a review of?

Why am I doing this review?

Worldview

Negotiation principles

Negotiation techniques

General concerns

What is this a review of?

Who is Chris Voss?

Christopher Voss worked at the FBI as a lead hostage negotiator. In 2008 he left to form his own company, The Black Swan Group, that offers coaching to people on negotiation techniques.

What is MasterClass?

MasterClass is a player in the growing edutainmment space, combining education and entertainment. There are 100+ "classes" in MasterClass. Each class is taught by a subject matter expert (the teacher), and includes several videos (ranging from 2 to 30 minutes, usually 5 to 20 minutes). The typical video format just includes the teacher speaking to the camera, but there are occasionally interactive sessions with other participants, or other co-instructors.

Course notes can be downloaded. There are also community features on. MasterClass.

You may have seen video ads for MasterClass on YouTube!

What does this cost?

A MasterClass annual subscription all-access pass costs $180. A single. MasterClass (such as Chris Voss's) can be purchased for $90.

How long is the Chris Voss MasterClass?

There are 18 videos and the total video time is 3 hours 4 minutes.

What aspects of the MasterClass does my review cover?

The MasterClass includes three live exercises, that I do not discuss here (since they are more for demonstration purposes of a range of techniques). I also do not discuss the historical hostage negotiation examples that Chris Voss discusses (a bank robbery, the Jill Carroll kidnapping case, and the. Dwight Watson (tobacco farmer) case). I also skip some other material, including discussion of bargaining (using the Ackerman model), that is outside of the main negotiation techniques.

The class guide is 14 pages long, including a cover page (this blog post overlaps quite a bit with the class guide).

Are there other ways of watching the MasterClass without paying the subscription fee?

You can sign up and cancel within 30 days to get a full refund. This gives you enough time to consume the Chris Voss MasterClass.

You may be able to sign up for a free week; however, I'm not sure what subset of the videos in the MasterClass are available. More details here.

Are there other ways of accessing the information without watching the MasterClass?

Many of the techniques discussed in the MasterClass are discussed online, including in videos from Chris Voss and others. I have more links for each of the individual techniques, so you can use my post to basically get a "free" version of the MasterClass. You can also Google around for more videos and written materials on the techniques.

Voss has also co-authored a book Never Split The Difference that covers these techniques. It's available on. Amazon; at $11.99 for the Kindle edition it is considerably cheaper than the. MasterClass.

Masterwiki.how can be a helpful free version of MasterClass, but it does not seem to have Chris Voss's. MasterClass.

What's the value-add of the MasterClass beyond the free material in and linked from this blog post?

My blog post didn't exist at the time I bought the MasterClass; I think in the a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Chris Voss negotiation MasterClass: review, published by VipulNaik on November 20, 2021 on LessWrong.

This post is about the negotiation MasterClass taught by Chris. Voss and more broadly about the techniques and worldviews shared in that. MasterClass. I watched this MasterClass in April. 2020.

My post discusses some of the relationship -- similarities and differences -- with rationalist advice. These portions should be of particular relevance to LessWrong and are part of the reason for posting to LessWrong.

This is a fairly lengthy post, so please use the sections to navigate to the portions you are most interested in.

I cover a lot in this review:

What is this a review of?

Why am I doing this review?

Worldview

Negotiation principles

Negotiation techniques

General concerns

What is this a review of?

Who is Chris Voss?

Christopher Voss worked at the FBI as a lead hostage negotiator. In 2008 he left to form his own company, The Black Swan Group, that offers coaching to people on negotiation techniques.

What is MasterClass?

MasterClass is a player in the growing edutainmment space, combining education and entertainment. There are 100+ "classes" in MasterClass. Each class is taught by a subject matter expert (the teacher), and includes several videos (ranging from 2 to 30 minutes, usually 5 to 20 minutes). The typical video format just includes the teacher speaking to the camera, but there are occasionally interactive sessions with other participants, or other co-instructors.

Course notes can be downloaded. There are also community features on. MasterClass.

You may have seen video ads for MasterClass on YouTube!

What does this cost?

A MasterClass annual subscription all-access pass costs $180. A single. MasterClass (such as Chris Voss's) can be purchased for $90.

How long is the Chris Voss MasterClass?

There are 18 videos and the total video time is 3 hours 4 minutes.

What aspects of the MasterClass does my review cover?

The MasterClass includes three live exercises, that I do not discuss here (since they are more for demonstration purposes of a range of techniques). I also do not discuss the historical hostage negotiation examples that Chris Voss discusses (a bank robbery, the Jill Carroll kidnapping case, and the. Dwight Watson (tobacco farmer) case). I also skip some other material, including discussion of bargaining (using the Ackerman model), that is outside of the main negotiation techniques.

The class guide is 14 pages long, including a cover page (this blog post overlaps quite a bit with the class guide).

Are there other ways of watching the MasterClass without paying the subscription fee?

You can sign up and cancel within 30 days to get a full refund. This gives you enough time to consume the Chris Voss MasterClass.

You may be able to sign up for a free week; however, I'm not sure what subset of the videos in the MasterClass are available. More details here.

Are there other ways of accessing the information without watching the MasterClass?

Many of the techniques discussed in the MasterClass are discussed online, including in videos from Chris Voss and others. I have more links for each of the individual techniques, so you can use my post to basically get a "free" version of the MasterClass. You can also Google around for more videos and written materials on the techniques.

Voss has also co-authored a book Never Split The Difference that covers these techniques. It's available on. Amazon; at $11.99 for the Kindle edition it is considerably cheaper than the. MasterClass.

Masterwiki.how can be a helpful free version of MasterClass, but it does not seem to have Chris Voss's. MasterClass.

What's the value-add of the MasterClass beyond the free material in and linked from this blog post?

My blog post didn't exist at the time I bought the MasterClass; I think in the a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: From language to ethics by automated reasoning, published by Michele Campolo on November 21, 2021 on The AI Alignment Forum.

Posted also on the EA Forum.

Introduction

In Naturalism and AI alignment I claimed that we might get an aligned AI by building an agent which derives its own preferences (a partial order over world-histories) from its knowledge of the natural world.

Now I am working on a formal (i.e. mathematical or algorithmic) model of that agent. When AI capabilities will reach human level in natural language understanding, the formal model will help turn a natural language AI system that understands the world at human level of intelligence into an AI that is able to reason about ethics and possibly do EA-style research.

Here, I elaborate on the idea that an AI which reasons like a human might be aligned. The basic argument is that:

Some humans act and think ethically

If we understand the causes of such behaviour and replicate them in AI, we get AI that is roughly aligned. If we also manage to eliminate some human cognitive biases, we should get AI that is more robustly aligned.

Factors underlying altruistic behaviour in humans

At least some humans act and think in ethical or altruistic terms. To explain why this happens, we can consider the different factors that may lead to such behaviour.

Positive and negative first-person experiences: if humans couldn’t perceive any kind of pleasure or pain, survival in general would be significantly harder, let alone making considerations about others’ subjective experiences.

Theory of mind: without it, humans probably could not reason about what is good or bad for others.

Empathy: in addition to theory of mind, humans can also experience a positive or negative response depending on what somebody else is feeling. Empathy incentives harm reduction and makes positive emotions spread.

Social and cultural drives: culture influences what actions are perceived as good or bad; and humans, as social animals, tend to adjust their behaviour accordingly, in order to be seen as good persons by the people around them.

Game-theoretical reasoning: one can end up acting altruistically and cooperatively simply because doing so provides the greatest benefit to themself.

Moral reasoning: some humans classify actions as good or bad according to a system of beliefs held to be true. Two examples: religious people often rely on a set of dogmas that guide their decisions during life; some philosophers wonder “Given what I know about the world, what matters most?”, and some of them act according to the answer they find.

If we were able to replicate all these factors in artificial minds, at least some of these minds would end up thinking ethically. However, we don’t yet know how to build an AI that possesses all the factors in the list. It would be nice if we could ignore some of the factors above and still get an aligned AI.

The last factor, moral reasoning, deserves special attention, since it could be less or particularly important depending on one’s view about morality.

Moral reasoning and subjective vs objective morality

As trivial as it sounds, the more subjective morality is, the less objectively true moral knowledge and reasoning are. If what makes moral statements true are opinions, feelings or attitudes of people, then our philosophical conclusions about good and bad are what they are simply because we are human. We should expect other rational minds to reach different conclusions regarding morality if they have different feelings, or cultural backgrounds, etc.

Thus: first-person experiences, empathy and theory of mind, social and cultural drives are the underlying cause of moral reasoning, which is not grounded in truth. Objective moral knowledge and objective moral progress are an illusion.

On the other hand, if morality is more obje...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: From language to ethics by automated reasoning, published by Michele Campolo on November 21, 2021 on The AI Alignment Forum.

Posted also on the EA Forum.

Introduction

In Naturalism and AI alignment I claimed that we might get an aligned AI by building an agent which derives its own preferences (a partial order over world-histories) from its knowledge of the natural world.

Now I am working on a formal (i.e. mathematical or algorithmic) model of that agent. When AI capabilities will reach human level in natural language understanding, the formal model will help turn a natural language AI system that understands the world at human level of intelligence into an AI that is able to reason about ethics and possibly do EA-style research.

Here, I elaborate on the idea that an AI which reasons like a human might be aligned. The basic argument is that:

Some humans act and think ethically

If we understand the causes of such behaviour and replicate them in AI, we get AI that is roughly aligned. If we also manage to eliminate some human cognitive biases, we should get AI that is more robustly aligned.

Factors underlying altruistic behaviour in humans

At least some humans act and think in ethical or altruistic terms. To explain why this happens, we can consider the different factors that may lead to such behaviour.

Positive and negative first-person experiences: if humans couldn’t perceive any kind of pleasure or pain, survival in general would be significantly harder, let alone making considerations about others’ subjective experiences.

Theory of mind: without it, humans probably could not reason about what is good or bad for others.

Empathy: in addition to theory of mind, humans can also experience a positive or negative response depending on what somebody else is feeling. Empathy incentives harm reduction and makes positive emotions spread.

Social and cultural drives: culture influences what actions are perceived as good or bad; and humans, as social animals, tend to adjust their behaviour accordingly, in order to be seen as good persons by the people around them.

Game-theoretical reasoning: one can end up acting altruistically and cooperatively simply because doing so provides the greatest benefit to themself.

Moral reasoning: some humans classify actions as good or bad according to a system of beliefs held to be true. Two examples: religious people often rely on a set of dogmas that guide their decisions during life; some philosophers wonder “Given what I know about the world, what matters most?”, and some of them act according to the answer they find.

If we were able to replicate all these factors in artificial minds, at least some of these minds would end up thinking ethically. However, we don’t yet know how to build an AI that possesses all the factors in the list. It would be nice if we could ignore some of the factors above and still get an aligned AI.

The last factor, moral reasoning, deserves special attention, since it could be less or particularly important depending on one’s view about morality.

Moral reasoning and subjective vs objective morality

As trivial as it sounds, the more subjective morality is, the less objectively true moral knowledge and reasoning are. If what makes moral statements true are opinions, feelings or attitudes of people, then our philosophical conclusions about good and bad are what they are simply because we are human. We should expect other rational minds to reach different conclusions regarding morality if they have different feelings, or cultural backgrounds, etc.

Thus: first-person experiences, empathy and theory of mind, social and cultural drives are the underlying cause of moral reasoning, which is not grounded in truth. Objective moral knowledge and objective moral progress are an illusion.

On the other hand, if morality is more obje...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Split and Commit, published by Duncan Sabien on November 21, 2021 on LessWrong.

This is an essay describing a basic sanity-inducing mental movement that I use approximately ten times per week, and suspect other people would benefit from adopting (and regularly reminding each other to do). I've talked about it elsewhere, but until now it didn't have its own linkable reference post.

P1: Things are often not what they seem.

This can be because the seeming itself is broken, e.g. "Well, here I am in 1700's America and it sure seems to me that these black folk are fundamentally intellectually and morally and spiritually inferior to white folk" or "Well, here I am in 1800's America and it sure seems to me that these women are constitutionally incapable of holding political office" or "Well, here I am in 1900's America and it sure seems to me that these gay men are an active threat to everyone's safety, including our women (somehow) and children" or "Well, here I am in 2000's America and it sure seems to me that ███ ██ █████ ████████████ █ ███ ██ ██████ ██ ████ █████ ████████ ██ ████ ███ ████ ████ ██████ ███."

(Or "Well, here I am inventing physics and it sure seems to me that force consistently equals mass multiplied by acceleration.")

It can also be because the seeming is generally correct, but there exists variance. If 95% of the marbles in a bag are red, and 5% of the marbles in the bag are green, it's correct to guess that the next randomly selected marble will be red, because betting on red makes you the least likely to be wrong. But that doesn't mean that it can't be green, or that you should be shocked if it's green. If you bet on red the whole way through the bag, you know you'll be wrong five percent of the time.

P2: Every observation is evidence in favor of more than one hypothesis.

If my friend tells me my hair looks good, this is an update in favor of my hair looking good and an update in favor of my hair not looking good but my friend lying to me for various reasons (kindness, prank). In order to know which update is larger, you need some other information about what kind of person my friend is, and what our relationship is like.

If I see some breaking news about some person having maybe done something horrible, this is an update in favor of that person having done something horrible, and an update in favor of that person being the target of a smear campaign/conspiracy, and an update in favor of the presented claim being technically correct but there being a ton of other relevant context and facts that will substantially change the gestalt of the story. In order to know which update is larger, you need some other information, etc.

(Many of my readers will be thinking "this is just Bayes," and yep, it's just Bayes.)

The upshot of these two premises is that it's wise to have at least two active hypotheses running at all times, for any question of import.

By default, it seems to me that most humans have only one, i.e. not a hypothesis at all but a singular belief.

This is true (in my experience) even of the type of person who is acutely aware of their own fallibility, and who acknowledges that they are occasionally (or often) mistaken. Even people who are savvy enough to try to put a number (like "85% confident") on it, such that they can track their calibration over time, tend to leave the remaining swath of possibility un- or under-specified.

They will admit that they might be wrong in some vague fashion about [COVID, Trump, existential risk from artificial intelligence, Kyle Rittenhouse, universal basic income, Leverage Research, what the person who wrote that comment was obviously really saying, never mind their actual words], etc., but that doesn't stop them from having only a single real guess in many situations where that seems (to me) to be wildly prematur...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Split and Commit, published by Duncan Sabien on November 21, 2021 on LessWrong.

This is an essay describing a basic sanity-inducing mental movement that I use approximately ten times per week, and suspect other people would benefit from adopting (and regularly reminding each other to do). I've talked about it elsewhere, but until now it didn't have its own linkable reference post.

P1: Things are often not what they seem.

This can be because the seeming itself is broken, e.g. "Well, here I am in 1700's America and it sure seems to me that these black folk are fundamentally intellectually and morally and spiritually inferior to white folk" or "Well, here I am in 1800's America and it sure seems to me that these women are constitutionally incapable of holding political office" or "Well, here I am in 1900's America and it sure seems to me that these gay men are an active threat to everyone's safety, including our women (somehow) and children" or "Well, here I am in 2000's America and it sure seems to me that ███ ██ █████ ████████████ █ ███ ██ ██████ ██ ████ █████ ████████ ██ ████ ███ ████ ████ ██████ ███."

(Or "Well, here I am inventing physics and it sure seems to me that force consistently equals mass multiplied by acceleration.")

It can also be because the seeming is generally correct, but there exists variance. If 95% of the marbles in a bag are red, and 5% of the marbles in the bag are green, it's correct to guess that the next randomly selected marble will be red, because betting on red makes you the least likely to be wrong. But that doesn't mean that it can't be green, or that you should be shocked if it's green. If you bet on red the whole way through the bag, you know you'll be wrong five percent of the time.

P2: Every observation is evidence in favor of more than one hypothesis.

If my friend tells me my hair looks good, this is an update in favor of my hair looking good and an update in favor of my hair not looking good but my friend lying to me for various reasons (kindness, prank). In order to know which update is larger, you need some other information about what kind of person my friend is, and what our relationship is like.

If I see some breaking news about some person having maybe done something horrible, this is an update in favor of that person having done something horrible, and an update in favor of that person being the target of a smear campaign/conspiracy, and an update in favor of the presented claim being technically correct but there being a ton of other relevant context and facts that will substantially change the gestalt of the story. In order to know which update is larger, you need some other information, etc.

(Many of my readers will be thinking "this is just Bayes," and yep, it's just Bayes.)

The upshot of these two premises is that it's wise to have at least two active hypotheses running at all times, for any question of import.

By default, it seems to me that most humans have only one, i.e. not a hypothesis at all but a singular belief.

This is true (in my experience) even of the type of person who is acutely aware of their own fallibility, and who acknowledges that they are occasionally (or often) mistaken. Even people who are savvy enough to try to put a number (like "85% confident") on it, such that they can track their calibration over time, tend to leave the remaining swath of possibility un- or under-specified.

They will admit that they might be wrong in some vague fashion about [COVID, Trump, existential risk from artificial intelligence, Kyle Rittenhouse, universal basic income, Leverage Research, what the person who wrote that comment was obviously really saying, never mind their actual words], etc., but that doesn't stop them from having only a single real guess in many situations where that seems (to me) to be wildly prematur...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Opportunity Costs of Technical Talent: Intuition and (Simple) Implications, published by Ozzie Gooen on November 19, 2021 on The Effective Altruism Forum.

TLDR

Strong technical talent has very high opportunity costs right now

Therefore, technical nonprofit projects need to do one of:

Create very large amounts of value per person

Find ways of usefully using people with low opportunity costs

These basic principles also apply to other areas

Meta

Epistemic Status. I've known this area for a while and am speaking from experience. This post is mainly to attempt to convey intuitions. If you want to know more about the specifics of (high) salaries in the tech sector, there's a lot of other materials out there.

Facebook Thread. An early version of this was written as a Facebook post. That post features some relevant comments.

Personal Take. I've been struggling to figure out how to best express these ideas. I think these metaphors will be useful to some, but get in the way for others. I think the ideas are useful. I wish the flow were better, but the opportunity cost of improving it further seemed too high.

Ambitious Software Engineering Efforts. This is on the same topic as the previous post, but the style is very different.

Updates. 6PM Nov 19: I changed the title from The Dreaded Beast Who Gobbles Up Technical Talent, after feeling sort of awkward about it and noticing the post seemed to be getting some downvotes.

The Horror

Software products take almost absurd amounts of labor and talent. The world has far fewer people capable of implementing these services than it does potentially useful services to be implemented. Market forces mean that the absolute most profitable sectors are eating up the big bulk of these capabilities, and everyone else gets the scraps.

Startup valuations are sky-high right now. Tech salaries are increasing. Recruiters are looming.

These bring a ton of pressure to non-elite employers.

It feels like a nefarious beast that eats up anyone who gets to be too good or makes loud noises.,_Ancient_Slime

Employers are like horror movie civilians. Hiding in corners. Trying to keep hires who won't attract the hoard. If anyone gets too experienced, they'll be noticed. Maybe if they provide some work-from-home or extra vocational benefits, they can mitigate damage this year.

It's like a chopping block that destroys (from the employer's view) anything that clears a nebulous bar.

Or a swarm of locusts that comes intermittently and eats up all of the promising opportunities and leaves dust in its path.

It's terrifying, and it's a constant worry.

Sometimes this effect is called Creative Destruction. [1] Creative Destruction sounds like accountant jargon for a minor inconvenience, not like a terror that devastates ambitious efforts.

Is Good Management a Solution?

One common criticism to the above complaints is something like,

"You just need to have great management. If you can inspire programmers, provide great working conditions, and find really ideal projects that provide solid returns.

I agree that if you have great management, you can greatly reduce the problems here.

SpaceX and Tesla are both interesting examples. I went to college in Engineering at Harvey Mudd (graduating in 2012), close to LA (roughly near both companies), and have friends who went to both companies. Pay was surprisingly low at both. Hours were often incredibly high, and the working environments seemed highly stressful. People seemed to work there due to a combination of company vision, ambition, professional status, and interest.

Few companies can figure out how to replicate these conditions. Elon Musk figured out how to, and now he's worth over $250 Billion.

So, if you are the next Elon Musk, you can probably make it work. You can outcompete the rest of the market at converting labor into ec...