Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Soft optimization makes the value target bigger, published by Jeremy Gillen on January 2, 2023 on LessWrong.

Produced As Part Of The SERI ML Alignment Theory Scholars Program 2.1, mentored by John Wentworth.

Summary

We can solve Goodhart's Curse by making the agent (1) know the reliability of its goal representation and (2) cap the amount of optimization power devoted to achieving its goals, based on this reliability measure. One simple way to formalize this is with a generalization bound on the quality of the value proxy and a quantilizer whose q value is chosen based on the generalization bound. For a competitive implementation of this algorithm, it's easy to create outer objectives that will push a learned planning algorithm toward soft optimization. I think this is a productive research avenue, and this approach should stack on top of value learning and interpretability approaches to reducing AGI risk.

Extremal Goodhart

Powerful artificial agents are dangerous if they do unbounded uninterruptible optimization. Unbounded optimization is dangerous because of Extremal Goodhart, which is demonstrated in the diagram below.

In this diagram, the potential policies taken by the agent are arranged in order of expected value on the x-axis. We can see that as we push further to the right (which is what an optimizer does), the proxy value becomes less correlated with the true value. If we completely trusted our distribution over possible utility functions, we would simply take the rightmost policy, since it has the highest expected value. But we can't trust this knowledge, because by choosing the rightmost policy we have applied optimization pressure to find the policy with the most upward error in its value estimate.

We can reduce this problem by randomizing our choice of policy. Randomizing the policy is equivalent to reducing the amount of optimization power. The problem is that randomizing inherently reduces the usefulness and capabilities of your agent.

This post is about an approach for setting the amount of optimization power such that the optimizer achieves as much expected value as possible, within the limits set by its knowledge about the utility function. In other words, it's dumber when in unfamiliar territory, i.e. out of distribution. If implemented well, I think this makes the alignment problem easier in two ways. One, it allows us to imperfectly specify the utility function, and two, it allows us to specify a single-task utility function and be uncertain about the utility of unrelated world states.

Setting q based on a generalization bound

First off, a q-quantilizer is an algorithm that samples randomly from the top q proportion of policies. A 40%-quantilizer looks like this:

Quantilizers are the optimal policy distribution if you have some cost function c:πR+, and have knowledge about the expected cost under the prior distribution γ(π) and want to upper bound the expected cost under the chosen policy distribution. Based on the required upper bound you can determine the q value. However, this isn't a good way to choose the q value. We don't know the cost function, and we don't know how much cost is too much.

Instead, it's possible to determine the optimal policy distribution if we have a representation of uncertainty about the true utility function and a generalization bound telling us how reliable that representation is. The optimal policy distribution will often be exactly the same as a quantilizer, but sometimes not exactly.

A generalization bound for a learning algorithm is simply a number that upper bounds the average on-distribution test loss. We can use a generalization bound to guarantee that a learning algorithm hasn't overfitted to the training data. We are going to assume we have a generalization bound on the quality of our proxy of the true utilit...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My first year in AI alignment, published by Alex Altair on January 2, 2023 on LessWrong.

2022 was a pretty surprising year for me. In the beginning, I had just finished going through the Recurse Center retreat, and was expecting to go back into software engineering. As it turned out, I spent almost all of it getting increasingly committed to working on technical AI alignment research full-time.

I think a lot of people are trying out this whole "independent alignment researcher" thing, or are considering it, and I think a lot of these people are encountering a lack of structure or information. So I figure it might be helpful if I just relay what my experience has been, to add to the available information. And of course, there's a ton of variance in people's experience, so this might be more of a data point about what the variance looks like than it is about what the average looks like.

Some personal background

I'm probably pretty unusual in that I've been involved in the rationalist community since 2010. I bought the arguments for AI x-risk virtually since I first read them in the sequences. I tried somewhat to do useful work back then, but I wasn't very productive.

I have a life-long attention problem, which has mostly prevented me from being intellectually productive. Because I couldn't really do research, I went into software engineering. (This was still a struggle but was sufficiently doable to be a stable living.) A lot of this year's transition was figuring out how to do alignment work despite the attention problem.

My technical background is that I majored in math and physics in college, took most of those required classes, and also dropped out (twice). I was well above average in school (but probably average for the rationalist community). I never really did math in the interim, but it always remained part of my identity, and I have always had a theory mindset.

Goals and plans

When I say I've been "working on AI alignment," what do I mean? What sort of goals or plans have I been following? This has evolved over the year as I got more information.

First, I went through AGI Safety Fundamentals course. In the beginning I was just happy to be catching up with how the field had developed. I was also curious to see what happened when I interacted with alignment content more intensely. It turned out that I enjoyed it a lot; I loved almost every reading, and dived way deeper into the optional content.

Next, I tried to see if I could figure out what optimization was. I choose this as my AGISF final project, which was obviously ambitious, but I wanted to see what would happen if I focused on this one research-flavored question for four whole weeks.

I made a lot of progress. I decided it was a good project to continue working on, and then I started alternating between writing up what I had done already, and doing more research. The content continued to evolve and grow.

After a few more months, it was clear to me that I could and wanted to do this longer-term. The optimization project was taking forever, but I had some inside and outside view evidence that I was spending my time reasonably. So while I continued spending some weeks focusing entirely on trying to get the optimization project closer to an end point, I also deliberately spent some weeks doing other tasks that a full-time researcher would do, like learning more about the field, others' agendas, the open problems, et cetera. I tried to improve my own models of the alignment problem, think about what the most critical components were, and what I could do to make progress on them, including what my next major project should be. I haven't made any decisions about this, but I do have a pretty solid shortlist.

For much of December, I decided to do more focused reflection on how the optimization project was going. It...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What are the most underrated posts & comments of 2022, according to you?, published by peterhartree on January 1, 2023 on The Effective Altruism Forum.

Share your views in the comments!

To make this clear and easy to follow, please use these guidelines:

Use the template below.

Post as many items as you want.

One item per comment, so that it's easy for people to read and react.

(Optional, encouraged) Highlight at least one of your own contributions.

If you need some inspiration, open your EA Forum Wrapped and scroll to the bottom of your "Strong Upvoted" list.

Template

Title:

Author:

URL:

Why it's good:

If you're sharing an underrated comment, set the title to "[Username] on [topic]".

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New book: The Tango of Ethics: Intuition, Rationality and the Prevention of Suffering, published by jonleighton on January 2, 2023 on The Effective Altruism Forum.

As mentioned in a recent post, I have a new book being published this week, titled The Tango of Ethics: Intuition, Rationality and the Prevention of Suffering. It’s rooted in reflections I’ve had on ethics and value since some of my earliest interactions with EAs ten years ago, and my observation that some specific ways of thinking about ethics that were already mainstream in the EA community could legitimately be challenged. These include common intuitions about notions like “good”, “bad” and “value” that have been imported into rational arguments about ethics, without necessarily being put into question or analysed more deeply. The project expanded into a broader reflection on ethics and the dance between intuition and rationality that I think is fundamental to ethical thinking and practice.

Some of the claims I make may appear counterintuitive or conflict with beliefs that are strongly held by many others in the EA community. However, I urge people to consider reading the book with an open mind. Over the years that it has taken shape, I’ve continued to reevaluate my arguments and I remain confident that they have merit. Many of the individual ideas aren’t novel, and are even subscribed to by a subset of EAs. But aside from offering some new perspectives, one of my main goals is to offer a more “holistic” way of thinking about ethics that integrates several core ideas, and that is aligned with solid truths about reality, including the content of subjective experience.

My hope is that the book will provoke reflection within the EA community about the foundations of our core values and how we think about “doing good”. Although I defend a form of negative utilitarianism I call “xNU+”, I show that it doesn’t need to lead to nihilism, especially within the framework I propose. It doesn’t negate self-preservation and the search for meaning, caring about the welfare of future sentient beings, or striving to realise an optimistic vision for the future. But I do argue for the importance of preventing intense and especially extreme/unbearable suffering as an essential ethical principle – and by extension, that only a future that encodes and reflects this principle is a reasonable one to try to preserve.

The book is available in paperback and e-book from Amazon (UK, US, Germany...), B&N, and also directly from the publisher in the UK (paperback only).

Description from the publisher’s page

Despite existing for thousands of years, the field of ethics remains strongly influenced by several largely unquestioned assumptions and cognitive biases that can dramatically affect our priorities. The Tango of Ethics: Intuition, Rationality and the Prevention of Suffering proposes a deep, rigorous reassessment of how we think about ethics. Eschewing the traditional language of morality, it places a central emphasis on phenomenological experience and the unique urgency of suffering wherever it occurs, challenges our existence bias and examines the consequences of a metaphysically accurate understanding of personal identity.

A key paradigm in The Tango of Ethics is the conflict and interplay between two fundamentally different ways of seeing and being in the world — that of the intuitive human being who wants to lead a meaningful life and thrive, and that of the detached, rational agent who wants to prevent unbearable suffering from occurring. Leighton aims to reconcile these two stances or motivations within a more holistic framework he labels 'xNU+' that places them at distinct ethical levels. This approach avoids some of the flaws of classical utilitarianism, including the notion that extreme suffering can be formally balanced ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Would it be good or bad for the US military to get involved in AI risk?, published by Grant Demaree on January 1, 2023 on LessWrong.

Meant as a neutral question. I'm not sure whether this would be good or bad on net:

Suppose key elements of the US military took x-risk from misaligned strong AI very seriously. Specifically, I mean:

Key scientists at the Defense Threat Reduction Agency. They have a giant budget (~$3B/year) and are literally responsible for x-risks. Current portfolio is focused on nuclear risks with some biosecurity

Influential policy folks at the Office of the Undersecretary of Defense for Policy. Think dignified career civil servants, 2 levels below the most senior political appointees

Commander's Initiative Group at USSTRATCOM. Folks who have the commander's ear, tend to be well-respected, and have a huge effect on which ideas are taken seriously

Why this would be good:

The military has far more discretionary budget than anyone else in the world. You could multiply the resources dedicated to AI safety research tenfold overnight

The military is a huge source of AI risk (in the sense that advancing AI capabilities faster obviously helps the US in competition with China). If key influencers took the risk seriously, they might be more judicious about their capabilities research

A key policy goal is preventing the sharing of AI capabilities research. The military is very good at keeping things secret and has policy levers to make private companies do the same

The military is a huge source of legitimacy with the general public. And it seems easier than other routes to legitimacy. I think less than 10 key people actually need to be persuaded on the merits, and everyone else will follow suit

If the right person agrees, it's literally possible to get one of the best researchers from this community appointed to lead AI safety research for a major government agency, in the same sense that Wernher von Braun led the space program. You just have to be really familiar with the civil service's intricate rules for hiring

Why this would be bad:

If someone presents the ideas badly, it's possible to poison the well for later. You could build permanent resistance in the civil service to AI safety ideas. And it's really easy to make that mistake: presentations that work in these agencies are VERY different from what works in the tech community

Even if the agency is persuaded, they could make a noisy and expensive but ultimately useless effort

A big government project (with lots of middle managers) adds a moral maze element to alignment research

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Thingness of Things, published by Tsvi Benson-Tilsen on January 1, 2023 on The AI Alignment Forum.

[Metadata: crossposted from. I'm fairly likely to not respond to comments promptly. If you're especially interested in chatting, my gmail address is: tsvibtcontact ]

What's a thing, in general? Minds deal with things, so this question comes up in trying to understand minds. Minds think about things, speak of things, manipulate things, care about things, create things, and maybe are made of things.

Things

Examples of things: table, elephant, carbon atom. France, Martin Luther King. Insertion sort. Chess. Unicorn. Learning. Seven. Towel, strand, wing, crystal, finger, space, diffraction. The laws of electromagnetism. The first World War.

Non-things

What about non-things? It might be hard to list non-things because what we have words for, tend to be things. Redness seems like sort of a thing, but less so. Laws of physics also. Also ghosts. One might say "the ideal gas law is totally a thing" or "ghosts aren't a thing", though I think ghosts are a thing. Events can be things; WWI seems like a thing to me. But a minute ago I picked up my bottle of water and drank from it; that's clearly an event, a real one, but it doesn't feel that much like a thing. The abstract [drinking from a water bottle] feels like a thing though.

(There are some usual critiques of thingness. Yes, there's no sharp dividing line between a wave and a trough, but clearly waves are a thing. Yes, seven is not a physical object you'll ever bump into, but it's clearly a thing. The ideal gas law isn't localized in space or time, but it's a thing. Unicorns aren't real, in that you'll bump into people speaking about them but you won't bump into unicorns themselves and won't be constrained by unicorns in the way you're constrained by seven, but they're things. The world of Ender's Game isn't real, though it's a remote possibility, and it's a thing.)

Features of things

Coherence. A thing usually has properties, parts, aspects that are coherent with each other. Some things are big, some things are small; but usually there's nothing like a table that's both big and small.

Prediction, homogeneity, constancy, constraint. A thing usually has some predictive meaning; its presence implies constraints on other things or the future, and it implies some constancy in some features.

Expression. Related to prediction, a name that names a thing or an idea that's about a thing contributes to expressing thoughts about situations involving the thing. Expressing thoughts about the thing contributes to further behavior such as successfully predicting or manipulating the thing.

Cluster. A thing that's an instance of a type of thing, has multiple features mostly shared by most things of that type and not mostly shared by most things not of that type.

Exterior, relations. A thing usually can "impinge on other things from the outside". "Reality is that which, when you stop believing in it, doesn't go away."

Indicatability. A thing can be indicated. A physical object can literally be pointed at, seven can be pointed at by showing how to count and showing sets of seven like things, and many things can be pointed at by saying the word for that thing.

Truth, solidity, fixed import. "True" comes from PIE deru- ("be firm, solid"), cognate with "trust" and "tree" and possibly "durable". Things have truth to them; a thing makes a promise of agreement between minds on questions about the thing upon further independent investigation. Beyond indicatability, the truth of a thing implies that the thing can be advantageously treated and used in a fixed way.

Hollowness, cavernousness. The more comprehensively the history, features, changes, components, causes, internal relations, overarching structures, and explanations of a thing are kept in min...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Investing for a World Transformed by AI, published by PeterMcCluskey on January 1, 2023 on LessWrong.

AI looks likely to cause major changes to society over the next decade.

Financial markets have mostly not reacted to this forecast yet. I expect it will be at least a few months, maybe even years, before markets have a large reaction to AI. I'd much rather buy too early than too late, so

I'm trying to reposition my investments this winter to prepare for AI.

This post will focus on scenarios where AI reaches roughly human levels sometime around 2030 to 2035, and has effects that are at most 10 times as dramatic as the industrial revolution. I'm not confident that such scenarios are realistic. I'm only saying that they're plausible enough to affect my investment strategies.

Companies working on AI

Google's DeepMind seems a bit more likely than any other company to produce valuable AI. That's a good enough reason to hold a modest position in Google. TPU and

Waymo are additional AI-related reasons for holding Google stock.

OpenAI is a serious contender. Microsoft has a $1 billion stake in

OpenAI. But profits from that seem to be capped at

100x, so the best case scenario seems to be that Microsoft shareholders get $13.40 per share from that stake. Maybe Microsoft will invest more in OpenAI. I only see Microsoft getting a modest boost from AI.

Conjecture sounds quite promising. I offered to invest $100k in them this summer. They don't seem interested in investments that small.

There are other new private companies that I haven't been able to evaluate. Here are some that seem worth paying some attention to:

Adept

Anthropic

Bubble City

Character

Cohere

Inflection

AI

Keen

Technologies

That's a dramatic pace of new startups being founded that look potentially important. VC's seem to have been throwing enough money at them in mid-2022 that they likely had little reason to talk to more ordinary investors such as myself. Investment in such startups has maybe cooled slightly now that FTX has stopped throwing money at them, but

I'm guessing it hasn't cooled much.

I see a 50% chance that one or more of these new startups will make

DeepMind and OpenAI irrelevant. I have little hope of being able to buy a diversified portfolio of such startups, so I don't expect to profit from direct investments in companies working on AI.

Computing Hardware Companies

The most promising investments involve the hardware needed to power AI. That mostly means semiconductors.

Semiconductor capital equipment companies seem more promising than companies that are more directly involved in making chips, as capital equipment is more cyclical, and less dependent on specific uses.

My current semiconductor-related investments are (in alphabetic order): AMKR, AOSL, ASML, ASYS, KLAC, MTRN, LSE:SMSN, TRT.

I'm likely to buy more sometime in 2023, but I'm being patient because we're likely at a poor part of an industry cycle.

Others that I'm considering buying: ACLS, AMAT, INTC, LRCX, MU, and

PLAB.

Some of you will be surprised that I didn't suggest NVDA. Its PE and price/sales ratio are high compared to my other semiconductor investments. I can imagine buying it if its stock price drops relative to other semiconductor stocks. But I suspect its fans underestimate the risk of competition from hardware that is optimized more specifically for deep learning.

I've invested in privately held Fathom

Radiant, which seems to have a shot at

NVIDIA-like success in AI-related hardware. I'm unsure whether they'll want any more investment in 2023.

I've seen some noise about neuromorphic computing. I suspect that such research is far enough from commercialization that it's hard to figure out how to invest in it. (Also, I'm nervous about helping to speed up

AI, as it's already coming fast enough to worry me.)

I have a tiny position i...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Löb's Theorem for implicit reasoning in natural language: Löbian party invitations, published by Andrew Critch on January 1, 2023 on The AI Alignment Forum.

Related to: Löb's Lemma: an easier approach to Löb's Theorem.

Natural language models are really taking off, and it turns out there's an analogue of Löb's Theorem that occurs entirely in natural language — no math needed. This post will walk you through the details in a simple example: a very implicit party invitation.

Motivation

(Skip this if you just want to see the argument.)

Understanding the structure here may be helpful for anticipating whether Löbian phenomena can, will, or should arise amongst language-based AI systems. For instance, Löb's Theorem has implications for the emergence of cooperation and defection in groups of formally defined agents (LaVictoire et al, 2014; Critch, Dennis, Russell, 2022). The natural language version of Löb could play a similar role amongst agents that use language, which is something I plan to explore in a future post. Aside from being fun, I'm hoping this post will make clear that the phenomenon underlying Löb's Theorem isn't just a feature of formal logic or arithmetic, but of any language that can talk about reasoning and deduction in that language, including English. And as Ben Pace points out here, invitations are often self-referential, such as when people say "You are hereby invited to the party": hereby means "by this utterance" (google search).

So invitations a natural place to explore the kind of self-reference happening in Löb's Theorem.

This post isn't really intended as an "explanation" of Löb's Theorem in its classical form, which is about arithmetic. Rather, the arguments here stand entirely on their own, are written in natural language, and are about natural language phenomena. That said, this post could still function as an "explanation" of Löb's Theorem because of the tight analogy with it.

Implicitness

Okay, imagine there's a party, and maybe you're invited to it. Or maybe you're implicitly invited to it. Either way, we'll be talking a bunch about things being implicit, with phrasing like this:

"It's implicit that X",

"Implicitly X", or

"X is implicit".

These will all mean "X is implied by things that are known (to you) (via deduction or logical inference)".

Explicit knowledge is also implicit. In this technical sense of the word, "implicit" and "explicit" are not actually mutually exclusive: X trivially implies X, so if you explicitly observed X in the world, then you also know X implicitly. If you find this bothersome or confusing, just grant me this anyway, or skip to "Why I don't treat 'implicit' and inexplicit' as synonyms here" at the end.

Abbreviations. To abbreviate things and to show there's a simple structure at play here, I'll sometimes use the box symbol "□" as shorthand to say things are implicit:

"□(cats love kittens)" will mean "It's implicit that cats love kittens"

"□X" will mean "It's implicit that X"

A peculiar invitation

Okay! Let p be the statement "You're invited to the party". You'd love to receive such a straightforward invitation to the party, like some people did, those poo poo heads, but instead the host just sends you the following intriguing message:

Abbreviation: □pp

Interesting! Normally, being invited to a party and being implicitly invited are not the same thing, but for you in this case, apparently they are. Seeing this, you might feel like the host is hinting around at implicitly inviting you, and maybe you'll start to wonder if you're implicitly invited by virtue of the kind of hinting around that the host is doing with this very message. Well then, you'd be right! Here's how.

For the moment, forget about the host's message, and consider the following sentence, without assuming its truth (or implicitness):

Ψ:

The sentenc...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Slack matters more than any outcome, published by Valentine on December 31, 2022 on LessWrong.

About a month ago Ben Pace invited me to expand on a point I'd made in a comment.

The gist is this:

Addictions can cause people to accomplish things they wouldn't accomplish otherwise.

But if the accomplishment were worthwhile, why would the addiction be helpful? Why wouldn't the clarity that it's worthwhile be enough?

I postulate that the reason is a kind of metaphorical heaviness in culture. A particular structural tendency to eat slack.

So I think it'd be net better to let some worthwhile things go unaccomplished in favor of lifting the metaphoric burden. Creating more slack.

And I'd even say that this is the main plausible pathway I see for creating a great future for humanity. I don't think we can get there by focusing on making worthwhile things happen.

I felt inspired to write up an answer. Then I spent a month working on it. I clarified my thinking a lot but slid into a slog of writing. Kind of a perfectionist thing.

So I'm scrapping all that. I'm going to write a worse version, since the option is (a) a quickly hacked together version or (b) nothing at all.

Addictions

My main point isn't really about addictions, but I need to clarify something about this topic anyway. They're also a great example cluster.

When I say "addiction", I'm not gesturing at a vague intuition. I mean a very particular structure:

There's some unwelcome experience that repeatedly arises.

There's a behavior pattern that can temporarily distract the person in question from the unwelcome experience.

But the behavior pattern doesn't address the cause of the unwelcome experience arising in the first place.

So when someone engages in the distraction, it provides temporary relief, but the unwelcome experience arises again — and now the distraction is a little more tempting. A little more habit-forming. When that becomes automatic, it can feel like you're trapped inside it, like you're powerless against some behavior momentum.

Which is to say, this structure eats slack.

Some rapid-fire examples:

Caffeine dependency becomes an addiction when you autopilot react to the withdrawal symptoms by reaching for another cup of coffee.

Alcoholism as an addiction is often (usually? always?) about avoiding emotional experiences. Since the causes of the emotions don't go away, sobriety can result in the unwelcome experience arising, which the alcoholic knows how to numb away.

I have a long-standing habit of feeling kind of listless, lonely, like I should be doing something more or different with my life but I'm not quite sure what it is or that I can do it. If I don't pay attention when that sensation/emotion/thought cluster arises, I find myself on my computer scrolling social media or watching YouTube or Netflix. Putting up blockers to these sites both (a) makes me good at disabling the blockers and (b) makes things like porn or Minesweeper more tempting.

I'm not saying that all addictions are like this. I can't think of any exceptions off the top of my head, but that might just be a matter of my lack of creativity or a poor filing system in my mind.

I'm saying that there's this very particular structure, that it's quite common, and that I'm going to use the word "addiction" to refer to it.

And yeah, I do think it's the right word, which is why I'm picking it. Please notice the framing effect, and adjust yourself as needed.

Imposing an idea

The main thing I want to talk about is a generalization of rationalization, in the sense of writing the bottom line.

Caffeine dependency

When I grab a cup of coffee for a pick-me-up, I'm basically asserting that I should have more energy than I do right now.

This is kind of odd if you think about it. If I found out my house were on fire, I wouldn't feel too tired to deal with...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Recursive Middle Manager Hell, published by Raemon on January 1, 2023 on LessWrong.

I think Zvi's Immoral Mazes sequence is really important, but comes with more worldview-assumptions than are necessary to make the points actionable. I conceptualize Zvi as arguing for multiple hypotheses. In this post I want to articulate one sub-hypothesis, which I call "Recursive Middle Manager Hell". I'm deliberately not covering some other components of his model.

tl;dr:

Something weird and kinda horrifying happens when you add layers of middle-management. This has ramifications on when/how to scale organizations, and where you might want to work, and maybe general models of what's going on in the world.

You could summarize the effect as "the org gets more deceptive, less connected to its original goals, more focused on office politics, less able to communicate clearly within itself, and selected for more for sociopathy in upper management."

You might read that list of things and say "sure, seems a bit true", but one of the main points here is "Actually, this happens in a deeper and more insidious way than you're probably realizing, with much higher costs than you're acknowledging. If you're scaling your organization, this should be one of your primary worries."

The Core Model

Say you have a two-layer company, a CEO and a widgetmaker. The CEO is directly in contact with reality – his company either is profitable or not. He can make choices about high-level-widgetmaking strategy, and see those choices play out in customers that buy his product.

The widgetmaker is also in direct contact with reality – he's got widgets to make. He can see them getting built. He can run into problems with widget-production, see that widgets are no longer getting made, or getting made worse. And then he can fix those problems.

Add one middle manager into the mix.

The middle manager is neither directly involved with widget-production, or the direct consequences of high-level widget strategy. Their feedback loop with reality is weaker. But, they do directly interact with the CEO, and the widgetmakers. So they get exposed to the object level problems of widget-making and company-profit-maximizing.

Hire a lot of widgetmakers, such that you need two middle managers. Now, the middle managers start talking to each other, and forming their own culture.

Scale the company enough that you need two layers of middle-managers. Now there's an upper layer who reports to the CEO, but the things they reports about are "what did the lower-middle-managers tell me?". The lower layer talks directly to the widgetmakers, and reports down what they hear about high level strategy from upper management.

Lower middle management wants promotions and raises. Upper management isn't directly involved with the process of widgetmaking, so they only have rough proxies to go on. Management begins constructing a culture about legible signals of progress, which begin to get goodharted in various ways.

Then, scale the company enough that you have three layers of middle management.

Now, in the center of the hierarchy are people who never talk to someone who's directly engaged with a real-world problem. And there are multiple levels, which create a ladder for career advancement. Middle management culture develops which is about career advancement – people rise through the ranks of that culture if they prioritize career advancement as their goal, trading off against other things. Those people end up in charge of how career advancement happens, and they tend to promote people who are like them.

You can fudge numbers to make yourself look good, and because nobody is in direct contact with the reality, it's hard to tell when the numbers are bullshit. It's really hard to evaluate middle managers, and even if you could, evaluation is expensive. If...