Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Scott Aaronson and Steven Pinker Debate AI Scaling, published by Liron on June 28, 2022 on LessWrong.

Before June 2022 was the month of the possible start of the Second American Civil War, it was the month of a lively debate between Scott Alexander and Gary Marcus about the scaling of large language models, such as GPT-3. Will GPT-n be able to do all the intellectual work that humans do, in the limit of large n? If so, should we be impressed? Terrified? Should we dismiss these language models as mere “stochastic parrots”?

I was privileged to be part of various email exchanges about those same questions with Steven Pinker, Ernest Davis, Gary Marcus, Douglas Hofstadter, and Scott Alexander. It’s fair to say that, overall, Pinker, Davis, Marcus, and Hofstadter were more impressed by GPT-3’s blunders, while we Scotts were more impressed by its abilities. (On the other hand, Hofstadter, more so than Pinker, Davis, or Marcus, said that he’s terrified about how powerful GPT-like systems will become in the future.)

Anyway, at some point Pinker produced an essay setting out his thoughts, and asked whether “either of the Scotts” wanted to share it on our blogs. Knowing an intellectual scoop when I see one, I answered that I’d be honored to host Steve’s essay—along with my response, along with Steve’s response to that. To my delight, Steve immediately agreed. Enjoy! –SA

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Kurzgesagt - The Last Human (Longtermist video), published by Lizka on June 28, 2022 on The Effective Altruism Forum.

Kurzgesagt's recent video, "The Last Human – A Glimpse Into The Far Future," invites viewers to consider the potential for a vast number of future people. "If we don't kill ourselves in the next few centuries or millennia, almost all humans that will ever exist will live in the future."

Here's the video:

Kurzgesagt has some other very relevant videos (and some other cool videos):

Good news case on climate change

The Egg - a short story

What if we nuke a city?

Note: If you were involved in making the video happen and want to author or coauthor this link-post, please let me know!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: It’s Probably Not Lithium, published by Natália Mendonça on June 28, 2022 on LessWrong.

A Chemical Hunger (a), a series by the authors of the blog Slime Mold Time Mold (SMTM) that has been received positively on LessWrong, argues that the obesity epidemic is entirely caused (a) by environmental contaminants. The authors’ top suspect is lithium (a), primarily because it is known to cause weight gain at the doses used to treat bipolar disorder.

After doing some research, however, I found that it is not plausible that lithium plays a major role in the obesity epidemic, and that a lot of the claims the SMTM authors make about the topic are misleading, flat-out wrong, or based on extremely cherry-picked evidence. I have the impression that reading what they have to say about this often leaves the reader with a worse model of reality than they started with, and I’ll explain why I have that impression in this post.

(Preamble) A brief summary of their hypotheses

The SMTM authors have recently (a) summarized their hypotheses on how lithium exposure could explain the obesity epidemic. The first hypothesis is that trace exposure is responsible:

One possibility is that small amounts of lithium are enough to cause obesity, at least with daily exposure.

And the second one is that people are intermittently exposed to therapeutic doses:

[E]ven if people aren’t getting that much lithium on average, if they sometimes get huge doses, that could be enough to drive their lipostat upward.

I am going to argue that neither of those is plausible. I address the plausibility of the second hypothesis in the next section, and the plausibility of the first one in the rest of the post.

Lithium exposure in the general population is extremely low, even at the tails, in the majority of countries for which we have data

A few days ago, the SMTM authors published a literature review (a) on the lithium content of food. They conclude that, whereas the existing literature isn’t great, “[i]t seems like most people get at least 1 mg [of lithium] a day from their food, and on many days, there’s a good chance you’ll get more.” They also say it seems plausible that people are intermittently exposed to doses of lithium within the therapeutic range through their diet.

However, their literature review pretty much only includes studies that are outliers in the literature. Moreover, they use a misleading threshold for the therapeutic range of lithium. I’ll explain.

The studies in SMTM’s literature review of lithium levels in food are pretty much all outliers

In 2006, France conducted its second Total Diet Study (henceforth TDS). Across 1,319 food samples, the highest lithium concentration found was 0.6 mg/kg, in water. That’s not the highest average concentration among food groups – it’s the highest concentration of any single sample they tested. (For context, a standard clinical dose of elemental lithium is about 200 mg/day, or 1 gram/day of lithium carbonate.)

Similarly, New Zealand’s 2016 TDS examined 1,056 food samples and the highest concentration it found in any single sample was 0.54 mg/kg (in mussels).

Canada makes the raw data of its Total Diet Study publicly available (a), and they too measure the lithium content of their food. The maximum level reported is 1.1 mg/kg (in table salt, which is presumably rarely consumed in kilogram quantities) across 479 food samples, with the mean being 25 µg/kg and the median 11 µg/kg. Here’s a histogram of the data:

Excluding table salt, the maximum value in the rest of the dataset (N = 476) is 0.4 mg/kg, in mineral water.

Total Diet Studies in other countries report similarly low levels. Using data from the UK’s 1994 TDS (which included 400 food samples), the mean daily lithium intake among adults was estimated to be 17 µg/day, more than 50 times lower than SMTM’s estima...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Are You Tracking In Your Head?, published by johnswentworth on June 28, 2022 on LessWrong.

A large chunk - plausibly the majority - of real-world expertise seems to be in the form of illegible skills: skills/knowledge which are hard to transmit by direct explanation. They’re not necessarily things which a teacher would even notice enough to consider important - just background skills or knowledge which is so ingrained that it becomes invisible.

I’ve recently noticed a certain common type of illegible skill which I think might account for the majority of illegible-skill-value across a wide variety of domains.

Here are a few examples of the type of skill I have in mind:

While operating a machine, track an estimate of its internal state.

While talking to a person, track an estimate of their internal mental state - emotions, engagement, thoughts/worries, true motivations, etc.

While writing an algorithm, track a Fermi estimate of runtime.

While reading or writing math, track a prototypical example of what the math is talking about.

While playing a competitive game, track an estimate of the other players’ plans, intentions and private information

While writing, track an estimate of the mental state of a future reader - confusion, excitement, eyes glossing over, etc.

While reasoning through a difficult search/optimization problem, track an estimate of which constraints are most taut.

While working on math, physics, or a program, track types/units

While working on math, physics, or a program, track asymptotic behavior

While in conversation, track ambiguous tokenization for potential jokes.

While presenting to a crowd, track engagement level.

While absorbing claims/information, track an estimate of the physical process which produced the information, and how that process entangles the information with physical reality.

The common pattern among all these is that, while performing a task, the expert tracks some extra information/estimate in their head. Usually the extra information is an estimate of some not-directly-observed aspect of the system of interest. From outside, watching the expert work, that extra tracking is largely invisible; the expert may not even be aware of it themselves. Rarely are these mental tracking skills explicitly taught. And yet, based on personal experience, each of these is a central piece of performing the task well - arguably the central piece, in most cases.

Let’s assume that this sort of extra-information-tracking is, indeed, the main component of illegible-skill-value across a wide variety of domains. (I won’t defend that claim much; this post is about highlighting and exploring the hypothesis, not proving it.) What strategies does this suggest for learning, teaching, and self-improvement? What else does it suggest about the world?

Pay Attention To Extra Information Tracking

I had a scheme, which I still use today when somebody is explaining something that I’m trying to understand: I keep making up examples. For instance, the mathematicians would come in with a terrific theorem, and they’re all excited. As they’re telling me the conditions of the theorem, I construct something which fits all the conditions. You know, you have a set (one ball) – disjoint (two balls). Then the balls turn colors, grow hairs, or whatever, in my head as they put more conditions on. Finally they state the theorem, which is some dumb thing about the ball which isn’t true for my hairy green ball thing, so I say, ‘False!’

- Feynman

A lot of people have heard Feynman’s “hairy green ball thing” quote. It probably sounds like a maybe-useful technique to practice, but not obviously more valuable than any of a dozen other things.

The hypothesis that extra-information-tracking is the main component of illegible-skill-value shines a giant spotlight on things like Feynman...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Units of Exchange, published by CFAR!Duncan on June 28, 2022 on LessWrong.

Epistemic status: Established and confirmed

The lessons taught in Units of Exchange are straightforward applications of extremely well-established principles from economics and sociology, such as supply and demand, Pareto curves, value of information, the sunk cost fallacy, and arbitrage. The causal relationships underlying by each of these principles have been robustly confirmed in a wide variety of domains, and the recommended actions we’ve derived from them are simple and fairly conventional.

A version of this course is taught near the beginning of every CFAR workshop, often as the very first class. That’s not because the concepts it covers are revelational or groundbreaking, but rather the opposite—they’re core concepts, fundamental prerequisites that underlie and inform much of the rest of our content.

If you’re already familiar with them—great! This is a quick-and-dirty overview—we don’t mean to condescend to people who’ve already had specific training in these fields, only to provide those same tools to all of our participants. If these are not the droids you’re looking for, feel free to skip ahead to Inner Simulator.

The Lego Principle: Bricks and bargains

If you’ve ever done proofs in math or logic, then you know that it’s possible to reach complex and interesting results by starting from a limited number of assumptions. The conclusions in Units of Exchange aren’t mathematically rigorous, but they do emerge from two key premises, which combine to form what we call the Lego Principle.

1. Things are made of parts

The first half of the Lego Principle is reductionism, or the idea that, having fully explained all of the components of a thing and how those components interrelate, there’s nothing left to say. Metaphorically, if one has described the trees, shrubs, and fauna in all of their relevant detail, one has fully explained the forest; there is no ephemeral “missing” property that is forest-ness.

Reductionism is a powerful concept, because it allows us to interface with complex phenomena by dealing with smaller, simpler sub-phenomena. Brains, emotions, societies, financial markets—these are all large, and sometimes daunting to engage with. But neurons, cognitive if-then patterns, social norms, and individual commodities are all at least relatively more tractable.

2. Parts may be exchanged

The technical term here is currencies, and the idea is that, just as we can trade dollars for pounds (and buy things with either, in a place like an international airport), so too can we trade money for effort or sleep for knowledge or respect for social influence. In particular, where things are made up of similar parts, we can make exchanges between them, swapping our resources around to prioritize what we think is important.

Most of the rest of this unit boils down to straightforward applications of these two premises. There are things that we want more of—time, money, motivation, pleasure, attention, energy, knowledge, stuff, sleep, respect, be- longing, accomplishment, the well-being of the people we care about. Some of these wants are instrumental—we want them because they will lead to other good things (money being the classic example). Others are more terminal— they’re good in and of themselves (such as happiness or satisfaction).

Since life isn’t perfect and most of us aren’t all-powerful, we make trade- offs. We skimp on sleep to get more work done, bail on a work party to spend time with a significant other, skip dessert because we’re trying to get in shape. To a great extent, making good decisions can be framed as paying close attention to the exchange rates between these various currencies.

Part I: Apples to oranges

Some people benefit from converting everything in their lives to a common...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CFAR Handbook: Introduction, published by CFAR!Duncan on June 28, 2022 on LessWrong.

The Center for Applied Rationality is a Bay Area non-profit that, among other things, ran lots of workshops to offer people tools and techniques for solving problems and improving their thinking. Those workshops were accompanied by a reference handbook, which has been available as a PDF since 2020.

The handbook hasn't been substantially updated since it was written in 2017, but it remains a fairly straightforward primer for a lot of core rationality content. The LW team, working with the handbook's author Duncan Sabien, have decided to republish it as a lightly-edited sequence, so that each section can be linked on its own.

In the workshop context, the handbook was a supplement to lectures, activities, and conversations taking place between participants and staff. Care was taken to emphasize the fact that each tool or technique or perspective was only as good as it was effectively applied to one's problems, plans, and goals. The workshop was intentionally structured to cause participants to actually try things (including iterating on or developing their own versions of what they were being shown), rather than simply passively absorb content. Keep this in mind as you read—mere knowledge of how to exercise does not confer the benefits of exercise!

Discussion is strongly encouraged, and disagreement and debate are explicitly welcomed. Many LWers (including the staff of CFAR itself) have been tinkering with these concepts for years, and will have developed new perspectives on them, or interesting objections to them, or thoughts about how they work or break in practice. What follows is a historical artifact—the rough state-of-the-art at the time the handbook was written, circa 2017. That's an excellent jumping-off point, especially for newcomers, but there's been a lot of scattered progress since then, and we hope some of it will make its way into the comments.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Community Builders Spend Too Much Time Community Building, published by emmawilliamson on June 28, 2022 on The Effective Altruism Forum.

For helpful comments I thank Koji Flynn-Do, Abie Rohrig, Harry Taussig, Emma Abele, Sabrina Chwalek, Matt Burtell, Trevor Levin, Jemima Jones, Michel Justen, Caleb Parikh, and Max Daniel.

Some of my observations come from my experience grantmaking with the EA Infrastructure Fund. I am contributing in a personal capacity however, not as an EAIF assistant fund manager. (In particular, you are welcome to apply for EAIF funding if your plans mostly consist of the kind of activities I’m questioning in this post.) All views are my own.

Epistemic Status: In this post I’m mainly referring to university group community builders. It’s possible that a lot of what I say will still apply to city / country / other groups, but I’m less confident of this. In my problem section, I give some percentage estimates of how much organizers are marketing (defined later) and how much they should be marketing. This is based off of some rough estimates, which I’m not confident in. I’d love to see someone better estimate this or run a survey.

Advisory Note: If you’re only going to read one section of this post, I encourage you to read the TLDR and the Solutions section.

TLDR: There are significant downsides to the current model of community building, which involves organizers spending a large portion of their time marketing EA, doing (often menial) operations work for EA, and explaining to other EAs how to market EA:

It trades off against some of the most promising EA students skilling up themselves

It means that EA groups aren’t actually learning about object-level EA issues, and instead are learning things about small-scale marketing and club organization, which is significantly less important.

It can give EA a “Ponzi scheme” vibe on campus, because EA groups are disproportionately focused on growing their group as opposed to skilling up and tackling object-level issues.

Instead, many group leaders should think about what would be most useful for them, and then invite others to join them in the activity they’re doing. For examples of this, skim the solutions section.

Outline:

The Problem (in 4 Claims)

Solutions

Caveats and Why I Might be Wrong

The problem

The problem is that organizers and others at university EA groups (and possibly other EA groups) are spending too much time marketing EA and doing basic operations work when they should be skilling up. I think this is bad for the organizers, the epistemics of the group, the members of the group, and the external view of the group.

By skilling up - also referred to as skill building - I mean taking action to improve one’s epistemics, knowledge of key global problems, or career. Most of the actions that fall under skilling up involve cognitively intensive learning. Some actions that fall within the category of skill building would be extremely useful for some and trivial for others (e.g. doing an ML bootcamp may be great for someone looking to do AI safety research, but useless for someone interested in GPR).

For the rest of this essay, I will define marketing as “marketing EA, doing (often menial) operations work for EA, and explaining to other EAs how to market EA”. I recognize this is a broader definition of marketing than is generally used. However, I want to be able to simply point at the issue I’m talking about, which is when individuals - often top organizers - spend too much time doing activities like: running fellowships, advertising at club fairs, creating marketing materials (posters, emails, websites, etc), pitching EA to others, 1:1 outreach, and some events.

Note: Some parts of organizing an event would be included in my definition of marketing and some would not. If hosting an event involves a ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What success looks like, published by mariushobbhahn on June 28, 2022 on The Effective Altruism Forum.

This article was written by Marius Hobbhahn, Max Räuker, Yannick Mühlhäuser, Jasper Götting and Simon Grimm. We are grateful for feedback from and discussions with Lennart Heim, SE and AO.

Summary

Thinking through scenarios where TAI goes well informs our goals regarding AI safety and leads to concrete action plans. Thus, in this post,

We sketch stories where the development and deployment of transformative AI go well. We broadly cluster them like

Alignment won’t be a problem, .

Because alignment is easy: Scenario 1

We get lucky with the first AI: Scenario 4

Alignment is hard, but .

We can solve it together, because .

We can effectively deploy governance and technical strategies in combination together: Scenario 2

Humanity will wake up due to an accident: Scenario 3

The US and China will realize their shared interests: Scenario 5

One player can win the race, by .

Launching an Apollo Project for AI: Scenario 6

We categorize central points of influence that seem relevant for causing the success of our sketches. The categories with some examples are:

Governance: domestic laws, international treaties, safety regulations, whistleblower protection, auditing firms, compute governance and contingency plans

Technical: Red teaming, benchmarks, fire alarms, forecasting and information security

Societal: Norms in AI, publicity and field-building

We lay out some central causal variables for our stories in the third chapter. They include the level of cooperation, AI timelines, take-off speeds, size of the alignment tax, type of actors and number of actors

Introduction

There are many posts in AI alignment on sketching out failure scenarios. However, there seems to be less (public) work that talks about possible pathways to success. Holden Karnofsky writes in the appendix of Important, actionable research questions for the most important century:

Quote (Holden Karnofsky): I think there’s a big vacuum when it comes to well-thought-through visions of what [a realistic best-case transition to transformative AI] could look like, and such a vision could quickly receive wide endorsement from AI labs (and, potentially, from key people in government). I think such an outcome would be easily worth billions of dollars of longtermist capital.

We want to sketch out such best-case scenarios for the transition to transformative AI. This post is inspired by Paul Christiano’s What Failure Looks Like. Our goals are

To better understand how positive scenarios could look like

To better understand what levers are available to make positive outcomes more likely

To better understand subgoals to work towards

To make our reasoning transparent and receive feedback on our misconceptions

Scenarios

In the following, we will sketch out what some of the success stories for relevant subcomponents of TAI could look like. We know that these scenarios often lack important detailed counterarguments. We are also aware that every single point here could be its own article but we want this post to merely give a broad overview.

Scenario 1: Alignment is much easier than expected

The alignment problem turns out much easier than expected. Increasingly better AI models have a better understanding of human values, and they do not naturally develop strong influence-seeking tendencies. Moreover, in cases of malfunctions and for preventative measures, interpretability tools now allow us to understand important parts of large models on the most basic level and ELK-like tools allow us to honestly communicate with AI systems.

We know of many systems where a more powerful actor is relatively well aligned to one or many powerless actors. Parents usually protect their children who couldn’t survive without them, democratic governments...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What success looks like, published by Marius Hobbhahn on June 28, 2022 on The AI Alignment Forum.

TL;DR: We wrote a post on possible success stories of a transition to TAI to better understand which factors causally reduce the risk of AI risk. Furthermore, we separately explain these catalysts for success in more detail and this post can thus be thought of as a high-level overview of different AI governance strategies.

Summary

Thinking through scenarios where TAI goes well informs our goals regarding AI safety and leads to concrete action plans. Thus, in this post,

We sketch stories where the development and deployment of transformative AI go well. We broadly cluster them like

Alignment won’t be a problem, .

Because alignment is easy: Scenario 1

We get lucky with the first AI: Scenario 4

Alignment is hard, but .

We can solve it together, because .

We can effectively deploy governance and technical strategies in combination together: Scenario 2

Humanity will wake up due to an accident: Scenario 3

The US and China will realize their shared interests: Scenario 5

One player can win the race, by .

Launching an Apollo Project for AI: Scenario 6

We categorize central points of influence that seem relevant for causing the success of our sketches. The categories with some examples are:

Governance: domestic laws, international treaties, safety regulations, whistleblower protection, auditing firms, compute governance and contingency plans

Technical: Red teaming, benchmarks, fire alarms, forecasting and information security

Societal: Norms in AI, publicity and field-building

We lay out some central causal variables for our stories in the third chapter. They include the level of cooperation, AI timelines, take-off speeds, size of the alignment tax, type of actors and number of actors

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nonprofit Boards are Weird, published by Holden Karnofsky on June 28, 2022 on The Effective Altruism Forum.

Disclaimer: This post was cross-posted with the author's permission, under his account. He may not notice comments. It was originally posted on June 23. You can find the audio version here.

Note: anything in this post that you think is me subtweeting your organization is actually about, like, at least 3 organizations. (I'm currently on 4 boards in addition to Open Philanthropy's; I've served on a bunch of other boards in the past; and more than half of my takes on boards are not based on any of this, but rather on my interactions with boards I'm not on via the many grants made by Open Philanthropy.)

Writing about ideal governance reminded me of how weird my experiences with nonprofit boards (as in "board of directors" - the set of people who formally control a nonprofit) have been.

I thought that was a pretty good intro. The rest of this piece will:

Try to articulate what's so weird about nonprofit boards, fundamentally. I think a lot of it is the combination of great power, unclear responsibility, and ~zero accountability; additionally, I haven't been able to find much in the way of clear, widely accepted statements of what makes a good board member.

Give my own thoughts on what makes a good board member: which core duties they should be trying to do really well, the importance of "staying out of the way" on other things, and some potentially helpful practices.

I am experienced with nonprofit boards but not with for-profit boards. I'm guessing that roughly half the things I say below will apply to for-profit boards, and that for-profit boards are roughly half as weird overall (so still quite weird), but I haven't put much effort into disentangling these things; I'm writing about what I've seen.

I can't really give real-life examples here (for reasons I think will be pretty clear) so this is just going to be me opining in the abstract.

Why nonprofit boards are weird

Here's how a nonprofit board works:

There are usually 3-10 people on the board (though sometimes much more). Most of them don't work for the nonprofit (they have other jobs).

They meet every few months. Nonprofit employees (especially the CEO1) do a lot of the agenda-setting for the meeting. Employees present general updates and ask for the board's approval on various things the board needs to approve, such as the budget.

A majority vote of the directors can do anything: fire the CEO, dissolve the nonprofit, add and remove directors, etc. You can think of the board as the "owner" of the nonprofit - formally, it has final say in every decision.

In practice, though, the board rarely votes except on matters that feel fairly "rubber-stamp," and the board's presence doesn't tend to be felt day-to-day at a nonprofit. The CEO leads the decision-making. Occasionally, someone has a thought like "Wait, who does the CEO report to? Oh, the board of directors ... who's on the board again? I don't know if I've ever really spoken with any of those people."

In my experience, it's common for the whole thing to feel extremely weird. (This doesn't necessarily mean there's a better way to do it - footnote has more on what I mean by "weird."2)

Board members often know almost nothing about the organization they have complete power over.

Board meetings rarely feel like a good use of time.

When board members are energetically asking questions and making demands, it usually feels like they're causing chaos and wasting everyone's time and energy.

On the rare occasions when it seems like the board should do something (like replacing the CEO, or providing an independent check on some important decision), the board often seems checked out and it's unclear how they would even come to be aware of the situation.

Everyone constantl...