Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is part one of: Yudkowsky and Christiano discuss "Takeoff Speeds", published by Eliezer Yudkowsky on the AI Alignment Forum

This is a transcription of Eliezer Yudkowsky responding to Paul Christiano's Takeoff Speeds live on Sep. 14, followed by a conversation between Eliezer and Paul. This discussion took place after Eliezer's conversation with Richard Ngo.

Color key:

Chat by Paul and Eliezer Other chat Inline comments

5.5. Comments on "Takeoff Speeds"

[Yudkowsky][10:14] (Nov. 22 follow-up comment)

(This was in response to an earlier request by Richard Ngo that I respond to Paul on Takeoff Speeds.)

[Yudkowsky][16:52]

maybe I'll try liveblogging some/ here in the meanwhile

Slower takeoff means faster progress

[Yudkowsky][16:57]

The main disagreement is not about what will happen once we have a superintelligent AI, it’s about what will happen before we have a superintelligent AI. So slow takeoff seems to mean that AI has a larger impact on the world, sooner.

It seems to me to be disingenuous to phrase it this way, given that slow-takeoff views usually imply that AI has a large impact later relative to right now (2021), even if they imply that AI impacts the world "earlier" relative to "when superintelligence becomes reachable".

"When superintelligence becomes reachable" is not a fixed point in time that doesn't depend on what you believe about cognitive scaling. The correct graph is, in fact, the one where the "slow" line starts a bit before "fast" peaks and ramps up slowly, reaching a high point later than "fast". It's a nice try at reconciliation with the imagined Other, but it fails and falls flat.

This may seem like a minor point, but points like this do add up.

In the fast takeoff scenario, weaker AI systems may have significant impacts but they are nothing compared to the “real” AGI. Whoever builds AGI has a decisive strategic advantage. Growth accelerates from 3%/year to 3000%/year without stopping at 30%/year. And so on.

This again shows failure to engage with the Other's real viewpoint. My mainline view is that growth stays at 5%/year and then everybody falls over dead in 3 seconds and the world gets transformed into paperclips; there's never a point with 3000%/year.

Operationalizing slow takeoff

[Yudkowsky][17:01]

There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles.

If we allow that consuming and transforming the solar system over the course of a few days is "the first 1 year interval in which world output doubles", then I'm happy to argue that there won't be a 4-year interval with world economic output doubling before then. This, indeed, seems like a massively overdetermined point to me. That said, again, the phrasing is not conducive to conveying the Other's real point of view.

I believe that before we have incredibly powerful AI, we will have AI which is merely very powerful.

Statements like these are very often "true, but not the way the person visualized them". Before anybody built the first critical nuclear pile in a squash court at the University of Chicago, was there a pile that was almost but not quite critical? Yes, one hour earlier. Did people already build nuclear systems and experiment with them? Yes, but they didn't have much in the way of net power output. Did the Wright Brothers build prototypes before the Flyer? Yes, but they weren't prototypes that flew but 80% slower.

I guarantee you that, whatever the fast takeoff scenario, there will be some way to look over the development history, and nod wisely and say, "Ah, yes, see, this was not unprecedented, here are these earlier systems which presaged the final system!" Maybe you could even look back to today and say that about GPT-3, yup, totally presaging stuff all over the place, great. But it isn't transforming soc...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is part one of: Yudkowsky and Christiano discuss "Takeoff Speeds", published by Eliezer Yudkowsky on the AI Alignment Forum

This is a transcription of Eliezer Yudkowsky responding to Paul Christiano's Takeoff Speeds live on Sep. 14, followed by a conversation between Eliezer and Paul. This discussion took place after Eliezer's conversation with Richard Ngo.

Color key:

Chat by Paul and Eliezer Other chat Inline comments

5.5. Comments on "Takeoff Speeds"

[Yudkowsky][10:14] (Nov. 22 follow-up comment)

(This was in response to an earlier request by Richard Ngo that I respond to Paul on Takeoff Speeds.)

[Yudkowsky][16:52]

maybe I'll try liveblogging some/ here in the meanwhile

Slower takeoff means faster progress

[Yudkowsky][16:57]

The main disagreement is not about what will happen once we have a superintelligent AI, it’s about what will happen before we have a superintelligent AI. So slow takeoff seems to mean that AI has a larger impact on the world, sooner.

It seems to me to be disingenuous to phrase it this way, given that slow-takeoff views usually imply that AI has a large impact later relative to right now (2021), even if they imply that AI impacts the world "earlier" relative to "when superintelligence becomes reachable".

"When superintelligence becomes reachable" is not a fixed point in time that doesn't depend on what you believe about cognitive scaling. The correct graph is, in fact, the one where the "slow" line starts a bit before "fast" peaks and ramps up slowly, reaching a high point later than "fast". It's a nice try at reconciliation with the imagined Other, but it fails and falls flat.

This may seem like a minor point, but points like this do add up.

In the fast takeoff scenario, weaker AI systems may have significant impacts but they are nothing compared to the “real” AGI. Whoever builds AGI has a decisive strategic advantage. Growth accelerates from 3%/year to 3000%/year without stopping at 30%/year. And so on.

This again shows failure to engage with the Other's real viewpoint. My mainline view is that growth stays at 5%/year and then everybody falls over dead in 3 seconds and the world gets transformed into paperclips; there's never a point with 3000%/year.

Operationalizing slow takeoff

[Yudkowsky][17:01]

There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles.

If we allow that consuming and transforming the solar system over the course of a few days is "the first 1 year interval in which world output doubles", then I'm happy to argue that there won't be a 4-year interval with world economic output doubling before then. This, indeed, seems like a massively overdetermined point to me. That said, again, the phrasing is not conducive to conveying the Other's real point of view.

I believe that before we have incredibly powerful AI, we will have AI which is merely very powerful.

Statements like these are very often "true, but not the way the person visualized them". Before anybody built the first critical nuclear pile in a squash court at the University of Chicago, was there a pile that was almost but not quite critical? Yes, one hour earlier. Did people already build nuclear systems and experiment with them? Yes, but they didn't have much in the way of net power output. Did the Wright Brothers build prototypes before the Flyer? Yes, but they weren't prototypes that flew but 80% slower.

I guarantee you that, whatever the fast takeoff scenario, there will be some way to look over the development history, and nod wisely and say, "Ah, yes, see, this was not unprecedented, here are these earlier systems which presaged the final system!" Maybe you could even look back to today and say that about GPT-3, yup, totally presaging stuff all over the place, great. But it isn't transforming soc...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Explanatory Obstacle of EA, published by GidonKadosh on November 27, 2021 on The Effective Altruism Forum.

This post discusses multiple issues relating to the way the EA movement is perceived (ranging from common misconceptions to unjustified strong opinions against EA) and suggests alternatives to the ways we describe EA.

Since I don’t have the resources to quantify this problem, I rely on my personal experience as a community builder and that of many other community builders and explain the rationale behind my suggestions.

Around 2013, a couple of mass media articles about EA (1,2,3) - specifically about Earning To Give - were published. These articles clearly missed most of the nuances behind the idea of Earning To Give, and heavily misrepresented the idea.

In light of such events, the EA movement at that time faced a critical question: Should we stay away from mass media?The answer the EA community arrived at was yes, and CEA formalized this as a part of its strategy:

Historically, spreading EA via the mass media was a key focus of CEA. Over time it became clear that the mass media is not particularly well suited to spreading ideas with high fidelity. Therefore, we have pivoted away from this focus and towards higher-fidelity methods like books and podcasts.

I think this was a good decision, but I think the movement should have asked itself another critical question to prevent future broad misunderstandings of EA:Are we explaining what EA is well enough?

From my experience, the answer is no, even today. EA Israel used and experimented with the common pitches and explanations from several EA pitch guides (1,2,3), but we’ve kept noticing how almost every new member needs additional explanations on the basics of EA. We’ve kept encountering newcomers who are highly excited about EA while thinking it’s something else (e.g. it’s about making people more altruistic, or specifically making charities more efficient), and then lose excitement through the onboarding process. We also find ourselves struggling with common misconceptions, as does the rest of the community (1,2).

If individuals who look into EA perceive it as something that is not sufficiently close to its meaning, we both attract individuals who are not a good fit for EA, and miss individuals who could have been a good fit.

Clarifying our explanation of EA is also a great way to recognize disagreements among the community about what EA is. For instance, does EA call individuals to spend more resources on doing good, or to do more good with the same amount of resources?

What is a good explanation?

For the purposes of this post, let’s make a distinction between three approaches to describing a concept, each with a different focus:

A definition: Should be as accurate as possible.

An explanation: Should be as clear as possible.

A pitch: Should be as convincing as possible.

While not true in all contexts, a pitch in our case should include a clear explanation of EA: Regardless of how convincing our pitch is, we still want newcomers to have a good understanding of what EA is, so they can later make better decisions. Therefore, crafting better explanations of EA is beneficial both for pitching and for explanations.

The EA community discussed the definition of EA many times before (1,2,3), and has many convincing and clever pitches (linked above). But I haven’t seen as many discussions on explanations, and accordingly, I think what’s most missing for newcomers is clarity.

Sidenote: It can be challenging to discuss what “convincing” is. I find it useful using the theory of motivational salience, and I’ll try to explain how certain phrases create reasons for being involved with EA (incentive salience) versus creating reasons against being involved in EA (aversive salience). For the purpose of this post, this mostly ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Explanatory Obstacle of EA, published by GidonKadosh on November 27, 2021 on The Effective Altruism Forum.

This post discusses multiple issues relating to the way the EA movement is perceived (ranging from common misconceptions to unjustified strong opinions against EA) and suggests alternatives to the ways we describe EA.

Since I don’t have the resources to quantify this problem, I rely on my personal experience as a community builder and that of many other community builders and explain the rationale behind my suggestions.

Around 2013, a couple of mass media articles about EA (1,2,3) - specifically about Earning To Give - were published. These articles clearly missed most of the nuances behind the idea of Earning To Give, and heavily misrepresented the idea.

In light of such events, the EA movement at that time faced a critical question: Should we stay away from mass media?The answer the EA community arrived at was yes, and CEA formalized this as a part of its strategy:

Historically, spreading EA via the mass media was a key focus of CEA. Over time it became clear that the mass media is not particularly well suited to spreading ideas with high fidelity. Therefore, we have pivoted away from this focus and towards higher-fidelity methods like books and podcasts.

I think this was a good decision, but I think the movement should have asked itself another critical question to prevent future broad misunderstandings of EA:Are we explaining what EA is well enough?

From my experience, the answer is no, even today. EA Israel used and experimented with the common pitches and explanations from several EA pitch guides (1,2,3), but we’ve kept noticing how almost every new member needs additional explanations on the basics of EA. We’ve kept encountering newcomers who are highly excited about EA while thinking it’s something else (e.g. it’s about making people more altruistic, or specifically making charities more efficient), and then lose excitement through the onboarding process. We also find ourselves struggling with common misconceptions, as does the rest of the community (1,2).

If individuals who look into EA perceive it as something that is not sufficiently close to its meaning, we both attract individuals who are not a good fit for EA, and miss individuals who could have been a good fit.

Clarifying our explanation of EA is also a great way to recognize disagreements among the community about what EA is. For instance, does EA call individuals to spend more resources on doing good, or to do more good with the same amount of resources?

What is a good explanation?

For the purposes of this post, let’s make a distinction between three approaches to describing a concept, each with a different focus:

A definition: Should be as accurate as possible.

An explanation: Should be as clear as possible.

A pitch: Should be as convincing as possible.

While not true in all contexts, a pitch in our case should include a clear explanation of EA: Regardless of how convincing our pitch is, we still want newcomers to have a good understanding of what EA is, so they can later make better decisions. Therefore, crafting better explanations of EA is beneficial both for pitching and for explanations.

The EA community discussed the definition of EA many times before (1,2,3), and has many convincing and clever pitches (linked above). But I haven’t seen as many discussions on explanations, and accordingly, I think what’s most missing for newcomers is clarity.

Sidenote: It can be challenging to discuss what “convincing” is. I find it useful using the theory of motivational salience, and I’ll try to explain how certain phrases create reasons for being involved with EA (incentive salience) versus creating reasons against being involved in EA (aversive salience). For the purpose of this post, this mostly ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: FEM’s Research: What we learned about our audience and how best to reach them, published by Anna Christina on November 26, 2021 on The Effective Altruism Forum.

Family Empowerment Media (FEM) is an evidence-driven nonprofit working to improve maternal and child health. We do this by broadcasting clear and compelling radio content that addresses barriers to the uptake of family planning services.

In previous posts, we explained why we think family planning is a cause worth prioritizing, and shared the results of our proof of concept campaign. Here, we outline some of the research we used to inform our pilot, which is now on air in Nigeria. We hope our approach and findings can be useful to other organisations undertaking evidence-based decision-making and intervention design. For those interested, we also close with some opportunities to support our work.

As an evidence-driven organisation, we aim to base all our key decisions on data. To achieve this, we needed to conduct research before launching our pilot campaign in Kano State. In particular, we needed to ensure that we had a proper understanding of the local context.

Our key questions were:

Who is our target audience? (what do they value, how do they live, and who are their role models.)

What factors encourage or discourage contraceptive use in Kano?

How should we structure our content?

To answer these questions, we took a cluster approach: drawing on multiple different research tools and combining the results to build a better picture of the landscape. Some of the methods we used included literature reviews, surveys, interviews, and focus group discussions. (For more information about the tools we used, and their strengths and weaknesses, see the appendix.)

Overall, we spent more than 190 hours on desk research, and interviewed more than 300 radio listeners. Our findings, outlined below, ultimately resulted in our pilot campaign, which launched on World Contraceptive Day, in collaboration with the Ministry of Health. The campaign is currently on air in Kano State, Nigeria, and is now reaching 5.6 million listeners through our partner station.

1. Who is our target audience?

Our pilot campaign in Kano State was primarily aimed at married couples with children. There were three main reasons for this. First, due to cultural concerns, our Nigerian stakeholders were not comfortable with us targeting unmarried women. Second, relevant studies demonstrated larger effect sizes within these groups. Third, experts had advised that our messaging was unlikely to impact newlywed couples, as the pressure to have a child soon after marriage is very strong. Given these factors, it seemed most impactful to focus on couples who already had children, and who wanted to space or limit future births.

However, to create content that resonated with this audience, we needed to know more about them. What was most important to them— what were their values and dreams? What were their daily routines and joys? Who were their role models?

To investigate, we drew on a mix of qualitative research (one-to-one interviews and focus group discussions with listeners,) expert opinion, barrier analysis surveys, and interviews with clinic workers. (See the appendix for further information about these research tools). Our findings suggested that:

Family, education, and economic success were important valuesProviding proper care to their children and others in the community was an important value for both men and women. Both men and women also cited the importance of peace and harmony. They wanted their children to become contributing members of the community and to have “good values.” Some women wished for more education for themselves, and men wanted to improve their business and earning prospects.

Work dominated daily routines, while religion and the ra...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: FEM’s Research: What we learned about our audience and how best to reach them, published by Anna Christina on November 26, 2021 on The Effective Altruism Forum.

Family Empowerment Media (FEM) is an evidence-driven nonprofit working to improve maternal and child health. We do this by broadcasting clear and compelling radio content that addresses barriers to the uptake of family planning services.

In previous posts, we explained why we think family planning is a cause worth prioritizing, and shared the results of our proof of concept campaign. Here, we outline some of the research we used to inform our pilot, which is now on air in Nigeria. We hope our approach and findings can be useful to other organisations undertaking evidence-based decision-making and intervention design. For those interested, we also close with some opportunities to support our work.

As an evidence-driven organisation, we aim to base all our key decisions on data. To achieve this, we needed to conduct research before launching our pilot campaign in Kano State. In particular, we needed to ensure that we had a proper understanding of the local context.

Our key questions were:

Who is our target audience? (what do they value, how do they live, and who are their role models.)

What factors encourage or discourage contraceptive use in Kano?

How should we structure our content?

To answer these questions, we took a cluster approach: drawing on multiple different research tools and combining the results to build a better picture of the landscape. Some of the methods we used included literature reviews, surveys, interviews, and focus group discussions. (For more information about the tools we used, and their strengths and weaknesses, see the appendix.)

Overall, we spent more than 190 hours on desk research, and interviewed more than 300 radio listeners. Our findings, outlined below, ultimately resulted in our pilot campaign, which launched on World Contraceptive Day, in collaboration with the Ministry of Health. The campaign is currently on air in Kano State, Nigeria, and is now reaching 5.6 million listeners through our partner station.

1. Who is our target audience?

Our pilot campaign in Kano State was primarily aimed at married couples with children. There were three main reasons for this. First, due to cultural concerns, our Nigerian stakeholders were not comfortable with us targeting unmarried women. Second, relevant studies demonstrated larger effect sizes within these groups. Third, experts had advised that our messaging was unlikely to impact newlywed couples, as the pressure to have a child soon after marriage is very strong. Given these factors, it seemed most impactful to focus on couples who already had children, and who wanted to space or limit future births.

However, to create content that resonated with this audience, we needed to know more about them. What was most important to them— what were their values and dreams? What were their daily routines and joys? Who were their role models?

To investigate, we drew on a mix of qualitative research (one-to-one interviews and focus group discussions with listeners,) expert opinion, barrier analysis surveys, and interviews with clinic workers. (See the appendix for further information about these research tools). Our findings suggested that:

Family, education, and economic success were important valuesProviding proper care to their children and others in the community was an important value for both men and women. Both men and women also cited the importance of peace and harmony. They wanted their children to become contributing members of the community and to have “good values.” Some women wished for more education for themselves, and men wanted to improve their business and earning prospects.

Work dominated daily routines, while religion and the ra...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Almost everyone should be less afraid of lawsuits, published by alyssavance on November 27, 2021 on LessWrong.

One sad feature of modern American society is that many people, especially those tied to big institutions, don't help each other out because a fear of lawsuits. Employers don't give meaningful references, or ever tell their rejected interviewees how they could improve their skills. Abuse victims keep silent, in case someone on their abuser's side files a defamation case. Doctors prescribe unnecessary, expensive tests as "defensive medicine". Inventions don't get built, in case there's a patent lawsuit. I'm not an attorney myself, but my best guess is that letting litigation fears stop you is often a mistake, and I've given this advice to friends several times before. Here's why:

Almost all lawsuit threats never happen

Threats are easy - anyone can threaten to sue anyone else, with two minutes of time and a smartphone. Actually suing is much harder. Outside of small claims court, hiring an attorney will usually cost tens of thousands of dollars, at least. Litigating a case takes months or even years, while angry feelings often go away after a few weeks. The person suing will have to give up a lot. Instead of playing games or taking a vacation or putting in extra hours at work, they will have to do legal research and give testimony. Most people are distracted by their families, their career, their hobbies, and their lives, and will (often rationally) eventually give up, rather than remaining obsessed with whatever the case was about.

Lawsuit mitigation can be expensive

Doctors as a profession are traditionally concerned with legal risk. But the total value of medical malpractice claims is around $5 billion per year in the US - compared to healthcare spending of $3,800 billion, malpractice lawyer fees of $3 billion, and "defensive medicine" costs of $45 billion according to one study. Likewise, the cost to media companies and journalists of not publishing articles to stop defamation suits surely exceeds that of the few dozen defamation cases against them every year (stats from the UK, but British defamation law is widely considered plaintiff-friendly). The cost of things like rape victims staying silent because of defamation threats is hard to quantify, but clearly also bad. Some of these costs will fall on you directly, but often many more are externalized; eg., the cost to a patient of not giving them the right treatment out of fear can be enormous.

Many legal risks are hypothetical or imaginary

Since most people aren't legal specialists, the "knowledge" that something is a legal risk can be passed around from place to place without ever being checked, like many urban legends or droplet disease transmission or stories about the amount of iron in spinach. For example, a lot of people "hear" that IQ tests are illegal in hiring. But not only is this not true, any company can easily buy one right now - "the CCAT is a pre-employment aptitude test that measures an individual's aptitude, or ability to solve problems, digest and apply information, learn new skills, and think critically. Individuals with high aptitude are more likely to be quick learners and high performers than are individuals with low aptitude." Likewise, it's easy to make up a story where something might be hypothetically risky, even if it's never happened in court, and hard to do enough research to make sure it isn't.

Most lawsuit types are rare

One study found around 750,000 civil lawsuits per year in big counties, with a population around 75 million, for a total annual risk of ~1% per person. But the distribution of case types is thin-tailed. Courts and lawyers are somewhat machine-like, and mostly handle the same mundane problems over and over again (this is also true for criminal cases). A maj...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Almost everyone should be less afraid of lawsuits, published by alyssavance on November 27, 2021 on LessWrong.

One sad feature of modern American society is that many people, especially those tied to big institutions, don't help each other out because a fear of lawsuits. Employers don't give meaningful references, or ever tell their rejected interviewees how they could improve their skills. Abuse victims keep silent, in case someone on their abuser's side files a defamation case. Doctors prescribe unnecessary, expensive tests as "defensive medicine". Inventions don't get built, in case there's a patent lawsuit. I'm not an attorney myself, but my best guess is that letting litigation fears stop you is often a mistake, and I've given this advice to friends several times before. Here's why:

Almost all lawsuit threats never happen

Threats are easy - anyone can threaten to sue anyone else, with two minutes of time and a smartphone. Actually suing is much harder. Outside of small claims court, hiring an attorney will usually cost tens of thousands of dollars, at least. Litigating a case takes months or even years, while angry feelings often go away after a few weeks. The person suing will have to give up a lot. Instead of playing games or taking a vacation or putting in extra hours at work, they will have to do legal research and give testimony. Most people are distracted by their families, their career, their hobbies, and their lives, and will (often rationally) eventually give up, rather than remaining obsessed with whatever the case was about.

Lawsuit mitigation can be expensive

Doctors as a profession are traditionally concerned with legal risk. But the total value of medical malpractice claims is around $5 billion per year in the US - compared to healthcare spending of $3,800 billion, malpractice lawyer fees of $3 billion, and "defensive medicine" costs of $45 billion according to one study. Likewise, the cost to media companies and journalists of not publishing articles to stop defamation suits surely exceeds that of the few dozen defamation cases against them every year (stats from the UK, but British defamation law is widely considered plaintiff-friendly). The cost of things like rape victims staying silent because of defamation threats is hard to quantify, but clearly also bad. Some of these costs will fall on you directly, but often many more are externalized; eg., the cost to a patient of not giving them the right treatment out of fear can be enormous.

Many legal risks are hypothetical or imaginary

Since most people aren't legal specialists, the "knowledge" that something is a legal risk can be passed around from place to place without ever being checked, like many urban legends or droplet disease transmission or stories about the amount of iron in spinach. For example, a lot of people "hear" that IQ tests are illegal in hiring. But not only is this not true, any company can easily buy one right now - "the CCAT is a pre-employment aptitude test that measures an individual's aptitude, or ability to solve problems, digest and apply information, learn new skills, and think critically. Individuals with high aptitude are more likely to be quick learners and high performers than are individuals with low aptitude." Likewise, it's easy to make up a story where something might be hypothetically risky, even if it's never happened in court, and hard to do enough research to make sure it isn't.

Most lawsuit types are rare

One study found around 750,000 civil lawsuits per year in big counties, with a population around 75 million, for a total annual risk of ~1% per person. But the distribution of case types is thin-tailed. Courts and lawyers are somewhat machine-like, and mostly handle the same mundane problems over and over again (this is also true for criminal cases). A maj...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: larger language models may disappoint you [or, an eternally unfinished draft], published by nostalgebraist on November 26, 2021 on LessWrong.

what this post is

The following is an incomplete draft, which I'm publishing now because I am unlikely to ever finish writing it.

I no longer fully endorse all the claims in the post. (In a few cases, I've added a note to say this explicitly.) However, there are some arguments in the post that I still endorse, and which I have not seen made elsewhere.

This post is the result of me having lots of opinions about LM scaling, at various times in 2021, which were difficult to write down briefly or independently of one another. This post, originally written in July 2021, is the closest I got to writing them all down in one place.

-nost, 11/26/21

0. caveat

This post will definitely disappoint you.

Or, anyway, it will definitely disappoint me. I know that even though I haven't written it yet.

My drafts folder contains several long, abandoned attempts to write (something like) this post. I've written (something like) this post many times in my head. I just can't seem to get it right, though. The drafts always sprawl out of control.

So, if I can't do it right, why not do it wrong? Here's the disorganized, incomplete, brain-dump version of the better post I wish I were writing. Caveat lector.

1. polarization

The topic of this post is large language models (LMs) like GPT-3. Specifically, what will happen as we make them larger and larger.

By my lights, everyone else seems either too impressed/scared by the concept of LM scaling, or not impressed/scared enough.

On LessWrong and related communities, I see lots of people worrying in earnest about whether the first superhuman AGI will be a GPT-like model. Both here and in the wider world, people often talk about GPT-3 like it's a far "smarter" being that it seems to me.

On the other hand, the people who aren't scared often don't seem like they're even paying attention. Faced with a sudden leap in machine capabilities, they shrug. Faced with a simple recipe that can make those machines even better -- with eerie, physics-like regularity -- they . . . still shrug. I wrote about the most infamous of these detractors here.

Meanwhile, I'm here in the middle. What do I think? Something like:

The newer (i.e. large transformer) LMs really are a huge advance in NLP over the prior state of the art

The prior state of the art was really bad, though. Before the new LMs, neural nets simply couldn't "do" language the way they could "do" images, something I noted back in 2017.

Most of the "huge advance" happened in the smallest of the new models, like BERT-Base and GPT-2-small.

The effect of scaling up these models is mostly to "de-noise" capabilities already evident in the small ones. It makes their strengths more robust and easier to access, but doesn't add fundamentally new strengths.

The larger language models of the future will be highly impactful, but banal.

They will probably allow us to fully automate all the routine linguistic tasks you could almost imagine automating with GPT-3.

People will make wonderful new things using them.

They won't be "smart" in any way that GPT-3 is not, or indeed, really in any way that GPT-2 was not.

They will get better at abstract reasoning -- in the sense that it will be easier to get them to spit out text that sounds like it is the product of abstract reasoning. (As even GPT-2 does frequently.) They will be weak at this relative to their other capabilities, as they are today, and little will come of it.

They might end up as sub-systems in an AGI one day.

The rest of the post will consist of some gestures where I try to make the above feel as natural to you as it does to me.

2. the enthusiast's argument

First, let's spell out the argument that has people thinking GPT wil...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: larger language models may disappoint you [or, an eternally unfinished draft], published by nostalgebraist on November 26, 2021 on LessWrong.

what this post is

The following is an incomplete draft, which I'm publishing now because I am unlikely to ever finish writing it.

I no longer fully endorse all the claims in the post. (In a few cases, I've added a note to say this explicitly.) However, there are some arguments in the post that I still endorse, and which I have not seen made elsewhere.

This post is the result of me having lots of opinions about LM scaling, at various times in 2021, which were difficult to write down briefly or independently of one another. This post, originally written in July 2021, is the closest I got to writing them all down in one place.

-nost, 11/26/21

0. caveat

This post will definitely disappoint you.

Or, anyway, it will definitely disappoint me. I know that even though I haven't written it yet.

My drafts folder contains several long, abandoned attempts to write (something like) this post. I've written (something like) this post many times in my head. I just can't seem to get it right, though. The drafts always sprawl out of control.

So, if I can't do it right, why not do it wrong? Here's the disorganized, incomplete, brain-dump version of the better post I wish I were writing. Caveat lector.

1. polarization

The topic of this post is large language models (LMs) like GPT-3. Specifically, what will happen as we make them larger and larger.

By my lights, everyone else seems either too impressed/scared by the concept of LM scaling, or not impressed/scared enough.

On LessWrong and related communities, I see lots of people worrying in earnest about whether the first superhuman AGI will be a GPT-like model. Both here and in the wider world, people often talk about GPT-3 like it's a far "smarter" being that it seems to me.

On the other hand, the people who aren't scared often don't seem like they're even paying attention. Faced with a sudden leap in machine capabilities, they shrug. Faced with a simple recipe that can make those machines even better -- with eerie, physics-like regularity -- they . . . still shrug. I wrote about the most infamous of these detractors here.

Meanwhile, I'm here in the middle. What do I think? Something like:

The newer (i.e. large transformer) LMs really are a huge advance in NLP over the prior state of the art

The prior state of the art was really bad, though. Before the new LMs, neural nets simply couldn't "do" language the way they could "do" images, something I noted back in 2017.

Most of the "huge advance" happened in the smallest of the new models, like BERT-Base and GPT-2-small.

The effect of scaling up these models is mostly to "de-noise" capabilities already evident in the small ones. It makes their strengths more robust and easier to access, but doesn't add fundamentally new strengths.

The larger language models of the future will be highly impactful, but banal.

They will probably allow us to fully automate all the routine linguistic tasks you could almost imagine automating with GPT-3.

People will make wonderful new things using them.

They won't be "smart" in any way that GPT-3 is not, or indeed, really in any way that GPT-2 was not.

They will get better at abstract reasoning -- in the sense that it will be easier to get them to spit out text that sounds like it is the product of abstract reasoning. (As even GPT-2 does frequently.) They will be weak at this relative to their other capabilities, as they are today, and little will come of it.

They might end up as sub-systems in an AGI one day.

The rest of the post will consist of some gestures where I try to make the above feel as natural to you as it does to me.

2. the enthusiast's argument

First, let's spell out the argument that has people thinking GPT wil...