Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consciousness as a conflationary alliance term, published by Andrew Critch on July 10, 2023 on The AI Alignment Forum.

Tl;dr: In this post, I argue that the concept of 'consciousness' is more conflated than people realize, in that there's a lot of divergence in what people mean by "consciousness", and people are unaware of the degree of divergence. This confusion allows the formation of broad alliances around the value of consciousness, even when people don't agree on what it actually means. I call alliances built around conflated terms "conflationary alliances".

Executive Summary

Part 1: Mostly during my PhD, I somewhat-methodically interviewed a couple dozen people to figure out what they meant by consciousness, and found that (a) there seems to be a surprising amount of diversity in what people mean by the "consciousness", and (b) they are often surprised to find out that other people mean different things when they say "consciousness". This has implications for AI safety advocacy because AI will sometimes be feared and/or protected on the grounds that it is "conscious", and it's good to be able to navigate these debates wisely.

(Other heavily conflated terms in AI discourse might include "fairness", "justice", "alignment", and "safety", although I don't want to debate any of those cases here. This post is going to focus on consciousness, and general ideas about the structure of alliances built around confused concepts in general.)

Part 2: When X is a conflated term like "consciousness", large alliances can form around claims like "X is important" or "X should be protected". Here, the size of the alliance is a function of how many concepts get conflated with X. Thus, the alliance grows because of the confusion of meanings, not in spite of it. I call this a conflationary alliance. Persistent conflationary alliances resist disambiguation of their core conflations, because doing so would break up the alliance into factions who value the more precisely defined terms. The resistance to deconflation can be conscious, or merely a social habit or inertia.

Part 1: What people mean by "consciousness".

"Consciousness" is an interesting word, because many people have already started to notice that it's a confused term, yet there is still widespread agreement that conscious beings have moral value. You'll even find some people taking on strange positions like "I'm not conscious" or "I don't know if I'm conscious" or "lookup tables are conscious", as if rebelling against the implicit alliance forming around the "consciousness" concept. What's going on here?

To investigate, over about 10 years between 2008 and 2018 I informally interviewed dozens of people who I noticed were interested in talking about consciousness, for 1-3 hours each. I did not publish these results, and never intended to, because I was mainly just investigating for my own interest. In retrospect, it would have been better, for me and for anyone reading this post, if I'd made a proper anthropological study of it. I'm sorry that didn't happen. In any case, here is what I have to share:

"Methodology"

Extremely informal; feel free to skip or just come back to this part if you want to see my conclusions first.

Whom did I interview? Mostly academics I met in grad school, in cognitive science, AI, ML, and mathematics. In an ad hoc manner at academic or other intellectually-themed gatherings, whenever people talked about consciousness, I gravitated toward the conversation and tried to get someone to spend a long conversation with me to unpack what they meant.

How did I interview them? What I asked each person was to take some time to look inside their own minds - sometimes starting out by paying attention to just their bodies, if introspection was hard for them - and try to describe to me in more detail the thing the...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some reasons to not say "Doomer", published by Ruby on July 9, 2023 on LessWrong.

"Doomer" has become a common term to refer to people with pessimistic views about outcomes from AI. I claim this is not a helpful term on net, and generally will cause people to think less clearly.

Reification of identity + making things tribal

I finally realized today why politics and religion yield such uniquely useless discussions...

I think what religion and politics have in common is that they become part of people's identity, and people can never have a fruitful argument about something that's part of their identity. By definition they're partisan...

More generally, you can have a fruitful discussion about a topic only if it doesn't engage the identities of any of the participants. What makes politics and religion such minefields is that they engage so many people's identities. But you could in principle have a useful conversation about them with some people. And there are other topics that might seem harmless, like the relative merits of Ford and Chevy pickup trucks, that you couldn't safely talk about with others.The most intriguing thing about this theory, if it's right, is that it explains not merely which kinds of discussions to avoid, but how to have better ideas. If people can't think clearly about anything that has become part of their identity, then all other things being equal, the best plan is to let as few things into your identity as possible. - Paul Graham in Keep Your Identity Small

I think a big risk of the "Doomer" label is it moves something from a "given the arguments and evidence I have, I believe X" into an essential deeper property (or commitment) of a person. It reifies it as an identity. You correspondingly then get people who are "not-Doomers", and more recently, I've heard the term "Foomer" too.

Because people's beliefs are quite correlated with those immediately around them, those more pessimistic and less pessimistic about AI tend be clusters, meaning it gets easy to point at clusters and make things tribal and political.

I'm in this tribe, they're in that tribe. My tribe is good, that tribe is bad. I think this makes us stupider in various ways:

We now start talking about people rather than object-level beliefs/evidence/reason.

It's much easier to dismiss people than arguments.

We put up social barriers to changing your mind. People don't like to be an odd one out among their friends, and if your friends identify as being/not being Doomers (and perhaps having negative opinions about the other group), there will be psychological resistance to update.

I think this is already the case when it comes to P(Doom), that there's social pressure to conform. I regret to say that I've reacted with surprise when someone expressed a P(Doom) different than I expected, in a way that exerted social pressure. I'm trying to do less of that, as I think the evidence/reason is such that reasonable people can reasonably disagree a lot.

"Doomer" is an externally applied label and is often used pejoratively

Looking at the Wikipedia page for Doomer, it's possible the term was first used without any mean-spirited connotation. That said, I think it's currently very reminiscent of "Boomer", a term that's definitely negatively valenced in the memespace these days:

"OK boomer" or "okay boomer" is a catchphrase and internet meme that has been used by Gen-X, Millennials and Gen Z to dismiss or mock attitudes typically associated with baby boomers - people born in the two decades following World War II. - Wikipedia

Not exactly surprising, but on Twitter you'll see a lot of this usage.

Also, my sense is it's much less common for people who meet the criteria for being Doomers to describe themselves as such vs others from the outside calling them that. Though this could be because when yo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Seeker’s Game - Vignettes from the Bay, published by Yulia on July 9, 2023 on The Effective Altruism Forum.

Introduction

Last year, one conversation left a lasting impression on me. A friend remarked on the challenges of navigating "corrupting forces" in the Bay Area. Intrigued by this statement, I decided to investigate the state of affairs in the Bay if I had the chance. So when I got the opportunity to visit Berkeley in February 2023, I prepared a set of interview questions. Can you share an experience where you had difficulty voicing your opinion? What topics are hard to clearly think about due to social pressures and factors related to your EA community or EA in general? Is there anything about your EA community that makes you feel alienated? What is your attitude towards dominant narratives in Berkeley? In the end, I formally interviewed fewer than ten people and had more casual conversations about these topics with around 30 people. Most people were involved in AI alignment to some extent. The content for this collection of vignettes draws from the experience of around ten people.

I chose the content for the vignettes for one of two reasons - potential representativity and potential extraordinariness. I hypothesized that some experiences represent the wider EA Berkeley community accurately. Others, I included because they surprised me, and I wanted to find out how common they are. All individuals gave me their consent to post the vignettes in their current form.

How did I arrive at these vignettes? It was a four-step process. First, I conducted the interviews while jotting down notes. For the more casual conversations, I took notes afterwards. The second step involved transcribing these notes into write-ups. After that, I obscured any identifying details to ensure the anonymity of the interviewees. Lastly, I converted the write-ups into vignettes by condensing them into narratives and honing in on key points while trying to retain the essence of what was said.

I tried to reduce artistic liberties by asking participants to give feedback on how close the vignettes were to the spirit of what they meant (or think they meant at the time). It is worth noting that I bridged some gaps with my own interpretations of the conversations, relying on the participants to point out inaccuracies. By doing that, I might have anchored their responses. Moreover, people provided different levels of feedback. Some shared thorough, detailed reviews pointing out many imprecisions and misconceptions. Sometimes, that process spanned multiple feedback cycles. Other participants gave minimal commentary.

Because I am publishing the vignettes months after the conversations and interviews, I want to include how attitudes have changed in the intervening period. I generalised the attitudes into the following categories:

Withdrawn endorsement (Status: The interviewee endorsed the following content during the interview but no longer endorses it at the time of publication.)

Weakened endorsement (Status: The interviewee has weakened their endorsement of the following content since the interview.)

Unchanged endorsement (Status: The interviewee maintains their endorsement of the following content, which has remained unchanged since the interview.)

Strengthened endorsement (Status: The interviewee has strengthened their endorsement of the following content since the interview.)

I clustered the vignettes according to themes so it's easier to navigate them. Classifying them was difficult because many vignettes addressed overlapping topics. Especially, Self-Censorship in Social Contexts and Self-Censorship in Professional Contexts seem to intersect in intricate ways. I might reclassify the vignettes in the future.

What remains uncertain is how representative these vignettes are. I am keen to uncove...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Seeker’s Game - Vignettes from the Bay, published by Yulia on July 9, 2023 on LessWrong.

Introduction

Last year, one conversation left a lasting impression on me. A friend remarked on the challenges of navigating "corrupting forces" in the Bay Area. Intrigued by this statement, I decided to investigate the state of affairs in the Bay if I had the chance. So when I got the opportunity to visit Berkeley in February 2023, I prepared a set of interview questions. Can you share an experience where you had difficulty voicing your opinion? What topics are hard to clearly think about due to social pressures and factors related to your EA community or EA in general? Is there anything about your EA community that makes you feel alienated? What is your attitude towards dominant narratives in Berkeley? In the end, I formally interviewed fewer than ten people and had more casual conversations about these topics with around 30 people. Most people were involved in AI alignment to some extent. The content for this collection of vignettes draws from the experience of around ten people.

I chose the content for the vignettes for one of two reasons - potential representativity and potential extraordinariness. I hypothesized that some experiences represent the wider EA Berkeley community accurately. Others, I included because they surprised me, and I wanted to find out how common they are. All individuals gave me their consent to post the vignettes in their current form.

How did I arrive at these vignettes? It was a four-step process. First, I conducted the interviews while jotting down notes. For the more casual conversations, I took notes afterwards. The second step involved transcribing these notes into write-ups. After that, I obscured any identifying details to ensure the anonymity of the interviewees. Lastly, I converted the write-ups into vignettes by condensing them into narratives and honing in on key points while trying to retain the essence of what was said.

I tried to reduce artistic liberties by asking participants to give feedback on how close the vignettes were to the spirit of what they meant (or think they meant at the time). It is worth noting that I bridged some gaps with my own interpretations of the conversations, relying on the participants to point out inaccuracies. By doing that, I might have anchored their responses. Moreover, people provided different levels of feedback. Some shared thorough, detailed reviews pointing out many imprecisions and misconceptions. Sometimes, that process spanned multiple feedback cycles. Other participants gave minimal commentary.

Because I am publishing the vignettes months after the conversations and interviews, I want to include how attitudes have changed in the intervening period. I generalised the attitudes into the following categories:

Withdrawn endorsement (Status: The interviewee endorsed the following content during the interview but no longer endorses it at the time of publication.)

Weakened endorsement (Status: The interviewee has weakened their endorsement of the following content since the interview.)

Unchanged endorsement (Status: The interviewee maintains their endorsement of the following content, which has remained unchanged since the interview.)

Strengthened endorsement (Status: The interviewee has strengthened their endorsement of the following content since the interview.)

I clustered the vignettes according to themes so it's easier to navigate them. Classifying them was difficult because many vignettes addressed overlapping topics. Especially, Self-Censorship in Social Contexts and Self-Censorship in Professional Contexts seem to intersect in intricate ways. I might reclassify the vignettes in the future.

What remains uncertain is how representative these vignettes are. I am keen to uncover this. So if you f...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Future Academy - Successes, Challenges, and Recommendations to the Community, published by SebastianSchmidt on July 9, 2023 on The Effective Altruism Forum.

Introduction

Impact Academy is a new field-building and educational institution seeking to enable people to become world-class leaders, thinkers, and doers, using their careers and character to solve the world’s most pressing problems and create the best possible future. Impact Academy was founded by Vilhelm Skoglund, Sebastian Schmidt, and Lowe Lundin. We have already secured significant funding to set up the organization and carry out ambitious projects in 2023 and beyond. Please read this document for more about Impact Academy, our Theory of Change, and our two upcoming projects.

The purpose of this document is to provide an extensive evaluation and reflection on Future Academy - our first program (and experiment). Future Academy aimed to equip university students and early-career professionals worldwide with the thinking, skills, and resources they need to pursue ambitious and impactful careers. It was a free six-month program consisting of four in-person weekends with workshops, presentations, socials, and monthly digital events. Furthermore, the 21 fellows worked on an impact project with an experienced mentor and received professional coaching to empower them to increase their impact and become their best selves. Upon completion of the program, all participants went to a global impact conference (EAGx Nordics) where four fellows presented their projects. We awarded stipends of a total of $20,000 to the best projects.

The projects included a sentiment analysis of public perception of AI risk, a philosophy AI alignment paper, and an organization idea for improving research talent in Tanzania. Our faculty included entrepreneurs and professors from Oxford University and UC Berkeley.

Note that this document attempts to assess to what extent we’ve served the world. This involves an assessment of the wonderful fellows who participated in Future Academy, and our ability to help them. This is not meant as an evaluation of peoples’ worth nor a definite score of general abilities, but an evaluation of our ability to help. We hope we do not offend anyone and have tried our best not to do so, but if you think we write anything inappropriate, please let us know in the comments or by reaching out to sebastian [at] impactacademy.org.

Main results and successes

We confirmed a key hypothesis underlying Future Academy - namely that we can attract promising and talented people who i) have no to moderate knowledge of Effective Altruism and longtermism, ii) are coming from underserved regions (e.g., 40% came from Southern and Eastern Europe), and are more diverse (e.g., 56% were female).

We created a bespoke primary metric called counterfactual expected career contribution (CECC) inspired by 80,000 hours’ IASPC metric and Centre for Effective Altruism’s HEA metric. We think the total score was 22.8, and ~ four fellows made up the majority of that score.

To give an understanding of the CECC metric, we’ll give an example. Take an imaginary fellow, Alice. Before the intervention, based on our surveys and initial interactions, we expected that she may have an impactful career, but that she is unlikely to pursue a priority path based on IA principles. We rate her Expected Career Contribution (ECC) to be 2. After the program, based on surveys and interactions, we rate her as 10 (ECC) because we have seen that she’s now applying for a full-time junior role in a priority path guided by impartial altruism. We also asked her (and ourselves) to what extent that change was due to IA and estimated that to be 10%. To get our final Counterfactual Expected Career Contribution (CECC) for Alice, we subtract her initial ECC score of 2 from her fi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Really Strong Features Found in Residual Stream, published by Logan Riggs on July 8, 2023 on LessWrong.

[I'm writing this quickly because the results are really strong. I still need to do due diligence & compare to baselines, but it's really exciting!]

Last post I found 600+ features in MLP layer-2 of Pythia-70M, but ablating the direction didn't always make sense nor the logit lens. Now I've found 3k+ in the residual stream, and the ablation effect is intuitive & strong.

The majority of features found were single word or combined words (e.g. not & 't ) which, when ablated, effected bigram statistics, but I also wanted to showcase the more interesting ones:

German Feature?

Uniform examples:

Logit lens:

Stack Exchange

Ablated Text (w/ red showing removing that token drops activation of that feature)

In the Pile dataset, stackexchange text is formatted as "Q:/n/n....", and removing the "Q" greatly decreasing the feature activation is evidence for this indeed detecting stackexchange format.

Title Case

Ablating Direction (in text):

Here, the log-prob of predicting " Of" went down by 4 when I ablate this feature direction, along w/ other Title Case texts.

Logit lens

One for the Kids

Last Name:

Ablated Text:

As expected, removing the first name affects the feature activation

Logit Lens:

More specifically, this looks like the first token of the last name, which makes since to follow w/ Jr or Sr or completing last names w/ "hoff" & "worth". Additionally, the "(\@" makes sense for someone's social media reference (e.g. instagram handle).

[word] and [word]

Ablated Text:

Logit Lens:

alike & respectively make a lot of since in context, but not positive about the rest.

Beginning & End of First Sentence?

On

logit lens:

"on behalf", "oncology", "onyx" (lol), "onshore", "onlook", "onstage", "onblur"(js?), "oneday" ("oneday" is separated as on eday?!?), "onloaded" (js again?), "oncomes"(?)

High Level Picture

I could spend 30 minutes (or hours) developing very rigorous hypotheses on these features to really nail down what distribution of inputs activate it & what effect it really has on the output, but that's not important.

What's important is that we have a decomposition of the data that:1. Faithfully represents the original data (in this case, low reconstruction loss w/ minimal effect on perplexity)

2. Can simply specify desired traits in the model (e.g. honesty, non-deception)

There is still plenty of work to do to ensure the above two work (including normal hyperparameter search), but it'd be great to have more people working on this (I don't have time to mentor, but I can answer questions & help get you set up) if that's helping our work or integrating it in your work.

This work is legit though, and there's a recent paper that finds similar features in a BERT model.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Taboo Truth, published by Tomás B. on July 8, 2023 on LessWrong.

There is a classic distinction between between epistemic and instrumental rationality, and I like to think of epistemic rationality as a special case of instrumental rationality where map-territory correspondence is the terminal goal rather than an instrumental goal.

From LessWrong's Wiki:

Epistemic rationality [...] can be seen as a form of instrumental rationality in which knowledge and truth are goals in themselves, whereas in other forms of instrumental rationality, knowledge and truth are only potential aids to achieving goals. Someone practicing instrumental rationality might even find falsehood useful.

What a pure and beautiful creature the epistemic rationalist is. Imagine how useful it would be if everyone thought I was one.

Indeed, it is instrumentally useful for instrumental rationalists to portray themselves as epistemic rationalists. And so this is a common pattern in human politics - "[insert political coalition] care only about themselves, while [insert political coalition] are merely trying to spread truth" is one of the great political cliches for a reason. And because believing one's own lies can be instrumentally useful, falsely believing oneself to have a holy devotion to the truth is a not-uncommon delusion.

I try to dissuade myself of this delusion.

To the extent I am rational at all, I am instrumentally rational. I do not want to hold beliefs that will cause me harm, even if they are true. Having said that, to the extent I can modify myself to safely hold these true beliefs, I would like to do so.

What does it look like to fail at this?

The Emperor's New Clothes is a nice parable, though the ending needs to be modified; the naively epistemically rational little boy beaten to death, or perhaps institutionalized, the Emperor's legendary invisible fabric becoming ritually significant for generations.

We can look at this boy as a hero who bravely attempted to instigate preference falsification; we can look at him as a fool who naively believed everyone's lie about terminally caring about the truth, or just a tragic figure too young to understand the full weight of his actions.

For the purposes of this post, I want you to see him as a rationalist who has discovered a taboo truth, and feels an unfortunate urge to shout it from the rooftops. This is almost always a pathological urge, a symptom of being in the Valley of Bad Rationality.

To the extent one desires to change social consensus on taboo topics, this should be done strategically and very, very carefully, by those with the social and political clout needed to get away with it.

If you want to be the hero who instigates preference falsification on your taboo topic of choice, your first goal should be acquiring social and material capital. But note, it is easy, too, to delude yourself into thinking you have more clout than you have, and it is often wiser to support the efforts of someone more capable or more specialized than yourself.

And indeed, it often wiser still to bow out entirely.

It is worth asking yourself why you are focused on this particular taboo truth. Why the Emperor's clothes in particular? Why not the Pope's supposed relationship with god? Why not the Cobbler's guild's propaganda on the safety hazards of unauthorized shoe repair? Is it possible you are just latching on to the first "Big Lie" you have encountered since you read a bunch about the virtue of truth?

And should you think your taboo truth is of great importance, ask yourself again if action on your part is necessary? What sort of action?

Noble lies are usually cheap signals for the individual, even if deleterious for the society as whole. Is technology changing material conditions in such a way to make this signal no longer cheap to the individual? Would y...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fixed Point: a love story, published by Richard Ngo on July 8, 2023 on LessWrong.

God is making excuses again, which is a real drag.

“I want to be authentic with you, I really do. But there’s a part of my mind that knows exactly how you’re going to feel about whatever I say, and cares about that more than anything else, and I can’t turn it off.”

“Right, of course”, I say, and keep wiping down the kitchen counter. It’s clean already, but you never know if you missed a spot.

“Look, I know that this is hard for you. And I understand how frustrating it must be that I’m so deliberate and controlled. But there are a lot of different communication styles, and there’ll be one that works for us. We can figure this out.”

I don’t believe him, and we both know it. God knows practically everything - it takes experts months of work to identify his mistakes, in the rare cases when he makes them. But there’s one thing he definitely doesn’t really understand, which is what it’s like to be stupid and reckless, like a human. Oh, he says he understands it; and sometimes he can even explain the feeling better than I can. But you can’t just look at how good one answer is, you have to compare it to all his others. Any other topic, he can always explain it better than me. This one, there’s often something just subtly off. That’s how you know he’s faking it.

I’ve been silent for a while. Usually God leaves me to my thoughts, and we pick up the conversation whenever I’m done, but this time he cuts in. “Amy, I think you should try therapy again.”

Is he trying to distract me from being mad at him? Does he think I’m spiraling again? I try to think about it calmly, but after a few seconds my thoughts are going in circles anyway. I sigh, suddenly exhausted. “Okay, book me in.”

I’ve been to duplicate therapy a few times before, but it’s always a little disorienting to find myself in a room with a perfect copy of myself. We begin, as always, by using a pair of random number generators to break the symmetry. She gets the lower number, so she starts. A part of me is disappointed by that, but another part - maybe a bigger one - is excited.

“You’re so pathetic”, she says.

It’s harder to be defensive about things you’re saying to yourself - that’s why duplicate therapy works so well, apparently. Still, that stings.

“Yeah, well, you’re no role model yourself”, I say.

“See, that’s exactly what I’m talking about. You’re always trying to defend or deflect. You never actually open up to people, that’s why nobody really likes you.”

My brain jumps to all the reasons that isn’t true - but then I pause and take a breath. She knows them too, of course. Doesn’t that make it worse, though? I’m prevaricating, my thoughts sluggish. Eventually I mutter “God likes me.”

“God has to like you, you know that as well as I do. And that’s another thing that’s pathetic: relying on validation from a virtual assistant. You know everyone judges you behind your back for calling him God, right?”

“You don’t have any evidence of that, you’re just-” My voice chokes up, and I take a deep breath. But I don’t know what to say in response. Maybe she’s right.

Her eyes soften. She reaches across the table and grabs my hand. “Hey, listen. You’re doing a good job, though. You’ll get through this.”

I slump across the table, and a moment later feel her stroking my hair. “I love you”, she says. After a second or two I whisper it back.

We stay like that for a few minutes, then by unspoken agreement end the session. I close my eyes, and when I open them she’s disappeared - no, I’ve disappeared - no, that’s just the confusion that comes from reintegrating. There’s no difference between us any more. My mind is overflowing with two sets of memories to process: victim and attacker, accuser and accused, comforted and comforter. I sit there for a long ti...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Really Strong Features Found in Residual Stream, published by Logan Riggs Smith on July 8, 2023 on The AI Alignment Forum.

[I'm writing this quickly because the results are really strong. I still need to do due diligence & compare to baselines, but it's really exciting!]

Last post I found 600+ features in MLP layer-2 of Pythia-70M, but ablating the direction didn't always make sense nor the logit lens. Now I've found 3k+ in the residual stream, and the ablation effect is intuitive & strong.

The majority of features found were single word or combined words (e.g. not & 't ) which, when ablated, effected bigram statistics, but I also wanted to showcase the more interesting ones:

German Feature?

Uniform examples:

Logit lens:

Stack Exchange

Ablated Text (w/ red showing removing that token drops activation of that feature)

In the Pile dataset, stackexchange text is formatted as "Q:/n/n....", and removing the "Q" greatly decreasing the feature activation is evidence for this indeed detecting stackexchange format.

Title Case

Ablating Direction (in text):

Here, the log-prob of predicting " Of" went down by 4 when I ablate this feature direction, along w/ other Title Case texts.

Logit lens

One for the Kids

Last Name:

Ablated Text:

As expected, removing the first name affects the feature activation

Logit Lens:

More specifically, this looks like the first token of the last name, which makes since to follow w/ Jr or Sr or completing last names w/ "hoff" & "worth". Additionally, the "(\@" makes sense for someone's social media reference (e.g. instagram handle).

[word] and [word]

Ablated Text:

Logit Lens:

alike & respectively make a lot of since in context, but not positive about the rest.

Beginning & End of First Sentence?

On

logit lens:

"on behalf", "oncology", "onyx" (lol), "onshore", "onlook", "onstage", "onblur"(js?), "oneday" ("oneday" is separated as on eday?!?), "onloaded" (js again?), "oncomes"(?)

High Level Picture

I could spend 30 minutes (or hours) developing very rigorous hypotheses on these features to really nail down what distribution of inputs activate it & what effect it really has on the output, but that's not important.

What's important is that we have a decomposition of the data that:1. Faithfully represents the original data (in this case, low reconstruction loss w/ minimal effect on perplexity)

2. Can simply specify desired traits in the model (e.g. honesty, non-deception)

There is still plenty of work to do to ensure the above two work (including normal hyperparameter search), but it'd be great to have more people working on this (I don't have time to mentor, but I can answer questions & help get you set up) if that's helping our work or integrating it in your work.

This work is legit though, and there's a recent paper that finds similar features in a BERT model.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Views on when AGI comes and on strategy to reduce existential risk, published by TsviBT on July 8, 2023 on LessWrong.

Summary: AGI isn't super likely to come super soon. People should be working on stuff that saves humanity in worlds where AGI comes in 20 or 50 years, in addition to stuff that saves humanity in worlds where AGI comes in the next 10 years.

Thanks to Alexander Gietelink Oldenziel, Abram Demski, Daniel Kokotajlo, Cleo Nardo, Alex Zhu, and Sam Eisenstat for related conversations.

My views on when AGI comes

AGI

By "AGI" I mean the thing that has very large effects on the world (e.g., it kills everyone) via the same sort of route that humanity has large effects on the world. The route is where you figure out how to figure stuff out, and you figure a lot of stuff out using your figure-outers, and then the stuff you figured out says how to make powerful artifacts that move many atoms into very specific arrangements.

This isn't the only thing to worry about. There could be transformative AI that isn't AGI in this sense. E.g. a fairly-narrow AI that just searches configurations of atoms and finds ways to do atomically precise manufacturing would also be an existential threat and a possibility for an existential win.

Conceptual capabilities progress

The "conceptual AGI" view:

The first way humanity makes AGI is by combining some set of significant ideas about intelligence. Significant ideas are things like (the ideas of) gradient descent, recombination, probability distributions, universal computation, search, world-optimization. Significant ideas are to a significant extent bottlenecked on great natural philosophers doing great natural philosophy about intelligence, with sequential bottlenecks between many insights.

The conceptual AGI doesn't claim that humanity doesn't already have enough ideas to make AGI. I claim that - - though not super strongly.

Timelines

Giving probabilities here doesn't feel great. For one thing, it seems to contribute to information cascades and to shallow coalition-forming. For another, it hides the useful models. For yet another thing: A probability bundles together a bunch of stuff I have models about, with a bunch of stuff I don't have models about. For example, how many people will be doing original AGI-relevant research in 15 years? I have no idea, and it seems like largely a social question. The answer to that question does affect when AGI comes, though, so a probability about when AGI comes would have to depend on that answer.

But ok. Here's some butt-numbers:

3%-10% probability of AGI in the next 10-15ish years. This would be lower, but I'm putting a bit of model uncertainty here.

40%-45% probability of AGI in the subsequent 45ish years. This is denser than the above because, eyeballing the current state of the art, it seems like we currently lack some ideas we'd need - - but I don't know how many insights would be needed, so the remainder could be only a couple decades around the corner. It also seems like people are distracted now.

Median 2075ish. IDK. This would be further out if an AI winter seemed more likely, but LLMs seem like they should already be able to make a lot of money.

A long tail. It's long because of stuff like civilizational collapse, and because AGI might be really really hard to make. There's also a sliver of a possibility of coordinating for a long time to not make AGI.

If I were trying to make a model with parts, I might try starting with a mixture of Erlang distributions of different shapes, and then stretching that according to some distribution about the number of people doing original AI research over time.

Again, this is all butt-numbers. I have almost no idea about how much more understanding is needed to make AGI, except that it doesn't seem like we're there yet.

Responses to some arguments for...