Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quick general thoughts on suffering and consciousness, published by Rob Bensinger on October 30, 2021 on LessWrong.

(Gathering these thoughts in one place for more discussion.)

1. Pain signaling

In September 2019, I wrote on my LW shortform:

Rolf Degen, summarizing part of Barbara Finlay's "The neuroscience of vision and pain":

Humans may have evolved to experience far greater pain, malaise and suffering than the rest of the animal kingdom, due to their intense sociality giving them a reasonable chance of receiving help.

From the paper:

Several years ago, we proposed the idea that pain, and sickness behaviour had become systematically increased in humans compared with our primate relatives, because human intense sociality allowed that we could ask for help and have a reasonable chance of receiving it. We called this hypothesis ‘the pain of altruism’ [68]. This idea derives from, but is a substantive extension of Wall’s account of the placebo response [43]. Starting from human childbirth as an example (but applying the idea to all kinds of trauma and illness), we hypothesized that labour pains are more painful in humans so that we might get help, an ‘obligatory midwifery’ which most other primates avoid and which improves survival in human childbirth substantially ([67]; see also [69]). Additionally, labour pains do not arise from tissue damage, but rather predict possible tissue damage and a considerable chance of death.

Pain and the duration of recovery after trauma are extended, because humans may expect to be provisioned and protected during such periods. The vigour and duration of immune responses after infection, with attendant malaise, are also increased. Noisy expression of pain and malaise, coupled with an unusual responsivity to such requests, was thought to be an adaptation.

We noted that similar effects might have been established in domesticated animals and pets, and addressed issues of ‘honest signalling’ that this kind of petition for help raised. No implication that no other primate ever supplied or asked for help from any other was intended, nor any claim that animals do not feel pain. Rather, animals would experience pain to the degree it was functional, to escape trauma and minimize movement after trauma, insofar as possible.

Finlay's original article on the topic: "The pain of altruism".

[Epistemic status: Thinking out loud]

If the evolutionary logic here is right, I'd naively also expect non-human animals to suffer more to the extent they're (a) more social, and (b) better at communicating specific, achievable needs and desires.

There are reasons the logic might not generalize, though. Humans have fine-grained language that lets us express very complicated propositions about our internal states. That puts a lot of pressure on individual humans to have a totally ironclad, consistent "story" they can express to others. I'd expect there to be a lot more evolutionary pressure to actually experience suffering, since a human will be better at spotting holes in the narratives of a human who fakes it (compared to, e.g., a bonobo trying to detect whether another bonobo is really in that much pain).

It seems like there should be an arms race across many social species to give increasingly costly signals of distress, up until the costs outweigh the amount of help they can hope to get. But if you don't have the language to actually express concrete propositions like "Bob took care of me the last time I got sick, six months ago, and he can attest that I had a hard time walking that time too", then those costly signals might be mostly or entirely things like "shriek louder in response to percept X", rather than things like "internally represent a hard-to-endure pain-state so I can more convincingly stick to a verbal narrative going forward about how hard-to-end...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tell the Truth, published by lsusr on October 30, 2021 on LessWrong.

My favorite thing about living in the 21ˢᵗ century is that nobody can stop me from publishing whatever I want.

I mean this in the most literal sense. I can log onto a free blogging platform via Tor and no government in the world could deanonymize me. I might even do this already and there would be no way for you (or anybody else) to know.

Different countries have different levels of freedom. I have friends who live under authoritarian regimes who would literally be disappeared by the secret police if they declared their political views outright. They say "it's dangerous to express my views publicly" but they never say "I'm an independent thinker". The only people I know who claim they're scared to express independent thought live in the United States where freedom of the press is protected under the First Amendment of the Constitution.

People tell me they're worried of being cancelled by woke culture. I think this is just a convenient excuse for laziness and cowardice. What are you afraid of saying? That there are personality differences between men and women? Women consistently score higher than men in agreeableness and neroticism.

These findings cannot easily be attributed to self-report artifacts, as McCrae and colleagues (2005) have replicated them in observer reports of FFM traits across 50 cultures.

―Gender Differences in Five Factor Model Personality Traits in an Elderly Cohort: Extension of Robust and Surprising Findings to an Older Generation

Are you afraid to say that there are significant heritable intelligence disparities between ethnic groups? It's the obvious conclusion if you think critically about US immigration policy.

"87.2% of Indian-American adults in 2010 were foreign-born[1]." US immigration policy makes it very hard for natives of India to immigrate into the United States. It's so hard to get an H1-B visa that a young hardworking ambitious Indian data scientist who reads this blog[2] told me about how working for a US company was "unrealistic". If most Indian-American families immigrated recently and recent Indian immigrants to the United States are smarter than average Indian people then one of the following statements must be true.

Recently-immigrated Americans from India are smarter than average Americans.

Average Americans are smarter than average Indians.

Intelligence is more than 50% heritable. US immigration policy does not distinguish between heritable intelligence and non-heritable intelligence. If there is a difference in intelligence between these populations then at least 50% is likely to be heritable. I am willing to bet money that the average IQ of descendants of H1-B visa holders is (or will be) higher than the IQs of both the average American and the average Indian. I'd be happy to be proven wrong. I would prefer to live in a world without ethnic differences in IQ. But I do not believe that is the world I live in.

I would love to be proven wrong about the gender-personality connection too but that won't happen if I keep my conclusions to myself.

If you work for Cyberdyne then you should use an alias when you say bad things about Cyberdyne so you don't get fired. If you live under an authoritarian regime then you should behave appropriately. Neither excuse applies to most readers of this blog. Denying reality hurts social justice. Epistemic rationality is a prerequisite to instrumental rationality.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tell the Truth, published by lsusr on October 30, 2021 on LessWrong.

My favorite thing about living in the 21ˢᵗ century is that nobody can stop me from publishing whatever I want.

I mean this in the most literal sense. I can log onto a free blogging platform via Tor and no government in the world could deanonymize me. I might even do this already and there would be no way for you (or anybody else) to know.

Different countries have different levels of freedom. I have friends who live under authoritarian regimes who would literally be disappeared by the secret police if they declared their political views outright. They say "it's dangerous to express my views publicly" but they never say "I'm an independent thinker". The only people I know who claim they're scared to express independent thought live in the United States where freedom of the press is protected under the First Amendment of the Constitution.

People tell me they're worried of being cancelled by woke culture. I think this is just a convenient excuse for laziness and cowardice. What are you afraid of saying? That there are personality differences between men and women? Women consistently score higher than men in agreeableness and neroticism.

These findings cannot easily be attributed to self-report artifacts, as McCrae and colleagues (2005) have replicated them in observer reports of FFM traits across 50 cultures.

―Gender Differences in Five Factor Model Personality Traits in an Elderly Cohort: Extension of Robust and Surprising Findings to an Older Generation

Are you afraid to say that there are significant heritable intelligence disparities between ethnic groups? It's the obvious conclusion if you think critically about US immigration policy.

"87.2% of Indian-American adults in 2010 were foreign-born[1]." US immigration policy makes it very hard for natives of India to immigrate into the United States. It's so hard to get an H1-B visa that a young hardworking ambitious Indian data scientist who reads this blog[2] told me about how working for a US company was "unrealistic". If most Indian-American families immigrated recently and recent Indian immigrants to the United States are smarter than average Indian people then one of the following statements must be true.

Recently-immigrated Americans from India are smarter than average Americans.

Average Americans are smarter than average Indians.

Intelligence is more than 50% heritable. US immigration policy does not distinguish between heritable intelligence and non-heritable intelligence. If there is a difference in intelligence between these populations then at least 50% is likely to be heritable. I am willing to bet money that the average IQ of descendants of H1-B visa holders is (or will be) higher than the IQs of both the average American and the average Indian. I'd be happy to be proven wrong. I would prefer to live in a world without ethnic differences in IQ. But I do not believe that is the world I live in.

I would love to be proven wrong about the gender-personality connection too but that won't happen if I keep my conclusions to myself.

If you work for Cyberdyne then you should use an alias when you say bad things about Cyberdyne so you don't get fired. If you live under an authoritarian regime then you should behave appropriately. Neither excuse applies to most readers of this blog. Denying reality hurts social justice. Epistemic rationality is a prerequisite to instrumental rationality.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: True Stories of Algorithmic Improvement, published by johnswentworth on October 29, 2021 on LessWrong.

In May 2020, OpenAI released a report on algorithmic efficiency improvements in deep learning. Main headline:

Compared to 2012, it now takes 44 times less compute to train a neural network to the level of AlexNet (by contrast, Moore’s Law would yield an 11x cost improvement over this period). Our results suggest that for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency.

A lot people were surprised by this; there’s a common narrative in which AI progress has come mostly from throwing more and more compute at relatively-dumb algorithms. (This is a common interpretation of The Bitter Lesson, though I would argue it is largely a misinterpretation.)

I’ve had various experiences over the years which made the result not-that-surprising. Algorithms beating compute is the sort of thing I expect by default, on a gut level. The point of this post is to tell a few of the stories which underlie that intuition, aimed especially toward people who don’t have much first-hand experience with software engineering, ML, or simulation. (There will still be some jargon, though.)

Disclaimer: this does not mean that you should put tons of confidence on this view. The goal is just to provide a possible lens through which “algorithmic progress has yielded more gains than classical hardware efficiency” makes sense; I want to raise that hypothesis from entropy. I’m not going to provide the sort of evidence which would justify very high confidence, I’m just going to point it out as a hypothesis to keep in the back of your mind, and update on when results like OpenAI’s come along.

Rewrite In C

Back in college, I spent a summer simulating an unusual type of biochemical oscillator, officially under the aegis of the Minnesota Supercomputing Institute. The algorithm was conceptually simple: every time a reaction occurs between two molecules, update the counts of each molecule, then randomly sample to figure out when the next reaction happens. (In a single cell, molecule counts are often small enough that we can simulate each individual reaction like this, at least for a particular reaction-type.) Early in the summer, I spent a few days studying the algorithm and coding up a simulation in python. In order to get decent statistics, we needed to run it a lot, so the professor overseeing the work recommended that I book some time on one of the supercomputer clusters.

I did not book time on the supercomputers. Instead, I spent another three days re-writing the algorithm in C and tweaking it a bit for speed. That sped it up by a factor of about a hundred, which was enough that I could get statistically significant results on my laptop in an hour. Now, that means I needed to sit around for an hour waiting for results whenever I changed something, but that’s still a lot faster than applying for a timeslot on the supercomputer!

I’ve also rewritten things in C outside of academia. At one company, we had a recommendation algorithm which gave great recommendations but took about a second to run. (It was an exponential-time Bayesian inference algorithm, but with only a handful of data points.) I rewrote it in C, and it went down to < 10 ms - fast enough that a user wouldn’t notice the page loading slowly.

Also: have you ever checked just how much faster numpy is, compared to naive matrix multiplication written in python?

I coded up a quick test just now, with two 1k by 1k random matrices. Numpy.dot ran in about 40ms (after warmup). My simple three-nested-for-loop ran in about 130-140 seconds (slow enough that warmup was not relevant). That’s a speedup factor of more than 3k. Now, that’s not just from running in C (or fortran); it’s also...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: True Stories of Algorithmic Improvement, published by johnswentworth on October 29, 2021 on LessWrong.

In May 2020, OpenAI released a report on algorithmic efficiency improvements in deep learning. Main headline:

Compared to 2012, it now takes 44 times less compute to train a neural network to the level of AlexNet (by contrast, Moore’s Law would yield an 11x cost improvement over this period). Our results suggest that for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency.

A lot people were surprised by this; there’s a common narrative in which AI progress has come mostly from throwing more and more compute at relatively-dumb algorithms. (This is a common interpretation of The Bitter Lesson, though I would argue it is largely a misinterpretation.)

I’ve had various experiences over the years which made the result not-that-surprising. Algorithms beating compute is the sort of thing I expect by default, on a gut level. The point of this post is to tell a few of the stories which underlie that intuition, aimed especially toward people who don’t have much first-hand experience with software engineering, ML, or simulation. (There will still be some jargon, though.)

Disclaimer: this does not mean that you should put tons of confidence on this view. The goal is just to provide a possible lens through which “algorithmic progress has yielded more gains than classical hardware efficiency” makes sense; I want to raise that hypothesis from entropy. I’m not going to provide the sort of evidence which would justify very high confidence, I’m just going to point it out as a hypothesis to keep in the back of your mind, and update on when results like OpenAI’s come along.

Rewrite In C

Back in college, I spent a summer simulating an unusual type of biochemical oscillator, officially under the aegis of the Minnesota Supercomputing Institute. The algorithm was conceptually simple: every time a reaction occurs between two molecules, update the counts of each molecule, then randomly sample to figure out when the next reaction happens. (In a single cell, molecule counts are often small enough that we can simulate each individual reaction like this, at least for a particular reaction-type.) Early in the summer, I spent a few days studying the algorithm and coding up a simulation in python. In order to get decent statistics, we needed to run it a lot, so the professor overseeing the work recommended that I book some time on one of the supercomputer clusters.

I did not book time on the supercomputers. Instead, I spent another three days re-writing the algorithm in C and tweaking it a bit for speed. That sped it up by a factor of about a hundred, which was enough that I could get statistically significant results on my laptop in an hour. Now, that means I needed to sit around for an hour waiting for results whenever I changed something, but that’s still a lot faster than applying for a timeslot on the supercomputer!

I’ve also rewritten things in C outside of academia. At one company, we had a recommendation algorithm which gave great recommendations but took about a second to run. (It was an exponential-time Bayesian inference algorithm, but with only a handful of data points.) I rewrote it in C, and it went down to < 10 ms - fast enough that a user wouldn’t notice the page loading slowly.

Also: have you ever checked just how much faster numpy is, compared to naive matrix multiplication written in python?

I coded up a quick test just now, with two 1k by 1k random matrices. Numpy.dot ran in about 40ms (after warmup). My simple three-nested-for-loop ran in about 130-140 seconds (slow enough that warmup was not relevant). That’s a speedup factor of more than 3k. Now, that’s not just from running in C (or fortran); it’s also...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Goodhart's Imperius, published by Duncan Sabien on October 29, 2021 on LessWrong.

Author's note: this essay was originally published pseudonymously in 2017. It's now being permanently rehosted at this link. I'll be rehosting a small number of other upper-quintile essays from that era over the coming weeks.

Claim 1: Goodhart’s Law is true.

Goodhart’s Law (which is incredibly appropriately named) reads “any measure which becomes a target ceases to be a good measure.” Another way to say this is “proxies are leaky,” i.e. the proxy never quite gets you the thing it was intended to get you.

If you want to be able to differentiate between promising math students and less-promising ones, you can try out a range of questions and challenges until you cobble together a test that the 100 best students (as determined by other assessments, such as teacher ratings) do well on, and the following 900 do worse on.

But as soon as you make that test into the test, it’s going to start leaking. In the tenth batch of a thousand students, the 100 best ones will still do quite well, but you’ll also get a bunch of people who don’t have the generalized math skill you're looking for, but who did get good at answering the specific, known questions. Your top 100 will no longer be composed only of the 100 actual-best math students—and things will just keep getting worse, over time.

This is analogous to what’s happened with Western diets and sugar. Prehistoric primates who happened to have a preference for sweet things (fruit) also happened to get a lot more vitamins and minerals and calories, and therefore they survived and thrived at higher rates than those sugar-ambivalent primates who failed to become our ancestors and died out. The process of natural selection turned a measure for nutrition (sweetness) into a target (a biologically hardwired "belief" that more sugar = more utility), which was fine right up until we learned to efficiently separate the sugar from the nutrients (teaching to the test) and discovered that our preferences were hardwired to the proxy rather than to the Actual Good Thing.

Claim 2: When attempting to do operant conditioning with a given reward or punishment, for any desired strength-of-conditioning-effect, there exists a sufficiently small delay between behavior and consequence that will produce that effect.

This one is not literally true. In order for it to be literally true, the hyperbolic nature of discounting (such that closer rewards are disproportionately more effective in creating reinforcement) would have to extend off to absurdity such that an infinitesimally small reward or response could produce an arbitrarily large conditioning effect if it was immediately proximal to the relevant behavior, and if that were true then clicker training (in which you use a click sound that’s been associated with treats and compliments and other rewards to signal to a dog that you like what it just did) wouldn’t reinforce the distant behavior of rolling over but would instead reinforce something like the last blink of the dog’s eye before the soundwave of the click reached the dog’s ear.

However, I claim it is effectively true, for rewards as small as fleeting thoughts or shifts in emotion, and for time scales as small as hundredths of a second. If I want an anti-Oreo conditioning effect that is as strong as the pleasure-burst I receive from eating an Oreo, I can in fact get it, even with a stimulus as small as a thought—provided that thought pops up quickly enough.

(This is actually why clicker training is a thing—because you literally cannot deliver a treat quickly enough to produce effects of the size you can get through the much-tighter feedback loop provided by the audio channel. If you can make a click into a positive reward for a dog, then you’re better off clicking than...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Goodhart's Imperius, published by Duncan Sabien on October 29, 2021 on LessWrong.

Author's note: this essay was originally published pseudonymously in 2017. It's now being permanently rehosted at this link. I'll be rehosting a small number of other upper-quintile essays from that era over the coming weeks.

Claim 1: Goodhart’s Law is true.

Goodhart’s Law (which is incredibly appropriately named) reads “any measure which becomes a target ceases to be a good measure.” Another way to say this is “proxies are leaky,” i.e. the proxy never quite gets you the thing it was intended to get you.

If you want to be able to differentiate between promising math students and less-promising ones, you can try out a range of questions and challenges until you cobble together a test that the 100 best students (as determined by other assessments, such as teacher ratings) do well on, and the following 900 do worse on.

But as soon as you make that test into the test, it’s going to start leaking. In the tenth batch of a thousand students, the 100 best ones will still do quite well, but you’ll also get a bunch of people who don’t have the generalized math skill you're looking for, but who did get good at answering the specific, known questions. Your top 100 will no longer be composed only of the 100 actual-best math students—and things will just keep getting worse, over time.

This is analogous to what’s happened with Western diets and sugar. Prehistoric primates who happened to have a preference for sweet things (fruit) also happened to get a lot more vitamins and minerals and calories, and therefore they survived and thrived at higher rates than those sugar-ambivalent primates who failed to become our ancestors and died out. The process of natural selection turned a measure for nutrition (sweetness) into a target (a biologically hardwired "belief" that more sugar = more utility), which was fine right up until we learned to efficiently separate the sugar from the nutrients (teaching to the test) and discovered that our preferences were hardwired to the proxy rather than to the Actual Good Thing.

Claim 2: When attempting to do operant conditioning with a given reward or punishment, for any desired strength-of-conditioning-effect, there exists a sufficiently small delay between behavior and consequence that will produce that effect.

This one is not literally true. In order for it to be literally true, the hyperbolic nature of discounting (such that closer rewards are disproportionately more effective in creating reinforcement) would have to extend off to absurdity such that an infinitesimally small reward or response could produce an arbitrarily large conditioning effect if it was immediately proximal to the relevant behavior, and if that were true then clicker training (in which you use a click sound that’s been associated with treats and compliments and other rewards to signal to a dog that you like what it just did) wouldn’t reinforce the distant behavior of rolling over but would instead reinforce something like the last blink of the dog’s eye before the soundwave of the click reached the dog’s ear.

However, I claim it is effectively true, for rewards as small as fleeting thoughts or shifts in emotion, and for time scales as small as hundredths of a second. If I want an anti-Oreo conditioning effect that is as strong as the pleasure-burst I receive from eating an Oreo, I can in fact get it, even with a stimulus as small as a thought—provided that thought pops up quickly enough.

(This is actually why clicker training is a thing—because you literally cannot deliver a treat quickly enough to produce effects of the size you can get through the much-tighter feedback loop provided by the audio channel. If you can make a click into a positive reward for a dog, then you’re better off clicking than...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Stuart Russell and Melanie Mitchell on Munk Debates, published by Alex Flint on October 29, 2021 on The AI Alignment Forum.

Financial status: This is independent research, now supported by a grant.

Epistemic status: Reflections on a debate about AI.

Earlier this year, The Munk Debates podcast organized a debate between Stuart Russell and Melanie Mitchell on the question of whether AI poses an existential risk. The motion was:

Be it resolved, the quest for true AI is one of the great existential risks of our time.

Stuart was for, Melanie was against.

A significant portion of the debate was about the nature of the term "intelligence". Melanie contended that a truly intelligent machine would understand what we really mean when we give it incomplete instructions, or else not deserve the mantle of "truly intelligent". She said that if we built a general-purpose AI and asked it to do some simple task, and the machine burned up all the energy in the universe in service of that simple task, then the AI would have to lack some common sense that even humans have, so it would be an error to say that such an AI had "superhuman intelligence".

Stuart responded by noting that powerful AI systems could plausibly pursue goals that are either beneficial or harmful to humans (the orthogonality thesis), and that it seems very difficult to specify what we mean by "good" in a single pre-specified objective function (the argument from fragility of human values).

It seemed to me that Stuart and Melanie were using the word "intelligent" to refer to different concepts. Start, I believe, was using the word to refer to systems that have the capacity to flexibly influence the future in service of a wide variety of goals, while Melanie, I think, was using the word to refer to systems that have a kind of friendly aliveness that I think is close to what we mean by "friendly AI" or "aligned superintelligence".

I am personally most used to using the word "intelligence" as Stuart used it, but I think Melanie was also pointing to a real phenomenon in the world, and perhaps even one that accords more closely with common English-language use of the word. For example, I might not intuitively think of "intelligent" as appropriate to describe a corporation that is highly effective at achieving a myopic and rigidly held goal, even if I did view that corporation as very powerful. In contrast, there is a certain sense of the word "intelligent" that is highly appropriate to describe the natural resonance that a dog or horse forms with me, even though such an animal does not steer the future very much. This is not the way that I most often use the word "intelligent" but it does, I think, point to something real.

In the same way, a paperclip maximizer might possess overwhelming power to convert matter and energy into paperclips, yet it would not be a completely unreasonable use of the English word "intelligent" to say that a paperclip maximizer is not "intelligent". This is a purely terminological point concerning only the use of the word "intelligent" in contemporary English.

If we taboo the word "intelligent" then there are two real-world phenomena that were being pointed at in the debate, one by Stuart, and one by Melanie. Stuart was, I think, pointing at the phenomenon of machines that exert flexible influence over the future. Melanie was, I think, pointing at the phenomenon of being friendly or benevolent or helpful. Both are extremely important concepts, and it seems to me that the whole problem of AI alignment is about bringing that which is powerful together with that which is friendly or benevolent or helpful.

I don’t think Stuart and Melanie were hitting on a real disagreement about the nature of powerful autonomous systems or whether they might present dangers to life on Earth, but rather were using th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Stuart Russell and Melanie Mitchell on Munk Debates, published by Alex Flint on October 29, 2021 on The AI Alignment Forum.

Financial status: This is independent research, now supported by a grant.

Epistemic status: Reflections on a debate about AI.

Earlier this year, The Munk Debates podcast organized a debate between Stuart Russell and Melanie Mitchell on the question of whether AI poses an existential risk. The motion was:

Be it resolved, the quest for true AI is one of the great existential risks of our time.

Stuart was for, Melanie was against.

A significant portion of the debate was about the nature of the term "intelligence". Melanie contended that a truly intelligent machine would understand what we really mean when we give it incomplete instructions, or else not deserve the mantle of "truly intelligent". She said that if we built a general-purpose AI and asked it to do some simple task, and the machine burned up all the energy in the universe in service of that simple task, then the AI would have to lack some common sense that even humans have, so it would be an error to say that such an AI had "superhuman intelligence".

Stuart responded by noting that powerful AI systems could plausibly pursue goals that are either beneficial or harmful to humans (the orthogonality thesis), and that it seems very difficult to specify what we mean by "good" in a single pre-specified objective function (the argument from fragility of human values).

It seemed to me that Stuart and Melanie were using the word "intelligent" to refer to different concepts. Start, I believe, was using the word to refer to systems that have the capacity to flexibly influence the future in service of a wide variety of goals, while Melanie, I think, was using the word to refer to systems that have a kind of friendly aliveness that I think is close to what we mean by "friendly AI" or "aligned superintelligence".

I am personally most used to using the word "intelligence" as Stuart used it, but I think Melanie was also pointing to a real phenomenon in the world, and perhaps even one that accords more closely with common English-language use of the word. For example, I might not intuitively think of "intelligent" as appropriate to describe a corporation that is highly effective at achieving a myopic and rigidly held goal, even if I did view that corporation as very powerful. In contrast, there is a certain sense of the word "intelligent" that is highly appropriate to describe the natural resonance that a dog or horse forms with me, even though such an animal does not steer the future very much. This is not the way that I most often use the word "intelligent" but it does, I think, point to something real.

In the same way, a paperclip maximizer might possess overwhelming power to convert matter and energy into paperclips, yet it would not be a completely unreasonable use of the English word "intelligent" to say that a paperclip maximizer is not "intelligent". This is a purely terminological point concerning only the use of the word "intelligent" in contemporary English.

If we taboo the word "intelligent" then there are two real-world phenomena that were being pointed at in the debate, one by Stuart, and one by Melanie. Stuart was, I think, pointing at the phenomenon of machines that exert flexible influence over the future. Melanie was, I think, pointing at the phenomenon of being friendly or benevolent or helpful. Both are extremely important concepts, and it seems to me that the whole problem of AI alignment is about bringing that which is powerful together with that which is friendly or benevolent or helpful.

I don’t think Stuart and Melanie were hitting on a real disagreement about the nature of powerful autonomous systems or whether they might present dangers to life on Earth, but rather were using th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I Really Don't Understand Eliezer Yudkowsky's Position on Consciousness, published by Jemist on October 29, 2021 on LessWrong.

From Twitter:

I'd say that I "don't understand" why the people who worry that chickens are sentient and suffering, don't also worry that GPT-3 is sentient and maybe suffering; but in fact I do understand, it's just not a charitable understanding. Anyway, they're both unsentient so no worries.

His overall thesis is spelt out in full here but I think the key passages are these ones:

What my model says is that when we have a cognitively reflective, self-modely thing, we can put very simple algorithms on top of that — as simple as a neural network having its weights adjusted — and that will feel like something, there will be something that it is like that thing to be, because there will be something self-modely enough to feel like there’s a thing happening to the person-that-is-this-person.

So I would be very averse to anyone producing pain in a newborn baby, even though I’d be truly shocked (like, fairies-in-the-garden shocked) to find them sentient, because I worry that might lose utility in future sentient-moments later.

I’m not totally sure people in sufficiently unreflective flow-like states are conscious, and I give serious consideration to the proposition that I am reflective enough for consciousness only during the moments I happen to wonder whether I am conscious.

I'm currently very confident on the following things, and I'm pretty sure EY is too:

Consciousness (having qualia) exists and humans have it

Consciousness isn't an epiphenomenon

Consciousness is a result of how information is processed in an algorithm, in the most general sense: a simulation of a human brain is just as conscious as a meat-human

EY's position seems to be that self-modelling is both necessary and sufficient for consciousness. But I don't ever see him putting forward a highly concrete thesis for why this is the case. He is correct that his model has more moving parts than other models. But having more moving parts only makes sense if it's actually good at explaining observed data. And we only have one datapoint, which is that adult humans are conscious. Or do we?

"Higher" Consciousness

We actually have a few datapoints here. An ordering of consciousness as reported by humans might be:

Asleep Human < Awake Human < Human on Psychedelics/Zen Meditation

I don't know if EY agrees with this. From his beliefs he might say something along the lines of "having more thoughts doesn't mean you're more conscious". Given his arguments about babies, I'm pretty sure he thinks that you can have memories of times when you weren't conscious, and then consciously experience those things in a sort of "second hand" way by loading up those memories.

Now a lot of Zen meditation involves focusing on your own experiences, which seems like self-modelling. However something else I notice here is the common experience of "ego death" while using psychedelics and in types of meditation. Perhaps EY has a strong argument that this in fact requires more self-modelling than previous states. On the other hand, he might argue that consciousness is on/off, and then amount of experience is unrelated to whether or not those experiences are being turned into qualia.

I'm trying to give potential responses to my arguments, but I don't want to strawman EY so I ought to point out that there are lots of other counter-arguments to this he might have, which might be more insightful than my imagined ones.

Inner Listeners

EY talks a lot about "inner listeners", and mentions that a good theory should be able to have them arise naturally in some way. I agree with this point, and I do agree that his views provide a possible explanation as to what produces an inner listener.

Where I disagree is that we 100% need a se...