Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The poetry of progress, published by jasoncrawford on November 15, 2021 on LessWrong.

I’ve said that Western culture, before the World Wars, had a positive philosophy of progress. How do we know that people in that era saw glory, romance, even poetry in progress?

One way we know is that they literally wrote poetry about it. Take, for example, Erasmus Darwin (grandfather of Charles). By vocation a doctor, he also wrote poems. In 1791, he published The Botanic Garden, of which Part 1, titled “The Economy of Vegetation,” “celebrates scientific progress and technological innovation, such as the forging of steel, the invention of the steam engine and the improvements to gunpowder. It depicts scientists and inventors, such as Benjamin Franklin, responsible for this progress as the heroes of a new age.” (Wikipedia) In one remarkable passage, he praises the power of steam and predicts the steamboat, the locomotive, and the airship:

Soon shall thy arm, Unconquer’d Steam! afarDrag the slow barge, or drive the rapid car;Or on wide-waving wings expanded bearThe flying-chariot through the fields of air.—Fair crews triumphant, leaning from above,Shall wave their fluttering kerchiefs as they move;Or warrior-bands alarm the gaping croud,And armies shrink beneath the shadowy cloud.

Or take this stanza from “The Engine-Driver to His Engine”, by engineer William J. M. Rankine, which reads less like a classical epic and more like a popular ballad:

My blessing on old George Stephenson! Let his fame for ever last,For he was the man that found the plan to make you run so fast.His arm was strong, his head was long, he knew not guile nor fear;When I think of him it makes me proud that I am an engineer!

Samuel Morse, telegraph inventor, was also praised in poetry. A poem read at his memorial service in 1872 begins by describing mankind as having conquered all of nature except for “the Ocean, the Mountain, and Time and Space”—until Morse came along:

But one morning he made him a slender wire, As an artist’s vision took life and form,While he drew from heaven the strange, fierce fire, That reddens the edge of the midnight storm;And he carried it over the Mountain’s crest,And dropped it into the Ocean’s breast;And Science proclaimed, from shore to shore,That Time and Space ruled man no more.

It wasn’t only amateurs and enthusiasts who wrote poetry praising machines and inventions. No less than Rudyard Kipling wrote of the beauty and glory of the steam engine in “McAndrew’s Hymn.” McAndrew is a Scottish engineer aboard a steamship, and in one passage he compares the engine to an orchestra, after one of the “first-class passengers” asks him, “don’t you think steam spoils romance at sea?”:

Romance! Those first-class passengers they like it very well,Printed an’ bound in little books; but why don’t poets tell?I’m sick of all their quirks an’ turns—the loves an’ doves they dream—Lord, send a man like Robbie Burns to sing the Song o’ Steam!To match wi’ Scotia’s noblest speech yon orchestra sublimeWhaurto—uplifted like the Just—the tail-rods mark the time.The Crank-throws give the double-bass; the feed-pump sobs an’ heaves:An’ now the main eccentrics start their quarrel on the sheaves.Her time, her own appointed time, the rocking link-head bides,Till—hear that note?—the rod’s return whings glimmerin’ through the guides.They’re all awa!

True beat, full power, the clangin’ chorus goesClear to the tunnel where they sit, my purrin’ dynamoes.Interdependence absolute, foreseen, ordained, decreed,To work, Ye’ll note, at any tilt an’ every rate o’ speed.Fra skylight-lift to furnace-bars, backed, bolted, braced an’ stayed,An’ singin’ like the Mornin’ Stars for joy that they are made;While, out o’ touch o’ vanity, the sweatin’ thrust-block says:“Not unto us the praise, or man—not unto us the praise!”Now, a’ toge...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simple comparison polling to create utility functions, published by NunoSempere on November 15, 2021 on The Effective Altruism Forum.

tl;dr:

The Utility Function Extractor is a small web application.

It guides individuals through a process of comparing different pairs of items, by asking them, “How much more valuable is the first option compared to the second one?”

This result is turned into a simple utility function that can be used to compare between all of the items in the set.

You can try the application here.

This technique can be used for comparing moral goods, research outputs, policy initiatives, and more.

Introduction

Say you have a list of items that you want to compare the value of. You just want to know how valuable every item is, in comparison to every other item. Perhaps these items are research documents, or the lives of people at different ages, or personal life improvements.

You could try to come up with one standard unit (QALY, "adjusted research paper", "adjusted life improvement"), but this can be messy.

Instead, all you really need to do is to compare them in pairs. In each case, you say, roughly, "The first item is X% as valuable as the second". If you do this enough times [1], you can produce a simple utility function of all of the items. This would allow you to successfully be able to calculate the value of any two items in the set, even items you haven't directly compared. [2]

One problem with a naive approach is that your preferences might be inconsistent. After valuing A vs. B, B vs. C, and A vs. C, it might be the case that your numbers don't completely match each other. However, you could at least see these discrepancies, and make manual changes upon reflection. You might also want to see the inconsistencies, rather than use some other method that smoothes them out from the beginning (like denominating all items in the same unit.)

I've developed a simple web tool to help people walk through the above steps. I've experimented with this on a few small lists of comparison items. This project is an early exploration of this idea. It would be great to later see much more sophisticated versions.

I originally built it while trying to be less confused about the value of research and about how to elicit information about it. But I recently realized that it also has a more immediate and less speculative application: the EA community could use it to elicit and aggregate GiveWell's moral weights by allowing users to input their own comparisons.

By default, the site asks about GiveWell moral weights. If you instead want to compare research outputs, you can do so here. If you'd instead like to compare your own set of elements, you can do so under "Advanced Options", or you can reach out, and I'm happy to create a page specifically for a given project. Note that I'm saving the user's inputs because I think it might be interesting to aggregate them later.

Potential use cases

Extract comparisons between different moral goods to use as an input to the giant GiveWell spreadsheet.

Poll associates about the value of projects one is considering doing.

Construct a utility function over different types of research outputs.

Construct a utility function over different types of policy initiatives.

Compare the moral weight of different types of sentient beings.

Allocate a prize fund proportionally to the value of entries.

Example

As an example, let's show the app working on comparing various research documents. Right now, we assume that all research is net-positive, and we assume that users feel comfortable estimating how valuable each document is versus each other document.

We're not trying to have a precise definition of "value". Instead, the app leaves this up to the user.

The user is at the start presented with the following screen:

The user then selects, say, t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simple comparison polling to create utility functions, published by NunoSempere on November 15, 2021 on The Effective Altruism Forum.

tl;dr:

The Utility Function Extractor is a small web application.

It guides individuals through a process of comparing different pairs of items, by asking them, “How much more valuable is the first option compared to the second one?”

This result is turned into a simple utility function that can be used to compare between all of the items in the set.

You can try the application here.

This technique can be used for comparing moral goods, research outputs, policy initiatives, and more.

Introduction

Say you have a list of items that you want to compare the value of. You just want to know how valuable every item is, in comparison to every other item. Perhaps these items are research documents, or the lives of people at different ages, or personal life improvements.

You could try to come up with one standard unit (QALY, "adjusted research paper", "adjusted life improvement"), but this can be messy.

Instead, all you really need to do is to compare them in pairs. In each case, you say, roughly, "The first item is X% as valuable as the second". If you do this enough times [1], you can produce a simple utility function of all of the items. This would allow you to successfully be able to calculate the value of any two items in the set, even items you haven't directly compared. [2]

One problem with a naive approach is that your preferences might be inconsistent. After valuing A vs. B, B vs. C, and A vs. C, it might be the case that your numbers don't completely match each other. However, you could at least see these discrepancies, and make manual changes upon reflection. You might also want to see the inconsistencies, rather than use some other method that smoothes them out from the beginning (like denominating all items in the same unit.)

I've developed a simple web tool to help people walk through the above steps. I've experimented with this on a few small lists of comparison items. This project is an early exploration of this idea. It would be great to later see much more sophisticated versions.

I originally built it while trying to be less confused about the value of research and about how to elicit information about it. But I recently realized that it also has a more immediate and less speculative application: the EA community could use it to elicit and aggregate GiveWell's moral weights by allowing users to input their own comparisons.

By default, the site asks about GiveWell moral weights. If you instead want to compare research outputs, you can do so here. If you'd instead like to compare your own set of elements, you can do so under "Advanced Options", or you can reach out, and I'm happy to create a page specifically for a given project. Note that I'm saving the user's inputs because I think it might be interesting to aggregate them later.

Potential use cases

Extract comparisons between different moral goods to use as an input to the giant GiveWell spreadsheet.

Poll associates about the value of projects one is considering doing.

Construct a utility function over different types of research outputs.

Construct a utility function over different types of policy initiatives.

Compare the moral weight of different types of sentient beings.

Allocate a prize fund proportionally to the value of entries.

Example

As an example, let's show the app working on comparing various research documents. Right now, we assume that all research is net-positive, and we assume that users feel comfortable estimating how valuable each document is versus each other document.

We're not trying to have a precise definition of "value". Instead, the app leaves this up to the user.

The user is at the start presented with the following screen:

The user then selects, say, t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Worst Commonsense Concepts?, published by abramdemski on November 15, 2021 on LessWrong.

Perhaps the main tool of rationality is simply to use explicity reasoning where others don't, as Jacob Falcovich suggests:

New York Times reporter Cade Metz interviewed me and other Rationalists mostly about how we were ahead of the curve on COVID and what others can learn from us. I told him that Rationality has a simple message: “people can use explicit reason to figure things out, but they rarely do”

However, I also think a big chunk of the value of rationality-as-it-exists-today is in its corrections to common mistakes of explicit reasoning. (To be clear, I'm not accusing Jacob of ignoring that.) For example, bayesian probability theory is one explicit theory which helps push a lot of bad explicit reasoning to the side.

The point of this question, however, is not to point to the good ways of reasoning. The point here is, rather, to point at bad concepts which are in widespread use.

For example:

Fact vs opinion. There are several reasons why this is an awful concept.

The common usage suggests that there are "matters of fact" vs "matters of opinion"; eg, I like hummus (opinion) but 1+1=2 (matter of fact). But common usage also suggests that probabilistic reasoning gives mere opinions, while other modes of reasoning (such as direct observation, and logical reasoning) yield facts. This is inconsistent; it suggests that we can tell whether a belief is an opinion or a fact by examining what it is about (beliefs about subjective things = opinions; beliefs about objective things = facts), while also seeming to need the mode of reasoning by which we arrived at the belief (eg, if I saw a black hole myself, it would be a fact, but if I derived one's existence from unproven physics, it would be opinion).

Calling something a fact generally indicates that others are epistemically obligated to believe it. But if it is contentious, then this is precisely what's at issue. So calling something a fact like this is generally useless.

We could take "fact" to mean something like "true opinion". But from the inside, this is no different from a strong belief. So again, to call something a fact rather than a strong opinion seems to add no information (whereas, I take it, it's supposed to according to common usage).

"Purpose" as an inherent property. In common usage, it makes sense to ask "the purpose of life" because a purpose is a property which lots of objects have. In reality, it only makes sense to think of "purpose" relative to some agent, as in "I made this for this purpose". Common usage allows purpose to be agent-independent because there are lots of things (tables, chairs, silverware, etc) which have purposes largely independent of agent (most people use tables to set things on for convenient reach, chairs to sit on, silverware to eat, etc). However, in cases which aren't like that, the language doesn't make sense without explanation (but people treat it like it does).

These are intended to be the sort of thing which people use unthinkingly -- IE, not popular beliefs like astrology. While astrology has some pretty bad concepts, it is explicitly bundled as a belief package which people consider believing/disbelieving. Very few people have mental categories like "fact-ist" for someone who believes in a fact/opinion divide. It's therefore useful to make explicit belief-bundles for these things, so that we can realize when we are choosing whether to use that belief-bundle.

My hope is that when you encounter a pretty bad (but common) concept out there in the wild, you'll think to return here and add it to the list as a new answer. (IE, as with all LW Questions, I hope this can become a timeless list, rather than just something people interact with once when it is on the front page.)

Properly diss...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Worst Commonsense Concepts?, published by abramdemski on November 15, 2021 on LessWrong.

Perhaps the main tool of rationality is simply to use explicity reasoning where others don't, as Jacob Falcovich suggests:

New York Times reporter Cade Metz interviewed me and other Rationalists mostly about how we were ahead of the curve on COVID and what others can learn from us. I told him that Rationality has a simple message: “people can use explicit reason to figure things out, but they rarely do”

However, I also think a big chunk of the value of rationality-as-it-exists-today is in its corrections to common mistakes of explicit reasoning. (To be clear, I'm not accusing Jacob of ignoring that.) For example, bayesian probability theory is one explicit theory which helps push a lot of bad explicit reasoning to the side.

The point of this question, however, is not to point to the good ways of reasoning. The point here is, rather, to point at bad concepts which are in widespread use.

For example:

Fact vs opinion. There are several reasons why this is an awful concept.

The common usage suggests that there are "matters of fact" vs "matters of opinion"; eg, I like hummus (opinion) but 1+1=2 (matter of fact). But common usage also suggests that probabilistic reasoning gives mere opinions, while other modes of reasoning (such as direct observation, and logical reasoning) yield facts. This is inconsistent; it suggests that we can tell whether a belief is an opinion or a fact by examining what it is about (beliefs about subjective things = opinions; beliefs about objective things = facts), while also seeming to need the mode of reasoning by which we arrived at the belief (eg, if I saw a black hole myself, it would be a fact, but if I derived one's existence from unproven physics, it would be opinion).

Calling something a fact generally indicates that others are epistemically obligated to believe it. But if it is contentious, then this is precisely what's at issue. So calling something a fact like this is generally useless.

We could take "fact" to mean something like "true opinion". But from the inside, this is no different from a strong belief. So again, to call something a fact rather than a strong opinion seems to add no information (whereas, I take it, it's supposed to according to common usage).

"Purpose" as an inherent property. In common usage, it makes sense to ask "the purpose of life" because a purpose is a property which lots of objects have. In reality, it only makes sense to think of "purpose" relative to some agent, as in "I made this for this purpose". Common usage allows purpose to be agent-independent because there are lots of things (tables, chairs, silverware, etc) which have purposes largely independent of agent (most people use tables to set things on for convenient reach, chairs to sit on, silverware to eat, etc). However, in cases which aren't like that, the language doesn't make sense without explanation (but people treat it like it does).

These are intended to be the sort of thing which people use unthinkingly -- IE, not popular beliefs like astrology. While astrology has some pretty bad concepts, it is explicitly bundled as a belief package which people consider believing/disbelieving. Very few people have mental categories like "fact-ist" for someone who believes in a fact/opinion divide. It's therefore useful to make explicit belief-bundles for these things, so that we can realize when we are choosing whether to use that belief-bundle.

My hope is that when you encounter a pretty bad (but common) concept out there in the wild, you'll think to return here and add it to the list as a new answer. (IE, as with all LW Questions, I hope this can become a timeless list, rather than just something people interact with once when it is on the front page.)

Properly diss...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A positive case for how we might succeed at prosaic AI alignment, published by Evan Hubinger on November 16, 2021 on The AI Alignment Forum.

This post is my attempt at something like a response to Eliezer Yudkowsky’s recent discussion on AGI interventions.

I tend to be relatively pessimistic overall about humanity’s chances at avoiding AI existential risk. Contrary to some others that share my pessimism, however—Eliezer Yudkowsky, in particular—I believe that there is a clear path forward for how we might succeed within the current prosaic paradigm (that is, the current machine learning paradigm) that looks plausible and has no fundamental obstacles.

In the comments on Eliezer’s discussion, this point about whether there exists any coherent story for prosaic AI alignment success came up multiple times. From Rob Bensinger:

I think it's pretty important here to focus on the object-level. Even if you think the goodness of these particular research directions isn't cruxy (because there's a huge list of other things you find promising, and your view is mainly about the list as a whole rather than about any particular items on it), I still think it's super important for us to focus on object-level examples, since this will probably help draw out what the generators for the disagreement are.

In that spirit, I’d like to provide my own object-level story for how prosaic AI alignment might work out well.

Of course, any specific story for how we might succeed is going to be wrong simply because it specifies a bunch of details, and this is such a specific story. The point, however, is that the fact that we can write plausible stories for how we might succeed that don’t run into any fundamental obstacles implies that the problem isn’t “we don’t even know how we could possibly succeed” but rather “we know some ways in which we might succeed, but they all require a bunch of hard stuff that we have to actually execute on,” which I think is a pretty different place to be.

One thing I do agree with Eliezer on, however, is that, when you’re playing from behind—as I think we are—you play for variance. That means embracing strategies that might not work in expectation, but that have long tails in the positive direction, and I definitely see my picture here as falling into that category.

Furthermore, as one should probably expect with any full roadmap for solving such a complicated problem, there’s still a lot left out of my picture—especially my intuitions for why I think each of these steps is actually plausible. I am currently working on a research agenda that will go into a lot more detail on that, but until it’s published, it might be best to just think of this post as an overview of what that agenda will look like.[1]

Alright, without further ado, here’s my concrete picture for how we might end up succeeding:

We produce an understanding of a simple, natural class of agents such that agents of this form are capable of doing all of the things that we might want a powerful, advanced AI to do—but such that no agents of this form will ever act deceptively.

My current best guess for what such a natural class might look like is a myopic agent—that is, an agent that only cares about its next action rather than the long-term consequences of its actions. I think it is possible to produce a simple, natural description of myopia such that myopic agents are still capable of doing all the powerful things we might want out of an AGI but such that they never have any reason to be deceptive.[2]

In the language of training stories, (1) gives us our training goal, the mechanistic description of what sort of model we’re trying to produce.

We develop some way of determining whether a given non-deceptive model falls into the natural class we developed in step (1). It’s fine for this not to work for all n...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A positive case for how we might succeed at prosaic AI alignment, published by Evan Hubinger on November 16, 2021 on The AI Alignment Forum.

This post is my attempt at something like a response to Eliezer Yudkowsky’s recent discussion on AGI interventions.

I tend to be relatively pessimistic overall about humanity’s chances at avoiding AI existential risk. Contrary to some others that share my pessimism, however—Eliezer Yudkowsky, in particular—I believe that there is a clear path forward for how we might succeed within the current prosaic paradigm (that is, the current machine learning paradigm) that looks plausible and has no fundamental obstacles.

In the comments on Eliezer’s discussion, this point about whether there exists any coherent story for prosaic AI alignment success came up multiple times. From Rob Bensinger:

I think it's pretty important here to focus on the object-level. Even if you think the goodness of these particular research directions isn't cruxy (because there's a huge list of other things you find promising, and your view is mainly about the list as a whole rather than about any particular items on it), I still think it's super important for us to focus on object-level examples, since this will probably help draw out what the generators for the disagreement are.

In that spirit, I’d like to provide my own object-level story for how prosaic AI alignment might work out well.

Of course, any specific story for how we might succeed is going to be wrong simply because it specifies a bunch of details, and this is such a specific story. The point, however, is that the fact that we can write plausible stories for how we might succeed that don’t run into any fundamental obstacles implies that the problem isn’t “we don’t even know how we could possibly succeed” but rather “we know some ways in which we might succeed, but they all require a bunch of hard stuff that we have to actually execute on,” which I think is a pretty different place to be.

One thing I do agree with Eliezer on, however, is that, when you’re playing from behind—as I think we are—you play for variance. That means embracing strategies that might not work in expectation, but that have long tails in the positive direction, and I definitely see my picture here as falling into that category.

Furthermore, as one should probably expect with any full roadmap for solving such a complicated problem, there’s still a lot left out of my picture—especially my intuitions for why I think each of these steps is actually plausible. I am currently working on a research agenda that will go into a lot more detail on that, but until it’s published, it might be best to just think of this post as an overview of what that agenda will look like.[1]

Alright, without further ado, here’s my concrete picture for how we might end up succeeding:

We produce an understanding of a simple, natural class of agents such that agents of this form are capable of doing all of the things that we might want a powerful, advanced AI to do—but such that no agents of this form will ever act deceptively.

My current best guess for what such a natural class might look like is a myopic agent—that is, an agent that only cares about its next action rather than the long-term consequences of its actions. I think it is possible to produce a simple, natural description of myopia such that myopic agents are still capable of doing all the powerful things we might want out of an AGI but such that they never have any reason to be deceptive.[2]

In the language of training stories, (1) gives us our training goal, the mechanistic description of what sort of model we’re trying to produce.

We develop some way of determining whether a given non-deceptive model falls into the natural class we developed in step (1). It’s fine for this not to work for all n...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quantilizer ≡ Optimizer with a Bounded Amount of Output, published by itaibn0 on November 16, 2021 on The AI Alignment Forum.

(Crossposted from my blog)

In 2015 MIRI proposed quantilizers as a decision theory criterion that an AI may use that allows it strive towards a goal while not optimizing too hard at the goal, so that if a superintelligent were a quantilizer there's a smaller chance it will do something catastrophic like kill all humans and turn all matter on Earth into paperclips because it's trying really hard to maximize the productivity of a paperclip factory. For a real number 0≤p≤1, a p-quantilizer is an agent which, given a goal and a space of possible strategies, picks randomly out of the top proportion p, i.e., randomly among any strategy that is better than a proportion (1−p) of all strategies. In fact, a quantilizer is for all practical purposes equivalent to a a perfect optimizer (an agent that always picks the best of all its possible strategies) that is further restricted to have its strategy specifies by a string of bits whose length must be less than log2(1/p).

Since the people at MIRI have already considered the possibility of controlling an AI by restricting its ability to perform input and output and have generally decided that they are not satisfied with this, I consider the idea of quantilizers to be a failure.

Roughly speaking, my claim is that for a natural number n, up to a constant additive factor, a perfect optimizer that is limited to strategies taking at most n bits performs as well in any goal as a 2−n-quantilizer. More precisely, to explain what I mean by "up to a constant additive factor", there is a reasonably-sized constant c such that an optimizer with n+c bits of outputs performs as well as a 2−n-quantilizer, and a 2−(n+c)-quantilizer performs as well as an optimizer with n bits of output. I also need to assume that the space of strategies is given a rich enough encoding that we can do things like specify a strategy by specifying a computer program whose output is that strategy, or that the environment is rich enough that this can be done implicitly with strategies like "Walk to a computer and type this program, and attach to the computer the accessories necessary to run the strategy specified by the program."

To really make the result true up to a constant factor and not a logarithmic factor or anything like that you need to careful about encoding using stuff like prefix-free encodings but I don't think that's really important and will gloss over it.

Quantilizers are at least as good as bounded-output optimizers: If a particular strategy is optimal among all strategies of length ≤n bits, then since there are at most 2n+1 such strategies this one optimizing strategy already makes up a proportion 2−(n+1) of all strategies with ≤n bits, so a 2−(n+1)-quantilizer over strategies with ≤n bits must use exactly this strategy or an equally-performing strategy. If we assume the strategies have a measure biased towards short strategies or use a prefix-free encoding or a subset of the strategies are implicitly something like that, a proportion ≥1c of all strategies will have length ≤n, so a proportion ≥2−(n+c) will be identical to this ≤n-bit optimal strategy, so a 2−(n+c)-quantilizer will perform at least as well as this strategy.

Bounded-output optimizers are at least as good as quantilizers: Using a deterministic pseudo-random number generator, it is possible to precisely specify a sequence s0,s1,s2,. of strategies that are fairly sampled from the space of all strategies. A proportion 2−n of these strategies are also in the top 2−n of performance among all strategies. So the smallest m such that sm is among the top 2−n in performance is statistically certain to have m≤C2n for a small constant C. It takes n+O(1) bits to specify the m, ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quantilizer ≡ Optimizer with a Bounded Amount of Output, published by itaibn0 on November 16, 2021 on The AI Alignment Forum.

(Crossposted from my blog)

In 2015 MIRI proposed quantilizers as a decision theory criterion that an AI may use that allows it strive towards a goal while not optimizing too hard at the goal, so that if a superintelligent were a quantilizer there's a smaller chance it will do something catastrophic like kill all humans and turn all matter on Earth into paperclips because it's trying really hard to maximize the productivity of a paperclip factory. For a real number 0≤p≤1, a p-quantilizer is an agent which, given a goal and a space of possible strategies, picks randomly out of the top proportion p, i.e., randomly among any strategy that is better than a proportion (1−p) of all strategies. In fact, a quantilizer is for all practical purposes equivalent to a a perfect optimizer (an agent that always picks the best of all its possible strategies) that is further restricted to have its strategy specifies by a string of bits whose length must be less than log2(1/p).

Since the people at MIRI have already considered the possibility of controlling an AI by restricting its ability to perform input and output and have generally decided that they are not satisfied with this, I consider the idea of quantilizers to be a failure.

Roughly speaking, my claim is that for a natural number n, up to a constant additive factor, a perfect optimizer that is limited to strategies taking at most n bits performs as well in any goal as a 2−n-quantilizer. More precisely, to explain what I mean by "up to a constant additive factor", there is a reasonably-sized constant c such that an optimizer with n+c bits of outputs performs as well as a 2−n-quantilizer, and a 2−(n+c)-quantilizer performs as well as an optimizer with n bits of output. I also need to assume that the space of strategies is given a rich enough encoding that we can do things like specify a strategy by specifying a computer program whose output is that strategy, or that the environment is rich enough that this can be done implicitly with strategies like "Walk to a computer and type this program, and attach to the computer the accessories necessary to run the strategy specified by the program."

To really make the result true up to a constant factor and not a logarithmic factor or anything like that you need to careful about encoding using stuff like prefix-free encodings but I don't think that's really important and will gloss over it.

Quantilizers are at least as good as bounded-output optimizers: If a particular strategy is optimal among all strategies of length ≤n bits, then since there are at most 2n+1 such strategies this one optimizing strategy already makes up a proportion 2−(n+1) of all strategies with ≤n bits, so a 2−(n+1)-quantilizer over strategies with ≤n bits must use exactly this strategy or an equally-performing strategy. If we assume the strategies have a measure biased towards short strategies or use a prefix-free encoding or a subset of the strategies are implicitly something like that, a proportion ≥1c of all strategies will have length ≤n, so a proportion ≥2−(n+c) will be identical to this ≤n-bit optimal strategy, so a 2−(n+c)-quantilizer will perform at least as well as this strategy.

Bounded-output optimizers are at least as good as quantilizers: Using a deterministic pseudo-random number generator, it is possible to precisely specify a sequence s0,s1,s2,. of strategies that are fairly sampled from the space of all strategies. A proportion 2−n of these strategies are also in the top 2−n of performance among all strategies. So the smallest m such that sm is among the top 2−n in performance is statistically certain to have m≤C2n for a small constant C. It takes n+O(1) bits to specify the m, ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA-Aligned Impact Investing: Mind Ease Case Study, published by Brendon Wong on November 15, 2021 on The Effective Altruism Forum.

Summary

We use the term “impact investing” to refer to any investment made for the dual purpose of having a positive impact and generating a financial return

Impact investing has received relatively little attention from an EA perspective

We review an investment case study produced by the Total Portfolio Project (TPP) leveraging research from Lionheart Ventures, Let’s Fund, and Rethink Priorities

The case study uses TPP’s Total Portfolio Return (TPR) framework, a rigorous framework which can be used to assess the value of an impact investment relative to other ways of generating impact

The case study estimates that an impact investment into Mind Ease averts DALYs at $150 per dollar invested in expectation, comparing favorably to GiveWell top charities before financial returns are taken into account

We show how the TPR framework can be used to determine that, for a donor who is already making the best possible use of donations and investing to give, investing in Mind Ease appears better than either additional donations or more investing to give

Given the lack of prior research in this area and the potential for impact investing to significantly benefit many cause areas, we recommend that the EA community direct more research and attention to this area.

We highlight several ways to contribute including: dialogue and movement building, evaluations, research, entrepreneurship, and shifting money towards impact investing.

Background and Implications

Impact investing has largely been neglected by the EA community. This is perhaps due to the perception that most impact investing is relatively low impact per dollar invested. In turn, EA principles are largely neglected within impact investing. We believe there could be benefits on both sides to exploring the potential of impact investing to produce effective results.

Complicating things is the fact that the term “impact investing” has many possible interpretations and connotations. It generally refers to one or more strategies within a disparate range of financial strategies that are associated (correctly or not) with “impact,” from buying (or not buying) certain stocks, to providing loans to nonprofits and disadvantaged communities, to funding early-stage startups with a specific social mission.

We only want to distinguish an investment as impactful if results are generated that would not occur in expectation without the investment. Among the various interpretations of the term impact investing, we believe that only some of these approaches offer potential for impact. Thus, we will use the term “impact investing” in this article to refer to any investment that is expected to have a positive impact on the world and generate financial returns. Such investments could take the form of anything from early stage startup investments to ESG-related strategies in liquid stock markets (but note that ESG does not imply impact).

There are many parallels between the impact investing space at present and the donation space before EA emerged. Most EAs believe the impact of charitable donations has a heavy-tailed distribution, meaning a small number of donations/charities have a much higher impact (sometimes orders of magnitude more) than donations/charities on average. We think impact investing likely has a distribution of impact similar to or even more extreme than donations (see here for a technical discussion). Analogous to the early days of EA, there is no evaluator like GiveWell using rigorous evaluation techniques to find the most impactful investing opportunities.

In this article, we share what we think is the first work in the EA community attempting to rigorously evaluate a specific impact investment ...