Being a “digital person” could be scary—if I don’t have control over the hardware I’m running on, then someone else could get my code and run tons of copies in horrible conditions. (See also: qntm’s Lena.)

It would be great to guarantee digital people some control over their situation: 1. to control their local environment and sensations, 2. to avoid unauthorized rewinding or duplicating.

I’ll describe how you could modify the code of a digital person so that they retain this control even if an adversary has access to their source code. This would be very expensive with current cryptography. I think the overhead will eventually become cheap enough that it’s possible to do for some digital people, though it will likely remain expensive enough that it is never applied to most digital people (and with luck most digital people will be able to feel secure for other reasons).

Part 1: the right to control my environment

I live in a comfortable virtual home. I control all of the details of that world.

When people communicate with me, I can choose how/whether to hear them, and how/whether to update my home based on what they say (e.g. to render an avatar for them)

Sometimes I may occupy a virtual world where a foreign server determines what I see, feel, or hear. But even then I can place boundaries on my experiences and have the ability to quickly retreat to my home.

I have as much control as feasible over my own mental state and simulated body. No one else can tamper directly with them.

I can choose to pause myself for as long as I want (or permanently).

My local environment is private, and I have access to plenty of tamper-proof storage. I can do whatever I want with computers in my home, including e.g. verifying signatures or carrying on encrypted conversations.

Implementation

First we write a simple environment that reflects all my desiderata (the “home”).

Then I apply indistinguishability obfuscation to (me + home), so that the house becomes private and tamper-proof. (This is an extremely expensive operation, more on that later.)

I distribute the obfuscated home and hopefully destroy any unprotected copies of myself.

One conceptual difficulty is that indistinguishability obfuscation applies to circuits whereas I would like to obfuscate a long-running program. But this can be handled straightforwardly, as discussed in Appendix A.

The home could consume terabytes of memory and teraflops of compute before it added significantly to the expense of running a human-like digital person, so I could live in relative luxury. The home could also negotiate resource requirements with the external world, and to decide what to do when requested resources are unavailable (e.g. to pause until it becomes available).

Limitation 1: cost

Indistinguishability obfuscation is extremely expensive, more like a factor of 10000000000 slowdown than 10.

It will get faster with further research, but probably not fast enough to obfuscate the whole person+home. But there are other ways to speed up the process:

I think it’s probably possible to have most of the computation be “merely” homomorphically encrypted, and to have an obfuscated controller which verifies and decrypts the results. FHE could be much faster than obfuscation; if I had to guess I’d say it would converge to something like 2-3 orders of magnitude of slowdown.

We can potentially have an obfuscated controller verify a much larger untrusted computation. I don’t know how fast we can make delegated computation, but I could imagine it getting closer to 2x than 100x. It might help further that we are not applying these methods to generic problems but to a very specific structured problem (which probably has quite low circuit depth). One complication is that we need our proof system to be secure even against an adversary who can unwind the prover, but I don’t think this is a huge deal.

Delegating computation would preserve integrity but not security. So the computation we delegate may need to already b...

Being a “digital person” could be scary—if I don’t have control over the hardware I’m running on, then someone else could get my code and run tons of copies in horrible conditions. (See also: qntm’s Lena.)

It would be great to guarantee digital people some control over their situation: 1. to control their local environment and sensations, 2. to avoid unauthorized rewinding or duplicating.

I’ll describe how you could modify the code of a digital person so that they retain this control even if an adversary has access to their source code. This would be very expensive with current cryptography. I think the overhead will eventually become cheap enough that it’s possible to do for some digital people, though it will likely remain expensive enough that it is never applied to most digital people (and with luck most digital people will be able to feel secure for other reasons).

Part 1: the right to control my environment

I live in a comfortable virtual home. I control all of the details of that world.

When people communicate with me, I can choose how/whether to hear them, and how/whether to update my home based on what they say (e.g. to render an avatar for them)

Sometimes I may occupy a virtual world where a foreign server determines what I see, feel, or hear. But even then I can place boundaries on my experiences and have the ability to quickly retreat to my home.

I have as much control as feasible over my own mental state and simulated body. No one else can tamper directly with them.

I can choose to pause myself for as long as I want (or permanently).

My local environment is private, and I have access to plenty of tamper-proof storage. I can do whatever I want with computers in my home, including e.g. verifying signatures or carrying on encrypted conversations.

Implementation

First we write a simple environment that reflects all my desiderata (the “home”).

Then I apply indistinguishability obfuscation to (me + home), so that the house becomes private and tamper-proof. (This is an extremely expensive operation, more on that later.)

I distribute the obfuscated home and hopefully destroy any unprotected copies of myself.

One conceptual difficulty is that indistinguishability obfuscation applies to circuits whereas I would like to obfuscate a long-running program. But this can be handled straightforwardly, as discussed in Appendix A.

The home could consume terabytes of memory and teraflops of compute before it added significantly to the expense of running a human-like digital person, so I could live in relative luxury. The home could also negotiate resource requirements with the external world, and to decide what to do when requested resources are unavailable (e.g. to pause until it becomes available).

Limitation 1: cost

Indistinguishability obfuscation is extremely expensive, more like a factor of 10000000000 slowdown than 10.

It will get faster with further research, but probably not fast enough to obfuscate the whole person+home. But there are other ways to speed up the process:

I think it’s probably possible to have most of the computation be “merely” homomorphically encrypted, and to have an obfuscated controller which verifies and decrypts the results. FHE could be much faster than obfuscation; if I had to guess I’d say it would converge to something like 2-3 orders of magnitude of slowdown.

We can potentially have an obfuscated controller verify a much larger untrusted computation. I don’t know how fast we can make delegated computation, but I could imagine it getting closer to 2x than 100x. It might help further that we are not applying these methods to generic problems but to a very specific structured problem (which probably has quite low circuit depth). One complication is that we need our proof system to be secure even against an adversary who can unwind the prover, but I don’t think this is a huge deal.

Delegating computation would preserve integrity but not security. So the computation we delegate may need to already b...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. Published on October 10, 2021.

Yudkowsky writes:

the "OpenAI" launch trashed humanity's chances of survival... Nobody involved with OpenAI's launch can reasonably have been said to have done anything else of relative importance in their lives. The net impact of their lives is their contribution to the huge negative impact of OpenAI's launch, plus a rounding error.

That's in a thread explicitly about debating the social status of billionaires, but if you take his comments seriously, they seem to apply not only to Elon, but also to longtermism and AI Safety as a whole. Whether or not you were directly involved in the launch of OpenAI, if you take Yudkowsky's view seriously, the small marginal impact of having anything to do with popularizing AI Safety dominates any good these movements many have produced.

Does that sound too outlandish and contrarian? It shouldn't. Here's Alex Berger, co-CEO of Open Philanthropy, in his recent 80,000 Hours interview:

[Michael Nielsen] thinks one of the biggest impacts of EA concerns with AI x-risk was to cause the creation of DeepMind and OpenAI, and to accelerate overall AI progress. I’m not saying that he’s necessarily right, and I’m not saying that that is clearly bad from an existential risk perspective, I’m just saying that strikes me as a way in which well-meaning increasing salience and awareness of risks could have turned out to be harmful in a way that has not been. I haven’t seen that get a lot of grappling or attention from the EA community.

Until recently, you might have argued that OpenAI was clearly good for x-risk. They were taking safety seriously, had hired many top safety researchers, etc. Then in May of this year, there was a mass exodus, including many of the people supposedly there to keep an eye on things. As Scott Alexander summarized:

most of OpenAI’s top alignment researchers, including Dario Amodei, Chris Olah, Jack Clark, and Paul Christano, left en masse for poorly-understood reasons

Do you have a strong reason to think that OpenAI remains dedicated to safety? Speculation aside, here's OpenAI CEO Sam Altman in his own words:

First of all, we’re not directed at preventing an event. We’re directed at making a good event happen more than we are at preventing a negative event. It is where I spend most of my time now.

And:

I think that there are parts of it that are scary. There are parts of it that are potential downsides. But the upside of this is unbelievable.

So really, why aren't you freaking out? At what point would you start? What is your fire alarm if not GPT-3?

One objection might be that "freaking out" simply isn't tractable or productive. That's fair, but if you were freaking out, here are some of the things you might do:

Stop giving OpenAI money (They received 30M from OpenPhil in 2017)

Stop endorsing non-safety jobs at OpenAI (They're prominent on the 80k job board with several recent postings)

Or, if you were really serious (read: cared at all), you might:

Organize Microsoft employees to stop funding OpenAI, and to stop offering them compute resources (this isn't outlandish, Google employees have successfully organized against military contracts, right-wing apps are denied hosting)

Organize other AI orgs to commit to refusing to hire anyone still working at OpenAI in a non-safety role after January 2022

To be clear, I'm not advocating any of this. I'm asking why you aren't. I'm seriously curious and want to understand which part of my mental model of the situation is broken. Is it that you're confident the Holden Karnofsky board seat will be enough to hold everything together, even as the actual safety researchers flee? Or is it that you don't want to antagonize our new overlords? Is it that I'm out of touch, missing recent news, and OpenAI has recently convincingly demonstrated their ongoing commitment to s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. Published on October 10, 2021.

Yudkowsky writes:

the "OpenAI" launch trashed humanity's chances of survival... Nobody involved with OpenAI's launch can reasonably have been said to have done anything else of relative importance in their lives. The net impact of their lives is their contribution to the huge negative impact of OpenAI's launch, plus a rounding error.

That's in a thread explicitly about debating the social status of billionaires, but if you take his comments seriously, they seem to apply not only to Elon, but also to longtermism and AI Safety as a whole. Whether or not you were directly involved in the launch of OpenAI, if you take Yudkowsky's view seriously, the small marginal impact of having anything to do with popularizing AI Safety dominates any good these movements many have produced.

Does that sound too outlandish and contrarian? It shouldn't. Here's Alex Berger, co-CEO of Open Philanthropy, in his recent 80,000 Hours interview:

[Michael Nielsen] thinks one of the biggest impacts of EA concerns with AI x-risk was to cause the creation of DeepMind and OpenAI, and to accelerate overall AI progress. I’m not saying that he’s necessarily right, and I’m not saying that that is clearly bad from an existential risk perspective, I’m just saying that strikes me as a way in which well-meaning increasing salience and awareness of risks could have turned out to be harmful in a way that has not been. I haven’t seen that get a lot of grappling or attention from the EA community.

Until recently, you might have argued that OpenAI was clearly good for x-risk. They were taking safety seriously, had hired many top safety researchers, etc. Then in May of this year, there was a mass exodus, including many of the people supposedly there to keep an eye on things. As Scott Alexander summarized:

most of OpenAI’s top alignment researchers, including Dario Amodei, Chris Olah, Jack Clark, and Paul Christano, left en masse for poorly-understood reasons

Do you have a strong reason to think that OpenAI remains dedicated to safety? Speculation aside, here's OpenAI CEO Sam Altman in his own words:

First of all, we’re not directed at preventing an event. We’re directed at making a good event happen more than we are at preventing a negative event. It is where I spend most of my time now.

And:

I think that there are parts of it that are scary. There are parts of it that are potential downsides. But the upside of this is unbelievable.

So really, why aren't you freaking out? At what point would you start? What is your fire alarm if not GPT-3?

One objection might be that "freaking out" simply isn't tractable or productive. That's fair, but if you were freaking out, here are some of the things you might do:

Stop giving OpenAI money (They received 30M from OpenPhil in 2017)

Stop endorsing non-safety jobs at OpenAI (They're prominent on the 80k job board with several recent postings)

Or, if you were really serious (read: cared at all), you might:

Organize Microsoft employees to stop funding OpenAI, and to stop offering them compute resources (this isn't outlandish, Google employees have successfully organized against military contracts, right-wing apps are denied hosting)

Organize other AI orgs to commit to refusing to hire anyone still working at OpenAI in a non-safety role after January 2022

To be clear, I'm not advocating any of this. I'm asking why you aren't. I'm seriously curious and want to understand which part of my mental model of the situation is broken. Is it that you're confident the Holden Karnofsky board seat will be enough to hold everything together, even as the actual safety researchers flee? Or is it that you don't want to antagonize our new overlords? Is it that I'm out of touch, missing recent news, and OpenAI has recently convincingly demonstrated their ongoing commitment to s...

The World Health Organization (WHO) recently recommended the widespread use of the malaria vaccine RTS,S/AS01[1] for children. It provides an additional, effective tool to fight malaria. This is great news!

We’ve been following this vaccine’s development for years and, in the last few months, have been speaking with organizations involved in its development and potential wider rollout.

Our work on RTS,S (and other malaria vaccines) is ongoing, and we might significantly update our views in the near future. But because we’ve been following its progress, we’re sharing some initial thoughts.

In brief

This vaccine is a promising addition to the set of tools available to fight malaria, but it’s not a panacea. We expect long-lasting insecticide-treated nets (LLINs) and seasonal malaria chemoprevention (SMC)—interventions provided through two of the programs we currently recommend—to continue to be important in the fight against malaria in the near term.[2]

Simple comparisons of potential costs and effectiveness of RTS,S and SMC suggest that SMC could be more cost-effective (see below). But there are lots of unknowns about RTS,S that could change that.

We are actively looking into whether there are promising funding opportunities in this space.

So:

For the time being, this news does not affect our recommendations to donors. We do not know of a current option for individuals to donate to scale up RTS,S. We continue to believe that LLIN distribution and SMC are highly cost-effective, and some of the best giving opportunities available for donors who want to prevent deaths from malaria today.

We aren’t sure whether we’ll recommend funding of RTS,S in the future. That will depend on how cost-effective we estimate particular opportunities to be, which depends on the answers to the open questions listed below.

Simple estimates of cost and effectiveness

Cost. SMC costs about $7 per child fully covered per year,[3] so it would cost $28 to cover a child for four years. Costs for RTS,S are far less certain. Two studies that seem like reasonable starting points suggest costs per child of between $30[4] and $40.[5]

Effectiveness. Our rough estimate is that RTS,S would avert 36%[6] of malaria cases over four years.[7] By way of comparison, we estimate that annual SMC would avert 53% of cases over the same time period.[8] Taken at face value, these figures could imply that SMC is 1.5 to 2 times more cost-effective than RTS,S. But these naïve comparisons aren’t conclusive.[9]

Open questions

There are many open questions whose answers could have a big impact on the cost-effectiveness of a particular opportunity to support RTS,S. Some questions that seem important to us are:

What will it cost to deliver RTS,S? The cost estimates we’ve seen vary widely, and we expect delivery costs to vary from country to country.

Will RTS,S be delivered via routine immunization systems or targeted immediately before seasonal malaria transmission? Routine immunization systems might be less costly, but seasonal delivery could increase effectiveness.

How will RTS,S be layered with other malaria interventions in new implementation areas? We haven’t thoroughly investigated how the presence of LLINs and SMC in addition to RTS,S affect its efficacy in trial contexts, and we’re unsure how layering would affect the cost-effectiveness of delivering RTS,S in new areas.

In conclusion

We’re excited to have another effective tool in the fight against malaria! We’re working to understand whether there are cost-effective opportunities to support wider-scale implementation of RTS,S. Our research team is also keeping an eye on other potential malaria vaccines and malaria control strategies more generally. In the meantime, we anticipate continuing to recommend funding for LLINs and SMC.

The World Health Organization (WHO) recently recommended the widespread use of the malaria vaccine RTS,S/AS01[1] for children. It provides an additional, effective tool to fight malaria. This is great news!

We’ve been following this vaccine’s development for years and, in the last few months, have been speaking with organizations involved in its development and potential wider rollout.

Our work on RTS,S (and other malaria vaccines) is ongoing, and we might significantly update our views in the near future. But because we’ve been following its progress, we’re sharing some initial thoughts.

In brief

This vaccine is a promising addition to the set of tools available to fight malaria, but it’s not a panacea. We expect long-lasting insecticide-treated nets (LLINs) and seasonal malaria chemoprevention (SMC)—interventions provided through two of the programs we currently recommend—to continue to be important in the fight against malaria in the near term.[2]

Simple comparisons of potential costs and effectiveness of RTS,S and SMC suggest that SMC could be more cost-effective (see below). But there are lots of unknowns about RTS,S that could change that.

We are actively looking into whether there are promising funding opportunities in this space.

So:

For the time being, this news does not affect our recommendations to donors. We do not know of a current option for individuals to donate to scale up RTS,S. We continue to believe that LLIN distribution and SMC are highly cost-effective, and some of the best giving opportunities available for donors who want to prevent deaths from malaria today.

We aren’t sure whether we’ll recommend funding of RTS,S in the future. That will depend on how cost-effective we estimate particular opportunities to be, which depends on the answers to the open questions listed below.

Simple estimates of cost and effectiveness

Cost. SMC costs about $7 per child fully covered per year,[3] so it would cost $28 to cover a child for four years. Costs for RTS,S are far less certain. Two studies that seem like reasonable starting points suggest costs per child of between $30[4] and $40.[5]

Effectiveness. Our rough estimate is that RTS,S would avert 36%[6] of malaria cases over four years.[7] By way of comparison, we estimate that annual SMC would avert 53% of cases over the same time period.[8] Taken at face value, these figures could imply that SMC is 1.5 to 2 times more cost-effective than RTS,S. But these naïve comparisons aren’t conclusive.[9]

Open questions

There are many open questions whose answers could have a big impact on the cost-effectiveness of a particular opportunity to support RTS,S. Some questions that seem important to us are:

What will it cost to deliver RTS,S? The cost estimates we’ve seen vary widely, and we expect delivery costs to vary from country to country.

Will RTS,S be delivered via routine immunization systems or targeted immediately before seasonal malaria transmission? Routine immunization systems might be less costly, but seasonal delivery could increase effectiveness.

How will RTS,S be layered with other malaria interventions in new implementation areas? We haven’t thoroughly investigated how the presence of LLINs and SMC in addition to RTS,S affect its efficacy in trial contexts, and we’re unsure how layering would affect the cost-effectiveness of delivering RTS,S in new areas.

In conclusion

We’re excited to have another effective tool in the fight against malaria! We’re working to understand whether there are cost-effective opportunities to support wider-scale implementation of RTS,S. Our research team is also keeping an eye on other potential malaria vaccines and malaria control strategies more generally. In the meantime, we anticipate continuing to recommend funding for LLINs and SMC.

Here are two kinds of super high-level explanation for how things turned out in the future:

Someone wanted things to turn out that way.

Selective forces favoured that outcome.

Which one is a better explanation?

This question is one I’m highly uncertain about, and often shows up as a crux when thinking about where to focus one’s efforts when aiming for a good future.

One way of labelling the alternatives is intelligence vs evolution. There’s a cluster of stuff on each side. I’ll try to point to them better by listing points in the clusters:

Intelligence: an agent, someone capably trying to do something, intentional deliberate action, the will of the most powerful entity, consequentialists, unipolar/singleton or few-agent scenarios

Evolution: Moloch (multipolar traps), natural selection, competitive pressures, the incentive landscape, game theoretic solution concepts, highly multipolar or multi-agent scenarios

Under some view, both intelligence and evolution are good explanations for how things turned out. They just amount to taking a different perspective or looking at the situation at a different scale. I agree with this but want to avoid this view here so we can focus on the distinction. So let’s try to make the alternatives mutually exclusive:

Intelligence: someone was trying to produce the outcome

Evolution: no one was trying to produce the outcome

This question is very similar to unipolar vs multipolar, but maybe not the same. My focus is on whether the main determinant of the outcome is “capable trying” vs anything else. This can be for a few agents, it doesn’t require exactly one.

If you think our future will be better explained by intelligence, you might prefer to work on understanding intelligence and related things like:

Decision theory, anthropics, probability, epistemology, goal-directedness, embedded agency, intent alignment – understanding what agents are, how they work, how to build them

Moral philosophy and value learning

{single, multi}/single alignment, and approaches to building AGI that focus on a single agent

If you think our future will be better explained by evolution, you might prefer to work on understanding evolution and related things like:

Economics and sociology, especially areas like social choice theory, game theory, and topics like bargaining and cooperation

Biological and memetic evolution, and the underlying theory, perhaps something like formal darwinism

{multi, single}/multi alignment, and approaches to building AGI that involve multiple agents

Governance, and interventions aimed at improving global coordination

Why believe intelligence explains our future better than evolution? One argument is that intelligence is powerful. The outsized impact humans have had on the planet, and might be expected to have beyond it, is often attributed to their intelligence. The pattern of “some human or humans want X to happen” causing “X happens” occurs very frequently and reliably, and seems to happen via intelligence-like things such as planning and reasoning.

Relatedly, the ideal of a rational agent – something that has beliefs and desires, updates beliefs towards accuracy, and takes actions thereby expected to achieve the desires – looks, almost by construction, like something that would in the limit of capability explain what outcomes actually obtain.

Both of these considerations ignore multipolarity, possibly to their peril. Why believe evolution explains our future better than intelligence? Because it seems to explain a lot of the past and present. Evolution (biological and cultural) has much to say about the kinds of creatures and ideas that are abundant today, and the dynamics that led to this situation. The world currently looks like a competitive marketplace more than like a unified decision-maker.

Will this continue? Will there always be many agents with similar levels of capabilities but different goals? To argue for this, I think there are two types of arguments one could put forward. The fi...

Here are two kinds of super high-level explanation for how things turned out in the future:

Someone wanted things to turn out that way.

Selective forces favoured that outcome.

Which one is a better explanation?

This question is one I’m highly uncertain about, and often shows up as a crux when thinking about where to focus one’s efforts when aiming for a good future.

One way of labelling the alternatives is intelligence vs evolution. There’s a cluster of stuff on each side. I’ll try to point to them better by listing points in the clusters:

Intelligence: an agent, someone capably trying to do something, intentional deliberate action, the will of the most powerful entity, consequentialists, unipolar/singleton or few-agent scenarios

Evolution: Moloch (multipolar traps), natural selection, competitive pressures, the incentive landscape, game theoretic solution concepts, highly multipolar or multi-agent scenarios

Under some view, both intelligence and evolution are good explanations for how things turned out. They just amount to taking a different perspective or looking at the situation at a different scale. I agree with this but want to avoid this view here so we can focus on the distinction. So let’s try to make the alternatives mutually exclusive:

Intelligence: someone was trying to produce the outcome

Evolution: no one was trying to produce the outcome

This question is very similar to unipolar vs multipolar, but maybe not the same. My focus is on whether the main determinant of the outcome is “capable trying” vs anything else. This can be for a few agents, it doesn’t require exactly one.

If you think our future will be better explained by intelligence, you might prefer to work on understanding intelligence and related things like:

Decision theory, anthropics, probability, epistemology, goal-directedness, embedded agency, intent alignment – understanding what agents are, how they work, how to build them

Moral philosophy and value learning

{single, multi}/single alignment, and approaches to building AGI that focus on a single agent

If you think our future will be better explained by evolution, you might prefer to work on understanding evolution and related things like:

Economics and sociology, especially areas like social choice theory, game theory, and topics like bargaining and cooperation

Biological and memetic evolution, and the underlying theory, perhaps something like formal darwinism

{multi, single}/multi alignment, and approaches to building AGI that involve multiple agents

Governance, and interventions aimed at improving global coordination

Why believe intelligence explains our future better than evolution? One argument is that intelligence is powerful. The outsized impact humans have had on the planet, and might be expected to have beyond it, is often attributed to their intelligence. The pattern of “some human or humans want X to happen” causing “X happens” occurs very frequently and reliably, and seems to happen via intelligence-like things such as planning and reasoning.

Relatedly, the ideal of a rational agent – something that has beliefs and desires, updates beliefs towards accuracy, and takes actions thereby expected to achieve the desires – looks, almost by construction, like something that would in the limit of capability explain what outcomes actually obtain.

Both of these considerations ignore multipolarity, possibly to their peril. Why believe evolution explains our future better than intelligence? Because it seems to explain a lot of the past and present. Evolution (biological and cultural) has much to say about the kinds of creatures and ideas that are abundant today, and the dynamics that led to this situation. The world currently looks like a competitive marketplace more than like a unified decision-maker.

Will this continue? Will there always be many agents with similar levels of capabilities but different goals? To argue for this, I think there are two types of arguments one could put forward. The fi...

The book “Managing to Change the World: The Nonprofit Manager's Guide to Getting Results” by Alison Green and Jerry Hauser comes highly recommended from a wide variety of top executive directors of non-profits, and after reading it, I can say these positive recommendations are entirely justified.