Radio Bostrom

By Nick Bostrom and Eliezer Yudkowsky.

Abstract:
The possibility of creating thinking machines raises a host of ethical issues. These questions relate both to ensuring that such machines do not harm humans and other morally relevant beings, and to the moral status of the machines themselves. The first section discusses issues that may arise in the near future of AI. The second section outlines challenges for ensuring that AI operates safely as it approaches humans in its intelligence. The third section outlines how we might assess whether, and in what circumstances, AIs themselves have moral status. In the fourth section, we consider how AIs might differ from humans in certain basic respects relevant to our ethical assessment of them. The final section addresses the issues of creating AIs more intelligent than human, and ensuring that they use their advanced intelligence for good rather than ill.

Read the full paper:
https://nickbostrom.com/ethics/artificial-intelligence.pdf

More episodes at:
https://radiobostrom.com/

---

Outline:

(01:19) Ethics in Machine Learning and Other Domain‐Specific AI Algorithms

(07:23) Artificial General Intelligence

(17:01) Machines with Moral Status

(28:32) Minds with Exotic Properties

(42:45) Superintelligence

(56:39) Conclusion

(57:59) Author biographies

By Nick Bostrom.

Abstract:
Information hazards are risks that arise from the dissemination or the potential dissemination of true information that may cause harm or enable some agent to cause harm. Such hazards are often subtler than direct physical threats, and, as a consequence, are easily overlooked. They can, however, be important. This paper surveys the terrain and proposes a taxonomy.

Read the full paper:
https://nickbostrom.com/information-hazards.pdf

More episodes at:
https://radiobostrom.com/

---

Outline:

(00:18) Abstract

(00:51) Introduction

(05:30) Six information transfer modes

(14:42) Table 1. Typology of Information Hazards

(16:57) Adversarial risks

(30:54) Risks to social organization and markets

(48:50) Risks of irrationality and error

(01:05:27) Risks to valuable states and activities

(01:17:33) Risks from information systems

(01:26:51) Risks from development

(01:38:20) Discussion

(01:47:50) Acknowledgements

By Nick Bostrom, Anders Sandberg, and Matthew van der Merwe.

This is an updated version of The Wisdom of Nature, first published in the book Human Enhancement (Oxford University Press, 2009).

Abstract:
Human beings are a marvel of evolved complexity. When we try to enhance poorly-understood complex evolved systems, our interventions often fail or backfire. It can appear as if there is a “wisdom of nature” which we ignore at our peril. A recognition of this reality can manifest as a vaguely normative intuition, to the effect that it is “hubristic” to try to improve on nature, or that biomedical therapy is ok while enhancement is morally suspect. We suggest that one root of these moral intuitions may be fundamentally prudential rather than ethical. More importantly, we develop a practical heuristic, the “evolutionary optimality challenge”, for evaluating the plausibility that specific candidate biomedical interventions would be safe and effective. This heuristic recognizes the grain of truth contained in “nature knows best” attitudes while providing criteria for identifying the special cases where it may be feasible, with present or near-future technology, to enhance human nature.

Read the full paper:
https://www.nickbostrom.com/evolutionary-optimality.pdf

More episodes at:
https://radiobostrom.com/

---

Outline:

(00:31) Abstract

(01:58) Introduction

(07:22) The Evolutionary Optimality Challenge

(11:13) Altered tradeoffs

(12:18) Evolutionary incapacity

(13:33) Value discordance

(14:47) Altered tradeoffs

(17:50) Changes in resources

(23:24) Changes in demands

(28:44) Evolutionary incapacity

(30:54) Fundamental inability

(32:54) Local optima

(34:17) Example: the appendix

(36:37) Example: the ε4 allele

(37:52) Example: the sickle-cell allele

(42:33) Lags

(45:51) Marker 17

(46:26) Example: lactase persistence

(47:18) Value discordance

(49:05) Example: contraceptives

(50:55) Good for the individual

(55:22) Example: happiness

(56:40) Good for society

(58:18) Example: compassion

(01:00:03) The heuristic

(01:00:30) Current ignorance prevents us from forming any plausible idea about the evolutionary factors at play

(01:01:43) We come up with a plausible idea about the relevant evolutionary factors, and they suggest that the intervention would be harmful

(01:02:31) We come up with several different plausible ideas about the relevant evolutionary factors

(01:03:26) We develop a plausible idea about the relevant evolutionary factors, and they imply we wouldn’t have evolved the enhanced capacity even if it were beneficial

(01:08:23) Conclusion

(01:09:11) References

(01:09:18) Thanks to

By Nick Bostrom.

Abstract:
When water was discovered on Mars, people got very excited. Where there is water, there may be life. Scientists are planning new missions to study the planet up close. NASA’s next Mars rover is scheduled to arrive in 2010. In the decade following, a Mars Sample Return mission might be launched, which would use robotic systems to collect samples of Martian rocks, soils, and atmosphere, and return them to Earth. We could then analyze the sample to see if it contains any traces of life, whether extinct or still active. Such a discovery would be of tremendous scientific significance. What could be more fascinating than discovering life that had evolved entirely independently of life here on Earth? Many people would also find it heartening to learn that we are not entirely alone in this vast cold cosmos.

But I hope that our Mars probes will discover nothing. It would be good news if we find Mars to be completely sterile. Dead rocks and lifeless sands would lift my spirit.

Conversely, if we discovered traces of some simple extinct life form—some bacteria, some algae—it would be bad news. If we found fossils of something more advanced, perhaps something looking like the remnants of a trilobite or even the skeleton of a small mammal, it would be very bad news. The more complex the life we found, the more depressing the news of its existence would be. Scientifically interesting, certainly, but a bad omen for the future of the human race.

Read the full paper:
https://nickbostrom.com/extraterrestrial.pdf

More episodes at:
https://radiobostrom.com

By Nick Bostrom and Toby Ord.

Abstract:
In this article we argue that one prevalent cognitive bias, status quo bias, may be responsible for much of the opposition to human enhancement in general and to genetic cognitive enhancement in particular. Our strategy is as follows: first, we briefly review some of the psychological evidence for the pervasiveness of status quo bias in human decision making. This evidence provides some reason for suspecting that this bias may also be present in analyses of human enhancement ethics. We then propose two versions of a heuristic for reducing status quo bias. Applying this heuristic to consequentialist objections to genetic cognitive enhancements, we show that these objections are affected by status quo bias. When the bias is removed, the objections are revealed as extremely implausible. We conclude that the case for developing and using genetic cognitive enhancements is much stronger than commonly realized.

Read the full paper:
https://nickbostrom.com/ethics/statusquo.pdf

More episodes at:
https://radiobostrom.com/

---

Outline:

(00:32) I. INTRODUCTION

(06:27) II. PSYCHOLOGICAL EVIDENCE OF STATUS QUO BIAS

(20:55) III. A HEURISTIC FOR REDUCING STATUS QUO BIAS

(27:32) Fig. 1

(29:11) The Argument from Evolutionary Adaptation

(34:27) The Argument from Transition Costs

(37:28) The argument from risk

(37:53) Fig. 2

(43:43) The Argument from Person-Affecting Ethics

(50:18) IV. THE DOUBLE REVERSAL TEST

(57:11) V. APPLYING THE REVERSAL TESTS TO OTHER CASES

(01:14:55) Thanks to...

By Nick Bostrom.

Abstract:
Existential risks are those that threaten the entire future of humanity. Many theories of value imply that even relatively small reductions in net existential risk have enormous expected value. Despite their importance, issues surrounding human-extinction risks and related hazards remain poorly understood. In this paper, I clarify the concept of existential risk and develop an improved classification scheme. I discuss the relation between existential risks and basic issues in axiology, and show how existential risk reduction (via the maxipok rule) can serve as a strongly action-guiding principle for utilitarian concerns. I also show how the notion of existential risk suggests a new way of thinking about the ideal of sustainability.

Read the full paper:
https://existential-risk.org/concept

More episodes at:
https://radiobostrom.com/

---

Outline:

(00:15) Abstract

(01:10) Policy Implications

(02:43) The maxipok rule

(10:45) Qualitative risk categories

(17:00) Magnitude of expected loss in existential catastrophe

(26:48) Maxipok

(29:13) Classification of existential risk

(31:19) Human extinction

(34:18) Permanent stagnation

(42:00) Flawed realisation

(45:37) Subsequent ruination

(48:44) Capability and value

(54:16) Some other ethical perspectives

(01:01:57) Existential risk and normative uncertainty

(01:06:07) Keeping our options alive

(01:14:38) Outlook

(01:15:15) Barriers to thought and action

(01:24:36) Grounds for optimism

(01:31:40) Author Information

By Nick Bostrom.

Abstract:
Recounts the Tale of a most vicious Dragon that ate thousands of people every day, and of the actions that the King, the People, and an assembly of Dragonologists took with respect thereto.

Read the full paper:
https://nickbostrom.com/fable/dragon

More episodes at:
https://radiobostrom.com/

By Carl Shulman & Nick Bostrom.

Abstract:
The minds of biological creatures occupy a small corner of a much larger space of possible minds that could be created once we master the technology of artificial intelligence. Yet many of our moral intuitions and practices are based on assumptions about human nature that need not hold for digital minds. This points to the need for moral reflection as we approach the era of advanced machine intelligence. Here we focus on one set of issues, which arise from the prospect of digital minds with superhumanly strong claims to resources and influence. These could arise from the vast collective benefits that mass-produced digital minds could derive from relatively small amounts of resources. Alternatively, they could arise from individual digital minds with superhuman moral status or ability to benefit from resources. Such beings could contribute immense value to the world, and failing to respect their interests could produce a moral catastrophe, while a naive way of respecting them could be disastrous for humanity. A sensible approach requires reforms of our moral norms and institutions along with advance planning regarding what kinds of digital minds we bring into existence.

Read the full paper:
https://nickbostrom.com/papers/digital-minds.pdf

More episodes at:
https://radiobostrom.com/

---

Outline:

(00:39) Abstract

(02:22) Introduction

(07:10) Paths to realizing super-beneficiaries

(09:31) Reproductive capacity

(13:24) Cost of living

(14:55) Subjective speed

(16:54) Hedonic skew

(19:23) Hedonic range

(21:37) Inexpensive preferences

(24:56) Preference strength

(27:49) Objective list goods and flourishing

(31:14) Mind scale

(35:07) Moral and political implications of digital super-beneficiaries

(37:18) Creating super-beneficiaries

(42:13) Sharing the world with super-beneficiaries

(48:37) Discussion

(59:24) Author information

By Nick Bostrom.

Abstract:
Scientific and technological progress might change people’s capabilities or incentives in ways that would destabilize civilization. For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill millions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate. This paper introduces the concept of a vulnerable world: roughly, one in which there is some level of technological development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the ‘semi-anarchic default condition’. Several counterfactual historical and speculative future vulnerabilities are analyzed and arranged into a typology. A general ability to stabilize a vulnerable world would require greatly amplified capacities for preventive policing and global governance. The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit balance of developments towards ubiquitous surveillance or a unipolar world order.

Read the full paper:
https://nickbostrom.com/papers/vulnerable.pdf

More episodes at:
https://radiobostrom.com/

---

Outline:

(00:20) Abstract

(01:45) Policy Implications

(03:48) Is there a black ball in the urn of possible inventions?

(09:05) A thought experiment: easy nukes

(23:21) The vulnerable world hypothesis

(35:44) Typology of vulnerabilities

(35:54) Type-1 (‘easy nukes’)

(41:26) Type-2a (‘safe first strike’)

(52:27) Type-2b (‘worse global warming’)

(59:21) Type-0 (‘surprising strangelets’)

(01:10:11) Achieving stabilization

(01:11:28) Technological relinquishment

(01:20:10) Preference modification

(01:30:26) Some specific countermeasures and their limitations

(01:39:18) Governance gaps

(01:42:09) Preventive policing

(01:55:53) Global governance

(02:01:32) Discussion

(02:24:08) Conclusions

(02:29:16) References

(02:29:25) Author Information