Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some alternative AI safety research projects, published by Michele Campolo on June 28, 2022 on The AI Alignment Forum.

These are some "alternative" (in the sense of non-mainstream) research projects or questions related to AI safety that seem both relevant and underexplored. If instead you think they aren't, let me know in the comments, and feel free to use the ideas as you want if you find them interesting.

Access-to-the-internet scenario and related topics

A potentially catastrophic scenario that appears somewhat frequently in AI safety discourse involves a smarter-than-human AI which gets unrestricted access to the internet, and then bad things happen. For example, the AI manages to persuade or bribe one or more humans so that they perform actions which have a high impact on the world.

What are the worst (i.e. with worst consequences) examples of similar scenarios that already happened in the past? Can we learn anything useful from them?

Considering these scenarios, why is it the case that nothing worse has happened yet? Is it simply because human programmers with bad intentions are not smart enough? Or because the programs/AIs themselves are not agentic enough? I would like to read well-thought arguments on the topic.

Can we learn something from the history of digital viruses? What's the role played by cybersecurity? If we assume that slowing down progress in AI capabilities is not a viable option, can we make the above scenario less likely to happen by changing or improving cybersecurity?

Intuitively, it seems to me that the relation of AI safety with cybersecurity is similar to the relation with interpretability: even though the main objective of the other fields is not the reduction of global catastrophic risk, some of the ideas in those fields are likely to be relevant for AI safety as well.

Cognitive and moral enhancement in bioethics

A few days ago I came across a bioethics paper that immediately made me think of the relation between AI safety and AI capabilities. From the abstract:

"Cognitive enhancement [...] could thus accelerate the advance of science, or its application, and so increase the risk of the development or misuse of weapons of mass destruction. We argue that this is a reason which speaks against the desirability of cognitive enhancement, and the consequent speedier growth of knowledge, if it is not accompanied by an extensive moral enhancement of humankind."

As far as I understand, some researchers in the field are pro cognitive enhancement—sometimes even instrumentally as a way to achieve moral enhancement itself. Others, like the authors above, are much more conservative: they see research into cognitive enhancement as potentially very dangerous, unless accompanied by research into moral enhancement.

Are we going to solve all our alignment problems by reading the literature on cognitive and moral enhancement in bioethics? Probably not. Would it be useful if at least some individuals in AI safety knew more than the surface-level info given here? Personally, I would like that.

Aiming at “acceptably safe” rather than “never catastrophic”

Let's say you own a self-driving car and you are deciding whether to drive or give control to the car. If all you care about is safety of you and others, what matters for your decision is the expected damage of you driving the car versus the expected damage of self-driving.

This is also what we care about on a societal level. It would be great if self-driving cars were perfectly safe, but what is most important is that they are acceptably safe, in the sense that they are safer than the human counterpart they are supposed to replace.

Now, the analogy with AI safety is not straightforward because we don't know to what extent future AIs will replace humans, and also because it will be a matter of “coexistence" (...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Military Artificial Intelligence as Contributor to Global Catastrophic Risk, published by MMMaas on June 27, 2022 on The Effective Altruism Forum.

Note: the below is a chapter by Matthijs Maas, Kayla Matteucci, and Di Cooke (CSER), for a forthcoming book on new avenues of research on Global Catastrophic Risk. The chapter explores whether, or in what ways, the military use of AI could constitute or contribute to Global Catastrophic Risk. A PDF of the chapter is also available at SSRN (see also CSER publication page).

TLDR: The chapter focuses primarily on two distinct proposed scenarios, (1) the use of swarms of Lethal Autonomous Weapons Systems, and potential barriers and disincentives; and (2) the intersection of military AI and nuclear weapons. We specifically outline six ways in which the use of AI systems in-, around-, or against- nuclear weapons and their command infrastructures could increase the likelihood of nuclear escalation and global catastrophe

This research dovetails with themes explored by others in the community, such as in Christian Ruhl's recent Founders Pledge report on 'risks from autonomous weapons and military AI'. We welcome your thoughts both on the content of the chapter, as well as suggestions for future research in this area could be valuable and interesting, as this chapter is a condensed version of a larger research project that dives into a broader set of ways in which military AI systems could intersect with, contribute to, or compound global catastrophic risk.

(Length note: half of this page's length consists of references & endnotes).

Abstract: Recent years have seen growing attention for the use of AI technologies in warfare, which has been rapidly advancing. This chapter explores in what ways such military AI technologies might contribute to Global Catastrophic Risks (GCR). After reviewing the GCR field’s limited previous engagement with military AI, and giving an overview of recent advances in military AI, this chapter focuses on two risk scenarios that have been proposed. First, we discuss arguments around the use of swarms of Lethal Autonomous Weapons Systems, and suggest that while these systems are concerning, they appear not yet likely to be a GCR in the near-term, on the basis of current and anticipated production limits and costs which make these systems still uncompetitive with extant systems for mass destruction. Second, we delve into the intersection of military AI and nuclear weapons, which we argue has a significantly higher GCR potential.

We review historical debates over when, where, and why nuclear weapons could lead to GCR, along with recent geopolitical developments that could raise these risks further. We then outline six ways in which the use of AI systems in-, around-, or against- nuclear weapons and their command infrastructures could increase the likelihood of nuclear escalation and global catastrophe. The chapter concludes with suggestions for a research agenda that can gain a more comprehensive and multidisciplinary understanding of the potential risks from military AI, both today and in the future.

1. Introduction

It should hardly be surprising that military technologies have featured prominently in public discussions of global catastrophic risk (GCR).1 The prospect of uncontrolled global war stands as one of the oldest and most pervasive scenarios of what total societal disaster would look like. Conflict has always been able to devastate individual societies: in the modern era, technological and scientific progress has steadily increased the ability of state militaries, and possibly others, to inflict catastrophic violence.2–4

There are many such technologies, with Artificial Intelligence (AI) becoming an even more notable one in recent years. Increasingly, experts from numerous fields have begun to focus ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Examples of someone admitting an error or changing a key conclusion, published by Lizka on June 27, 2022 on The Effective Altruism Forum.

I think that changing our minds and owning up to past mistakes are two of the key virtues of effective altruism. I’d love to acknowledge instances of this and try to collect such stories as examples to follow.

Here are some examples to start with:

Hauke Hillebrandt: [updated] Global development interventions are generally more effective than climate change interventions

David Manheim: A Personal (Interim) COVID-19 Postmortem

Holden Karnofsky: Three Key Issues I've Changed My Mind About

Many organizations in effective altruism have pages about their past mistakes

Andrew Gelman: I was looking through the wrong end of the telescope (this one isn’t about EA)

Do you know of others? Please feel free to share examples of yourself doing this.

I think the best examples would have the following properties:

An important error or mind-change

A public change of conclusion or explanation of the error

In a context or conversation that’s relevant to effective altruism

Also relevant:

How we can make it easier to change your mind about cause areas

Epistemic Legibility - EA Forum

Six Ways To Get Along With People Who Are Totally Wrong

If you don't have good evidence one thing is better than another, don't pretend you do

Discussion norms (topic page on the Forum)

P.S. This is partially prompted by the Criticisms Contest. I’d be excited to see people engage with criticisms or red teaming of their work — especially when this leads to real changes of position or conclusion.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are long-form dating profiles productive?, published by AABoyles on June 27, 2022 on LessWrong.

There's been a noticeable trend over the last few years of in this community of writing long-form dating profiles. They have the benefit of being slightly costly signals, requiring significantly more time and attention and a much greater degree of disclosure than can be devoted to a traditional online dating profile. But they are (at present) publicized and distributed manually, sharply limiting the possible audience.

Has anyone who's written one found a date (and subsequently, relationship) by doing so?

Separately, if you've written one, regardless of whether you met anyone because of it, do you believe it was worth the effort?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Seven ways to become unstoppably agentic, published by Evie Cottrell on June 26, 2022 on LessWrong.

Nearly all of the best things that have happened to me in the last year have been a result of actively seeking and asking for those things. I have slowly been developing the skill of agency.

By ‘agency’ I mean the ability to get what you want across different contexts – the general skill of coming up with ambitious goals and actually achieving them, whatever they are. In my experience, I’ve found this is learnable.

Neel Nanda explains the concept of agency really well in this blog post, so I won’t repeat it here. Instead, I’ll focus on how to learn it.

It’s worth acknowledging that agency is often socially discouraged in different minority groups. Speaking from my own experience, I used to feel shameful around being agentic – I associated it with being entitled or ‘too much’ or bossy – which seems to be a phenomenon that many women experience. I sometimes still have a voice in my head saying, “who do you think you are???” But, fortunately, I’m paying less and less attention to it.

In no particular order, here are seven ways I’ve found to become unstoppably agentic:

1. Figure out what you need, figure out who can help you get it, ask them for it

There will (nearly) always be people who can help you achieve your goals better. Find them and ask them to help you. They might say yes!

If you don’t ask, the answer is always no.

When I was self-studying for my A-Levels (UK high school exams), I reached out to an EA who I’d never met, asking to chat about why I was finding self-studying so hard. We ended up having a call, which I found extremely helpful. At the time, I had a huge ugh-field around my Anki flashcards, which he completely helped me clear up. I then started sending him my daily plan (after checking this was ok), which turned out to be a really valuable accountability system. He gave support and encouragement countless times over the few months. These daily check-ins were the best thing that could have happened to my A-Levels, and led me to feel way more supported throughout my time self-studying.

Having a low bar for asking for help is a learnable mindset. Now, whenever I encounter something difficult or aversive, my first thought is, “who would be the best person to have on my side here?” Then I begin to reach out to people who could help.

Things that make this hard in practice:

It’s hard to actually know what you want or need;

It’s hard to be creative about solutions and imagine how another person would help solve it;

It can be intimidating to ask the person;

You are making yourself vulnerable to rejection, which can feel crushing and bad.

Some tentative solutions:

Questions that help me figure out what I need: If this was easy, what would it look like? What is the bottleneck here and what can I do to solve it? What would it look like if I could do this with 10% of the time/ effort? What’s weighing on my mind? Whose advice would help me solve this problem? See this list of questions for more.

One suggestion is that you can ask people if they know anyone who can help you. I recently asked an EA for help with this, and they generated nearly ten people who I could reach out to. (Also see point number 7 about considering a wide option space to be creative about solutions.)

People can be scary! One thing to bear in mind though, is that EAs really love agency. So if the person you are asking for help is an EA, they’re likely to be impressed that you were so proactive. But, yup, it’s scary to put yourself out there. I really think it’s worth doing anyway. Ultimately, there becomes a point where you have to choose to not let intimidation hold you back.

If you decide to experiment with this strategy, you could just be explicit about that. For example, caveating a request ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Germans' Opinions on Translations of "longtermism": Survey Results, published by Konstantin Pilz on June 27, 2022 on The Effective Altruism Forum.

TL;DR

I conducted a survey with 32 respondents via Mechanical Turk about which word for “longtermism” Germans prefer. Most think that the English word works fine and should not necessarily be translated. However, the word “Zukunftsschutz” (future protection) gets the highest ranking, even higher than “Longtermism” itself. As the word has a very positive connotation in German, it might work well for outreach purposes. However, since existing media articles already use “Longtermism”, I suppose publishing the translated content with the same framing.

Introduction

I am currently working part-time on translating more key longtermist content to German. Since this may shape the debate on the topic for many years, I want to do this carefully and be open to adapting it to the German culture and language. After all, the target group of the content is Germans that have not yet heard of longtermism.

I ran this competition for the best German translation of “longtermism”. Thank you to everyone who participated!

I ended up with more useful words than I had hoped. I thus picked the most promising ones and surveyed the German public’s preferences.

The words I used were:

Longtermism

Longtermismus (just a Germanification of the word)

Langzeitismus (a rather literary German translation)

Zukunftismus “futurism” (though futurism translate to Futurismus so it does not have that connotation)

Zukunftsschutz “future protection”

Langzeitdenken/ Langfristdenken “long-term thinking” Sorry, I confused the two words in the survey, though I think they are reasonably close to still make the results interpretable

Ganzzeitdenken “inclusive time thinking” “whole-time thinking” “thinking about all time”

There was some discussion on the term Zukunftsethik (“future ethics”) though I decided not to include it as it sounds more like a field than an ideology.

Methods

Consider skipping this part.

The survey was conducted on Mechanical Turk. Respondents were paid 2€ for spending ~ 5-10 minutes. The only requirement was having Germany as a location. This means respondents were not necessarily native Germans, but I have no data on this. (I could not include Austria and Switzerland though I would be surprised to see different results there.) Besides that, I collected no data on age, gender, education, etc.

Before launching the final version, a small version was tested to see whether the platform worked well and whether the answers were reasonable.

The final survey had 35 respondents.

You can find the original survey here and the results here (raw) and here (edited).

I presented the core idea of longtermism and then asked participants to summarize it.

Afterward, the first set of questions followed, presenting each German word in random order in the context of a constant German sentence, e.g. "Zukunftismus" ist die Einstellung, dass der Schutz künftiger Generationen stärker priorisiert werden soll.

Another set of questions put the same words in another order in the context of a member of the movement, e.g. Clara, überzeugte "Langzeitdenkerin", macht sich dafür stark, deutlich mehr Geld in die Pandemieprävention zu investieren.

The words used were: Longtermism, Longtermismus, Zukunftismus, Langzeitismus, Zukunftsschutz, Langfristdenken, Ganzzeitdenken

I realized I had mixed up the word Langfristdenken and used Langzeitdenkerin as the corresponding word.

Participants were asked to rate each word on a scale from 1 to 6.

In the end, participants were asked if they thought not translating the word at all would be better on a scale from 0 to 6.

Finally, I asked if they knew about Effective Altruism, though this question was not stated well and I suspect some participants did ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exploring Mild Behaviour in Embedded Agents, published by Megan Kinniment on June 27, 2022 on The AI Alignment Forum.

Thanks to Tristan Cook, Emery Cooper, Nicolas Macé and Will Payne for discussions, advice, and comments. All views are my own.

I think this post is best viewed as an effort to point at a potentially interesting research direction. The specific arguments I make centered around inducing mild behaviour in embedded agents are rough and exploratory.

Summary

If an embedded agent's utility function is heavily dependent on the parts of the environment which constitute its own thinking processes, we might see behaviour that does not have an equivalent in unembedded agents.

(This seems pretty likely to me. Of all the claims here I am most confident of this one.)

We might be able to design measures/utility functions that constrain the parts of the environment that constitute the agent’s thinking process itself, such that we can moderate the strength of the optimisation that an embedded agent exerts on the rest of the environment.

(This seems in theory possible to me but I am currently very uncertain about whether this is feasible in practice.)

Following this, we may be able to induce mild optimisation in EU maximising embedded agents through the specification of the agent’s utility function, and this induced ~mild optimisation could help protect that very same mildness-inducing specification from being gamed.

(Similar thoughts to above)

Measures that impose optimisation pressure on the thinking process might be more robust than usual to specification problems if they can induce some degree of mild optimisation even when they are not specified in a water-tight manner. To the extent that we can manage to specify a mediocre efficiency measure that induces even some small degree of mild optimisation, it will also slightly reduce the optimisation pressure the agent puts on the specification as a whole. In this way, there might be some potential for measures like these to be made somewhat self-protecting against specification problems like edge instantiation and nearest unblocked strategies.

(Again, similar thoughts to above, though I do currently find this general idea promising.)

Context

A fundamental property of embedded agents is that the agent’s thinking process is a part of the environment itself (for example, some process running on a physical chip). This can be a driver of certain ‘bad’ behaviours like wireheading, unwanted self-preservation, or extreme resource acquisition for the sake of self-improvement. However, perhaps we can also use the fact that embedded agents have physically instantiated ‘thinking processes’ to try and induce safer behaviour.

Two major things that make alignment so difficult are:

Avoiding small degrees of misspecification of ‘what we actually want’ is very difficult

The specification is likely to be put under an enormous amount of optimisation pressure by a powerful optimiser (which is likely to ‘find’ and ‘blow up the consequences’ of any small misspecification)

These considerations motivate work on approaches such as designing low-impact measures or methods of implementing mild optimisation. Low-impact measures mostly address the first point, by aiming to heuristically specify part of ‘what we actually want’ whilst also reducing the consequences of misspecifications. Mild optimisation mostly addresses the second point, by aiming to ‘tone down’ the optimisation pressure.

Embedded agents have physical instantiations of the optimisation processes that they use to decide how to optimise their environment (e.g. some algorithm running on a physical chip). What would happen if we attempted to construct a utility function that strongly incentivises the agent to constrain the physical instantiation of its own optimising process? Might it be...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Custom iPhone Widget to Encourage EA Forum Use, published by Will Payne on June 27, 2022 on The Effective Altruism Forum.

TLDR: I made a little custom iPhone widget for the EA Forum recently and have been getting a lot of value out of it. At the moment I don't feel bothered to make this into an app which is easy to install, but I think people could set it up themselves in ~5 mins or less, and if you don't use the forum much but want to use it more it might help.

What is it?

I set up a EA Forum widget using a little iPhone personalisation tool called Scriptable. It shows the details of a random front page forum post once per hour. And it looks like this...

At the time of writing I haven't read Ozymandias' post

Why I think it's good

I've noticed that when I interact with the forum I usually enjoy it. However, I usually don't reach for the forum over other easily available sources of entertainment like YouTube, Instagram, or TikTok (although don't worry, I deleted TikTok). I made this widget initially to try and make the Forum an easy reach alternative to these apps. I think it worked; my usage of the forum went from once per month (or less) to once or twice a day. At one point I removed the widget and didn't get around to re-adding it for a few months, and my forum usage dropped down again. Since reinstalling it a few days ago, it has jumped back up. I’d put this then at somewhere around a 10 - 100x increase in value out of the forum for me, but don’t know yet if it will work long term or just hold my attention for 1-2 week chunks.

How to set it up

This works for iPhone and presumably iPad (although I haven't been able to test it on iPad).

Download the Scriptable app then open it

Click the little plus icon in the top right corner

Paste this code into the editor

Go to your home screen

Long press on the home screen or any app to enter jiggle mode

Click the plus in the top left to add a widget

Find Scriptable (You can use the search bar)

Pick the midsized view and click "Add widget" at the bottom

Tap on the widget

Click choose script

Pick the script you just made (If you didn't actively change the name it should be the only option called "Untitled Script")

Tap anywhere else on the screen to exit jiggle mode.

You're done! I just ran through this process and it took a little over 1 minute. Here's a video.

I hope this helps you as much as it's helped me.

Since the EA Forum, LessWrong and the Alignment Forum all use the same code you can replace getForumPosts(EAForum) with getForumPosts(LessWrong) or getForumPosts(AlignmentForum).

For what it's worth, I don't advocate copying random code you see on the Internet and running it, but we're all friends here. (Plus I trust the community to have someone who feels like reading through the above and checking it's not horrifically unsafe and confirming in the comments.)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deliberation Everywhere: Simple Examples, published by Oliver Sourbut on June 27, 2022 on The AI Alignment Forum.

The analysis and definitions used here are tentative. My familiarity with the concrete systems discussed ranges from rough understanding (markets and parliaments), through abiding amateur interest (biology), to meaningful professional expertise (AI/ML things). The abstractions and terminology have been refined in conversation and private reflection, and the following examples are both generators and products of this conceptual framework.

We previously discussed a conceptual algorithmic breakdown of some aspects of goal-directed behaviour with the intention of inspiring insights and clarifying thought and discussion around these topics.

The examples presented here include some original motivating examples, some used to refine the concepts, and others drawn from the menagerie after the concepts were mostly refined[1]. Each example is subjected to the analysis, in several cases drawing out novel insights as a consequence.

Most of these examples, for all their intricacy in some cases, are relatively 'simple' as deliberators, and I am quite confident in the applicability of the framing. Analysis of more derived and sophisticated deliberative systems is reserved for upcoming posts.

Brief framework summary

We decompose 'deliberation' into 'proposal', 'promotion', and 'action'.

Propose:S→Δ{X}nonempty (generate candidate proposals)

Promote:S→{X}→{V} (promote and demote proposals according to some criterion)

Act:{X×V}→A (take outcome of promotion and demotion to activity in the environment)

We also identify as important whether a deliberator's actions are final, or give rise to relevantly-algorithmically-similar subsequent deliberators (iteration and replication), or create or otherwise condition heterogeneous deliberators (recursive deliberation).

Reaction examples

Chemical systems

Innumerable basic chemical reactions, like oxidation of iron, involve actions which change the composition or configuration of some material(s). For the purposes of this analysis these are, alone, mostly uninteresting, but serve to illustrate natural systems which do not preserve their essential algorithmic form and thus do not constitute iterated systems.

Some reactions, on the other hand, involve catalysis, wherein some reagents are essentially preserved or reconstituted in the action, producing the seeds of iterated algorithmic reaction, thus basic 'control'.

Despite pushing in a particular 'goal' direction, these systems are reactions rather than proper deliberations, because they occur without computing alternative pathways[2].

Biological systems

Even very simple organisms can react (that is, act or perform some function in response) to stimuli. No proper deliberation need be involved, no computation instantiated to consider alternatives.

A reaction can take place with or without a brain, or even a nervous system: consider many motions of single-celled organisms[3] or the snapping of a Venus flytrap. The cringe of many animals from intense heat goes via nervous circuitry but takes no deliberation.

Indeed, temperature changes are pervasive in nature, so it is no surprise that we find automatic heat-responsive behaviour at the protein-machinery level in every lineage of cellular life. These latter, along with other protein machinery and the organ-functions of multicellular organisms, demonstrate that, in nature, a single organism will be found to consist of many reactive and deliberative systems.

The class of 'systems undergoing a transformation' in chemical or physical interactions often does not preserve the essential algorithmic characteristics of the system, but most salient biological examples are iterated (the act essentially preserves the capacity to further act in an algorith...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deliberation, Reactions, and Control: Tentative Definitions and a Restatement of Instrumental Convergence, published by Oliver Sourbut on June 27, 2022 on The AI Alignment Forum.

This analysis is speculative. The framing has been refined in conversation and private reflection and research. To some extent it feels vacuous, but at least valuable for further research and communication.

A cluster of questions fundamental to many concerns around risks from artificial systems regard the concepts of search, planning, and 'deliberateness'. How do these arise? What can we predict about their occurrence and their consequences? How strong are they? What are they anyway?

Here is laid out one part of a conceptual decomposition which maps well onto many known systems and may allow further work towards answering more of those questions. The ambition is to really get at the heart of what is algorithmically happening in 'optimising systems', including humans, animals, algorithmic optimisers like SGD, and contemporary and future computational artefacts. That said, I do not have any privileged insight into the source code (or its proper interpretation!) for the examples discussed, so while this framing has already generated new insights for me, it may or may not be 'the actual algorithmic truth'.

We start with the analysis: a definition of 'deliberation' and its components, then of 'reactions' and 'control'. Next we consider, in light of these, what makes a deliberator or controller 'good'. We find conceptual connections with discussions of instrumental convergence. Little attention is given here to how to determine what the goals are, which is obviously also important.

These concepts were generated by contemplating various aspects of many different goal-directed systems and pulling out commonalities.

Some readers may prefer to start with the examples, which include animals, plants, natural selection, gradient descent, bureaucracies, and others. Here in the conceptual section I'll footnote particularly relevant concrete examples where I anticipate them helping to convey my point.

A full treatment is absent, but two major deferences to embedded agency underlie this analysis. A Cartesian separation need not be assumed, except over 'actor-moments' rather than temporally-extended 'actors'. And a major driver for this sequence is a fundamental recognition that any goal-directed behaviour instantiated in the real world must have bounded computational capacity per time[1].

Inspiration and related

My (very brief) 'Only One Shot' intuition pump for embedded agency may help to convey some background assumptions (especially regarding how time and actor-moments fit into this picture).

Scott Garrabrant's (A→B)→A talks about 'agency' and 'doing things on purpose'. I'm trying to unpack that further. The (open) question Does Agent-like Behaviour Imply Agent-like Architecture? is related and I hope for the perspective here to be useful toward answering that question.

Alex Flint's excellent piece The ground of optimization informs some of the perspective here, especially a focus on scope of generalisation and robustness to perturbation.

Daniel Filan's Bottle Caps Aren't Optimisers and Abram Demski's Selection vs Control begin discussing the algorithmic internals of optimising systems, which is the intent here also. Risks from Learned Optimization is of course relevant.

John Wentworth's discussions of abstraction (for example What is Abstraction?) especially with regards its predictive properties for a non-omniscient computer, are central to the notion of abstraction employed here. Related are the good and gooder regulator theorems, which touch closely computationally upstream of the aspects discussed here while making fewer concessions to embedded agency.

Definitions

Deliberation

A deliberation is any p...