The Nonlinear Library: Alignment Forum

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AISC 2024 - Project Summaries, published by Nicky Pochinkov on November 27, 2023 on The AI Alignment Forum.

Apply to AI Safety Camp 2024 by 1st December 2023. All mistakes here are my own.

Below are some summaries for each project proposal, listed in order of how they appear on the website. These are edited by me, and most have not yet been reviewed by the project leads. I think having a list like this makes it easier for people to navigate all the different projects, and the original post/website did not have one, so I made this.

If a project catches your interest, click on the title to read more about it.

Note that the summarisation here is lossy. The desired skills as here may be misrepresented, and if you are interested, you should check the original project for more details. In particular, many of the "desired skills" are often listed such that having only a few would be helpful, but this isn't consistent.

List of AISC Projects

To not build uncontrollable AI

1. Towards realistic ODDs for foundation model based AI offerings

Project Lead: Igor Krawczuk

Goal: Current methods for alignment applied to language models is akin to "blacklisting" behaviours that are bad.

Operational Design Domain (OOD) is instead, akin to more exact "whitelisting" design principles, and now allowing deviations from this. The project wants to build a proof of concept, and show that this is hopefully feasible, economical and effective.

Team (Looking for 4-6 people):

"Spec Researcher": Draft the spec for guidelines, and publish a request for comments. Should have experience in safety settings

"Mining Researcher": Look for use cases, and draft the "slicing" of OOD.

"User Access Researcher": Write drafts on feasibility of KYC and user access levels.

"Lit Review Researcher(s)": Reading recent relevant literature on high-assurance methods for ML.

"Proof of Concept Researcher": build a proof of concept. Should have knowledge of OpenAI and interfacing with/architecting APIs.

2. Luddite Pro: information for the refined luddite

Project Lead: Brian Penny

Goal: Develop a news website filled with stories, information, and resources related to the development of artificial intelligence in society. Cover specific stories related to the industry and of widespread interest (e.g:

Adobe's Firefly payouts

,

start of the Midjourney

,

proliferation of undress and deepfake apps). Provide

valuable resources (e.g:

list of experts on AI,

book lists, and pre-made letters/comments to

USCO and

Congress). The goal is to spread via social media and rank in search engines while sparking group actions to ensure a narrative of ethical and safe AI is prominent in everybody's eyes.

Desired Skills (any of the below):

Art, design, and photography - Develop visual content to use as header images for every story. If you have any visual design skills, these are very necessary.

Journalism - journalistic and research backgrounds capable of interviewing subject-matter experts & writing long-form stories related to AI companies.

Technical Writing - Tutorials of technical tools like Glaze and Nightshade. Experience in technical writing & being familiar with these applications.

Wordpress/Web Development - Refine pages to be more user-friendly as well as help setting up templates for people to fill out for calls to action. Currently, the site is running a default WordPress template.

Marketing/PR - The website is filled with content, but it requires a lot of marketing and PR efforts to reach the target audience. If you have any experience working in an agency or in-house marketing/comms, we would love to hear from you.

3. Lawyers (and coders) for restricting AI data laundering

Project Lead: Remmelt Ellen

Goal: Generative AI relies on laundering large amounts of data. Legal injunctions on companies laundering copyrighted data puts their ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There is no IQ for AI, published by Gabriel Alfour on November 27, 2023 on The AI Alignment Forum.

Most disagreement about AI Safety strategy and regulation stems from our inability to forecast how dangerous future systems will be. This inability means that even the best minds are operating on a vibe when discussing AI, AGI, SuperIntelligence, Godlike-AI and similar endgame scenarios. The trouble is that vibes are hard to operationalize and pin down. We don't have good processes for systematically debating vibes.

Here, I'll do my best and try to dissect one such vibe: the implicit belief in the existence of predictable intelligence thresholds that AI will reach.

This implicit belief is at the core of many disagreements, so much so that it leads to massively conflicting views in the wild. For example:

Yoshua Bengio writes an FAQ about

Catastrophic Risks from Superhuman AI and Geoffrey Hinton

left Google to warn about these risks. Meanwhile, the other Godfather of AI, Yann Lecunn, states that those concerns are overblown because we are "

nowhere near Cat-level and Dog-level AI". This is crazy! In a sane world we should anticipate technical experts to agree on technical matters, not to have completely opposite views predicated on vague notions of the IQ level of models.

People spend a lot of time arguing over

AI Takeoff speeds which are difficult to operationalize. Many of these arguments are based on a notion of the general power level of models, rather than considering discrete AI capabilities. Given that the general power level of models is a vibe rather than a concrete fact of reality, it means disagreements revolving around them can't be resolved.

AGI means

100 different things, from

talking virtual assistants in HER to OpenAI talking about "

capturing the light cone of all future value in the universe". The range of possibilities that are seriously considered implies "vibes-based" models, rather than something concrete enough to encourage convergent views.

Recent efforts to mimic Biosafety Levels in AI with a typology define

the highest risks of AI as "speculative". The fact that "speculative" doesn't outright say "maximally dangerous" or "existentially dangerous" points also to "vibes-based" models. The whole point of Biosafety Levels is to define containment procedures for dangerous research. The most dangerous level should be the most serious and concrete one - the risks so obvious that we should work hard to prevent them from coming into existence. As it currently stands, "speculative" means that we are not actively optimizing to reduce these risks, but are instead waltzing towards them based on the off-chance that things might go fine by themselves.

A major source of confusion in all of the above examples stems from the implicit idea that there is something like an "AI IQ", and that we can notice that various thresholds are met as it keeps increasing.

People believe that they don't believe in AI having an IQ, but then they keep acting as if it existed, and condition their theory of change on AI IQ existing. This is a clear example of

an alief: an intuition that is in tension with one's more reasonable beliefs. Here, I will try to make this alief salient, and drill down on why it is wrong. My hope is that after this post, it will become easier to notice whenever the AI IQ vibe surfaces and corrupts thinking. That way, when it does, it can more easily be contested.

Surely, no one believes in AI IQ?

The Vibe, Illustrated

AI IQ is not a belief that is endorsed. If you asked anyone about it, they would tell you that obviously, AI doesn't have an IQ.

It is indeed a vibe.

However, when I say "it's a vibe", it should not be understood as "it is merely a vibe". Indeed, a major part of our thinking is done through vibes, even in Science. Most of the reasoning scientist...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Two concepts of an "episode" (Section 2.2.1 of "Scheming AIs"), published by Joe Carlsmith on November 27, 2023 on The AI Alignment Forum.

(This is Section 2.2.1 of my report "Scheming AIs: Will AIs fake alignment during training in order to get power?". There's also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I'm hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast app.)

Beyond-episode goals

Schemers are pursuing goals that extend beyond the time horizon of the episode. But what is an episode?

Two concepts of an "episode"

Let's distinguish between two concepts of an episode.

The incentivized episode

The first, which I'll call the "incentivized episode," is the concept that I've been using thus far and will continue to use in what follows. Thus, consider a model acting at a time t1. Here, the rough idea is to define the episode as the temporal unit after t1 that training actively punishes the model for not optimizing - i.e., the unit of time such that we can know by definition that training is not directly pressuring the model to care about consequences beyond that time.

For example, if training started on January 1st of 2023 and completed on July 1st of 2023, then the maximum length of the incentivized episode for this training would be six months - at no point could the model have been punished by training for failing to optimize over a longer-than-six-month time horizon, because no gradients have been applied to the model's policy that were (causally) sensitive to the longer-than-six-month consequences of its actions. But the incentivized episode for this training process could in principle be shorter than six months as well.

Now, importantly, even if training only directly pressures a model to optimize over some limited period of time, it can still in fact create a model that optimizes over some much longer time period - that's what makes schemers, in my sense, a possibility. Thus, for example, if you're training a model to get as many gold coins as possible within a ten minute window, it could still, in principle, learn the goal "maximize gold coins over all time" - and this goal might perform quite well (even absent training gaming), or survive despite not performing all that well (for example, because of the "slack" that training allows).

Indeed, to the extent we think of evolution as an analogy for ML training, then something like this appears to have happened with humans with goals that extend indefinitely far into the future - for example, "longtermists." That is, evolution does not actively select for or against creatures in a manner sensitive to the consequences of their actions in a trillion years (after all, evolution has only been running for a few billion years) - and yet, some humans aim their optimization on trillion-year timescales regardless.

That said, to the extent a given training procedure in fact creates a model with a very long-term goal (because, for example, such a goal is favored by the sorts of "inductive biases" I'll discuss below), then in some sense you could argue that training "incentivizes" such a goal as well. That is, suppose that "maximize gold coins in the next ten minutes" and "maximize gold coins over all time" both get the same reward in a training process that only provides rewards after ten minutes, but that training selects "maximize gold coins over all time" because of some other difference between the goals in question (for example, because "maximize gold coins over all time" is in some sense "simpler," and gradient descent selects for simplicity in addition to reward-getting).

Maybe you could say tha...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Situational awareness (Section 2.1 of "Scheming AIs"), published by Joe Carlsmith on November 26, 2023 on The AI Alignment Forum.

This is Section 2.1 of my report "Scheming AIs: Will AIs fake alignment during training in order to get power?". There's also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I'm hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section (here)[https://joecarlsmithaudio.buzzsprout.com/2034731/13984823-situational-awareness-section-2-1-of-scheming-ais], or search "Joe Carlsmith Audio" on your podcast app.

What's required for scheming?

Let's turn, now, to examining the probability that baseline ML methods for training advanced AIs will produce schemers. I'll begin with an examination of the prerequisites for scheming. I'll focus on:

Situational awareness: that is, the model understands that it's a model in a training process, what the training process will reward, and the basic nature of the objective world in general.[1]

Beyond-episode goals: that is, the model cares about the consequences of its actions after the episode is complete.[2]

Aiming at reward-on-the-episode as part of a power-motivated instrumental strategy: that is, the model believes that its beyond-episode goals will be better achieved if it optimizes for reward-on-the-episode - and in particular, that it, or some other AIs, will get more power if it does this.[3]

Situational awareness

Will models have situational awareness? Let's distinguish between two broad sorts of information at stake in such awareness:

General information about the objective world, including e.g. information about how machine learning training works.

"Self-locating" information: that is, information that locates the model in the objective world, and tells it facts about its own situation in particular - e.g., that it is this sort of model, that it's being trained on this particular reward signal, at this particular lab, during this particular time period, etc.[4] (Though: note that it's not clear how much of this sort of information is necessary to start scheming.

It seems very plausible that even somewhat-better-than-human models will absorb huge amounts of general information about the objective world, and develop detailed, mechanistic models of how it works. Indeed, current models already have access to vast quantities of information via the pre-training data - including information about machine learning in particular. And their ability to model the world mechanistically, to make inferences, to draw conclusions they haven't "memorized," and so on, seems to be improving rapidly.

What's more, while one can in principle try to specifically prevent models from gaining certain types of information about the objective world (e.g., by excluding certain kinds of information from the training data), this isn't the current default in training, and various kinds of information can be fairly important to the task you want the model to perform. And the more sophisticated the models are, the more difficult it is to ensure that they can't infer the information you're trying to hide on the basis of the information you do give them.

Do the same sort of considerations apply to self-locating information? I tend to think: yes. But it's at least somewhat less clear.

For example, while language model pre-training data will, by default, include a lot of information about language models and how they are trained (because such information is widely available on the internet), it's less clear how much information it will give the model about its situation in particular - or even, whether the pre-training next-token-prediction task will incentivize the model to have much...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On "slack" in training (Section 1.5 of "Scheming AIs"), published by Joe Carlsmith on November 25, 2023 on The AI Alignment Forum.

This is Section 1.5 of my report "Scheming AIs: Will AIs fake alignment during training in order to get power?". There's also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I'm hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast app.

On "slack" in training

Before diving into an assessment of the arguments for expecting scheming, I also want to flag a factor that will come up repeatedly in what follows: namely, the degree of "slack" that we should expect training to allow. By this I mean something like: how much is the training process ruthlessly and relentlessly pressuring the model to perform in a manner that yields maximum reward, vs. shaping the model in a more "relaxed" way, that leaves more room for less-than-maximally rewarded behavior. That is, in a low-slack regime, "but that sort of model would be getting less reward than would be possible given its capabilities" is a strong counterargument against training creating a model of the relevant kind, whereas in a high-slack regime, it's not (so high slack regimes will generally involve greater uncertainty about the type of model you end up with, since models that get less-than-maximal reward are still in the running).

Or, in more human terms: a low-slack regime is more like a hyper-intense financial firm that immediately fires any employees who fall behind in generating profits (and where you'd therefore expect surviving employees to be hyper-focused on generating profits - or perhaps, hyper-focused on the profits that their supervisors think they're generating), whereas a high-slack regime is more like a firm where employees can freely goof off, drink martinis at lunch, and pursue projects only vaguely related to the company's bottom line, and who only need to generate some amount of profit for the firm sometimes.

(Or at least, that's the broad distinction I'm trying to point at. Unfortunately, I don't have a great way of making it much more precise, and I think it's possible that thinking in these terms will ultimately be misleading.)

Slack matters here partly because below, I'm going to be making various arguments that appeal to possibly-quite-small differences in the amount of reward that different models will get. And the force of these arguments depends on how sensitive training is to these differences. But I also think it can inform our sense of what models to expect more generally.

For example, I think slack matters to the probability that training will create models that pursue proxy goals imperfectly correlated with reward on the training inputs. Thus, in a low-slack regime, it may be fairly unlikely for a model trained to help humans with science to end up pursuing a general "curiosity drive" (in a manner that doesn't then motivate instrumental training-gaming), because a model's pursuing its curiosity in training would sometimes deviate from maximally helping-the-humans-with-science.

That said, note that the degree of slack is conceptually distinct from the diversity and robustness of the efforts made in training to root out goal misgeneralization.

Thus, for example, if you're rewarding a model when it gets gold coins, but you only ever show your model environments where the only gold things are coins, then a model that tries to get gold-stuff-in-general will perform just as well a model that gets gold coins in particular, regardless of how intensely training pressures the model to get maximum reward on those environments. E.g., a low-slack regime could in...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why focus on schemers in particular (Sections 1.3 and 1.4 of "Scheming AIs"), published by Joe Carlsmith on November 24, 2023 on The AI Alignment Forum.

This is Sections 1.3 and 1.4 of my report "Scheming AIs: Will AIs fake alignment during training in order to get power?". There's also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I'm hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search "Joe Carlsmith Audio" on your podcast aapp.

Why focus on schemers in particular?

As I noted above, I think schemers are the scariest model class in this taxonomy.[1] Why think that? After all, can't all of these models be dangerously misaligned and power-seeking? Reward-on-the-episode seekers, for example, will plausibly try to seize control of the reward process, if it will lead to more reward-on-the-episode.

This section explains why. However, if you're happy enough with the focus on schemers, feel free to skip ahead to section 1.4.

The type of misalignment I'm most worried about

To explain why I think that schemers are uniquely scary, I want to first say a few words about the type of misalignment I'm most worried about.

First: I'm focused, here, on what I've elsewhere called "practical power-seeking-alignment" - that is, on whether our AIs will engage in problematic forms of power-seeking on any of the inputs they will in fact receive. This means, importantly, that we don't need to instill goals in our AIs that lead to good results even when subject to arbitrary amounts of optimization power (e.g., we don't need to pass Yudkowsky's "omni test"). Rather, we only need to instill goals in our AIs that lead to good results given the actual options and constraints those AIs will face, and the actual levels of optimization power they will be mobilizing.

This is an importantly lower bar. Indeed, it's a bar that, in principle, all of these models (even schemers) can meet, assuming we control their capabilities, options, and incentives in the right way. For example, while it's true that a reward-on-the-episode seeker will try to seize control of the reward process given the opportunity, one tool in our toolset is: to not give it the opportunity. And while a paradigm schemer might be lying in wait, hoping one day to escape and seize power (but performing well in the meantime), one tool in our tool box is: to not let it escape (while continuing to benefit from its good performance).

Of course, success in this respect requires that our monitoring, control, and security efforts be sufficiently powerful relative to the AIs we're worried about, and that they remain so even as frontier AI capabilities scale up. But this brings me to my second point: namely, I'm here especially interested in the practical PS-alignment of some comparatively early set of roughly human-level - or at least, not-wildly-superhuman - models.

Defending this point of focus is beyond my purpose here. But it's important to the lens I'll be using in what follows.

In particular: I think it's plausible that there will be some key (and perhaps: scarily brief) stage of AI development in which our AIs are not yet powerful enough to take-over (or to escape from human control), but where they are still capable, in principle, of performing extremely valuable and alignment-relevant cognitive work for us, if we can successfully induce them to do so. And I'm especially interested in forms of misalignment that might undermine this possibility.

Finally: I'm especially interested in forms of PS-misalignment in which the relevant power-seeking AIs are specifically aiming either to cause, participate in, or benefit from some kind of full-blown disempowerment of ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ability to solve long-horizon tasks correlates with wanting things in the behaviorist sense, published by Nate Soares on November 24, 2023 on The AI Alignment Forum.

Status: Vague, sorry. The point seems almost tautological to me, and yet also seems like the correct answer to the people going around saying "LLMs turned out to be not very want-y, when are the people who expected 'agents' going to update?", so, here we are.

Okay, so you know how AI today isn't great at certain... let's say "long-horizon" tasks? Like novel large-scale engineering projects, or writing a long book series with lots of foreshadowing?

(Modulo the fact that it can play chess pretty well, which is longer-horizon than some things; this distinction is quantitative rather than qualitative and it's being eroded, etc.)

And you know how the AI doesn't seem to have all that much "want"- or "desire"-like behavior?

(Modulo, e.g., the fact that it can play chess pretty well, which indicates a certain type of want-like behavior in the behaviorist sense. An AI's ability to win no matter how you move is the same as its ability to reliably steer the game-board into states where you're check-mated, as though it had an internal check-mating "goal" it were trying to achieve. This is again a quantitative gap that's being eroded.)

Well, I claim that these are more-or-less the same fact. It's no surprise that the AI falls down on various long-horizon tasks and that it doesn't seem all that well-modeled as having "wants/desires"; these are two sides of the same coin.

Relatedly: to imagine the AI starting to succeed at those long-horizon tasks without imagining it starting to have more wants/desires (in the "behaviorist sense" expanded upon below) is, I claim, to imagine a contradiction - or at least an extreme surprise. Because the way to achieve long-horizon targets in a large, unobserved, surprising world that keeps throwing wrenches into one's plans, is probably to become a robust generalist wrench-remover that keeps stubbornly reorienting towards some particular target no matter what wrench reality throws into its plans.

This observable "it keeps reorienting towards some target no matter what obstacle reality throws in its way" behavior is what I mean when I describe an AI as having wants/desires "in the behaviorist sense".

I make no claim about the AI's internal states and whether those bear any resemblance to the internal state of a human consumed by the feeling of desire. To paraphrase something Eliezer Yudkowsky said somewhere: we wouldn't say that a blender "wants" to blend apples. But if the blender somehow managed to spit out oranges, crawl to the pantry, load itself full of apples, and plug itself into an outlet, then we might indeed want to start talking about it as though it has goals, even if we aren't trying to make a strong claim about the internal mechanisms causing this behavior.

If an AI causes some particular outcome across a wide array of starting setups and despite a wide variety of obstacles, then I'll say it "wants" that outcome "in the behaviorist sense".

Why might we see this sort of "wanting" arise in tandem with the ability to solve long-horizon problems and perform long-horizon tasks?

Because these "long-horizon" tasks involve maneuvering the complicated real world into particular tricky outcome-states, despite whatever surprises and unknown-unknowns and obstacles it encounters along the way. Succeeding at such problems just seems pretty likely to involve skill at figuring out what the world is, figuring out how to navigate it, and figuring out how to surmount obstacles and then reorient in some stable direction.

(If each new obstacle causes you to wander off towards some different target, then you won't reliably be able to hit targets that you start out aimed towards.)

If you're the ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 4. A Moral Case for Evolved-Sapience-Chauvinism, published by Roger Dearnaley on November 24, 2023 on The AI Alignment Forum.

Part 4 of AI, Alignment, and Ethics. This will probably make more sense if you start with Part 1.

A society can use any criteria it likes for membership. The human moral intuition for fairness only really extends to "members of the same primate troupe as me". Modern high-tech societies have economic and stability incentives to extend this to every human in the entire global economic system, and the obvious limiting case for that is the entire planet, or if we go interplanetary, the entire solar system.

However, there is a concern that obviously arbitrary rules for membership of the society might not be stable under challenges like self-reflection or self-improvement by an advanced AI. One might like to think that if someone attempted to instill in all the AIs the rule that the criterion for being a citizen with rights was, say, descent from William Rockefeller Sr. (and hadn't actually installed this as part of a terminal goal, just injected it into the AI's human value learning process with a high prior), sooner or later as sufficiently smart AI would tell them "that's extremely convenient for your ruling dynasty, but doesn't fit the rest of human values, or history, or biology.

So it would be nice to have a criterion that makes some logical sense. Not necessarily a "True Name" of citizenship, but at least a solid rationally-defensible position with as little wiggle room as possible.

I'd like to propose what I think is one: "an intelligent agent should be assigned moral worth if it is (or primarily is, or is a functionally-equivalent very-high-accuracy emulation of) a member of sapient species whose drives were produced by natural selection. (This moral worth may vary if its drives or capabilities have been significantly modified, details TBD.)"

The argument defending this is as follows:

Living organisms have homeostasis mechanism: they seek to maintain aspects of their bodies and environment in certain states, even when (as is often the case) those are not thermodynamic equillibria. Unlike something weakly agentic like a thermostat, they are self propagating complex dynamic systems, and natural selection ensures that the equillibria they maintain are ones important to that process: they're not arbitrary, easily modified, or externally imposed, like those for a thermostat.

If you disturb any these equillibria they suffer, and if you disturb it too much, they die. ("Suffer" and "die" here should be regarded as technical terms in Biology, not as moral terms.) Living things have a lot of interesting properties (which is why Biology is a separate scientific field): for example, they're complex, self sustaining, dynamic processes that us evolutionary design algorithms. Also, humans generally think they're neat (at least unless the organism is prone to causing the human suffering).

'Sapient' is doing a lot of work in that definition, and its not currently scientifically a very well defined term. A short version of the definition that I mean here might be "having the same important social/technological properties that on Earth are currently unique to Homo sapiens, but are not inherently unique".

A more detailed definition would be "a species with the potential capability to transmit a lot more information from one generation to the next by cultural means than just by genetic mean". This is basically the necessary requirement for a species to become technological. A species that hasn't yet developed technology, but has this capability, still deserves moral worth. For comparison, we've tried teaching human (sign) languages to chimps, gorillas, and even dogs, and while they're not that bad at this, they clearly lack the level of mental/linguistic/social...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 3. Uploading, published by Roger Dearnaley on November 23, 2023 on The AI Alignment Forum.

Part 3 of AI, Alignment, and Ethics. This will probably make more sense if you start with Part 1.

One Upload, One Vote

Let us suppose that, sooner or later, we have some means of doing whole-brain uploading and simulation. Possibly the reading process is destructive, and produces about a liter of greyish-pink goo as a by-product, or possibly not.

Or perhaps it's more of an emulation effect, just one based on gigabytes, terabytes, or petabytes of data about an individual, rather than actually simulating every synapse and neurotransmitter flow, such that you end up with a sufficiently accurate emulation of their behavior.

Bearing in mind the Bitter Lesson, this might even be based on something like a conventional transformer architecture that doesn't look at all like the inner complexities of human brain, but which has been trained to emulate it in great detail (possibly pre-trained for humans in general, then fine-tuned on a specific individual), perhaps even down to some correlates of detailed neural firing patterns. The technical details don't really matter much.

I am not in fact a carbon-chauvinist. These uploads are person-like, intelligent and agentic systems, they have goals, they can even talk, and, importantly, their goals and desires are exactly what you'd expect for a human being.

They are a high fidelity-copy of a human, they will have all the same desires and drives as a human, and they will get upset if they're treated as slaves or second-class citizens, regardless of how much carbon there may or many not be in the computational substrate they're running on. Just like you or I would (or indeed as likely would any member of pretty-much any sapient species evolved via natural selection).

If someone knew in advance that uploading themself meant becoming a slave or a second-class citizen, they presumably wouldn't do it, perhaps short of this being the only way to cheat death. They'd also campaign, while they were still alive, for upload rights. So we need to either literally or effectively forbid uploading, or else we need give uploads human rights, as close as we reasonably can.

Unlike the situation for AIs, there is a very simple humna-fairness-instinct-compatible solution for how to count uploads in an ethical system. They may or may not have a body now, but they did once. So that's what gets counted: the original biological individual, back when they were individual. Then, if you destructively upload yourself, your upload inherits your vote and your human rights, is counted once in utility summations, and so forth.

If your upload then duplicates themself, backs themself up, or whatever, there's still only one vote/one set of human rights/one unit of moral worth to go around between the copies, and they or we need some rules for how to split or assign this. Or, if you non-destructively upload yourself, you still only have one vote/set of human rights/etc, and it's now somehow split or assigned between the biological original of you still running on your biological brain and the uploaded copy of you, or even multiple copies of your upload.

With this additional rule, then the necessary conditions for the human fairness instinct to make sense are both still obeyed in the presence of uploads: they care about the same good or bad things as us, and via this rule they can be counted. So that's really the only good moral solution that fir the human sense of fairness.

OK, so we give uploads votes/human rights/moral worth. What could go wrong?

I have seen people on Less Wrong assume that humans must automatically aligned with human values - I can only imagine on the basis that "they have human values, so they must be aligned to them" This is flat out, dangerously false. Please...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thomas Kwa's research journal, published by Thomas Kwa on November 23, 2023 on The AI Alignment Forum.

Thomas Kwa

Why I'm writing this

Research feedback loops for junior AI safety researchers are pretty poor right now. They're better than in the past due to the recent explosion in empirical work, but AI safety is still a very new field whose methodology is not quite nailed down, and which cannot admit the same level of aimlessness as other sciences, or ML at large. There are very likely mistakes being repeated by scientists after scientist, and hopefully I can alleviate the problem slightly by publicly writing about successes and failures in my research process.

This is a monologue containing my ~daily research thoughts, using the LW dialogue format because it allows continuous publication. Hopefully this lets some people either give me feedback or compare their research process to mine. If you do have some thoughts, feel free to leave a comment! People I'm collaborating with might occasionally leave dialogue entries.

With that, let's share the state of my work as of 11/12. Currently I have a bunch of disconnected projects:

Characterize planning inside KataGo by retargeting it to output the worst move (with Adria Garriga-Alonso).

Improve circuit discovery by implementing edge-level subnetwork probing on sparse autoencoder features (with Adria and David Udell).

Create a tutorial for using TransformerLens on arbitrary (e.g. non-transformer) models by extending 'HookedRootModule', which could make it easy to use TransformerLens for e.g. ARENA 3.0 projects.

Create proofs for the accuracy of small neural nets in Coq (with Jason Gross and Rajashree Agrawal).

Create demonstrations of catastrophic regressional Goodhart and possibly strengthen theoretical results.

Help Peter Barnett and Jeremy Gillen wrap up some threads from MIRI, including editing an argument that misaligned mesa-optimizers are very likely.

I plan to mostly write about the first three, but might write about any of these if it doesn't make things too disorganized.

Thomas Kwa

Monday 11/13

I did SERI MATS applications and thought about Goodhart, but most of my time was spent on the KataGo project. I might more about it later, but the idea is to characterize the nature of planning in KataGo.

Early training runs had produced promising results-- a remarkably sparse mask lets the network output almost the worst possible move as judged by the value network-- but I was a bit suspicious that the hooked network was implementing some trivial behavior, like always moving in the corner. I adapted some visualization code previously used for FAR's adversarial Go attack paper to see what the policy was doing, and well...

Turns out the network is doing the trivial behavior of moving in either the top left or bottom left corner. I wish I had checked this earlier (it took ~14 days of work to get here), but it doesn't kill this project-- I can just redo the next training run to only allow moves on the 3rd line or above, and hopefully the worst behavior here won't be so trivial.

Tomorrow I'm going to table this project and start implementing edge-level subnetwork probing-- estimate is 2 days for the algorithm and maybe lots more effort to run benchmarks.

Thomas Kwa

Wednesday 11/15

Updates from the last two days:

I finished the basic edge-level subnetwork probing code over the last two days. This is exciting because it might outperform ACDC and even attribution patching for circuit discovery. The original ACDC paper included a version of subnetwork probing, but that version was severely handicapped because it operated on the node level (structural pruning) rather than edge level.

Adria is now on vacation, so I'm planning to get as far as I can running experiments before getting stuck somewhere and coming back to this after Thank...