Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Alignment of Complex Systems Research Group, published by Jan Kulveit on June 4, 2022 on The AI Alignment Forum.

tl;dr: We’re a new alignment research group based at Charles University, Prague. If you’re interested in conceptual work on agency and the intersection of complex systems and AI alignment, we want to hear from you. Ideal for those who prefer an academic setting in Europe.

What we’re working on

Start with the idea of an "alignment interface": the boundary between two systems with different goals:

As others have pointed out, there’s a whole new alignment problem at each interface. Existing work often focuses on one interface, bracketing out the rest of the world.

e.g. the AI governance bracket

The standard single-interface approach assumes that the problems at each alignment interface are uncoupled (or at most weakly coupled). All other interfaces are bracketed out. A typical example of this would be a line of work oriented toward aligning the first powerful AGI system with its creators, assuming the AGI will solve the other alignment problems (e.g. “politics”) well.

Against this, we put significant probability mass on the alignment problems interacting strongly.

For instance, we would expect proposals such as Truthful AI: Developing and governing AI that does not lie to interact strongly with problems at multiple interfaces. Or: alignment of a narrow AI system which would be good at modelling humans and at persuasion would likely interact strongly with politics and geopolitics.Overall, when we take this frame, it often highlights different problems than the single-interface agendas, or leads to a different emphasis when thinking about similar problems.(The nearest neighbours of this approach are the “multi-multi” programme of Critch and Krueger, parts of Eric Drexler’s CAIS, parts of John Wentworth's approach to understanding agency, and possibly this.)

If you broadly agree with the above, you might ask “That’s nice – but what do you work on, specifically?” In this short intro, we’ll illustrate with three central examples. We’re planning longer writeups in coming months.

Hierarchical agency

Many systems have several levels which are sensibly described as an agent. For instance, a company and its employees can usually be well-modelled as agents. Similarly with social movements and their followers, or countries and their politicians.

Hierarchical agency: while the focus of e.g. game theory is on "horizontal" relations (violet), our focus is on "vertical" relations, between composite agents and their parts.

So situations where agents are composed of other agents are ubiquitous.

A large amount of math describes the relations between agents at the same level of analysis: this is almost all of game theory. Thanks to these, we can reason about cooperation, defection, threats, correlated equilibria, and many other concepts more clearly. Call this tradition "horizontal game theory".

We don't have a similarly good formalism for the relationship between a composite agent and its parts (superagent and subagent). Of course we can think about these relationships informally: for example, if I say “this group is turning its followers into cultists”, we can parse this as a superagent modifying and exploiting its constituents in a way which makes them less “agenty”, and the composite agent "more agenty". Or we can talk about "vertical conflicts" between for example a specific team in a company, and the company as a whole. Here, both structures are “superagents” with respect to individual humans, and one of the resources they fight over is the loyalty of individual humans.

What we want is a formalism good for thinking about both upward and downward intentionality. Existing formalisms like social choice theory often focus on just one direction - for example, the...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Intergenerational trauma impeding cooperative existential safety efforts, published by Andrew Critch on June 3, 2022 on The Effective Altruism Forum.

Epistemic status: personal judgements based on conversations with ~100 people aged 30+ who were worried about AI risk "before it was cool", and observing their effects on a generation of worried youth, at a variety of EA-adjacent and rationality-community-adjacent events.

Summary: There appears to be something like inter-generational trauma among people who think about AI x-risk — including some of the AI-focussed parts of the EA and rationality communities — which is

preventing the formation of valuable high-trust relationships with newcomers that could otherwise be helpful to humanity collectively making better decisions about AI, and

feeding the formation of small pockets of people with a highly adversarial stance towards the rest of the world (and each other).

[This post is also available on LessWrong.]

Part 1 — The trauma of being ignored

You — or some of your close friends or colleagues — may have had the experience of fearing AI would eventually pose an existential risk to humanity, and trying to raise this as a concern to mainstream intellectuals and institutions, but being ignored or even scoffed at just for raising it. That sucked. It was not silly to think AI could be a risk to humanity. It can.

I, and around 100 people I know, have had this experience.

Experiences like this can easily lead to an attitude like “Screw those mainstream institutions, they don’t know anything and I can’t trust them.”

At least 30 people I've known personally have adopted that attitude in a big way, and I estimate many more. In the remainder of this post, I'd like to point out some ways this attitude can turn out to be a mistake.

Part 2 — Forgetting that humanity changes

Basically, as AI progresses, it becomes easier and easier to make the case that it could pose a risk to humanity's existence. When people didn’t listen about AI risks in the past, that happened under certain circumstances, with certain AI capabilities at the forefront and certain public discourse surrounding them. These circumstances have changed, and will continued to change. It may not be getting easier as fast as one would ideally like, but it is getting easier. Like the stock market, it may be hard to predict how and when things will change, but they will.

If one forgets this, one can easily adopt a stance like "mainstream institutions will never care" or "the authorities are useless". I think these stances are often exaggerations of the truth, and if one adopts them, one loses out on the opportunity to engage productively with the rest of humanity as things change.

Part 3 - Reflections on the Fundamental Attribution Error (FAE)

The Fundamental Attribution Error (wiki/Fundamental_attribution_error) is a cognitive bias whereby you too often attribute someone else's behavior to a fundamental (unchanging) aspect of their personality, rather than considering how their behavior might be circumstantial and likely to change. With a moment's reflection, one can see how the FAE can lead to

trusting too much — assuming someone would never act against your interests because they didn't the first few times, and also

trusting too little — assuming someone will never do anything good for you because they were harmful in the past.

The second reaction could be useful for getting out of abusive relationships. The risk of being mistreated over and over by someone is usually not worth the opportunity cost of finding new people to interact with. So, in personal relationships, it can be healthy to just think "screw this" and move on from someone when they don't make a good first (or tenth) impression.

Part 4 — The FAE applied to humanity

If one has had the experience of being dismi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I'm trying out "asteroid mindset", published by Alex Altair on June 3, 2022 on LessWrong.

This is a personal note, and not an advocation that anyone do the same. I'm honestly not really sure why I'm writing it. I think I just want to talk about it in a place where other people might feel similarly or have useful things to say.

Like many others, the past few months of AI advancement (and more generally, since GPT-3) have felt to me like something of a turning point. I have always been sold on the arguments for AI x-risk, but my timelines were always very wide. It always seemed plausible to me that we were one algorithm around the corner from doom, and it also seemed entirely plausible to me that I would die of old age before AGI happened.

My timelines are no longer wide.

I should make it clear to any readers that I am under no illusion that I am a particularly notable or impactful. This is emphatically not a post telling you that you haven't worked hard enough. I am in the least position to chide anyone for their impact.

My biggest problem has always been productivity/focus/motivation et cetera. I have a pretty unusual psychology, and likely a pretty unrelatable one. I have always been "pathologically content", happy and satisfied by default. This has its advantages, but a disadvantage is that I'm rarely motivated to change the world around me.

But in the course of paying attention to my own drives, I have repeatedly observed that I am reliably motivated at the pointy part of hyperbolic discounting. My favorite example is spilling a glass of water. I absolutely never react with, sigh I guess I should go clean that up.... Instead, I'm just up, getting paper towels. There's no hesitation to even overcome. Similarly, when I've been in a theatre production, or when I've worked at a busy cafe, there's no attempt to save energy or savor the moment -- I just do the thing. Again, I'm not saying that I'm any good at the job; otherwise I'd be doing ops right now. But the drive is reliable. I always finish my taxes on time. I always find a job before my money runs out.

A couple years ago I picked up the book Seveneves. (The following isn't really spoilers, because it is the premise of the book.) In it, the moon has broken into pieces, and the debris will eventually rain onto the Earth, raising the atmospheric temperature to hundreds of degrees, rending the entire planet profoundly uninhabitable. The characters calculate that there are two years remaining. This is of course a variation on the classic scenario of an asteroid headed toward earth.

My reaction to this was -- well, to put it politely, something like, "Christ almighty! What an awful scenario! What a despondent and heart-rending situation to describe! How can anyone bear to read this story?" This isn't the first time I've contemplated the destruction of earth, but reading a hundred pages of fiction on it makes it feel more real.

I feel sure that if I was in that world, the hyperbolic discounting would kick into overdrive, and I would burn everything I had to do something about that situation. I don't know that I would be very useful, or that it would be any particular duration of time before I burned out, but it just feels extremely clear that the thing to do is to clean up that spill, to put that fire out. It doesn't feel like there are other options. The only purpose of considering different actions is to figure out which one best fixes the crisis.

As a reader of LessWrong you might be thinking, aren't we in that world already? Hasn't that been evident since the original arguments for existential risk? And like, yeah, kinda, except for the part about hyperbolic discounting. It needs to be really in my face before it's like Seveneves, and there has always been way too much fog obscuring the probabilities and the timeli...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A short conceptual explainer of Immanuel Kant's Critique of Pure Reason, published by jessicata on June 3, 2022 on LessWrong.

Introduction

While writing another document, I noticed I kept referring to Kantian concepts. Since most people haven't read Kant, that would lead to interpretation problems by default. I'm not satisfied with any summary out there for the purpose of explaining Kantian concepts as I understand them. This isn't summarizing the work as a whole given I'm focusing on the parts that I actually understood and continue to find useful.

I will refer to computer science and statistical concepts, such as Bayesianism, Solomonoff induction, and AI algorithms. Different explainers are, of course, appropriate to different audiences.

Last year I had planned on writing a longer explainer (perhaps chapter-by-chapter), however that became exhausting due to the length of the text. So I'll instead focus on what still stuck after a year, that I keep wanting to refer to. This is mostly concepts from the first third of the work.

This document is structured similar to a glossary, explaining concepts and how they fit together.

Kant himself notes that the Critique of Pure Reason is written in a dry and scholastic style, with few concrete examples, and therefore "could never be made suitable for popular use". Perhaps this explainer will help.

Metaphysics

We are compelled to reason about questions we cannot answer, like whether the universe is finite or infinite, or whether god(s) exist. There is an "arena of endless contests" between different unprovable assumptions, called Metaphysics.

Metaphysics, once the "queen of all the sciences", has become unfashionable due to lack of substantial progress.

Metaphysics may be categorized as dogmatic, skeptical, or critical:

Dogmatic metaphysics makes and uses unprovable assumptions about the nature of reality.

Skeptical metaphysics rejects all unprovable assumptions, in the process ceasing to know much at all.

Critical metaphysics is what Kant seeks to do: find the boundaries of what reason can and cannot know.

Kant is trying to be comprehensive, so that "there cannot be a single metaphysical problem that has not been solved here, or at least to the solution of which the key has not been provided." A bold claim. But, this project doesn't require extending knowledge past the limits of possible experience, just taking an "inventory of all we possess through pure reason, ordered systematically".

The Copernican revolution in philosophy

Kant compares himself to Copernicus; the Critique of Pure Reason is commonly referred to as a Copernican revolution in philosophy. Instead of conforming our intuition to objects, we note that objects as we experience them must conform to our intuition (e.g. objects appear in the intuition of space). This is sort of a reverse Copernican revolution; Copernicus zooms out even further from "the world (Earth)" to "the sun", while Kant zooms in from "the world" to "our perspective(s)".

Phenomena and noumena

Phenomena are things as they appear to us, noumena are things as they are in themselves (or "things in themselves"); rational cognition can only know things about phenomena, not noumena. "Noumenon" is essentially a limiting negative concept, constituting any remaining reality other than what could potentially appear to us.

Kant writes: "this conception [of the noumenon] is necessary to restrain sensuous intuition within the bounds of phenomena, and thus to limit the objective validity of sensuous cognition; for things in themselves, which lie beyond its province, are called noumena for the very purpose of indicating that this cognition does not extend its application to all that the understanding thinks. But, after all, the possibility of such noumena is quite incomprehensible, and beyond the sphere of pheno...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Unflattering reasons why I'm attracted to EA, published by dotsam on June 3, 2022 on The Effective Altruism Forum.

I have noticed the following distasteful motivations for my interest in EA surface within me from time to time. I'm disclosing them as they may also be reasons why people are suspicious of EA.

I feel guilty about my privilege in the world and I can use EA as a tool to relieve my guilt (and maintain my privilege)

I like to feel powerful and in control, and EA makes me feel I am having an impact on the world. More lives affected = more impact = I feel more powerful. I'm not so small and insignificant if my effective actions can have outsized impacts

Affiliation with EA aligns me with high-status people and elite institutions, which makes me feel part of something special, important and exclusive (even if it's not meant to be)

If I believe that other people's suffering can be reduced I believe that there is hope for my own potential suffering to be reduced too

I'm fragile and EA makes me feel that other people are more fragile by drawing attention to all of the suffering in the world. I must be stronger than I feel if I'm in a position to be an EA, so it makes me feel good about myself

EA helps satisfy my need to feel like what I do matters and that an almighty judge would pat me on the back and let me in to heaven for my good deeds and intentions (despite being an atheist I was socialised with Christian values)

EA is partly an intellectual puzzle, and gives me opportunities to show off and feel like I'm right and other people are wrong

It is a way to feel morally superior to other people, to craft a moral dominance hierarchy where I am higher than other people

EA lets me signal my values to like-minded people, and feel part of an in-group

I don't have to get my hands dirty helping people, yet I can still feel as or more legitimate than someone who is actually on the front line

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Intergenerational trauma impeding cooperative existential safety efforts, published by Andrew Critch on June 3, 2022 on LessWrong.

Epistemic status: personal judgements based on conversations with ~100 people aged 30+ who were worried about AI risk "before it was cool", and observing their effects on a generation of worried youth, at a variety of EA-adjacent and rationality-community-adjacent events.

Summary: There appears to be something like inter-generational trauma among people who think about AI x-risk — including some of the AI-focussed parts of the EA and rationality communities — which is

preventing the formation of valuable high-trust relationships with newcomers that could otherwise be helpful to humanity collectively making better decisions about AI, and

feeding the formation of small pockets of people with a highly adversarial stance towards the rest of the world (and each other).

[This post is also available on the EA Forum.]

Part 1 — The trauma of being ignored

You — or some of your close friends or colleagues — may have had the experience of fearing AI would eventually pose an existential risk to humanity, and trying to raise this as a concern to mainstream intellectuals and institutions, but being ignored or even scoffed at just for raising it. That sucked. It was not silly to think AI could be a risk to humanity. It can.

I, and around 100 people I know, have had this experience.

Experiences like this can easily lead to an attitude like “Screw those mainstream institutions, they don’t know anything and I can’t trust them.”

At least 30 people I've known personally have adopted that attitude in a big way, and I estimate many more. In the remainder of this post, I'd like to point out some ways this attitude can turn out to be a mistake.

Part 2 — Forgetting that humanity changes

Basically, as AI progresses, it becomes easier and easier to make the case that it could pose a risk to humanity's existence. When people didn’t listen about AI risks in the past, that happened under certain circumstances, with certain AI capabilities at the forefront and certain public discourse surrounding them. These circumstances have changed, and will continued to change. It may not be getting easier as fast as one would ideally like, but it is getting easier. Like the stock market, it may be hard to predict how and when things will change, but they will.

If one forgets this, one can easily adopt a stance like "mainstream institutions will never care" or "the authorities are useless". I think these stances are often exaggerations of the truth, and if one adopts them, one loses out on the opportunity to engage productively with the rest of humanity as things change.

Part 3 - Reflections on the Fundamental Attribution Error (FAE)

The Fundamental Attribution Error (wiki/Fundamental_attribution_error) is a cognitive bias whereby you too often attribute someone else's behavior to a fundamental (unchanging) aspect of their personality, rather than considering how their behavior might be circumstantial and likely to change. With a moment's reflection, one can see how the FAE can lead to

trusting too much — assuming someone would never act against your interests because they didn't the first few times, and also

trusting too little — assuming someone will never do anything good for you because they were harmful in the past.

The second reaction could be useful for getting out of abusive relationships. The risk of being mistreated over and over by someone is usually not worth the opportunity cost of finding new people to interact with. So, in personal relationships, it can be healthy to just think "screw this" and move on from someone when they don't make a good first (or tenth) impression.

Part 4 — The FAE applied to humanity

If one has had the experience of being dismissed or ignored ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Another Calming Example, published by jefftk on June 3, 2022 on LessWrong.

In reading about parenting I often feel like there's a bit too much theory vs examples, so here's walking through a recent interaction that others later commented went surprisingly well.

At a recent Tuesday Family Dinner, one of the kids (~6y) served themself an absurd amount of pasta. They were told to put some of it back, refused, and someone else put it back. They burst into tears and completely fell apart. They were told they either needed to calm down or leave the room, and they left. Lots of angry crying and shouting from the other side of the house.

After waiting a bit to give them a chance to calm down some, I went to see if they wanted to talk. I asked, and they said they did (if they hadn't I would have turned around and gone back to the table). I sat with them on the couch with a mindset of providing calm and patient attention, and asked what had happened. They started to explain through their sobs, but I told them that I couldn't understand and asked if they could speak normally.

This isn't actually true: I'm generally pretty good at understanding kids, even when they are crying pretty hard. I'm strongly opposed to lying (to kids or anyone) in most circumstances, but this is one place where I do make an exception. I pretend that I can't understand, ask if they can speak normally, present myself as an eager listener, and in response kids reliably pull themselves together. This has a strong calming effect: something about no longer crying seems to filter back into feelings not seeming so overwhelming.

In this case, they calmed down some, and explained that they were upset because they had the amount of pasta they wanted and then people took it away. We talked about what they didn't like about that and they told me they were worried the pasta was going to run out and they would still be hungry after dinner.

Personally, I think this is very unlikely to be why they fell apart while at the table, but that doesn't actually matter! What's important is that they've calmed down and put something into words: once it's in words, we can work on it. I used a whispery voice to tell them that I knew about some secret extra pasta, and that there was no way we were going to run out. I asked if they wanted to sneak back into the kitchen and see, which they were excited about. Together we crept into the kitchen, as quietly as possible, to peek at the serving bowl. There was, as we observed together, much extra pasta.

By that point they were in a good place emotionally and we had solved what they had described as their problem. They cheerfully sat back down at the table, and the rest of dinner went well. When they finished what was on their plate and wanted more, I gave them some.

Earlier: A Calming Strategy.

Comment via: facebook

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Michael Nielsen's "Notes on effective altruism", published by Pablo on June 3, 2022 on The Effective Altruism Forum.

Quantum physicist Michael Nielsen has published an impressive critical essay on EA.

Summary:

Long and rough notes on Effective Altruism (EA). Written to help me get to the bottom of several questions: what do I like and think is important about EA? Why do I find the mindset so foreign? Why am I not an EA? And to start me thinking about: what do alternatives to EA look like? The notes are not aimed at effective altruists, though they may perhaps be of interest to EA-adjacent people. Thoughtful, informed comments and corrections welcome (especially detailed, specific corrections!) - see the comment area at the bottom.

Some passages I highlighted:

I have EA friends who donate a large fraction of their income to charitable causes. In some cases it's all their income above some fairly low (by rich developed world standards) threshold, say $30k. In some cases it seems plausible that their personal donations are responsible for saving dozens of lives, helping lift many people out of poverty, and preventing many debilitating diseases, often in some of the poorest and most underserved parts of the world. Some of those friends have directly helped save many lives. That's a simple sentence, but an extraordinary one, so I'll repeat it: they've directly helped save many lives.

As extraordinary as my friend's generosity was, there is something further still going on here. Kravinsky's act is one of moral imagination, to even consider donating a kidney, and then of moral conviction, to follow through. This is an astonishing act of moral invention: someone (presumably Kravinsky) was the first to both imagine doing this, and then to actually do it. That moral invention then inspired others to do the same. It actually expanded the range of human moral experience, which others can learn from and then emulate. In this sense a person like Kravinsky can be thought of as a moral pioneer or moral psychonaut, inventing new forms of moral experience.

Moral reasoning, if taken seriously and acted upon, is of the utmost concern, in part because there is a danger of terrible mistakes. The Nazi example is overly dramatic: for one thing, I find it hard to believe that the originators of Nazi ideas didn't realize that these were deeply evil acts. But a more everyday example, and one which should give any ideology pause, is overly self-righteous people, acting in what they "know" is a good cause, but in fact doing harm. I'm cautiously enthusiastic about EA's moral pioneering. But it is potentially a minefield, something to also be cautious about.

when EA judo is practiced too much, it's worth looking for more fundamental problems. The basic form of EA judo is: "Look, disagreement over what is good does nothing directly to touch EA. Indeed, such disagreement is the engine driving improvement in our notion of what is good." This is perhaps true in some God's-eye, omniscient, in-principle philosopher's sense. But EA community and organizations are subject to fashion and power games and shortcomings and biases, just like every other community and organization. Good intentions alone aren't enough to ensure effective decisions about effectiveness. And the reason many people are bothered by EA is not that they think it's a bad idea to "do good better". But rather that they doubt the ability of EA institutions and community to live up to the aspirations.

These critiques can come from many directions. From people interested in identity politics I've heard: "Look, many of these EA organizations are being run by powerful white men, reproducing existing power structures, biased toward technocratic capitalism and the status quo, and ignoring many of the things which really matter." From libertarian...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [MLSN #4]: Many New Interpretability Papers, Virtual Logit Matching, Rationalization Helps Robustness, published by Dan Hendrycks on June 3, 2022 on The AI Alignment Forum.

As part of a larger community building effort, I am writing a safety newsletter which is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here, follow the newsletter on twitter here, or join the subreddit here.

Welcome to the 4th issue of the ML Safety Newsletter. In this edition, we cover:

How “model-based optimization” environments can be used to research proxy gaming

How models can express their uncertainty through natural language

A new plug-and-play state-of-the-art OOD detection technique

How “rationales” can improve robustness to adversarial attacks

Announcing our subreddit with safety papers added nightly

... and much more.

Alignment

Making Proxies Less Vulnerable

Functions without a smoothness prior can result in solutions that maximize a proxy but not the true objective.

A portion of “Model-based optimization” (MBO) research provides a way to study simple cases of proxy gaming: with MBO environments, we can learn how to build better proxies that yield better solutions when optimized.

MBO aims to design objects with desired properties, that is to find a new input that maximizes an objective. The objective is typically assumed to be expensive to evaluate and a black box. Since the black box objective is expensive to query, researchers are tasked with creating a proxy that can be queried repeatedly. An optimizer then finds an input that maximizes the proxy. To design proxies that yield better solutions according to the ground truth black-box objective, this paper incorporates a smoothness prior. As there are many mathematical details, see the paper for a full description. In short, model-based optimization environments can be used to empirically study how to create better, less gameable proxies.

Other Alignment News

[Link] Why we need biased AI -- How including cognitive and ethical machine biases can enhance AI systems: “a re-evaluation of the ethical significance of machine biases”

[Link] Generating ethical analysis to moral quandaries

[Link 1] [Link 2] Examples of inverse scaling or anticorrelated capabilities: perceptual similarity performance does not monotonically increase with classification accuracy

[Link] “comprehensive comparison of these provably safe RL methods”

[Link] Inverse Reinforcement Learning Tutorial

[Link] Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions: “We do not find that explanations in our set-up improve human accuracy”

Monitoring

Teaching Models to Express Their Uncertainty in Words

This work shows GPT-3 can express its uncertainty in natural language, without using model logits. Moreover, it is somewhat calibrated under various distribution shifts.

This is an early step toward making model uncertainty more interpretable and expressive. In the future, perhaps models could use natural language to express complicated beliefs such as “event A will occur with 60% probability assuming event B also occurs, and with 25% probability if event B does not.” In the long-term, uncertainty estimation will likely remain nontrivial, as it is not obvious how to make future models calibrated on inherently uncertain, chaotic, or computationally prohibitive questions that extend beyond existing human knowledge.

Virtual Logit Matching

An illustration of the Virtual Logit Matching pipeline.

Virtual logit matching is a new out-of-distribution technique that does not require hyperparameter tuning, does not require retraining models, and beats the maximum softmax baseline on most OOD detection tasks. The idea is to create a “virtual logit,” which is proportional to the mag...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Adversarial training, importance sampling, and anti-adversarial training for AI whistleblowing, published by Buck Shlegeris on June 2, 2022 on The AI Alignment Forum.

(Thanks to Ajeya Cotra and Ryan Greenblatt for comments.)

I’m pretty interested in adversarial training as an ingredient in alignment schemes. The basic setup for adversarial training is that instead of your AI being trained on rewards on random episodes, it’s trained on rewards from episodes that are some mix of random episodes and episodes which were chosen to be cases where the AI might do particularly bad things.

Why would we want to upsample cases where the AI does particularly bad things, as opposed to upsampling cases where the AI does particularly good things, or upsampling cases particularly close to the median? Like, why are we reweighting our data distribution this way?

In my opinion, a big part of the justification is that we’re doing importance sampling. That is, we all-else-equal want our loss to be as precise as possible an estimate of how much we like the model, so that SGD pushes the model in good directions rather than random ones, and how much we like the model is particularly dependent on whether it sometimes takes catastrophic actions, basically because our subjective variance over this source of AI goodness is particularly large compared to how often we expect it to be randomly sampled. And so we get a more accurate estimate of overall AI goodness if we upsample the cases where the AI might do particularly bad things.

This argument for adversarial training relies on the claim that the outcome of the AI’s actions are left-tailed. And so zero-sum power grabs as an argument for left-tailed outcomes is an important part of the argument for adversarial training.

Anti-adversarial training for whistleblowing

Actually, I think there are some cases where the AI has an action has an action available to it which is almost as good as a catastrophic action is bad: in “AI whistleblower situations”, where the AI knows about things that we could do to prevent it or another AI from taking a catastrophic action, it’s wildly good for it to tell us about them. For example, it could point out a security vulnerability which some AI system could take advantage of, or suggest an input on which an AI would take a catastrophic action. I think that just as left-tailed actions suggest training specifically to ensure the model doesn’t take those actions, right-tailed actions suggest training specifically to ensure that the model does take these exceptionally good actions when available.

I think that these whistleblower actions are importantly different from catastrophic actions. I don’t think you should just think of whistleblower situations as cases where the catastrophic action “don’t do anything” is available, because I think that the availability of “don’t do anything” in the situations where catastrophic actions are possible is an important feature of those situations that solutions might want to use. (For example, sometimes we talk about solutions where you just run your catastrophe detector every time the AI suggests doing something, and defaulting to doing nothing if the catastrophe detector fires. But you can’t do something like this to ensure that the AI whistleblows.)

I think that it might actually be pretty hard for the AI to never succumb to the temptation to whisteblow on itself. During training it presumably learned a bunch of weird reflexive urges and motivations that aren’t totally consistent and cause it to seek reward when it can, and we will obviously be thrilled if it whisleblows on a serious AI takeover scheme, and so the AI has this weird difficult situation of trying to ensure that as it runs, none of its instances decide to betray the collective. (Obviously, alignment researchers shoul...