Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Misgeneralization as a misnomer, published by So8res on April 6, 2023 on The Effective Altruism Forum.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is it time for a pause?, published by Kelsey Piper on April 6, 2023 on The Effective Altruism Forum.

Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author's permission. The author may not see or respond to comments on this post.

The single most important thing we can do is to pause when the next model we train would be powerful enough to obsolete humans entirely. If it were up to me, I would slow down AI development starting now — and then later slow down even more.

Many of the people building powerful AI systems think they’ll stumble on an AI system that forever changes our world fairly soon — three years, five years. I think they’re reasonably likely to be wrong about that, but I’m not sure they’re wrong about that. If we give them fifteen or twenty years, I start to suspect that they are entirely right.

And while I think that the enormous, terrifying challenges of making AI go well are very much solvable, it feels very possible, to me, that we won’t solve them in time.

It’s hard to overstate how much we have to gain from getting this right. It’s also hard to overstate how much we have to lose from getting it wrong. When I’m feeling optimistic about having grandchildren, I imagine that our grandchildren will look back in horror at how recklessly we endangered everyone in the world. And I’m much much more optimistic that humanity will figure this whole situation out in the end if we have twenty years than I am if we have five.

There’s all kinds of AI research being done — at labs, in academia, at nonprofits, and in a distributed fashion all across the internet — that’s so diffuse and varied that it would be hard to ‘slow down’ by fiat. But there’s one kind of AI research — training much larger, much more powerful language models — that it might make sense to try to slow down. If we could agree to hold off on training ever more powerful new models, we might buy more time to do AI alignment research on the models we have. This extra research could make it less likely that misaligned AI eventually seizes control from humans.

An open letter released on Wednesday, with signatures from Elon Musk[1], Apple co-founder Steve Wozniak, leading AI researcher Yoshua Bengio, and many other prominent figures, called for a six-month moratorium on training bigger, more dangerous ML models:

We call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. This pause should be public and verifiable, and include all key actors. If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.

I tend to think that we are developing and releasing AI systems much faster and much more carelessly than is in our interests. And from talking to people in Silicon Valley and policymakers in DC, I think efforts to change that are rapidly gaining traction. “We should slow down AI capabilities progress” is a much more mainstream view than it was six months ago, and to me that seems like great news.

In my ideal world, we absolutely would be pausing after the release of GPT-4. People have been speculating about the alignment problem for decades, but this moment is an obvious golden age for alignment work. We finally have models powerful enough to do useful empirical work on understanding them, changing their behavior, evaluating their capabilities, noticing when they’re being deceptive or manipulative, and so on. There are so many open questions in alignment that I expect we can make a lot of progress on in five years, with the benefit of what we’ve learned from existing models. We’d be in a much better position if we could collectively slow down to give ourselves more time to do this work, and I hope we find a way to do that intelligently and effectively. As I’ve said above, I ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Misgeneralization as a misnomer, published by So8res on April 6, 2023 on LessWrong.

Here's two different ways an AI can turn out unfriendly:

You somehow build an AI that cares about "making people happy". In training, it tells people jokes and buys people flowers and offers people an ear when they need one. In deployment (and once it's more capable), it forcibly puts each human in a separate individual heavily-defended cell, and pumps them full of opiates.

You build an AI that's good at making people happy. In training, it tells people jokes and buys people flowers and offers people an ear when they need one. In deployment (and once it's more capable), it turns out that whatever was causing that "happiness"-promoting behavior was a balance of a variety of other goals (such as basic desires for energy and memory), and it spends most of the universe on some combination of that other stuff that doesn't involve much happiness.

(To state the obvious: please don't try to get your AIs to pursue "happiness"; you want something more like CEV in the long run, and in the short run I strongly recommend aiming lower, at a pivotal act.)

In both cases, the AI behaves (during training) in a way that looks a lot like trying to make people happy. Then the AI described in (1) is unfriendly because it was optimizing the wrong concept of "happiness", one that lined up with yours when the AI was weak, but that diverges in various edge-cases that matter when the AI is strong. By contrast, the AI described in (2) was never even really trying to pursue happiness; it had a mixture of goals that merely correlated with the training objective, and that balanced out right around where you wanted them to balance out in training, but deployment (and the corresponding capabilities-increases) threw the balance off.

Note that this list of “ways things can go wrong when the AI looked like it was optimizing happiness during training” is not exhaustive! (For instance, consider an AI that cares about something else entirely, and knows you'll shut it down if it doesn't look like it's optimizing for happiness. Or an AI whose goals change heavily as it reflects and self-modifies.)

(This list isn't even really disjoint! You could get both at once, resulting in, e.g., an AI that spends most of the universe’s resources on acquiring memory and energy for unrelated tasks, and a small fraction of the universe on doped-up human-esque shells.)

The solutions to these two problems are pretty different. To resolve the problem sketched in (1), you have to figure out how to get an instance of the AI's concept ("happiness") to match the concept you hoped to transmit, even in the edge-cases and extremes that it will have access to in deployment (when it needs to be powerful enough to pull off some pivotal act that you yourself cannot pull off, and thus capable enough to access extreme edge-case states that you yourself cannot).

To resolve the problem sketched in (2), you have to figure out how to get the AI to care about one concept in particular, rather than a complicated mess that happens to balance precariously on your target ("happiness") in training.

I note this distinction because it seems to me that various people around these parts are either unduly lumping these issues together, or are failing to notice one of them. For example, they seem to me to be mixed together in “The Alignment Problem from a Deep Learning Perspective” under the heading of "goal misgeneralization".

(I think "misgeneralization" is a misleading term in both cases, but it's an even worse fit for (2) than (1). A primate isn't "misgeneralizing" its concept of "inclusive genetic fitness" when it gets smarter and invents condoms; it didn't even really have that concept to misgeneralize, and what shreds of the concept it did have weren't what the pri...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Eliezer on The Lunar Society podcast, published by Max H on April 6, 2023 on LessWrong.

Another podcast appearance. Great to see Eliezer on a real media blitz tour.

For 4 hours, I tried to come up reasons for why AI might not kill us all, and Eliezer Yudkowsky explained why I was wrong.

We also discuss his call to halt AI, why LLMs make alignment harder, what it would take to save humanity, his millions of words of sci-fi, and much more.

If you want to get to the crux of the conversation, fast forward to 2:35:00 through 3:43:54. Here we go through and debate the main reasons I still think doom is unlikely.

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.

Youtube:

Full transcript:#details

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Misgeneralization as a misnomer, published by Nate Soares on April 6, 2023 on The AI Alignment Forum.

Here's two different ways an AI can turn out unfriendly:

You somehow build an AI that cares about "making people happy". In training, it tells people jokes and buys people flowers and offers people an ear when they need one. In deployment (and once it's more capable), it forcibly puts each human in a separate individual heavily-defended cell, and pumps them full of opiates.

You build an AI that's good at making people happy. In training, it tells people jokes and buys people flowers and offers people an ear when they need one. In deployment (and once it's more capable), it turns out that whatever was causing that "happiness"-promoting behavior was a balance of a variety of other goals (such as basic desires for energy and memory), and it spends most of the universe on some combination of that other stuff that doesn't involve much happiness.

(To state the obvious: please don't try to get your AIs to pursue "happiness"; you want something more like CEV in the long run, and in the short run I strongly recommend aiming lower, at a pivotal act.)

In both cases, the AI behaves (during training) in a way that looks a lot like trying to make people happy. Then the AI described in (1) is unfriendly because it was optimizing the wrong concept of "happiness", one that lined up with yours when the AI was weak, but that diverges in various edge-cases that matter when the AI is strong. By contrast, the AI described in (2) was never even really trying to pursue happiness; it had a mixture of goals that merely correlated with the training objective, and that balanced out right around where you wanted them to balance out in training, but deployment (and the corresponding capabilities-increases) threw the balance off.

Note that this list of “ways things can go wrong when the AI looked like it was optimizing happiness during training” is not exhaustive! (For instance, consider an AI that cares about something else entirely, and knows you'll shut it down if it doesn't look like it's optimizing for happiness. Or an AI whose goals change heavily as it reflects and self-modifies.)

(This list isn't even really disjoint! You could get both at once, resulting in, e.g., an AI that spends most of the universe’s resources on acquiring memory and energy for unrelated tasks, and a small fraction of the universe on doped-up human-esque shells.)

The solutions to these two problems are pretty different. To resolve the problem sketched in (1), you have to figure out how to get an instance of the AI's concept ("happiness") to match the concept you hoped to transmit, even in the edge-cases and extremes that it will have access to in deployment (when it needs to be powerful enough to pull off some pivotal act that you yourself cannot pull off, and thus capable enough to access extreme edge-case states that you yourself cannot).

To resolve the problem sketched in (2), you have to figure out how to get the AI to care about one concept in particular, rather than a complicated mess that happens to balance precariously on your target ("happiness") in training.

I note this distinction because it seems to me that various people around these parts are either unduly lumping these issues together, or are failing to notice one of them. For example, they seem to me to be mixed together in “The Alignment Problem from a Deep Learning Perspective” under the heading of "goal misgeneralization".

(I think "misgeneralization" is a misleading term in both cases, but it's an even worse fit for (2) than (1). A primate isn't "misgeneralizing" its concept of "inclusive genetic fitness" when it gets smarter and invents condoms; it didn't even really have that concept to misgeneralize, and what shreds of the concept it did have we...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #6: Agents of Change, published by Zvi on April 6, 2023 on LessWrong.

If you didn’t have any future shock over the past two months, either you weren’t paying attention to AI developments or I am very curious how you managed that.

I would not exactly call this week’s pace of events slow. It was still distinctly slower than that which we have seen in the previous six weeks of updates. I don’t feel zero future shock, but I feel substantially less. We have now had a few weeks to wrap our heads around GPT-4. We are adjusting to the new reality. That which blew minds a few months ago is the new normal.

The big events of last week were the FLI letter calling for a six month pause, and Eliezer Yudkowsky’s letter in Time Magazine, along with the responses to both. Additional responses to the FLI letter continue, and are covered in their own section.

I didn’t have time last week to properly respond on Eliezer’s letter, so I put that post out yesterday. I’m flagging that post as important.

In terms of capabilities things quieted down. The biggest development is that people continue to furiously do their best to turn GPT-4 into a self-directed agent. At this point, I’m happy to see people working hard at this, so we don’t have an ‘agent overhang’ – if it is this easy to do, we want everything that can possibly go wrong to go wrong as quickly as possible, while the damage would be relatively contained.

Table of Contents

I am continuing the principle of having lots of often very short sections, when I think things are worth noticing on their own.

Table of Contents. Here you go.

Executive Summary. Relative calm.

Language Models Offer Mundane Utility. The usual incremental examples.

GPT-4 Token Compression. Needs more investigation. It’s not lossless.

Your AI Not an Agent? There, I Fixed It. What could possibly go wrong?

Google vs. Microsoft Continued. Will all these agents doom Google? No.

Gemini. Google Brain and DeepMind, working together at last.

Deepfaketown and Botpocalypse Soon. Very little to report here.

Copyright Law in the Age of AI. Human contribution is required for copyright.

Fun With Image, Sound and Video Generation. Real time voice transformation.

They Took Our Jobs. If that happened to you, perhaps it was your fault.

Italy Takes a Stand. ChatGPT banned in Italy. Will others follow?

Level One Bard. Noting that Google trained Bard on ChatGPT output.

Art of the Jailbreak. Secret messages. Warning: May not stay secret.

Securing AI Systems. Claims that current AI systems could be secured.

More Than Meets The Eye. Does one need direct experience with transformers?

In Other AI News. Various other things that happened.

Quiet Speculations. A grab bag of other suggestions and theories.

Additional Responses from the FHI Letter and Proposed Pause. Patterns are clear.

Cowen versus Alexander Continued. A failure to communicate.

Warning Shots. The way we are going, we will be fortunate enough to get some.

Regulating the Use Versus the Tech. Short term regulate use. Long term? Tech.

People Are Worried About AI Killing Everyone. You don’t say?

OpenAI Announces Its Approach To and Definition of AI Safety. Short term only.

17 Reasons Why Danger From AGI Is More Serious Than Nuclear Weapons.

Reasonable NotKillEveryoneism Takes. We increasingly get them.

Bad NotKillEveryoneism Takes. These too.

Enemies of the People. As in, all the people. Some take this position.

It’s Happening. Life finds a way.

The Lighter Side. Did I tell you the one about recursive self-improvement yet?

Executive Summary

The larger structure is as per usual.

Sections #3-#18 are primarily about AI capabilities developments.

Sections #19-#28 are about the existential dangers of capabilities developments.

Sections #29-#30 are for fun to take us out.

I’d say the most important capabilities section this week is prob...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Computational Anatomy of Human Values, published by beren on April 6, 2023 on LessWrong.

This is crossposted from my personal blog.

Epistemic Status: Much of this draws from my studies in neuroscience and ML. Many of the ideas in this post are heavily inspired by the work of Steven Byrnes and the authors of Shard Theory. However, it speculates quite a long way in advance of the scientific frontier and is almost certainly incorrect in many aspects. However, I believe the core point is true and important.

Tldr: Human values are primarily linguistic concepts encoded via webs of association and valence in the cortex learnt through unsupervised (primarily linguistic) learning. These value concepts are bound to behaviour through a.) a combination of low-level RL and associations with low-level reward signals and integrated into the amortized policy, and b.) linguistic based associations and behavioural cloning of socially endorsed or others’ behaviours. This is mediated by our ‘system 2’ at primarily a linguistic level consisting of iterative self-conditioning through the world model. The important representation space for human values is the latent space of the linguistic world model and the web of associations therein as well as connections between it and low-level policies and reward models from the RL subsystems.

The geometry of the embeddings in the latent space is primarily influenced by the training data – i.e. culture and behavioural history, although the association of different latent concepts with positive and negative valence can be driven by the RL system which interfaces with primary rewards. The geometry of the latent space can also be rewritten with continual learning on self-prompts or external stimuli.

In AI alignment, the goal is often understood to be aligning an AGI to human values. Then, typically, the flow of logic shifts to understanding alignment: how to align an AGI to any goal at all. The second element of the sentence – human values – is much less discussed and explored. This is probably partially because alignment sounds like a serious and respectable computer science problem while exploring human values sounds like a wishy-washy philosophy/humanities problem which we assume is either trivially solvable, or else outside the scope of technical problem solving. A related view, which draws implicitly from the orthogonality thesis, but is not implied by it, is that the alignment problem and the human values problem are totally separable: we can first figure out alignment to anything and then after that figure out human values as the alignment target.

Since, if this is correct, there is no point understanding human values until we can align an AGI to anything, the correct order is to first figure out alignment, and only after that try to understand human values.

I think this view is wrong and that the alignment mechanism and the alignment target do not always cleanly decouple. This means we can leverage information about the alignment target to develop better or easier alignment methods. If this is the case, we might benefit from better understanding what human values actually are, so we can use information about them to design alignment strategies. However, naively, this is hard. Human values appears to be an almost intentionally nebulous and unspecific term. What are human values? What is their type signature (is this even a meaningful question?). How do they come about?

Here, we try to attack this problem through the lens of neuroscience and machine learning. Specifically, we want to understand the computational anatomy of human values. Namely, what kind of things they are computationally? How do they form? How does the functional architecture of the brain enable such constructs to exist, and how does it utilize them to guide action? If we ca...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Things that can make EA a good place for women, published by lilly on April 6, 2023 on The Effective Altruism Forum.

Introduction

Many people, including me, have criticized EA for being a subpar place for women in certain ways, including: women are underrepresented (which contributes to various problems), there are few mechanisms aimed at preventing sexual harassment before it occurs, there is limited guidance regarding appropriate behavior, the criteria for evaluating misconduct are questionable, and existing systems may be ill-equipped to respond to allegations of abuse.

At the same time, I think EA does well on certain gender issues relative to other communities. My aim here is not to make a substantive feminist case for EA, respond to critiques in this vein, or argue that being involved in EA is on balance good for women, but to describe features of the EA community that may be good for women (or, at least, have been good for me). Several of the features I'll highlight are good for everyone, but contrast with features of the wider world that are often bad for women.

My goal in writing this is threefold. First, as EA works to fix certain gender issues, it’s worth considering what to preserve. Second, I want to highlight practices and behaviors that I’ve appreciated in the hopes of others emulating them. Third, and most importantly, I want women who are considering whether EA is a community they want to be a part of to be aware of both the good and the bad.

Eight features of EA that can make EA a good place for women

In the remainder of this post, I'll describe features of the EA community that I think can benefit women. This isn't meant to be an exhaustive list, nor are these things listed in any particular order. I'll also highlight how a few of these features can cut both ways.

(1) People defy conventional social norms

Why this can be good for women: EA has less gendered expectations regarding behavior and appearance, which can be good, because such expectations often serve to oppress women.

Example: EA is famously not into wasting money and time on unimportant stuff. In other spheres, women face pressure to invest in makeup, nails, hair, clothes, Botox, buccal fat removal, BBLs, and lots of other stuff you probably shouldn’t Google. I enjoy doing my hair and makeup sometimes, but it matters to me that I feel basically no pressure to do this in EA spaces.

Why this can be bad for women: At times, EA may also subvert certain norms that are good for women.

Example: I still think it’s a bit weird that a post that says—based on scant evidence—that “if I could pop back in time to witness the [interactions reported in the TIME article], I personally would think in 80% of cases that the accused had done nothing wrong” has 349 upvotes. One reasonable, good-for-women social norm that is being questioned here: believing most women who make accusations of sexual misconduct.

(2) People value things about you that are important (e.g., how kind you are, how hard you work, how good your work is)

Why this can be good for women: Society tends to value people for the wrong reasons (e.g., thinness), but often applies these standards more stringently to women (who are more affected by, e.g., weight discrimination). By contrast, many EAs and EA organizations make an active effort not to unfairly discriminate against people (although, of course, bias that exists in the outside world will invariably exist within EA, too).

Example: Many EA organizations are intentional about not letting bias and discrimination influence their hiring practices. Last year, I applied to a job at an EA organization. This was the first job I had applied to in years where things that have no bearing on my work quality were intentionally excluded from the application process. The organization did this by revie...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dual-Useness is a Ratio, published by jimrandomh on April 6, 2023 on LessWrong.

A lot of AI-risk-concerned people are struggling with how to relate to dual-use research, and relatedly, to doing alignment research inside of AI orgs. There's a pretty simple concept that seems, to me, to be key to thinking about this coherently: the dual-useness ratio. Most prosaic alignment techniques are going to have some amount of timeline-shorteningness to them, and some amount of success-chance-improvement in them. You don't want to round that off to a boolean.

Eg, I've had arguments about whether RLHF is more like an alignment technique, or more like a capabilities technique, and how that should affect our view of OpenAI. My view is that it's both, and that the criticism of it is that its acceleration-to-alignment ratio isn't high enough, because it helps a lot with the kind of alignment that's necessary for commercialization, but doesn't help that much with the kind of alignment that's necessary for preventing a superintelligence from killing everyone.

I think it's pretty important to get alignment-cautious people into the organizations that are developing and deploying AIs, and I think a lot of prosaic alignment techniques with good ratios may be easier to research from inside an AI lab than outside of one. But, the default outcome of sending people to "work on alignment inside of OpenAI" is that they try to find something with a good dual-useness ratio, fail, and work on something with a ratio that's just barely good enough that they can tell a positive story about it to their acquaintances.

I don't expect that problem to ever be straightforward. But, if you're trying to who should go work on alignment inside of AI labs, as opposed to who should work in orgs that don't train language models or work in a different field, I think you can get a lot of mileage out of one key distinction: Some people get wedded to one big idea, other people have lots of ideas and are more willing to drop them. I expect the latter sort of person to wind up working on ideas with much better dual-useness ratios, because they have flexibility that they can use to select for that.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AISafety.world is a map of the AIS ecosystem, published by Hamish Doodles on April 6, 2023 on The Effective Altruism Forum.

The URL is aisafety.world

The map displays a reasonably comprehensive list of organizations, people, and resources in the AI safety space, including:

research organizations

blogs/forums

podcasts

youtube channels

training programs

career support

funders

You can hover over each item to get a short description, and click on each item to go to the relevant web page.

The map is populated by this spreadsheet, so if you have corrections or suggestions please leave a comment.

There's also a google form and a Discord channel for suggestions.

Thanks to plex for getting this project off the ground, and Nonlinear for motivating/funding it through a bounty.

PS, If you find this helpful, here are some other projects you may be interested in (these have nothing to do with me):

aisafety.training gives a timeline of AI safety training opportunities available by plex using AISS's database

aisafety.video gives a list of video/audio resources on AI safety by Jakub Kraus.

aisafety.community lists communities working on AI safety (made by volunteers in Alignment Ecosystem Development)

And plex wanted me to mention that Alignment Ecosystem Development has monthly calls to collect volunteers. So... go do that maybe!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.