Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We're Aligned AI, we're aiming to align AI, published by Stuart Armstrong on February 21, 2022 on The Effective Altruism Forum.

Aligned AI is an Oxford based startup focused on applied alignment research. Our goal is to implement scalable solutions to the alignment problem, and distribute these solutions to actors developing powerful transformative artificial intelligence (related Alignment Forum post here).

We are lead by Stuart Armstrong and Rebecca Gorman, and advised by Dylan Hadfield-Menell, Adam Gleave, Justin Shovelain, Charles Pattison, and Anders Sandberg.

Our Premises

We think AI poses an existential risk to humanity, and that reducing the chance of this risk is one of the most impactful things we can do with our lives. Here we focus not on the premises behind that claim, but rather on why we're particularly excited about Aligned AI's approach to reducing AI existential risk.

We believe AI Safety research is bottle-necked by a core problem: how to extrapolate values from one context to another.

We believe solving value extrapolation is necessary and almost sufficient for alignment.

Value extrapolation research is neglected, both in the mainstream AI community and the AI safety community. Note that there is a lot of overlap between value extrapolation and many fields of research (e.g. out of distribution detection, robustness, transfer learning, multi-objective reinforcement learning, active reward learning, reward modelling...) which provide useful research resources. However, we've found that we've had to generate our most of the key concepts ourselves.

We believe value extrapolation research is tractable (and we've had success generating the key concepts).

We believe distributing (not just creating) alignment solutions is critical for aligning powerful AIs.

How we'll do this

Solving value extrapolation will involve solving multiple subproblems. Therefore the research groups will iterate through sub-projects, like the ones presented here. The aim is to generate sub-projects that are close to current published research in machine learning, but whose solutions are designed to generalise. Our groups will take these projects, implement them in code, and build solutions for the relevant problem. At that point, we will either extend the project to investigate it in more depth, or write up the results and move on - passing the results to the library development team as needed.

Research methodology

At a high level, our research is structured around a linear pipeline, starting from theory and becoming progressively more applied. Each stage of the pipeline has tight feedback loops, and also inform the other stages of the pipeline (e.g. theory leads to experiments leads to revised theory). The following sections describe how such a process might go.

Sub-project generation

Once a sub component is deemed sufficiently promising, we will want to test it in code. To do so, we will generative "sub-project" ideas designed to be simple to implement but scalable to larger environments and models.

Minimum viable (sub-)project

We will start a sub-project with a "MVP", implementing the simplest project that captures the core of our approach.

Test sub-projects in higher dimensional settings

After implementing a successful MVP, we will iteratively experiment in increasingly high dimensional settings (think Deep Reinforcement Learning from Human Feedback to Learning to Summarize from Human Feedback).

Red-teaming

We will employ a "red-teaming" methodology similar to that of the Alignment Research Center, considering worst case scenarios and how a given approach handles them.

What we plan to produce

Software library

If we believe we can commercialize a successful sub-project responsibly (without differential enhancing AI capabilities), it will be incorporated into our product and mar...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Favorite / most obscure research on understanding DNNs?, published by Vivek Hebbar on February 21, 2022 on The AI Alignment Forum.

Looking for:

Any results / lines of research I haven't heard of in areas like interpretability, inductive biases, and ML theory

Obscure papers or posts on things like double descent, simplicity bias, induction heads, and grokking

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Big Picture Of Alignment (Talk Part 1), published by johnswentworth on February 21, 2022 on The AI Alignment Forum.

I recently gave a two-part talk on the big picture of alignment, as I see it. The talk is not-at-all polished, but contains a lot of stuff for which I don't currently know of any good writeup. Major pieces in part one:

Some semitechnical intuition-building for high-dimensional problem-spaces.

Optimization compresses information "by default"

Resources and "instrumental convergence" without any explicit reference to agents

A frame for thinking about the alignment problem which only talks about high-dimensional problem-spaces, without reference to AI per se.

The central challenge is to get enough bits-of-information about human values to narrow down a search-space to solutions compatible with human values.

Details like whether an AI is a singleton, tool AI, multipolar, oracle, etc are mostly irrelevant.

Fermi estimate: just how complex are human values?

Coherence arguments, presented the way I think they should be done.

Also subagents!

Note that I don't talk about timelines or takeoff scenarios; this talk is just about the technical problem of alignment.

Here's the video for part one:

Big thanks to Rob Miles for editing! Also, the video includes some good questions and discussion from Adam Shimi, Alex Flint, and Rob Miles.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prizes for the 2020 Review, published by Raemon on February 20, 2022 on LessWrong.

The third annual review is complete. We spent two months nominating, reviewing, and voting on the best posts of 2020. After lots of thoughtful evaluation, it's time to award some prizes!

As described in the voting results post, there are two prize pools for posts. Lightcone Infrastructure contributed $10k to the Review Vote prize pool, and LessWrong users donated $1770 to the Unit of Caring prize pool (which Lightcone matched 1:1, bringing it up to $3540).

Lightcone Infrastructure is also awarding $3600 in prizes for people who wrote reviews for posts.

Prizes for Posts

The Review Vote Prizes

We've awarded a $1000 Top Prize to each of the following authors:

catherio, for microCOVID.org, a tool to estimate covid risk

Ajeta Cotra, for Draft report on AI Timelines

evhub, for An Overview of 11 Proposals for Building Safe Advanced AI

johnswentworth for When Money is Abundant, Knowledge is the Real Wealth, Alignment by Default, The Pointers Problem, and Coordination as Scarce Resource

We've awarded a $500 Honorable Mention to each of:

Alkjash, for Pain is not the unit of effort

Mark Xu, for The Solomonoff Prior is Malign

Jacob Falcovich, for Seeing the Smoke

Alex Flint, for The Ground of Optimization

Zvi, for Simulacra Levels and their Interactions

Richard Ngo, for AGI Safety from First Principles

Paul Christiano, for Inaccessible Information

Steven Byrnes for My Computational Framework for the Brain

abramdemski for Most Prisoner's Dilemmas are Stag Hunts; Most Stag Hunts are Schelling Problems, and An Orthodox Case Against Utility Functions

Daniel Kokotajlo, for Cortés, Pizarro, and Afonso as Precedents for Takeover

Scott Garrabrant, for Introduction to Cartesian Frames

Rafael Harth, for Inner Alignment, Explain Like I'm 12 Edition

How were prizes determined?

The LessWrong moderation team looked over the results of the vote, sliced four different ways:

Total votes from all users;

Total votes from 1000+ karma users;

Total votes from Alignment Forum users;

Weighted votes from all users, but with 1000+ karma users weighted 3x.

That gave us some sense of how different users related to posts, and what they got out of them. In all cases, approximately the same posts showed up at the top of the rankings, albeit in somewhat different orders. I'm excited to reward all of them prizes for contributing to important intellectual progress.

Two things stuck out between the various rankings:

In the Alignment Forum vote, Draft Report on AI Timelines won by a landslide.

In the "All Users" vote, the MicroCOVID.org announcement won by a landslide.

Thoughts on the microCOVID.org Announcement

Several people asked, upon seeing microCOVID.org in the review: "Is this what the Review is meant to reward? It sure was useful, but was it 'intellectual progress'? Was it 'timeless'? What would it mean to put it in a book?"

We haven't yet made our final call on whether/how to do books this year, but if I were to print microCOVID, I'd specifically be interested in printing the whitepaper. Unlike the announcement (which would be quite weird to read in a book), the whitepaper is clearly a piece of enduring intellectual labor.

I actually took some pretty timeless, valuable lessons from microCOVID – the schema that you can break "risk" into units that linearly add up was helpful for my overall thinking, and I expect to be relevant in other domains. And the conceptual breakdown of splitting up "person risk" and "activity risk" was useful.

I think microCOVID was a great instance of applied rationality that built off a lot of concepts in our ecosystem, and translated them into a concrete product.

Thoughts on Draft Report on AI Timelines

Ajeya's Draft Report on AI Timelines inspired a ton of discussion: Eliezer wrote a response post...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New EA cause area: Breeding really dumb chickens, published by Sam Enright on February 20, 2022 on The Effective Altruism Forum.

A friend at dinner asked me why EAs aren't working on breeding chickens (and other farm animals) to be less intelligent as a way to reduce the suffering caused by factory farming. I was embarrassed that I didn't have a good response for him. Some questions I had about this idea:

Is selective breeding capable of meaningfully changing an animal's capacity to suffer? Domestic animals have brains that are 15-35% smaller than their wild counterparts, so there is some interaction between domestication and intelligence.

Is anyone working on genetic engineering or gene drives to make farm animals less able to suffer?

Is this a hedge against clean meat? This strategy would be unnecessary if clean meat is successful, but it may be more tractable. Also, a small proportion of people may refuse to switch to clean meat even when it is widely available, and we can minimise their negative impact by breeding farm animals with minimal capacity to suffer.

Has anyone looked into this? Am I going crazy?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Microcovid Becoming Less Useful, published by jefftk on February 20, 2022 on LessWrong.

I want to start by saying how much I appreciate what the volunteers behind

Microcovid have done. Providing a risk calculator has been really helpful to a lot of people. There's also been a lot of value in running a centralized estimate of the impact of different mitigations.

On the other hand, at this point, I think it has become substantially less useful, and in many cases is leading people to seriously overestimate the level of precautions that are appropriate for their particular situation:

Its data on prevalence is stale, because the automatic updater last ran successfully on 2022-02-07.

To determine how risky things are in your area it uses the average number of cases from last week. When cases are rising, it extrapolates to predict growth. On the other hand, when cases are falling (as they are now here, and have been for a while) it does not extrapolate and assumes that cases have stayed flat.

It's models the people you interact with as being equally likely to currently be infectious, regardless of their vaccination status. Vaccinated people, however, are less likely to get covid and are infectious for a shorter time period. [EDIT: possibly it is modeling the "less likely to get covid" piece but not "less likely to transmit if infected" and "infectious for less time" pieces?]

It defaults to a risk budget of 200 microcovids per week, a 1% annual chance of getting covid, regardless of what you tell it for your vaccination status. The highest budget it offers is only 10x that, for people who "can't avoid risk, but still want to make smart choices". This was a reasonable way of looking at things earlier in the pandemic, but at this stage I think much higher budgets are generally appropriate for fully vaccinated people: the risk to the individual is generally extremely low and the risk they impose on others is relatively low. Altruistically, since most of society is operating on a much higher risk budget keeping to low one is a lot of sacrifice for minimal benefit.

For a vaccinated and boosted person I think a better approach is to use no budget most of the time, and then use a budget of perhaps

10,000 microcovids/week (~50% annual risk) when demand on the medical system is likely to be especially high.

Even just the first two have a very large impact. For example, it currently gives a random person in my county as having a 0.78% chance of having covid (780 cases per

100k). Let's walk through the basic method with current data.

For the most recent seven days we've averaged 379 cases/day. Population is 1.6M, so that's 24/100k. The positive test rate is

2.2%. They have some math I don't entirely follow the reasoning for, with:

Where

day_i is number of days since 2020-02-12 (738 as of today). They cite

Estimating

True Infections Revisited: A Simple Nowcasting Model to Estimate

Prevalent Cases in the US, but I'm just going to use it as is:

Using current data moves the risk down 14x, from 780/100k to 54/100k.

Now let's apply the extrapolation that they do when cases are rising. The seven days before that averaged 460 cases/day, or

29/100k. Test positivity was 4%, so prevalence_ratio is 2.33, and estimated true_infections is 68/100k. Dividing 54 by 68 we get 79%, so the extrapolated current number is 79% 54/100k.

Consistent downward extrapolation brings it down another 20%, from

54/100k to 43/100k, for a total of 18x. This extrapolation is warranted here, if you look at the (more current) wastewater numbers.

Comment via: facebook

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What The Foucault, published by adamShimi on February 19, 2022 on LessWrong.

Epistemic status: exploratory

Student: Rah, it’s so frustrating!

Master: What is?

Student: I keep stumbling back into Foucault, and I’m never able to decide if I should discard him or if there’s something in there.

Master: I see. Why do you think that he might be irrelevant?

Student: A bunch of reasons: people keep telling me that he’s either wrong or outright dangerous, he apparently made a specialty of historical mistakes and biased evidence, and post-modernism seems to always end in denying reality.

Master: Pretty damning. And yet you still think that he could teach you something?

Student: That’s the frustrating part! I have this intuition that Foucault built — or tried to build — some great epistemic tools, and that these could be an important part of the toolkit I’m building.

Master: Just try reading him then, and see where that leads you.

Student: I did.

Master: Oh. And it didn’t go well, given your look.

Student: I might have thrown the book in frustration.

Master: I see.

Student: And stomped on it. And insulted its ancestry. And burned it.

Master: Not reading him directly, then. Or at least not without some preparation. And maybe some anger counseling.

Student: That’s why I looked for a secondary source after that.

Master: Good call.

Student: And I found this book: “How to read Foucault”, by Johanna Oksala.

Master: How should you read him, then?

Student: Well, she does make a case for Foucault as breaking certainties. For example, she writes

While science and much of philosophy aim to decipher from among the confusion of events and experiences that which is necessary and can be articulated as universal law, Foucault's thought moves in exactly the opposite direction. He attempted to find among the necessities that which upon closer philosophical scrutiny turned out to be contingent, fleeting and arbitrary. For Foucault, the aim of philosophy is to question the ways in which we think, live and relate to other people and to ourselves in order to show how that-which-is could be otherwise.

Master: But you already knew Foucault’s aim was critical, didn’t you? That’s one of the issue with hardcore postmodernism: overcriticizing to the point of intellectual suicide.

Student: Yes, but I wasn’t sure that this was a fair reading. The fact that a Foucault scholar and overall defender says it has more weight than from critics.

Master: Fair enough. That still doesn’t tell you much about his tools though.

Student: That’s the other thing I learned about: that Foucault’s main tool was history. He used to reveal where the concepts and necessities came from, and in doing so take away from them their sacred nature of “natural” background assumptions.

Master: Hum, so history for finding out where our concepts and values come from, in order to question them?

Student: Exactly! And the historical path plays a big role in the criticism, especially if they’re here for the wrong reasons.

Master: Interesting. Reminds me somewhat of Inadequate Equilibria.

Student: That’s what I thought! Except the focus is more concrete — when you can beat the consensus — and Yudkowsky depends far more on game theory and evolutionary psychology than history.

Master: And I guess there’s also similarities with works like signaling theory and The Categories were made from Man, not Man for the Categories?

Student: That’s my guess.

Master: Yet all these approaches work well; why do we need Foucault?

Student: I don’t know! I mean, okay, maybe it’s that Foucault is more. epistemic?

Master: In what sense?

Student: Something like “Foucault wants to understand the underlying epistemic changes and breaks”.

Master: Wait a minute: that’s something completely different.

Student: Oh. You’re right. So there’s two things:

The historical continge...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simulation arguments, published by Joe Carlsmith on February 18, 2022 on LessWrong.

“If Nature herself proves artificial, where will you go to seek wildness? Where is the real out-of-doors?”

– C.S. Lewis

(Cross-posted from Hands and Cities. Content warning: weird)

Summary:

This post examines “simulation arguments” – i.e., arguments that we should assign significant probability to living in a computer simulation.

I distinguish between two types. The first type starts with our “standard story” about the world, and tries to argue that on this story, unless something prevents the creation of suitably many sims suitably like us, we’re probably sims. I think this version fails: on our standard story, none of the sims will make our observations. For example, they won’t observe the specific set of books on my bookshelf.

This sort of objection feels cheatsy – and ultimately, I think, it doesn’t work on the best version of the argument. But I use a series of cases to bring out its similarity to less cheatsy-feeling cases: for example, cases in which it appears that all the sims that will ever get made are squid-people with tentacle arms. Simulation arguments, I suggest, should say the same thing about all these cases.

But the “indifference principles” that classic simulation arguments rely on get confused about this. In particular, they end up having to hand-wave about which things we count as “knowing,” and/or about which “reference classes” are “admissible.” This confusion makes it seem like classic arguments require an unprincipled kind of “selective skepticism” – one that risks undermining itself.

I think we do better to focus on a second type of simulation argument, which relies centrally on a claim about what sorts of priors it’s reasonable to have over “structurally similar” worlds that contain your observations. In particular, instead of saying “on your standard story, you’re probably a sim,” this version says: “if that’s your story, you should put lots of weight on these alternative, structurally similar stories, too – on priors, such stories are similarly likely, and they contain your observations, too. But on those stories, you’re a sim.”

I think this second type of argument works better – and that it brings more to the table than a standard skeptical “what if.” Indeed, I think that classic formulations of the simulation argument are implicitly leaning on something in the vicinity. But bringing out the underlying assumptions about “reasonable priors” more explicitly helps to avoid the confusion that classic formulations often create.

That said, Type 2 arguments raise their own questions – for example, about how the relevant notion of “structural similarity” should be understood and applied, and about serious flirtations with “epistemic pascal’s muggings.” These questions (and other heuristics) leave me wary of Type 2 arguments, too.

I also discuss two further simulation arguments. The first is from Thomas (2021), who argues that we’re almost certainly sims, because conditional on being non-sims, the expected ratio of sims to non-sims is very high. I think this argument rests on learning the lesson of the classic argument in one breath, but forgetting this lesson in the next.

The other argument is that even conditional on finding yourself (apparently) living in earth’s early history, finding yourself as an “unusually something” person (e.g., unusually influential, entertaining, etc) should be some kind of substantive additional update in favor of being a sim. I don’t currently see how this argument is supposed to work; and on the classic argument, it seems like Donald Trump’s credence on “I’m a sim” should be basically the same as that of a janitor in Des Moines (even if there will be many more Trump sims than janitor sims).

I conclude with a few reflections on “simu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why you are psychologically screwed up, published by R Bruns on February 18, 2022 on LessWrong.

Linkpost for

Epistemic status: deliberately written in simplified tale/myth mode. I could go off on so many tangents on the actual complexity and nuance of everything, but I wanted to keep it tight. I encourage you to 'unpack' the points by comparing them to things you know, and/or framing what I say in different ways, to see if the main theme/message makes any sense to you.

1: Monkeys Become Ants

In the beginning there were monkeys. Individuals had their own models of physical reality, and desired things, and took actions based on their desires and reality models. This often involved fighting the other monkeys to get something. Life was solitary, poor, nasty, brutish, and short.

And then something happened. Some monkeys became more cooperative and social. They formed packs, and the packs were more successful than the individuals. The monkeys in packs coordinated their actions and learned from each other. Homo sapiens crushed its competition and filled the world.

Making the pack work required a lot of psychological engineering. People had to be programmed to replace their own desires and world models with the shared desires and models of the pack, at least when it was necessary for the pack to cooperate and do something. People who were unable or unwilling to do this were selected against.

Over time, cultures and religions and ideologies learned new tricks to enable cooperation at larger scales. They became more aggressive about replacing the desires and world-models of people with those of the group. People were acculturated to deeply internalize the group's desires and have faith in the group's description of reality, such that going against either was somewhere between morally abhorrent and unthinkable.

The groups that were capable of generating cooperation/brainwashing at scale crushed the ones that had less ability to do this. Eventually giant hiveminds formed, after figuring out how to brainwash millions of people into useful cooperation. They came to rule the world, and people who were unable or unwilling to be assimilated into these mega-hiveminds were selected against.

In short, monkeys were psychologically manipulated into sometimes acting like ants. It didn't matter how traumatizing this process was, because the hive minds that did it won the fight against those who did not, and filled the world with their genes and memes.

Because of this, most people usually operate on a level of social reality, rather than physical reality. The hivemind doesn't want you to have your own model of the physical world, unless you are one of the few ordained specialists in having that job. Having your own model is a rebellion, a rejection of the cooperation protocol, and marks you out as a cancerous or treasonous element.

Similarly, having any desires not programmed or approved by the hive marks you as a threat. You are rebelling by doing the abhorrent/unthinkable thing. If the hive learns about this, you are in trouble. If you make it common knowledge that you have this, and are proud and open about it, such that you are making a public challenge to the cooperation protocol, you are in very deep trouble.

2. Evolved Hypocrisy

But humans did not become completely eusocial; we continued to reproduce as individuals. And there was plenty of opportunity to steal from the hive for personal gain, for those clever enough to do so without getting caught.

In addition, being one of the people with power at the top of society requires you to act as an individual monkey. You have to see the world as it is, and take action and make actual decisions. Hiveminds that completely destroyed the ability of their leaders to act as agents did not last very long. In addition to the leaders, the successful...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment researchers, how useful is extra compute for you?, published by Lauro Langosco di Langosco on February 19, 2022 on The AI Alignment Forum.

TLDR

If you work in AI alignment / safety research, please fill out this form on how useful access to extra compute would be for your research. This should take under 10 minutes, and you don't need to read the rest of this post beforehand—in fact it would be great if you could fill out the form right now.

Introduction

I want to get an idea of how much demand there is for a university-independent organization that manages a compute cluster for academic AI alignment groups and independent researchers. Currently I don’t know anybody who is willing to run such an organization, but if demand is large one could either actively look for people to run such a project or find an existing organization that is willing to take it on.

Main idea

Non-industry AI safety research organizations have a hard time procuring compute. Groups spend many researcher-hours on managing servers on a relatively small scale. Common obstacles are 1) having to deal with university bureaucracy (e.g. regarding hiring, engineer wages, and procurement) and 2) missing out on economies of scale.

Proposal: A university-independent organization that provides access to compute for academic AI alignment research groups as well as independent researchers. Such an organization could pay high wages for its engineers (compared to academic labs) and benefit from economies of scale.

Potential benefits

Time saved: currently, researchers spend time applying for compute grants, setting up and maintaining servers, and setting up software environments after switching between systems. Easy access to large amounts of compute would avoid most of these time costs.

Expanded capability to do research: a centralized organization could afford to manage larger compute clusters than those usually used by individual labs. The difference is even larger for independent researchers, who might not have access to large-memory GPUs at all.

Potential problems

Gatekeeping: just like with other resources such as funding, deciding who gets access can be hard and risks becoming a political problem. OTOH, subject-specific grants are common / accepted within academia. Still, management of access would have to be done carefully.

Demand: some groups have access to large university clusters and may not need this service. I’m currently uncertain about how large the demand for this is.

Leadership: even if it were clear that this is a good idea, I don’t know of anyone who is willing to run this project. This seems like a solvable problem though, once we have a clear idea of what the demand is.

Form

If you haven't already, please fill out this form about how much extra compute might accelerate your research (<10 mins).

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.