Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transformative VR Is Likely Coming Soon, published by jimrandomh on October 13, 2022 on LessWrong.

Meta (formerly known as Oculus and as Facebook) just had their yearly conference. I think right now is a good time to take a brief break from worrying about AI timelines, and take a few minutes to notice the shortness of VR timelines. VR isn't going to determine whether humanity lives or dies, but I think it's close enough to have immediate-term impact on decisions about where to live, and how to structure organizations.

Concretely: I think we're 6 months from the crossover point where VR is better than in-person for a narrow subset of meetings, specifically, 1:1 meetings without laptops, where both sides are knowledgeable enthusiasts using the latest headset in combination with a high-end desktop PC. I estimate about 2.5 years until the crossover point where VR is better than in-person for most meetings.

I think most people are wildly underestimating how impactful, and how sudden, this is going to be. I think they'll continue underestimating how impactful it's going to be, right up until the second crossover happens.

Basically, everyone tried out Zoom meetings and discovered that they sucked compared to in-person meetings, for reasons that are slightly hard to put your finger on. Some people tried VR meetings, and discovered that they were worse. People then updated towards believing that remote work (including both videochat and VR chat) would not be transformative, but did so without a model of what concretely was going wrong and when/whether it would be fixed. But, zooming in on the technology more closely, I see a fairly short list of known problems that seems to fully explain why people prefer in-person meetings in practice, and I'm watching that list slowly shrink. I currently think every item on the VR-meetings-suck list has line-of-sight to a full solution, and that at that point, VR meetings will be straightforwardly better than in-person ones.

For context: Oculus announced a new headset, the Quest Pro, at their yearly conference yesterday. It ships in two weeks and costs $1500. Consumer sentiment about it is probably going to be negative; I expect a lot of grumbling about the price and the battery life, in particular. But this is going to be misleading. The important thing about it is that Quest Pro is that it contains a first-pass version of all but one item on my why-remote-meetings-suck list. These solutions are going to have problems at first, but the problems are not fundamental.

In particular:

Color passthrough cameras solve the problem where entering VR requires giving up the ability to see the real world, which makes it very costly compared to joining a call. There are some minor visual-quality caveats, and some major software-not-ready caveats about how the worlds mix.

Inward-facing cameras with face and eye tracking. Under research-prototype-demo conditions, it looks like this solves the problem of not being able to see peoples' facial expressions. Under realistic conditions, what I've read suggests that they've got framerate problems, which is why I specified "1:1 with gaming PCs" as a setup that will work before other setups do.

Audio latency. I believe Oculus is currently pretty far ahead of Zoom on this one, due to a combination of console-like known hardware, expertise crossover from video-latency being a core requirement of VR, and engineering prioritization.

The one caveat that this doesn't attempt to address is that the panel resolution isn't high enough to have small text, which in turn means you can't bring a normal-sized laptop with you into VR. As far as I can tell this is not a fundamental problem, denser panels do exist elsewhere on the market.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please vis...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Help out Redwood Research’s interpretability team by finding heuristics implemented by GPT-2 small, published by Haoxing Du on October 12, 2022 on LessWrong.

Some of Redwood’s current research involves finding specific behaviors that language models exhibit, and then doing interpretability to explain how the model does these behaviors. One example of this is the indirect object identification (IOI) behavior, investigated in a forthcoming paper of ours: given the input When John and Mary went to the store, Mary gave a flower to, the model completes John instead of Mary. Another example is the acronym generation task: given the input In a statement released by the Big Government Agency (, the model completes BGA).

We are considering scaling up this line of research a bunch, and that means we need a lot more behaviors to investigate! The ideal tasks that we are looking for have the following properties:

The task arises in a subset of the training distribution. Both the IOI and the acronym tasks are abundant in the training corpus of language models. This means that we are less interested in tasks specific to inputs that never appear in the training distribution.

The ideal task can even be expressed as a regular expression that can be run on the training corpus to obtain the exact subset, as is the case for acronyms. The IOI task is less ideal in this sense, since it is harder to identify the exact subset of the training distribution that involves IOI.

There is a simple heuristic for the task. For IOI, the heuristic is “fill in the name that appeared only once so far”. For acronyms, the heuristic is “string together the first letter of each capitalized word, and then close the parentheses”. Note that the heuristic does not have to apply to every instance of the task in the training distribution, e.g. sometimes an acronym is not formed by simply taking the first letter of each word.

The gold standard here is if the heuristic can be implemented in an automated way, e.g. as a Python function, but we would also consider the task if a human is needed to supply the labels.

GPT-2 small implements this heuristic. We are focusing on the smallest model in the GPT-2 family right now, which is a 117M parameter model that is frankly not that good at most things. We are moderately interested in tasks that bigger models can do that GPT-2 small can’t, but the bar is at GPT-2 small right now.

Examples

The following is a list of tasks that we have found so far/are aware of. Induction and acronym generation remain the tasks that best meet all of the above desiderata.

Induction: If the model has seen . A B . in the input, then it is more likely to output B immediately after A when encountering A again. A common use case of induction is remembering complicated names: T|cha|ik|ovsky . T| -> cha|ik|ovsky

Acronym generation

IOI

Email address generation: Hi! My name is John Smith, and I work at Apple. My email address is -> [email protected]

Subject-verb agreement: The keys to the cabinet -> are and not is, even though the singular “cabinet” is also present in the prompt. (Credit to Nate Thomas)

Incrementing citation numbers: Most metals are excellent conductors [1]. However, this is not always the case [ -> 2] (Credit to Lucas Sato)

Variations of induction and IOI

Extended IOI: identify the right person out of a group with more than two names

Inferring someone’s last name from name of relative

dog_in_house.jpg”> (Credit to Rob Miles)

Some examples that we are less excited about include:

Closing doc strings, parentheses, etc

Close variations of known examples

We would love for interested people to contribute ideas! Below are some resources we put together to make the search as easy as possible:

You can play with GPT-2 small here in a web app that we built specifically for investigating model b...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Niceness is unnatural, published by Nate Soares on October 13, 2022 on The AI Alignment Forum.

When I'm arguing points like orthogonality and fragility of value, I've occasionally come across rejoinders that I'll (perhaps erroneously) summarize:

Superintelligences are not spawned fully-formed; they are created by some training process. And perhaps it is in the nature of training processes, especially training processes that involve multiple agents facing "social" problems or training processes intentionally designed by humans with friendliness in mind, that the inner optimizer winds up embodying the spirit of niceness and compassion.

Like, perhaps there just aren't all that many ways for a young mind to grow successfully in a world full of other agents with their own desires, and in the face of positive reinforcement for playing nicely with those agents, and negative reinforcement for crossing them. And perhaps one of the common ways for such a young mind to grow, is for it to internalize into its core goals the notions of kindness and compassion and respect-for-the-desires-of-others, in a manner broadly similar to humans. And, sure, this isn't guaranteed, but perhaps it's common enough that we can get young AI minds into the right broad basin, if we're explicitly trying to.

One piece of evidence for this view is that there aren't simple tweaks to human psychology that make them significantly more reproductively successful. Sociopathy isn't at fixation. Humans can in fact sniff out cheaters, and can sniff out people who want to make deals but who don't actually really care about you — and those people do less well. Actually caring about people in a readily-verifiable way is robustly useful in the current social equilibrium.

If it turns out to be easy-ish to instill similar sorts of caring into AI, then such an AI might not share human tastes in things like art or humor, but that might be fine, because it might embody broad cosmopolitan virtues — virtues that inspire it to cooperate with us to reach the stars, and not oppose us when we put a sizable portion of the stars toward Fun.

(Or perhaps we'll get even luckier still, and large swaths of human values will turn out to have pretty-wide basins that we can get the AI into if we're trying, so that it does share our sense of humor and laughs alongside us as we travel together to the stars!)

This view is an amalgam of stuff that I tentatively understand Divia Eden, John Wentworth and the shard theory advocates to be gesturing at.

I think this view is wrong, and I don't see much hope here. Here's a variety of propositions I believe that I think sharply contradict this view:

There are lots of ways to do the work that niceness/kindness/compassion did in our ancestral environment, without being nice/kind/compassionate.

The specific way that the niceness/kindness/compassion cluster shook out in us is highly detailed, and very contingent on the specifics of our ancestral environment (as factored through its effect on our genome) and our cognitive framework (calorie-constrained massively-parallel slow-firing neurons built according to DNA), and filling out those details differently likely results in something that is not relevantly "nice".

Relatedly, but more specifically: empathy (and other critical parts of the human variant of niceness) seem(s) critically dependent on quirks in the human architecture. More generally, there are lots of different ways for the AI's mind to work differently from how you hope it works.

The desirable properties likely get shredded under reflection. Once the AI is in the business of noticing and resolving conflicts and inefficiencies within itself (as is liable to happen when its goals are ad-hoc internalized correlates of some training objective), the way that its objectives ultimately shake out ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment 201 curriculum, published by richard ngo on October 12, 2022 on The Effective Altruism Forum.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Responding to recent critiques of iron fortification in India, published by e19brendan on October 12, 2022 on The Effective Altruism Forum.

Two recent EA Forum posts offered critiques on iron fortification in India and particularly the work of Fortify Health, given its relationship with EA: Targeted Treatment of Anemia in Adolescents in India as a Cause Area and Cost-effectiveness of iron fortification in India is lower than GiveWell's estimates, both authored by Akash Kulgod.

We heartily welcome the inquiry and critique, and Fortify Health will remain open minded in pursuit of understanding the true impact of its work and in adjusting course accordingly. I am one of the co-founders of Fortify Health, and although I have since stepped back from day-to-day operations, I offered to address the concerns raised so that the active core team can continue to focus on their implementation and partnership work.

In this post, I intend to provide initial responses to key points made in Kulgod’s articles:

Is iron fortification expected to increase diabetes prevalence in India?

Short answer: This should not be inferred from available evidence.

Is the proportion of anemia in India attributable to iron deficiency lower than we thought?

Short answer: At first glance, more recent estimates of iron deficiency among children cited are on average roughly similar to those incorporated in GiveWell’s 2019 CEA, and state and age specific prevalence could be incorporated into future models. Givewell’s 2019 model largely relies on iron deficiency prevalence rather than iron deficiency anemia prevalence, but I’m uncertain as to whether or for what outcomes use of one or both parameters would lead to the most accurate model.

Should hemoglobin cutoff values be changed, and how does this affect the expected impact of fortification efforts?

Short answer: The distribution of hemoglobin levels among the Indian sample cited does not support the inference that outcomes associated with anemia are equivalent between populations at different threshold hemoglobin levels. The new information presented by the cited study does not weaken the expected impact of fortification.

Is a targeted intervention preferable to widespread fortification?

Short answer: Targeted screening and treatment should be available as part of comprehensive primary care and possibly school-based programs, but does not preclude rapidly scaling up fortification efforts. Further exploration of this intervention would be worthwhile.

Is micronutrient fortification ethical?

Short answer: Extending the benefits of fortification is one part of an ethical imperative for health equity, and absence of fortification does not provide greater autonomy.

I recognize that this post only scratches the surface of complex issues, and it does not provide a comprehensive review of all available arguments and evidence that may be relevant. Its scope is substantially more limited, merely contributing to an evolving conversation started by Kulgod’s posts. I’m eager to get feedback from Kulgod and other readers, and hope we can collaboratively advance our own understanding and the EA community’s understanding of fortification efforts.

I. Is 10mg/day per capita iron fortification expected to raise prevalence of diabetes by 2-14% as apparently suggested by Ghosh et al. (preprint 2021)?

Initial reaction: the cited study’s conclusion does not seem to be supported by its results. Claims of correlation between high fasting blood sugar with discrete elevation in serum ferritin are supported by only a small effect size in a subgroup analysis of the highest wealth quintile, which is sharply discontinuous from the next wealth quintile. Furthermore, their model of marginal increase in fasting blood sugar by serum ferritin levels demonstrates minimal effect size, even in ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [MLSN #6]: Transparency survey, provable robustness, ML models that predict the future, published by Dan Hendrycks on October 12, 2022 on The AI Alignment Forum.

You can subscribe to the newsletter here, follow the newsletter on twitter here, or join the subreddit here.

Welcome to the 6th issue of the ML Safety Newsletter. In this edition, we cover:

A review of transparency research and future research directions

A large improvement to certified robustness

“Goal misgeneralization” examples and discussion

A benchmark for assessing how well neural networks predict world events (geopolitical, industrial, epidemiological, etc.)

Surveys that track what the ML community thinks about AI risks

$500,000 in prizes for new benchmarks

And much more.

Monitoring

Transparency Survey

A taxonomy of transparency methods. Methods are organized according to what part of the model they help to explain (weights, neurons, subnetworks, or latent representations). They can be intrinsic (implemented during training), post hoc (implemented after training), or can rely on a mix of intrinsic and post hoc techniques. ‘Hazards’ (in orange) are phenomena that make any of these techniques more difficult.

This survey provides an overview of transparency methods: what’s going on inside of ML models? It also discusses future directions, including:

Detecting deception and eliciting latent knowledge. Language models are dishonest when they babble common misconceptions like “bats are blind” despite knowing that this is false. Transparency methods could potentially indicate what the model ‘knows to be true’ and provide a cheaper and more reliable method for detecting dishonest outputs.

Developing rigorous benchmarks. These benchmarks should ideally measure the extent to which transparency methods provide actionable insights. For example, if a human implants a flaw in a model, can interpretability methods reliably identify it?

Discovering novel behaviors. An ambitious goal of transparency tools is to uncover why a model behaves the way it does on a set of inputs. More feasibly, transparency tools could help researchers identify failures that would be difficult to otherwise anticipate.

Other Monitoring News

[Link] This paper discusses the sudden emergence of capabilities in large language models. This unpredictability is naturally a safety concern, especially when many of these capabilities could be hazardous or discovered after deployment. It will be difficult to make models safe if we do not know what they are capable of.

[Link] This work attributes emergent capabilities to “hidden progress” rather than random discovery.

[Link] Current transparency techniques (e.g., feature visualization) generally fail to distinguish the inputs that induce anomalous behavior

Robustness

Mathematical Guarantees of Model Performance

The current state-of-the-art method for certified robustness (denoised smoothing) combines randomized smoothing with a diffusion model for denoising. In randomized smoothing, an input is perturbed many times and the most commonly assigned label is selected as the final answer, which guarantees a level robust accuracy within a certain perturbation radius. To improve this method, the perturbed inputs are denoised with a diffusion model after the perturbation step so that they can be more easily classified (from Salman et al.)

A central concern in the robustness literature is that empirical evaluations may not give performance guarantees. Sometimes the test set will not find important faults in a model, and some think empirical evidence is insufficient for having high confidence. However, robustness certificates enable definitive claims for how a model will behave in some classes of situations.

In this paper, Carlini et al. recently improved ImageNet certified robustness by 14 percentage points by s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cultural EA considerations for Nordic folks, published by Ada-Maaria Hyvärinen on October 12, 2022 on The Effective Altruism Forum.

I wrote this to list some cultural EA considerations from a Finnish/Nordic perspective. I guess some other people might find it useful too. And some others might feel that all of this is very obvious to them.

If you spot any mistakes or disagree on something please let me know in the comments!

US/UK cultural stuff

Many EA folks are located in the UK, especially London and Oxford

Even more EA folks are located in the US, especially in the Bay Area, which is an area in California and has places like San Francisco and Berkeley in it. This area is also known for having many IT companies (if you have heard of Silicon Valley this is the same place). This place is so common for EAs to live in that if you see posts like “is anyone up for going to the park” in an EA group they probably live there. (Kind of like people from Helsinki forget there are other places in Finland.)

And some US EA folks are in Boston which is on the other side of the country. Of course there are EAs also in other US places but these are some of the biggest ones.

It is more common to move to countries within the anglophone world (for example from Australia to the US) for work or study. People in EA might assume you’d be willing to do that (or that what is stopping you from doing that are external conditions and not for example “wanting to live in Finland”).

Many cities many EAs live in have very high living costs. Thus, even full-time working adults often do flat sharing. This is why some people have set up EA (or rationalist) group houses: they’d prefer to live with like-minded folks.

Since the state is not that great at taking care of people in the US, many people feel a social expectation to give money to their struggling relatives or acquaintances, for example if they don’t have enough to pay for a medical operation.

Because of this, US non-EA folks might think they are already being altruistic in their daily lives – they are, but it is not the impartial and cost-effective type.

In Finland it might be more common for non-EAs to feel like they should not be asked to donate, because helping all people should be outsourced to the state and helping should not be the responsibility of an individual. (I mostly agree with the sentiment but we don’t live in an ideal world, so help from individuals is still needed.)

Other geographical stuff

There are quite some EA/rationalist folks in Central Europe as well, in particular in Germany, Switzerland and Austria.

Also, Prague has a lot of EA/rationalist folks and established locations that are used for rationalist workshops and EA coworking.

Many other Western places have EA groups, some of them bigger than others.

There are also EAs in non-Western places but many non-Western groups are still small. But not all, for example EA Philippines is quite active.

Out of Nordic countries, the probably most well-established EA activity happens in Norway, but all Nordic countries have active EA national organizations and are catching up.

In the EA context Estonia is sometimes counted as a Nordic country. It has more EA activity than other Baltics and participates in some Nordic collaboration.

If you are interested in the cross-cultural aspect of EA, you are in luck

A lot of EAs like meeting (or even housing) other EAs when traveling, so if you are going abroad anyway, a good option is to check if there is a local group where you could meet new interesting people.

You can also connect to other EAs across the world through online EA stuff such as EA Anywhere or the EA Gathertown Coworking Space (you don’t need to be working on anything directly EA related to spend time on the latter!)

Donations

In the US, it is quite common to donate large sums ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Calibration of a thousand predictions, published by KatjaGrace on October 12, 2022 on LessWrong.

I’ve been making predictions in a spreadsheet for the last four years, and I recently got to a thousand resolved predictions. Some observations:

I’m surprisingly well calibrated for things that mostly aren’t my own behavior1. Here’s the calibration curve for 630 resolved predictions in that class:

I don’t know what’s up with the 80% category, but the average miscalibration of the eleven categories is <3%.

At risk of bragging, this seems wild to me. My experience of making these predictions is fairly well described as ‘pulling a number out of thin air’2. But apparently if you take all these conjured numbers, and look at the 45 of them that fell in the vicinity of 40%, then I implicitly guessed that 17.28 of those events would happen. And in fact 18 of them happened. WTF? Why wasn’t it eight of them or thirty-five of them? And that was only the fifth most accurate of the eleven buckets shown above! For predictions in the vicinity of 70%, I was off by 0.15%—I said 54.88 of those 80 things would happen, and in fact 55 of them happened.

Possibly people overall are just better calibrated than I thought. I had some remembered view that people’s asserted 90% confidence intervals were really 50% confidence intervals or something, but I can’t immediately find such evidence now, and I can find various graphs of groups of people being on average fairly calibrated. And the handful of PredictionBook users I could find with more than a thousand predictions are not hugely worse.

If you are curious about what I predicted, I put examples at the end of this post.

For the entire thousand predictions—the above plus 370 about my own behavior— I’m off by 6.25% on average (up from 2.95%) over the same eleven buckets.

As you may infer, I’m pretty bad overall at predicting my own behavior!

This is more what I expected of a calibration curve—broadly overconfident. And perhaps its general worseness is explained by the appeal of optimism in predicting oneself. But it’s a pretty weird shape, which seems less explicable. If I think there’s a 40% chance that I’ll do something, apparently it’s not happening. If you want it to happen, you should hope I change my mind and put 5% on it!

I’m not sure what is up with this particular strange shape. But note that making predictions about one’s own behavior has particular complication, if one likes to be right. If you put a number below 50% on taking an action, then you have a disincentive to doing it. So you should then put a lower probability on it than you would have, which would make you even more wrong if you took the action, so you have a further disincentive to doing it, etc. I do generally look for a fixed point where given that I put probability p on something (and the incentive consequences of that), I do think it will happen with probability p. But this is a different process than the usual predicting process, and I could imagine it going wrong in strange ways. For instance, if I’m more motivated by being right than I thought, then 40% predictions which might have been 50% predictions originally should really be 5% predictinons.

This theory doesn’t really work though, because then shouldn’t the lower categories also be especially overconfident? Whereas in fact they are okay.

(Maybe I just have free will? The kind of free will that manifests as being about 15% less likely to do anything than one might have expected seems disappointing, but the menu of possible free will options was never that inspiring.)

Example predictions

Here are some typical predictions, arbitrarily pulled from my spreadsheet and lightly edited:

I will be invited to play ONUW today: 0.45 (true)

The trial bank transfers come through to my IBKR account by the end of Monday: 0.34 (fal...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Parfit + Singer + Aliens = ?, published by Maxwell Tabarrok on October 12, 2022 on The Effective Altruism Forum.

Summary:

If you

Have a wide moral circle that includes non human animals and

Have a low or zero moral discount rate

Then the discovery of alien life should radically change your views on existential risk.

A grad student plops down in front of their computer where their code has been running for the past few hours. After several months of waiting, she had finally secured time on the James Webb Space Telescope to observe a freshly discovered exoplanet: Proxima Centauri D.

“That’s strange.” Her eyes dart over the results. Proxima D’s short 5 day orbit meant that she could get observations of both sides of the tidally locked planet. But the brightness of each side doesn’t vary nearly as much as it should. The dark side is gleaming with light.

Northern Italy at night

The Argument

This argument requires a few assumptions.

Strong evidence of intelligent alien life on a nearby planet

Future moral value is not inherently less important than present moral value

Many types of beings contain moral value including nonhuman animals and aliens

I will call people who have a Singer-style wide moral circle and a Parfit-style concern for the long term future “Longtermist EAs.” Given these assumptions, lets examine the basic argument given by Longtermist EAs for why existential risks should be a primary concern.

Start with assumption 2. The lives of future human beings are not inherently less important than the lives of present ones. Should Cleopatra eat an ice cream that causes a million deaths today?

Then consider that humanity may last for a very long time, or may be able to greatly increase the amount of moral value it sustains, or both.

Therefore, the vast majority of moral value in the universe lies along these possible future paths where humanity does manage to last for a long time and support a lot of moral value.

Existential risks make it less likely or impossible to end up on these paths so they are extremely costly and important to avoid.

But now introduce assumptions 1 and 3 and the argument falls apart. The link between the second and third points is broken when we discovery another morally valuable species which also has a chance to settle the galaxy.

Discovering aliens nearby means that there are likely billions of planetary civilizations in our galaxy. If, like Singer, you believe that alien life is morally valuable, then humanity’s future is unimportant to the sum total of moral value in the universe. If we are destroyed by an existential catastrophe, another civilization will fill the vacuum. If humanity did manage to preserve itself and expand, most of the gains would be zero-sum; won at the expense of another civilization that might have taken our place. Most of the arguments for caring about human existential risk implicitly assume a morally empty universe if we do not survive. But if we discover alien life nearby, this assumption is probably wrong and humanity’s value-over-replacement goes way down.

Holding future and alien life to be morally valuable means that, on the discovery of alien life, humanity’s future becomes a vanishingly small part of the morally valuable universe. In this situation, Longtermism ceases to be action relevant. It might be true that certain paths into far future contain the vast majority of moral value but if there are lots of morally valuable aliens out there, the universe is just as likely to end up one of these paths whether humans are around or not so Longtermism doesn’t help us decide what to do. We must either impartially hope that humans get to be the ones tiling the universe or go back to considering the nearer term effects of our actions as more important.

Consider Parfit’s classic thought experiment:

Option A: Peace

O...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment 201 curriculum, published by Richard Ngo on October 12, 2022 on The AI Alignment Forum.

This curriculum, a follow-up to the Alignment Fundamentals curriculum (the ‘101’ to this 201 curriculum), aims to give participants enough knowledge about alignment to understand the frontier of current research discussions. It assumes that participants have read through the Alignment Fundamentals curriculum, taken a course on deep learning, and taken a course on reinforcement learning (or have an equivalent level of knowledge).

Although these are the basic prerequisites, we expect that most people who intend to work on alignment should only read through the full curriculum after they have significantly more ML experience than listed above, since upskilling via their own ML engineering or research projects should generally be a higher priority for early-career alignment researchers.

When reading this curriculum, it’s worth remembering that the field of alignment aims to shape the goals of systems that don’t yet exist; and so alignment research is often more speculative than research in other fields. You shouldn’t assume that there’s a consensus about the usefulness of any given research direction; instead, it’s often worth developing your own views about whether techniques discussed in this curriculum might plausibly scale up to help align AGI.

The curriculum was compiled, and is maintained, by Richard Ngo. For now, it’s primarily intended to be read independently; once we’ve run a small pilot program, we’ll likely extend it to a discussion-based course.

Curriculum overview

Week 1: Further understanding the problemWeek 2: Decomposing tasks for better supervisionWeek 3: Training on unrestricted adversarial examplesWeek 4: InterpretabilityWeek 5: Reasoning about ReasoningWeeks 6 & 7 (Track 1): Eliciting Latent KnowledgeWeeks 6 & 7 (Track 2): Agent FoundationsWeeks 6 & 7 (Track 3): Science of Deep LearningWeeks 8 & 9: Literature Review or Project Proposal

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.