Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Building a transformer from scratch - AI safety up-skilling challenge, published by Marius Hobbhahn on October 12, 2022 on The AI Alignment Forum.

It is not always obvious whether your skills are sufficiently good to work for one of the various AI safety and alignment organizations. There are many options to calibrate and improve your skills including just applying to an org or talking with other people within the alignment community.

One additional option is to test your skills by working on projects that are closely related to or a building block of the work being done in alignment orgs. By now, there are multiple curricula out there, e.g. the one by Jacob Hilton or the one by Gabriel Mukobi.

One core building block of these curricula is to understand transformers in detail and a common recommendation is to check if you can build one from scratch. Thus, my girlfriend and I have recently set ourselves the challenge to build various transformers from scratch in PyTorch. We think this was a useful exercise and want to present the challenge in more detail and share some tips and tricks. You can find our code here.

Building a transformer from scratch

The following is a suggestion on how to build a transformer from scratch and train it. There are, of course, many details we omit but I think it covers the most important basics.

Goals

From the ground up we want to

Build the attention mechanism

Build a single-head attention mechanism

Build a multi-head attention mechanism

Build an attention block

Build one or multiple of a text classification transformer, BERT or GPT. The quality of the final model doesn’t have to be great, just clearly better than random.

Train the model on a small dataset.

We used the polarity dataset for binary text sentiment classification.

We used the AG_NEWS dataset (PyTorch built-in) for BERT and GPT.

Test that the model actually learned something

We looked at the first batch of the test data to see if the model predicted something plausible.

We compared the test loss of a random network with the test loss of the trained network to see if our model is better.

Bonus goals

Visualize one attention head

Visualize how multiple attention heads attend to the words of an arbitrary sentence

Reproduce the grokking phenomenon (see e.g. Neel’s and Tom’s piece).

Answer some of the questions in Jacob Hilton's post.

Soft rules

For this calibration challenge, we used the following rules. Note, that these are “soft rules” and nobody is going to enforce them but it’s in your interest to make some rules before you start. We were

allowed to read papers such as Attention is all you need or the GPT-3 paper.

allowed to read tutorials on attention such as The illustrated transformer (as long as they don’t contain code snippets).

allowed to look at tutorials to build generic models in PyTorch as long as they don’t contain NLP architectures.

allowed to watch videos such as the ones from Yannic Kilcher on NLP

not allowed to look at the source code of any transformer or attention mechanism before you have implemented it ourselves. In case we struggle a lot, we can take a peek after we tried and failed to implement one building block ourselves.

We found Andrej Karpathy’s code helpful for the GPT implementation.

allowed to replace a part with a PyTorch implementation once we have demonstrated that it is equivalent. For example, once we have shown that our attention mechanism produces the same output for the same input as the PyTorch attention mechanism, we can use the PyTorch code block.

allowed to use generic PyTorch functions that are not directly related to the task. For example, we don’t have to write the embedding layer, linear layer or layer-norm from scratch.

Things to look out for

Here are some suggestions on what to look out for during the project

Do I understand the tut...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Instrumental convergence: single-agent experiments, published by Edouard Harris on October 12, 2022 on The AI Alignment Forum.

Thanks to Alex Turner and Vladimir Mikulik for pointers and advice, and for reviewing drafts of this sequence. Thanks to Simon Suo for his invaluable suggestions, advice, and support with the codebase, concepts, and manuscript. And thanks to David Xu, whose comment inspired this work.

Work was done while at Gladstone AI, which Edouard is a co-founder of.

🎧 This research has been featured on an episode of the Towards Data Science podcast. You can listen to the episode here.

Summary of the sequence

Over the past few months, we’ve been investigating instrumental convergence in reinforcement learning agents. We started from the definition of single-agent POWER proposed by Alex Turner et al., extended it to a family of multi-agent scenarios that seemed relevant to AI alignment, and explored its implications experimentally in several RL environments.

The biggest takeaways are:

Alignment of terminal goals and alignment of instrumental goals are sharply different phenomena, and we can quantify and visualize each one separately.

If two agents have unrelated terminal goals, their instrumental goals will tend to be misaligned by default. The agents in our examples tend to interact competitively unless we make an active effort to align their terminal goals.

As we increase the planning horizon of our agents, instrumental value concentrates into a smaller and smaller number of topologically central states — for example, positions in the middle of a maze.

Overall, our results suggest that agents that aren’t competitive with respect to their terminal goals, nonetheless tend on average to become emergently competitive with respect to how they value instrumental states (at least, in the settings we looked at). This constitutes direct experimental evidence for the instrumental convergence thesis.

We’ll soon be open-sourcing the codebase we used to do these experiments. We’re hoping to make it easier for other folks to reproduce and extend them. If you’d like to be notified when it’s released, email Edouard at [email protected], or DM me here or on Twitter at @harris_edouard.

1. Introduction

One major concern for AI alignment is instrumental convergence: the idea that an intelligent system will tend to pursue a similar set of sub-goals (like staying alive or acquiring resources), independently of what its terminal objective is. In particular, it’s been hypothesized that intelligent systems will seek to acquire power — meaning, informally, “ability”, “control”, or “potential for action or impact.” If you have a lot of power, then whatever your terminal goal is, it’s easier to accomplish than if you have very little.

Recently Alex Turner et al. have formalized the concept of POWER in the single-agent RL context. Roughly speaking, formal POWER is the normalized optimal value an agent expects to receive in the future, averaged over all possible reward functions the agent could have.

Alex has explored many of the implications of this definition for instrumental convergence. He and Jacob Stavrianos have also looked at how POWER behaves in a limited multi-agent setting (Bayesian games). But, as far as we know, formal POWER hasn’t yet been investigated experimentally. The POWER definition also hasn’t yet been extended yet to a multi-agent RL setting — and this could offer a promising framework to investigate more general competitive dynamics.

In this sequence, we’ll explore how formal POWER behaves in experimental RL environments, on both single-agent and multi-agent gridworlds. We’ll propose a multi-agent scenario that models the learning dynamics between a human (which we’ll call “Agent H” and label in blue) and an AI (which we’ll call “Agent A” and label in red) u...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Actually, All Nuclear Famine Papers are Bunk, published by Lao Mein on October 12, 2022 on LessWrong.

After doing a bit of digging, I'm pretty convinced that nuclear famine was never a thing to begin with. Maybe back during the Cold War, when the Soviets were seriously considering ground-bursting thousands of warheads across the American corn belt to knock out missile silos, but that's not the world we live in anymore. With modern C&C , they simply can't realistically expect to destroy those sites before launch. I've heard Peter Zaihan and alarmists say things like "the world only has 2 months' worth of food in reserve" and assumed they were right, but the numbers say otherwise. For this analysis, I'm using this USDA report.

This is dried grain stored across the United States. Soon, after the autumn harvest, the US will have more than 400 million tons of dried grains in storage. They range from 3000-4000 kcal per kg. Assuming an average daily consumption of 2000 kcal per person per day and a population of 330 million, that's around half a decade's worth of food from this stockpile alone. Most of it goes to animal feed/ethanol under normal circumstances. Even a worst-case scenario in September still gives a 90 million-ton stockpile, which is enough to feed America for more than a year. And at that point the grain is in the field, almost ready for harvest. This isn't an emergency stockpile, it's just what farmers have lying around to arbitrage against harvest-related price fluctuations.

There are also plenty of other foodstuffs in the American food supply chain, stockpiles of other foods, the FEMA food stockpile, and so on, but it is pretty incredible most analyses of nuclear famine don't take even this into account. At this point, I'm convinced the entire field is based on active misrepresentations for anti-nuclear messaging and was never at any point interested in honest inquiry.

The nuclear famine paper I analyzed earlier lied about this, by the way, implying that existing food stockpiles would only increase food availability in the first year. I am awestruck that this isn't more well-known when people talk about casualty projections from nuclear war.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: That one apocalyptic nuclear famine paper is bunk, published by Lao Mein on October 12, 2022 on LessWrong.

I've seen a few commentors and bloggers cite this study from Nature Food to explain why they think nuclear war may lead to the collapse of civilization in event of a full nuclear exchange between the US and Russia. I read it and updated towards Nature Food being a publication with bad terrible editorial standards more than anything else.

The Apocalypse is Bad at Math

Wow, 5 Billion people dead. That's pretty bad. It actually projects 99% population reduction in China, the US, and Russia due to a 90% reduction in global farm yields. But let's break down those numbers. Firstly, how did they even get those crop yield numbers? Well, it turns out they assumed a 10 degrees C decrease in temperatures in the northern hemisphere, and then assumed no change in the crop selection and area under cultivation.

This isn't just bad statistics. It's intentionally misleading. Of course, if temperatures drop 10 degrees and people do literally nothing in response crop yields will decrease massively. But that's not how human beings work. They will increase land under cultivation in the tropics, clear-cut the Amazon, and do whatever it takes to get food production going again. Obviously, they will switch to more cold-resistant crops rather than lose 100% of their harvest to frost. These assumptions are terrible. Garbage in, garbage out.

This is a bad paper. Other scientists in the field know it's a bad paper, which is why almost no one has cited it. It's a scaremongering piece designed to push a denuclearization agenda and is used as clickbait by journalists. Its bad assumptions make it basically useless. Please don't take any apocalyptic nuclear starvation scenarios based on it seriously. I am utterly astounded that a journal like Nature let this one through. I will definitely try to publish in Nature Food in the future.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I think nuclear war triggered by Russian tactical nukes in Ukraine is unlikely, published by Dave Orr on October 11, 2022 on LessWrong.

This is a response to Max's post Why I think there's a one-in-six chance of an imminent global nuclear war.

Max's thesis is that Putin is likely to use nuclear weapons if the war in Ukraine goes badly for Russia, that the west will retaliate, and both sides will continue to escalate until we have a full nuclear exchange. In particular, he estimates a 30% chance that Ukraine is nuked, an 80% chance of a significant conventional retaliation, and a 70% chance that things further escalate from there all the way to KABOOM.

I think the first and last terms in that equation are much too high. I think the center term is perhaps slightly too low (it's hard to believe that nuclear weapon use will be tolerated because it sets up terrible incentives for future nuclear weapon use), but I won't explore that here in depth.

Initial nuclear strike

An initial use of tactical nuclear weapons is unlikely because:

There is no military purpose served by nuclear weapon use

There are very large downsides to using nuclear weapons, including the scenario Max outlines

Because of points 1 and 2, such a use will not win Putin political points at home

The war against Ukraine can't really be won by bigger explosions. The Ukrainian military is broadly dispersed across a huge front, operating on both sides of the line. Similarly their equipment is not stored in a small number of supply depots.

Infrastructure and logistical networks can be disrupted with conventional explosives, and both Ukraine and Russia are attacking enemy logistics (Ukraine with rather more success than Russia). But the Russian ineffectiveness is due to missiles getting shot down and poor targeting, neither of which is helped by nukes.

And of course there are significant downsides to using nukes, which is that it risks a serious escalation with the west, one that could endanger Russia's future as a country and Putin's leadership and life, as well as losing support domestically, allowing Putin's political rivals to argue (perhaps correctly!) that he is a madman taking irrational and highly risky actions, and needs to be stopped.

There is exactly the case that he is a madman to worry about. I think this is unlikely; perhaps he has been corrupted by power, or is succumbing to dementia, but absent those Putin has showed himself to be a careful strategic actor in the past.

Overall I put the likelihood of an initial tactical nuclear strike by Putin to be south of 5%. Most of the probability mass is the possibility that Putin is mentally unstable, or that there's a series of mistakes / bad judgments that lead to a very ill-considered decision.

Strategic considerations

I think my fundamental disagreement with Max on an apocalyptic cycle of escalation is this: Putin and his advisors don't want a nuclear war (that's terrible for Russian and Putin personally), and are very familiar with game theory, since that's the framework that's governed strategic nuclear weapon use for many decades. Similarly, nobody in the west wants a nuclear war either. The logic of nuclear standoffs is such that there are strong incentives to go to the brink, but not to go over, so I think in general tense situations feel scarier than they actually are. And we should expect lots of threats and posturing, because signaling that you are willing to go over the edge gives you negotiating leverage.

Therefore the most likely analysis is that Putin is essentially bluffing -- his nuclear threats are not credible.

(To be clear, I think this dynamic is awful, because it puts us in a position where mistakes are much more likely to have terrible consequences; therefore escalatory situations do make nuclear war more likely. However 80% is a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper summary: The Epistemic Challenge to Longtermism (Christian Tarsney), published by Global Priorities Institute on October 11, 2022 on The Effective Altruism Forum.

Note: The Global Priorities Institute (GPI) has started to create summaries of some working papers by GPI researchers with the aim to make our research more accessible to people outside of academic philosophy (e.g. interested people in the effective altruism community). We welcome any feedback on the usefulness of these summaries.

Summary: The Epistemic Challenge to Longtermism

This is a summary of the GPI Working Paper "The epistemic challenge to longtermism" by Christian Tarsney. The summary was written by Elliott Thornley.

According to longtermism, what we should do mainly depends on how our actions might affect the long-term future. This claim faces a challenge: the course of the long-term future is difficult to predict, and the effects of our actions on the long-term future might be so unpredictable as to make longtermism false. In “The epistemic challenge to longtermism”, Christian Tarsney evaluates one version of this epistemic challenge and comes to a mixed conclusion. On some plausible worldviews, longtermism stands up to the epistemic challenge. On others, longtermism’s status depends on whether we should take certain high-stakes, long-shot gambles.

Tarsney begins by assuming expectational utilitarianism: roughly, the view that we should assign precise probabilities to all decision-relevant possibilities, value possible futures in line with their total welfare, and maximise expected value. This assumption sets aside ethical challenges to longtermism and focuses the discussion on the epistemic challenge.

Persistent-difference strategies

Tarsney outlines one broad class of strategies for improving the long-term future: persistent-difference strategies. These strategies aim to put the world into some valuable state S when it would otherwise have been in some less valuable state ¬S, in the hope that this difference will persist for a long time. Epistemic persistence skepticism is the view that identifying interventions likely to make a persistent difference is prohibitively difficult — so difficult that the actions with the greatest expected value do most of their good in the near-term. It is this version of the epistemic challenge that Tarsney focuses on in this paper.

To assess the truth of epistemic persistence skepticism, Tarsney compares the expected value of a neartermist benchmark intervention N to the expected value of a longtermist intervention L. In his example, N is spending $1 million on public health programmes in the developing world, leading to 10,000 extra quality-adjusted life years in expectation. L is spending $1 million on pandemic-prevention research, with the aim of preventing an existential catastrophe and thereby making a persistent difference.

Exogenous nullifying events

Persistent-difference strategies are threatened by what Tarsney calls exogenous nullifying events (ENEs), which come in two types. Negative ENEs are far-future events that put the world into the less valuable state ¬S. In the context of the longtermist intervention L, in which the valuable target state S is the existence of an intelligent civilization in the accessible universe, negative ENEs are existential catastrophes that might befall such a civilization. Examples include self-destructive wars, lethal pathogens, and vacuum decay. Positive ENEs, on the other hand, are far-future events that put the world into the more valuable state S. In the context of L, these are events that give rise to an intelligent civilization in the accessible universe where none existed previously. This might happen via evolution, or via the arrival of a civilization from outside the accessible universe.

What unites negative...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prettified AI Safety Game Cards, published by abramdemski on October 11, 2022 on LessWrong.

The AI Safety Card Game has simple cards you can print out on cardstock and cut out, but I wanted a nicer print-on-demand version with illustrated card backs. You can order it here. Major thanks to Mati Roy for helping make this happen, along with Jim Babcock, and Daniel Recinto.

Card-back illustrations are AI-generated. We did this around December-January, when the state of the art for AI-generated illustrations was much worse! I would be excited for someone to re-do it now. There's a lot of room for improvement; for example, illustrated card fronts rather than just card backs.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anonymous advice: If you want to reduce AI risk, should you take roles that advance AI capabilities?, published by Benjamin Hilton on October 11, 2022 on The Effective Altruism Forum.

This is a cross-post of an article from the 80,000 Hours website with only minor changes for this audience. See the original here.We’ve argued that preventing an AI-related catastrophe may be the world’s most pressing problem, and that while progress in AI over the next few decades could have enormous benefits, it could also pose severe, possibly existential risks. As a result, we think that working on some technical AI research — research related to AI safety — may be a particularly high-impact career path.

But there are many ways of approaching this path that involve researching or otherwise advancing AI capabilities — meaning making AI systems better at some specific skills — rather than only doing things that are purely in the domain of safety. In short, this is because:

Capabilities work and some forms of safety work are intertwined.

Many available ways of learning enough about AI to contribute to safety are via capabilities-enhancing roles.

So if you want to help prevent an AI-related catastrophe, should you be open to roles that also advance AI capabilities, or steer clear of them?

We think this is a hard question! Capabilities-enhancing roles could be beneficial or harmful. For any role, there are a range of considerations — and reasonable people disagree on whether, and in what cases, the risks outweigh the benefits.

So we asked the 22 people we thought would be most informed about this issue — and who we knew had a range of views — to write a summary of their takes on the question. We received 10 really interesting responses, and think that these are likely a reasonable representation of the range of views held by the broader set of people.

We hope that these responses will help inform people making difficult decisions about working in roles that might advance AI capabilities.

The following are written by people whose work we respect and who would like to remain anonymous. These quotes don’t represent the views of 80,000 Hours, and in some cases, individual pieces of advice may explicitly contradict our own. Nonetheless, we think it’s valuable to showcase the range of views on difficult topics where reasonable people might disagree.

We’ve included the responses from these 10 experts in full. We’ve only made minor edits for clarity and ease of reading.

Advice from 10 anonymous experts

Expert 1

Right now I wish AI labs would slow down on the margin, but I don’t think it’s obvious that capabilities work is net negative and reasonable people can disagree on this point. What I say below would probably change if I were highly confident it’s extremely bad to advance capabilities (especially if I believed that advancing capabilities by a month is much more bad than advancing alignment research by a month is good). With that said:

AI labs are a great place to pick up good skills, especially for people doing technical roles (ML engineer, ML researcher) at those labs. If you’re early career and can get into one and think it’d be more interesting and a better personal fit for you than other roles you’re considering, you should probably go for it — on a pretty wide distribution of views (including those that think capabilities enhancement is pretty net negative), the investment in your human capital probably creates more good than your contribution to capabilities (at a junior level) creates harm.

It’s worth specifically asking about working on the safety teams of capabilities organizations and not assuming you have to choose between a pure capabilities role and no role; this won’t always work but I’d expect a reasonable fraction of the time you can tilt your role toward more safety projec...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why So Many Cookie Banners?, published by jefftk on October 10, 2022 on LessWrong.

Sometimes you'll see people saying things like:

Using cookies to track state on a website, that is only used for that website, is fine. You don't need to ask for consent.—rrwo

Or:

You don't need a cookie banner to be allowed to create cookies. You only need them if you're using them for something like tracking.—y4mi

Something like, "as long as you design your site properly and don't abuse storage you don't need to ask your European visitors for permission." While I'm not working in this area anymore, am not a lawyer, and am not attempting to give you legal advice, if you read the regulation this interpretation is completely off.

Cookie banners are a response to the 2002 ePrivacy

Directive (full text, guidance). While the ePrivacy Directive may be superseded soon by the (pretty similar) ePrivacy

Regulation, it's still the current rule. It requires you to get consent from visitors before you store information on their computer

(cookies, localStorage, etc) unless this behavior is "strictly necessary in order to provide an information society service explicitly requested by the subscriber or user" [1]. This isn't "in order to" or even "necessary in order to", it's "strictly necessary in order to". Which is quite firm!

This excludes, for example, using a cookie for basic single-site analytics (4.3), where you want to figure out where users are getting stuck on your site or to populate a "users who viewed this product ended up buying this other product" box. Even though this information helps you improve your site for future visitors, including potentially this one, it isn't 'strictly necessary' for serving this user right now.

If the user puts an item in their shopping cart you can set a cookie, because that's how you honor their request, but it's still quite restrictive (2.3):

a merchant could set the cookie either to persist past the end of the browser session or for a couple of hours in the future to take into account the fact that the user may accidentally close his browser and could have a reasonable expectation to recover the contents of his shopping basket when he returns to the merchant's website in the following minutes.

Maintaining a shopping cart across days isn't "strictly necessary" and so requires explicit consent. Despite it being a useful thing users may be expecting: if I put things in my cart, don't check out, and come back the next day, I'm going to be frustrated if the site has forgotten my selections!

Similarly, say you have a "language" dropdown or a "dark mode" checkbox". Unless you have explicitly marked the UI control with text like "uses cookies" (3.6) you can't persist this setting for future visits.

The overall effect of this is that most sites will not be ePrivacy-compliant unless they either (a) get cookie consent from users or (b) hire a lawyer to review each of the things they do in the context of ePrivacy, and make careful changes to keep everything within the tight bounds of "strictly necessary". It's not surprising we see so many cookie banners!

[1] Technically it's also allowed if it's "for the sole purpose of carrying out the transmission in the electronic communications network and provided that the information is not stored for any period longer than is necessary for the transmission and for traffic management purposes, and that during the period of storage the confidentiality remains guaranteed". But the "strictly necessary" criterion covers almost everything in practice.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ask Charity Entrepreneurship Anything, published by Ula on October 11, 2022 on The Effective Altruism Forum.

We invite you to ask us anything about Charity Entrepreneurship’s work. As examples, you might want to ask questions related to:

Our Incubation Program for starting new high-impact charities:

The application process (stages, preparation, etc.)

Who is the best fit for the program (personality traits, relevant experience, etc.)

Details about the 2-month training and co-founder pairing

Seed funding and financial support (during and after the program)

Our new Foundations Program

Our top ideas for the 2023 Incubation Programs:

Animal-focused interventions

Policy-based ideas

Current research on biosecurity

The research process we use for selecting the top interventions

Our track record, knowledge base, expertise, how we do stuff, etc.

Entrepreneurship-focused, career-advice questions

Our whole team will be engaging with your questions to provide the best answers. Deadline for asking questions is: October 16, 2022. We will try to answer all the questions by October 20, 2022.

How to ask questions:

Please post each question as a separate comment.

Don’t be discouraged from asking niche questions. We’re happy to address them, there are a lot of new people on the forum who may benefit from the answers.

Small reward for your time:

We will send out a copy of our Peter Singer endorsed handbook, How to Launch a High-Impact Nonprofit, to the authors of the five most interesting questions (as picked by the CE team).

About CE:

We launch high-impact nonprofits by connecting potential founders with effective ideas, training, and funding. This means we spend thousands of research hours to identify highly-effective interventions in chosen cause areas. We then provide a two-month intensive training program (all costs covered) to teach participants how to run effective charities. We help them pair with a co-founder that will best complement their skills and personality. They finish the program with a proposal for funders that we deliver to our seed network. They grant up to $200,000 USD per project. You can learn more about the program at our Incubation Program website.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.