Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Think wider about the root causes of progress, published by jasoncrawford on December 21, 2022 on LessWrong.

Too much discussion of the Industrial Revolution is myopic, focused narrowly on a few highlights such as steam and coal. The IR was part of broader trends that are wider in scope and longer in time than its traditional definition encompasses. To understand anything, it is crucial to get the correct scope for the phenomenon in question.

Here are some ways in which we have to widen our focus in order to see the big picture.

Wider than coal and steam

Some explanations of the IR focus on steam engines, and especially on the coal that fueled them. Economist Robert Allen, for instance, has one of the best-researched and most convincing arguments for cheap coal as a requirement for the development of steam power.

A weak version of this claim, such as “coal was a crucial factor in the IR,” is certainly true. But sometimes a much stronger claim is made, to the effect that the IR couldn’t have happened without abundant, accessible coal, and that this is the main factor explaining why it happened in Europe and especially in Britain. Or more broadly, that all of material progress is driven by fossil fuels (with the implication that once we are, sooner or later, forced to transition away from fossil fuels—whether by geology, economics, or politics—growth will inevitably slow).

But much of the early IR wasn’t dependent on steam power:

Textile machinery, such as Arkwright’s spinning machines, were originally powered by water.

Henry Maudslay’s earliest machine tools were made to manufacture locks; another key application of machine tools was guns with interchangeable parts. This was not motivated by steam power.

Improvements in factory organization, such as the arrangement of Wedgwood’s pottery manufacturing operation, were management techniques, independent of power sources.

The reaper was pulled by horses—even after the development of steam power, because steam tractors were too heavy for use in fields. Even stationary agricultural machines such as for threshing or winnowing were often muscle-powered.

You could argue that all of these inventions would have reached a plateau and would not have had as much economic impact without eventually being hooked up to steam or gas engines. But the fact remains that they were initially powered by water or muscle, and they were economically useful in those first incarnations, often achieving 10x or more gains in productivity. They didn’t need coal or gas for that, nor were they invented in anticipation that such power would soon be available.

So either it was an amazing coincidence that all of this mechanical invention was going on at the same time—or there was some wider, underlying trend.

Wider than industry

Further, the IR itself only represents a subset of the broader technical innovations that were going on in this period. Here are a few key things that aren’t considered “industrial” and so are often left out of the story of the IR:

Improvements to agriculture other than mechanization: for instance, new crop rotations

Improvements to maritime navigation, such as the marine chronometer that helped solve the longitude problem

Immunization techniques against smallpox: inoculation and later vaccination

To me, the fact of all these inventions happening in roughly the same time period indicates a general acceleration of progress during this time, reflecting some deep cause, not a simple playing out of the consequences of one specific resource or invention.

Wider than invention

In many areas, incremental improvements were being made even before the major inventions that make the history books:

Roads and canals were improved in the 17th and 18th centuries, speeding up transportation even before railroads

Experiments by engineers like John ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: US Policy Master's Degrees: Why and When? (Part 1), published by US Policy Careers on December 21, 2022 on The Effective Altruism Forum.

About this post

Working in policy is among the most effective ways to have a positive impact in areas like AI, biosecurity, animal welfare, or global health. Getting a policy master’s degree (e.g. in security studies or public policy) can help you pivot into or accelerate your policy career.

This two-part overview explains why, when, where, and how to get a policy master’s degree. Part 1 (this post) focuses on the “why” and the “when” and alternatives to policy master’s. Part 2 focuses on general criteria for choosing where to apply, specific degrees we recommend, how to apply, and how to secure funding. We also have a US policy master's database if you want to compare program options.

Part 1 Part 2

US Policy Master's Degrees: Why and When?

What are policy master’s degrees?

Why do a master’s if you want to work in policy?

Why not do a master’s for policy work?

When should you go to grad school—right after college or after working for a few years?

What are the alternatives to policy master’s?

US Policy Master's Degrees: Which Schools, How to Apply, and Funding (forthcoming)

Deciding which policy master’s programs to apply to

Application timelines

How to apply: Getting into policy master’s programs

Funding graduate school

If you are interested in applying for a policy master’s program—including if you are still unsure or plan to apply in future years—we encourage you to fill in this form so that we can potentially support your application and connect you with others who have gone through the program.

These posts are based on our personal experience working on policy in DC for several years, background reading, and conversations with more than two dozen policy professionals.

Please note that this two-part series focuses on:

Master’s degrees for those aiming to work in policy. Thus, it will be much less relevant for people pursuing technical or other non-policy careers.

Master’s degrees rather than other graduate degrees, like law degrees or PhDs. A later section briefly compares policy master’s programs with these alternatives.

Master’s programs for policy careers in the US, especially in/with federal government. Some of the advice may apply to policy at the US state-level or other countries, but much of it will not since DC policy institutions and paths may differ substantially from those not in DC or the US.

Summary

What are policy master’s degrees? They are typically two-year programs in subjects like public policy/administration (MPP/MPA), international relations/security studies, or more technical programs like public health (MPH). Policy master’s fall on a continuum from highly academic to highly practitioner-oriented, with the latter typically being better preparation for a policy career. [read more]

Do I need a master’s to work in policy? Doing a master’s (or other graduate degree) is generally valuable for policy work and often necessary, depending on the institution and role (especially in executive agencies and think tanks). As a policy professional, you’ll most likely want to do a master’s eventually, with limited exceptions such as some career tracks in Congress. [read more]

What’s the value of a master’s for policy work? A master’s builds your career capital for specific paths like policy. The credential is useful and often necessary to get a policy job. Master’s degrees also provide value through learning, skill-building, networking, exploration, and more. The relative importance of these factors depends on your background and goals, and may influence what degree to get (e.g. subject, location, type of graduate degree). [read more]

If I want to do a master’s, when should I do it? We recommend most people to work for 1-3 ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding the diffusion of large language models: summary, published by Ben Cottier on December 21, 2022 on The Effective Altruism Forum.

5-minute summary

Time from GPT-3’s publication (May 2020) to.TimeDateModelOrganizations involvedA better model is produced7 monthsDec 2020GopherDeepmindMay 2020OPT-175BMeta AI Research

A better or equally good model is open-sourced

AND

A successful, explicit attempt at replicating GPT-3 is completed

23 months

(equally good model)

Table 1: Key facts about the diffusion of GPT-3-like models

I present my findings from case studies on the diffusion of nine language models that are similar to OpenAI’s GPT-3 model, including GPT-3 itself. By “diffusion”, I mean the spread of artifacts among different actors, where artifacts include trained models, code, datasets, and algorithmic insights. Diffusion can occur through different mechanisms, including open publication and replication. Seven of the models in my case studies are “GPT-3-like” according to my definition, which basically means they are similar to GPT-3 in design and purpose, and have similar or better capabilities. Two models have clearly worse capabilities but were of interest for other reasons. (more)

I think the most important effects of diffusion are effects on (1) AI timelines—the leading AI developer can get to TAI sooner by using knowledge shared by other developers, (2) who leads AI development, (3) by what margin they lead, and (4) how many actors will plausibly be contenders to develop transformative AI (TAI). The latter three effects in turn affect AI timelines and the competitiveness of AI development. Understanding cases of diffusion today may improve our ability to predict and manage the effects of diffusion in the lead up to TAI being developed. (more)

See Table 1 for key facts about the timing of GPT-3-like model diffusion. Additionally, I’m 90% confident that no model exists which is (a) uncontroversially better than GPT-3 and (b) has its model weights immediately available for download by anyone on the internet (as at November 15, 2022). However, GLM-130B (Tsinghua University, 2022)—publicized in August 2022 and developed by Tsinghua University and the Chinese AI startup Zhipu.AI—comes very close to meeting these criteria: it is probably better than GPT-3, but still requires approval to download the weights. (more)

I’m 85% confident that in the two years since the publication of GPT-3 (in May 2020), publicly known GPT-3-like models have only been developed by (a) companies whose focus areas include machine learning R&D and have more than $10M in financial capital, or (b) a collaboration between one of these companies and either academia, or a state entity, or both. That is, I’m 85% confident that there has been no publicly known GPT-3-like model developed solely by actors in academia, very small companies, independent groups, or state AI labs. (more)

In contrast, I think that hundreds to thousands of people have enough resources and talent to use a GPT-3-like model through their own independent setup (rather than just an API provided by another actor). This is due to wider access to the model weights of GPT-3-like models such as OPT-175B and BLOOM since May 2022. (more)

I estimate that the cost of doing the “largest viable deployment” with a GPT-3-like model would be 20% of the cost of developing the model (90% CI: 10 to 68%), in terms of the dollar cost of compute alone. This means that deployment is most likely much less prohibitive than development. For people aiming to limit/shape diffusion, this analysis lends support to targeting interventions at the development stage rather than the deployment stage. (more)

Access to compute appears to have been the main factor hindering the development of GPT-3-like models. The next biggest hindering factor appears t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More notes from raising a late-talking kid, published by Steven Byrnes on December 20, 2022 on LessWrong.

My previous post on the topic was here. These are just some quick notes and observations, in no particular order.

Professionals pathologizing normal kid behaviors

Here are a couple descriptions of a thing that I have definitely experienced.

Source 1: Late-Talking Children by Camarata [great book!] page 8 (emphasis added)

When children are late in beginning to speak, many people begin to scrutinize their every action, seeing “symptoms” and “red flags” of something wrong in things that might otherwise be seen as typical of the odd things toddlers often do, which are regarded as cute when the child is not a late talker.

As an example, one little preschooler that I was treating for his speech delay liked to pick fuzz off a carpet and play with it. A psychologist said that this was one of the signs of autism. I then showed him a videotape of 13 children in that toddler’s preschool class, 6 of whom were picking fuzz off the carpet and playing with it. But none of the other 5 children were regarded as having a symptom of autism, because they were talking.

Source 2: A random twitter thread (lightly copyedited):

When a child is struggling at school, often they are seen by professionals (like me!) who assess, write reports and make recommendations. As this process goes on, something strange happens. Here’s what it is:

The things that the child does are scrutinised, compared to others and framed as in need of intervention. Do they get annoyed when the internet stops working? Maybe they need anger management or to work on their emotion regulation?

The child is compared to some fantasy person, who never shouts when they lose their progress in a game and definitely doesn't refuse to put their shoes on or leave the house when their parent asks.

Then everything is framed in the language of reports, which exists in an odd world of its own. “Elodie accesses her local community with support” when another child just goes to the shops and playground with her mum.

Things which make life easier for the child's school are framed as being for the child's own good. Following adult instructions is one of my (not) favourites; a goal in many reports. Useless in adult life, and dangerous when applied indiscriminately. Not all adults are right.

And let's not forget the recommendations to make children less different. More eye contact, confine activity to movement breaks, squashing themselves more efficiently to fit in with everyone else.

When I read reports, I wonder how I’d measure up. I get frustrated when I lose my work or the internet goes down. I get bored with small talk - maybe I need social skills training? I’m really bad at following instructions when I can't see the point (what's wrong with that?)

I worked with one boy who just couldn't stop moving. He would tap his pencil on the desk and it irritated the other children. One of the goals was for him to stop and “listen quietly in class”. He couldn't do it. He got in trouble instead.

He moved school and the next school took a different approach. They wrapped a piece of foam around his pencil and he tapped away. No one minded anymore and the goal wasn't necessary.

So let’s ask ourselves, who are these goals really for? Are we actually asking for compliance, rather than thinking about the long term interests of this young person? Are the goals those of the child, or those of the adults around them?

And are we pathologising this child, by framing everything they do through a lens of deficit and ignoring context? Could we use plain English - and might our reports start to make a different sense if we did?

“Kai gets angry when his game crashes” sounds quite different (and more logical) than “Kai becomes dysregulated when he is allowed scre...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Flooding is not a promising cause area - shallow investigation, published by Oscar Delaney on December 21, 2022 on The Effective Altruism Forum.

Summary

Floods are important - they on average kill 5,000 people and cause $35B of damage each year.

Compared to other problems in global health, however, floods are relatively minor.

Flood prevention - and especially response - are less neglected than other global health problem areas.

This may be because floods harm rich people too, who are politically empowered and cause governments to invest in mitigation.

I estimate that to beat GiveWell’s top charities, we would need to prevent all flood damage using $400M per year.

This seems very difficult.

I investigate several interventions in more detail, which show some promise but are unlikely to be competitive with GiveWell’s top charities.

This report was produced with roughly 50 hours of research and writing, as part of the Cause Innovation Bootcamp fellowship program.

Introduction

Floods are defined as land that is normally dry being submerged underwater. There are several quite distinct types of floods, the most important being those caused by:[1]

Rain: intense rainfall in a localised area can exceed the capacity of the ground to absorb water, and lead to a buildup of water in low-lying areas.

Rivers: distributed rainfall anywhere in a river’s catchment area, rapid snow-melt, or the collapse or quick release of dam water, can all cause a river to overflow its banks.

Storm surges: intense winds can blow ocean water onto land, and the lower air pressure of large storms causes the local sea level to rise by up to about 1m. Particularly if combined with a high tide this can inundate coastal areas.

Importance

Floods are a big deal. Of natural disasters occurring between 1998 and 2017, a UN report estimated that 43% of disasters were floods (next most common: storm), floods accounted for 45% of disaster-impacted people, at 2 billion (next highest: drought), 11% of deaths (earthquakes caused 56%), and 23% of the economic damage (storms were 46%).[2]

While the toll of floods is generally well recorded in terms of deaths and financial cost, at least for larger disasters, data is scarcer and less systematic for other harms.

Floods can have many negative impacts, ranging from direct to indirect and immediate to long-term:[3]

Deaths: the majority of flood deaths are drownings, with other causes including injuries from falling objects or debris in fast-moving water, and electrocution from fallen powerlines.[4] Floods caused an average of 5,000 deaths per year in 2001-2020, with the bulk of the deaths caused by many smaller events: the deadliest 104 floods accounted for half the deaths in this period.[5]

Injuries: as well as injuries during the flood, reconstruction and clean-up causes various injuries including sprains and strains, falling off roofs or ladders, and lacerations from sharp debris.[6] Floods caused an average of 15,000 injuries per year in 2001-2020, though injuries are probably significantly underreported.[7]

Water-borne diseases: particularly if sewage systems have been compromised, contact with floodwater can cause a variety of diseases notably diarrhoea, cholera and leptospirosis.

Mosquito-borne diseases: the abundance of stagnant water as floodwaters recede may cause a surge in mosquito populations and an associated spike in malaria, dengue and other vector-borne diseases.[8]

Malnutrition: lost crops and reduced purchasing power can lead to impaired nutrition in the years following a major flood event in a poor region. One study found wasting to be more than twice as common in children who were flooded twice in the last three years than in control households.[9]

Mental health: many studies have found increased incidences of PTSD, anxiety and depression in individuals ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A can of worms: the non-significant effect of deworming on happiness in the KLPS, published by Samuel Dupret on December 21, 2022 on The Effective Altruism Forum.

Summary

Mass deworming, where many people are provided drugs to treat parasitic worms, has long been considered a highly cost-effective intervention to improve lives in low-income countries. GiveWell directed over $163 million to deworming charities since 2010. Nevertheless, there are long-running debates about its impact and cost-effectiveness. In this report, we summarise the debate about the efficacy of deworming, present the first analysis of deworming in terms of subjective wellbeing (SWB), and compare the cost-effectiveness of deworming to StrongMinds (our current top recommended charity).

Analysing SWB data from the Kenyan Life Panel Survey (KLPS; Hamory et al., 2021), we find that deworming has a small, statistically non-significant effect on long-term happiness that seems (surprisingly) to become negative over time (see Figure 1). We conclude that the effect of deworming in the KLPS is either non-existent or too small to estimate with certainty. Typically, an academic analysis could stop here and not recommend deworming. However, the non-significant effects of deworming could be cost-effective in practice because it is extremely cheap to deliver. Because the effect of deworming is small and becomes negative over time, our best guess finds that the overall cost-effectiveness of deworming is negative. Even under more generous assumptions (but still plausible according to this data), deworming is less cost-effective than StrongMinds. Therefore, we do not recommend any deworming charities at this time.

To overturn this conclusion, proponents of deworming would either need to (1) appeal to different SWB data (we’re not aware of any) or (2) appeal to a non-SWB method of comparison which concludes that deworming is more cost-effective than StrongMinds.

Figure 1: Differences in happiness between treatment and control groups over time

1. Background and literature

In this section, we present the motivation for this analysis, the work by GiveWell that preceded this, and the broader literature on deworming. We then present the details and context of the dataset we use for this analysis – the Kenyan Life Panel Survey (KLPS).

1.1 Our motivation for this analysis

The Happier Lives Institute evaluates charities and interventions in terms of subjective wellbeing (SWB) - how people think and feel about their lives. We believe that wellbeing is what ultimately matters and we take self-reports of SWB to be the best indicator of how much good an intervention does. If deworming improves people’s lives, those treated for deworming should report greater SWB than those who aren’t. SWB should capture and integrate the overall benefits from all of the instrumental goods provided by an intervention. For example, if deworming makes people richer, and this makes them happier, they will report higher SWB (the same is true for improvements to health or education). Although we are not the first to use SWB as an outcome for decision-making (e.g., UK Treasury, Frijters et al., 2020, Birkjaer et al., 2020, Layard & Oparina, 2021), we are the first to use it to compare the impact of charities.

See McGuire et al. (2022b) for more detail about why we prefer the SWB approach to evaluate charities.

To determine whether the SWB approach changes which interventions we find the most cost-effective, we have been re-evaluating the charity recommendations of GiveWell (a prominent charity evaluator that recommends charities based on their mortality and economic impacts). For a review of our recent research, see this post. We present our findings in wellbeing-adjusted years (WELLBYs), where 1 WELLBY is the equivalent of a 1-point change on a 0-10...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Local Memes Against Geometric Rationality, published by Scott Garrabrant on December 21, 2022 on LessWrong.

Earlier in the sequence, I presented the claim that humans are evolved to be naturally inclined towards geometric rationality over arithmetic rationality, and that around here, the local memes have moved us too far off this path. In this post, I will elaborate on that claim. I will argue for this claim very abstractly, but I think the case is far from sound, and some of the arguments will be rather weak.

Naturally Occurring Geometric Rationality

When arguing for Kelly betting, I emphasized respecting your epistemic subagents, and not descending into internal politics. There is a completely different, more standard, argument for Kelly betting, which is that given enough time Kelly bettors will almost certainly end up with all the money. It is thus natural to expect natural selection might select for geometric rationality. This hypothesis is backed up by the way people seem naturally risk averse. Further, people have a tendency towards probability matching, as discussed here, which is a bug that seems more natural if people are naturally inclined towards geometric rationality.

I think the hypothesis that evolution favors geometric rationality is probably best considered empirically, and I haven't gone deep into the data. If anyone else wants to dig into this, I would be curious to hear what they find.

This post, however, will focus on local memes that pull us away from geometric rationality. I will discuss four reasons that I think we are pulled away from geometric rationality: the end of the world, astronomical stakes, utility theory, and low probability of success in our main shared endeavor.

The Endgame

Kelly betting is usually justified by looking at long run outcomes. If you make enough bets, an expected wealth maximizer will almost certainly end up with no money. My fairness orientation to Kelly betting is a nonstandard one.

People around here tend to think the world is ending. If there are only a few rounds of betting left, it is hard to justify optimizing for long run outcomes.

Similarly, Thompson sampling is about exploration. If you don't have time to learn from new things, you should be in exploit mode, and just do the best thing.

Further, cooperative and fair equilibria are a lot easier to maintain in a repeated game. People actually often defect at the end of a finitely repeated prisoners' dilemma.

You don't actually need the end of everything to make end-of-the-world style arguments against geometric rationality. You can also reach the same conclusions with a causal singularity. If there is an upcoming place/time that is the primary thing that determines how good the future is, we are approaching a causal end of the world, where while the world might not literally be ending, we should direct all our optimization towards this one place/time. If it is coming up soon, maybe we shouldn't waste our time with exploration.

Counterarguments

I think it is important to note that while this might take out one of the justifications for Kelly betting, I think there is good reason to Kelly bet even if there is a single round, which is part of why I emphasized nonstandard reasons in my Kelly betting post.

Also, one should be careful when trying to change the strategy they have practiced both evolutionarily and throughout their life in the last round. You should expect you are better at geometric rationality than you are at arithmetic rationality, if it is the thing you have been naturally doing so far.

Large Caring

Let's say that you want to geometrically maximize the number of paperclips in the quantum multiverse. Equivalently, you are trying to maximize the logarithm of the measure of paperclips. If we approximate to say that you are in one of a very large finit...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Decisions: Ontologically Shifting to Determinism, published by Chris Leong on December 21, 2022 on The AI Alignment Forum.

In a previous post, I posited that a significant portion of the confusion surrounding counterfactuals can be traced back to an ontological shift. In this post, I will seek to elaborate on this argument and explain how we should reinterpret decision theory problems in light of this insight.

Our Naive Ontology Assumes Libertarian Free Will

Consider our naive ontology, by which I mean how humans tend to think without substantial philosophical reflection. In our naive ontology, we assume both that:

That we possess libertarian free will, ie. that our choices are not predetermined before we make them.

That when we're making a decision, both the past and present state of the universe are fixed independent of that decision.

Unfortunately, scientific evidence seems to indicate:

3. At the point of a decision, each brain-state only corresponds to one possible choice

It is impossible for all of these statements to be true simultaneously. If they were, there would be multiple possible choices we could make, each of which would necessitate a different brain-state. However, this would also require the current state of the universe, and therefore our brain state, to be fixed independent of these choices.

From my perspective, it is clear that the mistake was in assuming the existence of libertarian free will (assumption 1). Given this, I'll take it for granted that we have to revise our naive conception of decisions so as to be rooted in a deterministic ontology.

Adapting to Determinism by Augmenting with Counterfactuals

Determinism dictates that whatever decision is ultimately made was the only decision that could have ever been made. This immediately poses a challenge: how do we avoid any decision from ending up as a trivial one?

The obvious answer is that we must find a way to augment the factual with a counterfactual for each option we desire beyond the first. But how should we construct this and what kind of object should it even be?

In Favour of Consistent Counterfactuals

As I previously argued in my post Counterfactuals are an Answer, Not a Question, the definition of a counterfactual will depend on the types of questions we seek to use them to answer. For example, while Causal Decision Theory results in counterfactual scenarios that are inconsistent, they can still be useful in practice, because the inconsistencies are often not relevant to the decisions we are attempting to make.

To illustrate, we typically disregard the fact that making a decision would result in a different brain state, which would then lead to numerous changes in the past. However, when considering Newcomb-like problems within a deterministic framework, these are exactly the types of facts about the world that we cannot ignore. While it may not be immediately obvious that we should handle Newcomb-like problems by backpropagating the full effects of our decision backwards in time, it is clear that we must backpropagate at least some of these consequences. It is possible that someone may devise a concept of counterfactuals that allows for the selective backpropagation of certain consequences. However, at present, I can't see any obvious ways of accomplishing this, leading me to favour full propagation at least until a better theory arises.

Furthermore, once we have begun backpropagating even some consequences, there is a compelling argument to be made for fully embracing this approach as we have already relinquished the primary benefit of not backpropagating consequences. This benefit is that since the past was fixed between counterfactuals, it is easier to justify that the counterfactuals were a fair comparison of the different choices.

However, once we've accepted the need for some de...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Comprehensive Mechanistic Interpretability Explainer & Glossary, published by Neel Nanda on December 21, 2022 on The AI Alignment Forum.

This is a linkpost for a very long doc defining, explaining, and giving intuitions and conceptual frameworks for all the concepts I think you should know about when engaging with mechanistic interpretability. If you find the UI annoying, there's an HTML version here

Why does this doc exist?

The goal of this doc is to be a comprehensive glossary and explainer for Mechanistic Interpretability (focusing on transformer language models), the field of studying how to reverse engineer neural networks.

There's a lot of complex terms and jargon in the field! And these are often scattered across various papers, which tend to be pretty well-written but not designed to be an introduction to the field as a whole. The goal of this doc is to resolve some research debt and strives to be a canonical source for explaining concepts in the field

I try to go beyond just being a reference that gives definitions, and to actually dig into how to think about a concept. Why does it matter? Why should you care about it? What are the subtle implications and traps to bear in mind? What is the underlying intuition, and how it fits into the rest of the field?

I also go outside pure mechanistic interpretability, and try to define what I see as the key terms in deep learning and in transformers, and how I think about them. If you want to reverse engineer a system, it's extremely useful to have a deep model of what's going on inside of it. What are the key components and moving parts, how do they fit together, and how could the model use them to express different algorithms?

How to read this doc?

The first intended way is to use this a reference. When reading papers, or otherwise exploring and learning about the field, coming here and looking up any terms and trying to understand them.

The second intended way is to treat this as a map to the field. My hope is that if you're new to the field, you can just read through this doc from the top, get introduced to the key ideas, and be able to dig into further sources when confused. And by the end of this, have a pretty good understanding of the key ideas, concepts and results!

It's obviously not practical to fully explain all concepts from scratch! Where possible, I link to sources that give a deeper explanation of an idea, or to learn more.

More generally, if something’s not in this glossary, you can often find something good by googling it or searching on alignmentforum.org. If you can’t, let me know!

I frequently go on long tangents giving my favourite intuitions and context behind a concept - it is not at all necessary to understand these (though hopefully useful!), and I recommend moving on if you get confused and skimming these if you feel bored.

Table of Contents

Introduction

Why does this doc exist?

How to read this doc

Mechanistic Interpretability

General

Representations of Features & Superposition

Superposition

A Toy Model of Superposition

The Broader Interpretability Field

Linear Algebra

Circuits As Computational Subgraphs

Machine Learning

Basic Concepts

Training Concepts

Training Dynamics

Misc

Transformers

Transformer Basics

Transformer Components

Attention Heads

Misc Transformer Words

Training

Transformer Circuits

Language Modelling

A Mathematical Framework for Transformer Circuits

Induction Circuits

SoLU

The Indirect Object Identification Circuit

Techniques

Mechanistic Interpretability Techniques

Non-MI Techniques

Tooling

Notable Models

Open Source GPT-Style Models

My Interpretability-Friendly Models

Other Open Source Models

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sazen, published by Duncan Sabien on December 21, 2022 on LessWrong.

Purpose of post: describe and (hopefully) popularize a concept I've found highly useful.

Last year, my partner Logan Strohl wrote a sequence to introduce the "naturalism" concept they've been developing and teaching for the past decade or so.

That sequence was structured around a single, short sentence. The first essay introduced the sentence, and the remaining essays were primarily about explaining what each of the important concepts in that short sentence actually meant.

So, for the sentence "knowing the territory takes direct and patient observation," there was a full essay on what was intended (and, more crucially, what was not intended) by the word "knowing," and another on "the territory," and another on "observation," and so on.

This format was largely inspired by a conversation in which I asked Logan to describe naturalism briefly, and they said "I totally can, but you'll get the wrong idea."

Together, we realized that there is a curious one-way sort of property to many sentences, in which they work as pointers or summaries after the fact, but fail to generate the-thing-they're-summarizing if used as standalone seeds.

(One could argue that every sentence has some of this property, but some sentences have a lot of it.)

I'd like to be able to point directly at this property, and as a result of historical accident that I'll explain in a footnote, the handle I've ended up with in my own head is sazen.

Example 1: "Duncan Sabien is a teacher and a writer."

This is a true sentence. People who know me very, very well, upon hearing this sentence, will nod. It's a good fit, retrospectively, for the data.

However, if you are attempting to give someone a sense of me up-front, saying "Duncan Sabien is a teacher and a writer" is an unusually bad start. The thing that most people will think of when they hear "teacher" or "writer" is specifically unlike me—I'm a very weird sort of teacher and a very weird sort of writer, and so anchoring people on the representative stereotypes is almost actively misleading.

The sentence "Duncan Sabien is a teacher and a writer" is a sazen.

Example II: Peanut butter and jelly sandwiches

There's a classic challenge in which a teacher asks a bunch of students to write down unambiguous and complete instructions for how to make a peanut butter and jelly sandwich. The gimmick is that, to grade each paper, the teacher will follow only the actions written on the page, which usually results in something very unlike a normal sandwich.

This activity is ... somewhat arbitrary and infuriating, because the teacher usually has to make a bunch of fluid judgment calls about where they draw the line, and there's usually not a clear and consistent standard for what level of detail is required, so the lesson often ends up being less about the complexity of background information and more about the fickle dickishness of teachers.

But purely as a sort of plug-and-play teachable moment, it's an interesting way to take a close and practical look at all of the little bits of context we take for granted, via the teacher pretending not to know them.

A sentence like "get a couple of slices of bread, and put peanut butter on one and jelly on the other and then stick 'em together" is a sazen. After the fact, you can look back and say "sure, those bones match the shape of what just happened."

But up front, they're woefully insufficient. They also match (for example) putting the entire jar of peanut butter atop one slice, and the entire jar of jelly atop the other, and then sliding the two smushed rectangles of bread into each other, flat on the counter. Or failing to use bread, peanut butter, or jelly at all, because the instructions didn't say to get those items and have them on hand, or didn't specify...