The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: About my job: "Content Specialist", published by Lizka on September 9, 2023 on The Effective Altruism Forum.

I've been the Content Specialist on the Online Team at CEA since early 2022. People often don't know what "Content Specialist" means and it seems useful to have more "about my job" posts, so here's mine.

This got long, so I encourage you to skip to whatever you want to see. I include my background, what I actually work on, and some quick reflections (what I value about the role, what's been hard, and the skills that I develop).

My background & how I got here

This ended up longer than expected, so tl;dr: math and comparative literature double major, helping to organize 3 summers of Canada/USA Mathcamp, getting into EA, research fellowship at Rethink Priorities, "Events Generalist" at CEA, and then being invited to apply to Aaron's job and ending up here. Non-tl;dr:

I studied math and comparative literature in college (and graduated in 2021). I really wasn't sure what I wanted to do after college, and was somewhat defaulting to math academia or something education-related. I was spending my summers doing ~math research, working at Canada/USA Mathcamp (in a mostly non-academic role where we ran events, helped make camp run, and more - I think this made me better at making-things-happen and made me a lot more confident), and teaching math to excited kids.

Meanwhile, a friend introduced me to EA, which clicked pretty quickly. The issue was that I didn't see myself contributing; I didn't want to do CS, didn't particularly want to study economics, didn't think I would last long earning-to-give, and didn't think I had other real skills. But I read a bunch, listened to the 80K Podcast, went to the 2020 Student Summit, and had a call with 80,000 Hours, where they told me to apply to Rethink Priorities (and defer graduate school). I followed that advice.

I wrote about my experience at RP here. This experience was great. It made me feel that fairly generalist research was actually a viable/useful path. It also got me over my Forum-intimidation and got me to post my first Forum post (and some others).

While I was at Rethink Priorities, I had applied to an "Events Generalist" role at CEA - largely because it seemed relatively interesting, CEA had "effective altruism" in its name (so I thought it was going to be at least somewhat useful), I wanted to test out something like ops, and I really wanted to have a job. I got and took the job, which I was expecting to be pretty low-responsibility since I was so junior.

I got more responsibility than I expected, which was both scary and very motivating. On my second week, I ended up being one of two(?) people from the Events Team at the Coordination Forum (I was there to help make it go smoothly and to take notes in some sessions), during which we also made the decision to double the size of EA Global London 2021 (happening in around a month). It felt like a lot of the work that I was doing just wouldn't get done if I wasn't there (or maybe something else important wouldn't get done). After EA Global: London 2021, things calmed down a bit. I was still writing sometimes, and I sometimes contributed a bit to side-projects, some of which grew out a bit.

Then Aaron reached out to say that he was leaving, and he was wondering if it would be ok to send me the job application. Skimming Aaron's messages from the time, it looks like Aaron reached out because of a combination of my Forum writing, my performance on the Events Team, and encouragement from Linch when Aaron checked with him. I checked with my then-manager Amy to see if she would be ok with me applying. I was very stressed about this, but Amy was very supportive. I went into the application pretty confident that I wouldn't get far (and I cried after the interview). That's the role...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Have Attention Spans Been Declining?, published by niplav on September 9, 2023 on LessWrong.

I investigate whether the attention span of individual humans has been falling over the last two decades (prompted by curiosity about whether the introduction of the internet may be harmful to cognitive performance). I find little direct work on the topic, despite its wide appeal. Reviewing related research indicates that individual attention spans might indeed have been declining65%.

Have Attention Spans Been Declining?

In what might be just the age-old regular

ephebiphobia, claims have been raised that individual attention spans have been declining - not just among adolescents, but among the general population. If so, this would be quite worrying: Much of the economy in industrialized societies is comprised of knowledge work, and knowledge work depends on attention to the task at hand: switching between tasks too often might prevent progress on complicated and difficult problems.

I became interested in the topic after

seeing several claims that e.g. Generation Z allegedly has lower attention spans and that clickbait has disastrous impacts on civilizational sanity, observing myself and how I struggled to get any work done when connected to the internet, and hearing reports from others online and in person having the same problem. I was finally convinced to actually investigate™ the topic after making a comment on

LessWrong asking the question and receiving a surprisingly large amount of upvotes.

The exact question being asked is:

"Have the attention spans of individuals on neutral tasks (that is, tasks that are not specifically intended to be stimulating) declined from 2000 to the present?"

(One might also formulate it as "Is there an equivalent of the "Reversed

Flynn Effect" for attention span?") I am not particularly wedded to the specific timeframe, though the worries mentioned above assert that this has become most stark during the last decade or so, attributing the change to widespread social media/smartphone/internet usage. Data from before

2000 or just the aughts would be less interesting. The near-global COVID-19 lockdows could provide an especially enlightening natural experiment: Did social media usage increase (my guess: yes90%), and if so, did attention spans decrease at the same time (or with a lag) (my guess: also yes75%), but I don't think anyone has the data on that and wants to share it.

Ideally want to have experiments from ~2000 up to 2019: close enough to the present to see whether there is a downward trend (a bit more than a decade after the introduction of the iPhone in

2007), but before the COVID-19 pandemic which might be a huge confounder, or just have accelerated existing trends (which we can probably check in another 2 years).

I am mostly interested in the attention span of individual humans and not groups: Lorenz-Spreen et al. 2019 investigate the development of a construct they call "collective attention" (and indeed find a decline), but that seems less economically relevant than individual attention span. I am also far less interested in self-perception of attention span, give me data from a proper power- or speed-test, cowards!

So the question I am asking is not any of the following:

"Does more social media/internet usage cause decreased attention spans?"

"Does more social media/internet usage correlate with decreased attention spans?"

"Does more social media/internet usage correlate with people reporting having shorter attention spans?"

"Did collective attention spans decrease?"

"Are people on average spending less time on webpages than they used to?"

How Is Attention Span Defined?

Attention is generally divided into three distinct categories: sustained attention, which is the consistent focus on a specific task or piece of information over ti...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is the optimal frontier for due diligence?, published by RobertM on September 9, 2023 on LessWrong.

The title isn't quite right. My current take is that:

There is a ~10k word post sharing some damaging information about someone else.

3-4k words are describing accusations that are disputed.

However, a few have already had substantial evidence provided against them.

The post author doesn't think the accuracy of those accusations is particularly cruxy for their major takeaways.

Those accusations generally sound pretty bad.

The post doesn't explain why the (disputed) accusations are salient, if they aren't cruxy for the post author's major takeaways.

The post author may not find those accusations cruxy, but I would be surprised if nobody else found them cruxy for deciding between "there's some bad stuff in here, but it seems possible that most of the harm was caused by some combination of large cultural differences and poor communication", and "nope nope nope".

This is to say, our actions have effects beyond the EA ecosystem. I think it's wrong to decide that, because other people may update incorrectly based on evidence you've provided, the harm that comes from those updates doesn't matter much.

The thing I'm noticing is some kind of missing mood. I think that there's been a failure to inhabit the least convenient possible world, and the general distribution over possible outcomes, and correspondingly attempt to move to the pareto-frontier of outcomes assuming that distribution. This comment in particular seems to be operating in the frame of "You weren't behaving the way I'd update positively on if the accusations were true". In the world where the disputed accusations are in fact false, for the specific reasons they provided, I think it would be pretty strange to take the suggested course of action.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Who Is a Good Fit for a Career in Nonprofit Entrepreneurship? (CCW2023), published by CE on September 8, 2023 on The Effective Altruism Forum.

TLDR: This post is part of a sequence we've prepared for the Career Conversations Week on the EA Forum. This article provides a concise overview of the traits that Charity Entrepreneurship deems most valuable for a career in nonprofit entrepreneurship. We intend to provide you with a deeper understanding of our criteria when evaluating applications for our Incubation Program. Additionally, we hope this post assists you in self-assessing whether this career path aligns with your aspirations and abilities.

It's been five years since we started our Incubation Program. We've since incubated 27 charities, some of which are now GiveWell incubated or ACE recommended, with many others on a path to becoming leaders in their fields. This would be impossible without dedicated, smart, resilient, and ambitiously altruistic individuals like you. Of course, not everyone is suited to be a successful nonprofit entrepreneur, and so we've spent the last few years refining our vetting process to identify people for whom this career would be an especially good fit.

There is no single blueprint for successful founders. Many of our alumni hadn't even considered this career path until someone encouraged them to explore it! What surprises many people is the incredible diversity among our successful charity founders in terms of age, educational background, and experience.

So far, people from 28 different countries joined our program. We've had many participants come directly from bachelor's or master's programs, as well as welcomed individuals with PhDs from recognized institutions like Cambridge, Oxford, and Harvard. They've specialized in fields as diverse as philosophy, geology, machine learning, and engineering. Some have chosen entrepreneurship over formal education, like our youngest participant, who, at the age of 19, left university to join our program!

Our program has also attracted individuals who boldly chose to transition from lucrative, secure, or high-status careers such as consulting, real estate, engineering at NASA or medicine. We've seen passionate street activists and seasoned nonprofit professionals alike join the program, each bringing their unique perspectives and skills to the table.

More than specific experience or expertise, they have each brought in an open mind and focused desire to launch a high-impact nonprofit. If it's not credentials, age, or specialized experience that make you a great fit for nonprofit entrepreneurship, what is it? Here is what we've learned in the last five years and who we are looking for to join our Incubation Programs.

Competence:

Previous Project Experience: People who have initiated their own projects or taken the lead in previous initiatives. Examples might be organizing impactful events or conferences, managing and expanding student groups, cultivating a newsletter or blog following, developing a mobile app with peers, serving as an editor for a university newspaper, or mobilizing an activist group. These instances serve as evidence of an entrepreneurial mindset and a proactive approach.

Academic or Personal Achievement: People who have excelled academically or demonstrated exceptional accomplishments in their career, hobbies, or interests. A wide range of things fit into this category, from academic distinction to playing an instrument really well, taking part in debate competitions, or winning local RPG contests. High achievement indicates a strong work ethic and a drive for excellence.

Scientific/Analytical/Empirical Mindset: Individuals with experience in empirical methods who value scientific evidence and are eager to test assumptions and change their minds. The skeptical, evidence-based mi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What happens on an 80,000 Hours call?, published by Abby Hoskin on September 8, 2023 on The Effective Altruism Forum.

A lot of people aren't sure what to expect when they apply for 80,000 Hours coaching calls. We thought it might be helpful to give you some context.

Who can we help?

We love when people form inside views on issues, and question ideas that don't make sense to them. So we encourage you to apply for advising even if you don't agree with everything on the 80k website.

However, we're most able to help people who are open to working on some of the top problems we list on our site, or something in a related area. We think most of our impact comes from helping people who end up working in these areas, and since they're the topics we focus on, our advice is best for applicants who could potentially take relevant career paths.

We're best able to help people who are mid or early career, with some idea of what they want to do, but open to changing directions and building new skills. These people are often between the ages of 20 and 40. But we definitely also talk to people who are outside of this range, and can offer them valuable guidance and point them to great opportunities.

Ways you can get value from an 80k call

Setting aside time to reflect deeply on your career trajectory is really valuable.

Running your ideas by somebody else with reasonable judgement is helpful.

If you already have a great plan, checking it over with somebody else can make you feel a lot better.

We can tell potential collaborators and mentors what you're doing in case they want to be involved or help you out.

Advisors tend to know a lot about the people and orgs working on important problems and can point you to resources/orgs/causes you might not know about.

Advisors have different backgrounds and can give expert advice in specific sectors (more on this below).

Speaking with an 80k advisor can help you grow your professional network (though this isn't always possible in all cases).

You can opt into being recommended for roles that are a match for your plans and skills. We are often asked to recommend candidates for roles at high impact organisations, and you can give us permission to recommend you for roles we think would suit you.

After your call, you will be invited to join the 80,000 Hours Alumni Slack. This is a great place to network with other people focused on doing good via their careers. The 80,000 Hours team also periodically updates our alumni about opportunities and resources they may find helpful.

Before the call

We ask you to fill out a call prep document that prompts you to reflect on how you define positive impact, which areas you want to work on, and which careers seem most appealing or attractive to you.

Filling out the call prep document has been consistently identified as one of the most valuable parts of advising. In other words, you can gain a lot of clarity on your career just by setting aside an hour to write down your answers to these questions, even without speaking to an advisor!

You can make a copy of our call prep doc and fill it out now if you'd like. (We won't be able to see your answers, this would be for your personal reflection.)

During the call

For people early in their career, we usually cover these topics:

What is your current situation and default career path?

Let's take your skills/experience out of the picture completely and think about what the world needs.

How do you think about choosing which pressing world problem to work on?

What qualities make these things problems?

What are your values, and how can your career align with them?

Which of these problems would you be able to make progress on based on your personal aptitudes, existing skill sets, and interests?

Options?

Long-run paths

Which skills/experiences/career capital will help you do impa...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Debate series: should we push for a pause on the development of AI?, published by Ben West on September 8, 2023 on The Effective Altruism Forum.

In March of this year, 30,000 people, including leading AI figures like Yoshua Bengio and Stuart Russell, signed a letter calling on AI labs to pause the training of AI systems. While it seems unlikely that this letter will succeed in pausing the development of AI, it did draw substantial attention to slowing AI as a strategy for reducing existential risk.

While initial work has been done on this topic (this sequence links to some relevant work), many areas of uncertainty remain. I've asked a group of participants to discuss and debate various aspects of the value of advocating for a pause on the development of AI on the EA Forum, in a format loosely inspired by Cato Unbound.

On September 16, we will launch with three posts:

David Manheim will share a post giving an overview of what a pause would include, how a pause would work, and some possible concrete steps forward

Nora Belrose will post outlining some of the risks of a pause

Thomas Larson will post a concrete policy proposal

After this, we will release one post per day, each from a different author

Many of the participants will also be commenting on each other's work

Responses from Forum users are encouraged; you can share your own posts on this topic or comment on the posts from participants. You'll be able to find the posts by looking at this tag (remember that you can subscribe to tags to be notified of new posts).

I think it is unlikely that this debate will result in a consensus agreement, but I hope that it will clarify the space of policy options, why those options may be beneficial or harmful, and what future work is needed.

People who have agreed to participate

These are in random order, and they're participating as individuals, not representing any institution:

David Manheim (Technion Israel)

Matthew Barnett (Epoch AI)

Zach Stein-Perlman (AI Impacts)

Holly Elmore (AI pause advocate)

Buck Shlegeris (Redwood Research)

Anonymous researcher (Major AI lab)

Anonymous professor (Major University)

Rob Bensinger (Machine Intelligence Research Institute)

Nora Belrose (EleutherAI)

Thomas Larsen (Center for AI Policy)

Quintin Pope (Oregon State University)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sum-threshold attacks, published by TsviBT on September 8, 2023 on LessWrong.

How do you affect something far away, a lot, without anyone noticing?

(Note: you can safely skip sections. It is also safe to skip the essay entirely, or to read the whole thing backwards if you like.)

The frog's lawsuit

Attorney for the defendant: "So, Mr. Frog. You allege that my client caused you grievous bodily harm. How is it that you claim he harmed you?"

Frog: "Ribbit RIBbit ribbit."

Attorney: "Sir..."

Frog: "Just kidding. Well, I've been living in a pan for the past two years. When I started, I was the picture of health, and at first everything was fine. But over the course of the last six months, something changed. By last month, I was in the frog hospital with life-threatening third-degree burns."

Attorney: "And could you repeat what you told the jury about the role my client is alleged to have played in your emerging medical problems?"

Frog: "Like I said, I don't know exactly. But I know that when my owner wasn't away on business, every day he'd do something with the stove my pan was sitting on. And then my home would seem to be a bit hotter, always a bit hotter."

Attorney: "Your owner? You mean to say..."

Judge: "Let the record show that Mr. Frog is extending his tongue, indicating the defendant, Mr. Di'Alturner."

Attorney: "Let me ask you this, Mr. Frog. Is it right to say that my client - - your owner - - lives in an area with reasonably varied weather? It's not uncommon for the temperature to vary by ten degrees over the course of the day?"

Frog: "True."

Attorney: "And does my client leave windows open in his house?"

Frog: "He does."

Attorney: "So I wonder, how is it that you can tell that a slight raise in temperature that you experience - - small, by your own admission - - how can you be sure that it's due to my client operating his stove, and not due to normal fluctuations in the ambient air temperature?"

Frog: "I can tell because of the correlation. I tend to feel a slight warming after he's twiddled the dial."

Attorney: "Let me rephrase my question. Is there any single instance you can point to, where you can be sure - - beyond a reasonable doubt - - that the warming was due to my client's actions?"

Frog: "Ah, um, it's not that I'm sure that any one increase in temperature is because he turned the dial, but..."

Attorney: "Thank you. And would it be fair to say that you have no professional training in discerning temperature and changes thereof?"

Frog: "That would be accurate."

Attorney: "And are you aware that 30% of frogs in your state report spontaneous slight temperature changes at least once a month?"

Frog: "But this wasn't once a month, it was every day for weeks at a ti - - "

Attorney: "Sir, please only answer the questions I ask you. Were you aware of that fact?"

Frog: "No, I wasn't aware of that, but I don't see wh - - "

Attorney: "Thank you. Now, you claim that you were harmed by my client's actions, which somehow put you into a situation where you became injured."

Frog: "¡I have third degree burns all ov - - "

Attorney: "Yes, we've seen the exhibits, but I'll remind you to only speak in response to a question I ask you. What I'd like to ask you is this: Why didn't you just leave the frying pan? If you were, as you allege, being grievously injured, wasn't that enough reason for you to remove yourself from that situation?"

Frog: "I, I didn't notice that it was happening at the time, each change was so subtle, but..."

Attorney: "Thank you. As your counsel would have advised you, the standard for grievous bodily harm requires intent. Now are we really expected to conclude, beyond a reasonable doubt, that my client intended to cause you harm, via a method that you didn't even notice? That even though you can't point to so much as a single instance where my ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Incomplete List of Things I Think EAs Probably Shouldn't Do, published by Rockwell on September 8, 2023 on The Effective Altruism Forum.

There has already been ample discussion of what norms and taboos should exist in the EA community, especially over the past ten months. Below, I'm sharing an incomplete list of actions and dynamics I would strongly encourage EAs and EA organizations to either strictly avoid or treat as warranting a serious - and possibly ongoing - risk analysis.

I believe there is a reasonable risk should EAs:

Live with coworkers, especially when there is a power differential and especially when there is a direct report relationship

Date coworkers, especially when there is a power differential and especially when there is a direct report relationship

Promote drug use among coworkers, including legal drugs, and including alcohol and stimulants

Live with their funders/grantees, especially when substantial conflict-of-interest mechanisms are not active

Date their funders/grantees, especially when substantial conflict-of-interest mechanisms are not active

Date the partner of their funder/grantee, especially when substantial conflict-of-interest mechanisms are not active

Retain someone as a full-time contractor or grant recipient for the long term, especially when it might not adhere to legal guidelines

Offer employer-provided housing for more than a predefined and very short period of time, thereby making an employee's housing dependent on their continued employment and allowing an employer access to an employee's personal living space

Potentially more controversial, two aspects of the community I believe have substantial downsides that the community has insufficiently discussed or addressed:

EA™ Group Houses and the branding of private, personal spaces as "EA"

"Work trials" that require interruption of regular employment to complete, such that those currently employed full-time must leave their existing job to be considered for a prospective job

As said, this list is far from complete and I imagine people may disagree with portions of it. I'm hoping to stake this as a position held by some EAs and I'm hoping this post can serve as a prompt for further discussion and assessment.

"Promote" is an ambiguous term here. I think this is true to life in that one person's enthusiastic endorsement of a drug is another person's peer pressure.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AMA: 80,000 Hours Career Advising Team, published by Abby Hoskin on September 8, 2023 on The Effective Altruism Forum.

We're the 80,000 Hours Career Advising team, and you should ask us anything!

We are advisors at 80,000 Hours! Ask us anything about careers and career advising! (If you're interested in us personally or other things about 80k we might also answer those.) We'll answer questions on September 13th (British time), so please post questions before then. Unfortunately, we might not be able to get to all the questions, depending on how much time we end up having and what else is asked. So be sure to upvote the questions you most want answered :)

Logistics/practical instructions:

Please post your questions as comments on this post. The earlier you share your questions, the easier it will be for us to get to them.

We'll probably answer questions on September 13th. Questions posted after that aren't likely to get answers.

Some context:

You have 80,000 hours in your career. This makes it your best opportunity to have a positive impact on the world. If you're fortunate enough to be able to use your career for good, but aren't sure how, our website helps you:

Get new ideas for fulfilling careers that do good

Compare your options

Make a plan you feel confident in

You can also check out our free career guide. We are excited to recently launch the second edition! It's based on 10 years of research alongside academics at Oxford.

We're a nonprofit, and everything we provide, including our one-on-one career advising, is free.

Curious about what happens on a 1:1 career advising call? Check out our EA Forum post on what happens during calls here.

If you're ready to use your career to have a greater positive impact on the world, apply for career advising with us!

Who are we?

I (Abigail Hoskin) have a PhD in psychology and neuroscience and can talk about paths into and out of academia. I can also discuss balancing having an impact while parenting (multiple!) kids. I will be taking the lead on answering questions in this AMA, but other advisors might chime in, especially on questions in their specific areas of expertise.

Huon Porteous has a background in philosophy and experience in management consulting. He has run a huge number of useful "cheap tests" to test out his aptitudes for different careers and is always running self experiments to optimise his workflow.

Matt Reardon is a lawyer who can talk in depth about paths to value in law, government, and policy, especially in the US. He also works on product improvements and marketing for our team.

Sudhanshu Kasewa was a machine learning engineer at a start-up and has experience doing ML research in academia. He's also worked in human resources and consulting.

Anemone Franz is a medical doctor who worked for a biotech startup on pandemic preparedness. She particularly enjoys discussing careers in biosecurity, biotech, or global health.

We are led by Michelle Hutchinson! Michelle co-founded the Global Priorities Institute, ran Giving What We Can, was a fund manager for EA Funds, and is a top contributor to our cute animals Slack channel. Michelle is the director of the 1on1 team and does not take calls, but she'll be chiming in on the AMA.

We have a special guest, Benjamin Hilton, who will be on deck to answer questions about our website's written content. Ben is a researcher at 80,000 Hours, who has written many of our recent articles, including on AI technical career paths, in addition to helping write our updated career guide.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Explaining grokking through circuit efficiency, published by Vikrant Varma on September 8, 2023 on The AI Alignment Forum.

This is a linkpost for our paper ''Explaining grokking through circuit efficiency", which provides a general theory explaining when and why grokking (aka delayed generalisation) occurs, and makes several interesting and novel predictions which we experimentally confirm (introduction copied below). You might also enjoy our explainer on X/Twitter.

Abstract

One of the most surprising puzzles in neural network generalisation is grokking: a network with perfect training accuracy but poor generalisation will, upon further training, transition to perfect generalisation. We propose that grokking occurs when the task admits a generalising solution and a memorising solution, where the generalising solution is slower to learn but more efficient, producing larger logits with the same parameter norm. We hypothesise that memorising circuits become more inefficient with larger training datasets while generalising circuits do not, suggesting there is a critical dataset size at which memorisation and generalisation are equally efficient. We make and confirm four novel predictions about grokking, providing significant evidence in favour of our explanation.

Most strikingly, we demonstrate two novel and surprising behaviours: ungrokking, in which a network regresses from perfect to low test accuracy, and semi-grokking, in which a network shows delayed generalisation to partial rather than perfect test accuracy.

Introduction

When training a neural network, we expect that once training loss converges to a low value, the network will no longer change much. Power et al. (2021) discovered a phenomenon dubbed grokking that drastically violates this expectation. The network first ''memorises'' the data, achieving low and stable training loss with poor generalisation, but with further training transitions to perfect generalisation. We are left with the question: why does the network's test performance improve dramatically upon continued training, having already achieved nearly perfect training performance?

Recent answers to this question vary widely, including the difficulty of representation learning (Liu et al., 2022), the scale of parameters at initialisation (Liu et al., 2023), spikes in loss (''slingshots") (Thilak et al., 2022), random walks among optimal solutions (Millidge et al., 2022), and the simplicity of the generalising solution (Nanda et al., 2023, Appendix E). In this paper, we argue that the last explanation is correct, by stating a specific theory in this genre, deriving novel predictions from the theory, and confirming the predictions empirically.

We analyse the interplay between the internal mechanisms that the neural network uses to calculate the outputs, which we loosely call ''circuits'' (Olah et al., 2020). We hypothesise that there are two families of circuits that both achieve good training performance: one which generalises well (Cgen) and one which memorises the training dataset (Cmem). The key insight is that when there are multiple circuits that achieve strong training performance, weight decay prefers circuits with high ''efficiency'', that is, circuits that require less parameter norm to produce a given logit value.

Efficiency answers our question above: if Cgen is more efficient than Cmem, gradient descent can reduce nearly perfect training loss even further by strengthening Cgen while weakening Cmem, which then leads to a transition in test performance. With this understanding, we demonstrate in Section 3 that three key properties are sufficient for grokking: (1) Cgen generalises well while Cmem does not, (2) Cgen is more efficient than Cmem, and (3) Cgen is learned more slowly than Cmem.

Since Cgen generalises well, it automatically works ...