The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - Open problems in activation engineering by Alex Turner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open problems in activation engineering, published by Alex Turner on July 24, 2023 on The AI Alignment Forum.
    Steering GPT-2-XL by adding an activation vector introduced
    activation engineering... techniques which steer models by modifying their activations. As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime.
    These results were recently complemented by Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, which doubled TruthfulQA performance by adding a similarly computed activation vector to forward passes!
    We think that activation engineering has a bunch of low-hanging fruit for steering and understanding models. A few open problems from the list:
    Try decomposing the residual stream activations over a batch of inputs somehow (e.g. PCA). Using the principal directions as activation addition directions, do they seem to capture something meaningful?
    Take a circuit studied from existing literature on GPT2, or find another one using ACDC. Targeting the nodes in these circuits, can you learn anything more about them and generally about how activation additions interact with circuits?
    What's the mechanism by which adding a steering vector with too large a coefficient breaks the model? (Credit: Thomas Kwa; see also @Ulisse Mini's initial data/explanation.)
    If you want to work on activation engineering, come by the Slack server to coordinate research projects and propose new ideas.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - A reformulation of Finite Factored Sets by Matthias G. Mayer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A reformulation of Finite Factored Sets, published by Matthias G. Mayer on July 24, 2023 on LessWrong.
    I present my bachelor's thesis that reformulates Finite Factored Sets:
    Causality with Deterministic Relationships
    Abstract:We present a reformulation of Finite Factored Sets that uses functions and cartesian products instead of partitions and factorizations. In the new setting, we motivate assuming conditional orthogonality from observed conditional independence and show that inference is decidable. We show that Pearlian causal structures have analogs in the developed framework and propose an extension to the infinite unconditional case using measurable spaces.
    I will quote Section 1.3 to give an overview of the additions to the original paper.
    Most of the concepts in this work were first introduced in [Finite Factored Sets (FFS)], where the author starts with a set S and factorizes it using a set of partitions {Fi:i∈I} of S. Then partitions on S are analyzed using this factorization. This is called a (finite) factored set.
    In this work, we instead start from the factors Fi as setsand construct F=×i∈IFi as a cartesian product.Here we call F a (finite) factor space.Instead of a partition on S, we analyze features X:FΩX,where a feature would correspond to the partition {X-1(ω):ω∈ΩX} of F.
    In the same manner, we can translate all concepts using functions.In the authors' opinion, this is a more natural way to frame things.We hope that this reframing is helpful to understand the frameworkand for doing further work.
    To compare this work to the original paper [FFS], we give a detailed list of the main additions, apart from the reformulation.
    In Section 2.4, we provide an alternative way to define conditional histories using a factor space induced on a subset A⊆F.This is helpful to understand the concepts more deeply and leads, in some cases,to shorter and more intuitive proofs.
    In Section 3.3, we generalize the examples provided in [FFS] (Example 1 and 2)to groups.
    In Section 3.2 and the appendix, we spell out the implications carried bythe fundamental theorem of finite factored sets [FFS] (Theorem 3)to justify assuming conditional orthogonality from observed conditional independence.
    In Section 3.4, we prove that inference is decidable, thus solvingthis open problem raised in [FFS] (Section 7.1).
    In Section 4, we convert Pearlian causal structures and models to factor spaces.We prove that d-separation in a causal structure is equivalent toconditional orthogonality in the corresponding factor space.We further give a definition of interventions in such a factor space and prove that they lead to the same probabilities a causal structure provides.
    In Section 5, we provide a generalization of the framework to arbitrarymeasure spaces in the unconditional case.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Slowing down AI progress is an underexplored alignment strategy by Norman Borlaug
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Slowing down AI progress is an underexplored alignment strategy, published by Norman Borlaug on July 24, 2023 on LessWrong.
    The emotional burden of watching the world end
    My current beliefs about AI timelines have made my life significantly worse. I find thoughts about the ever-shrinking timeline to AGI invading nearly every aspect of my life. Every choice now seems to be affected by the trajectory of this horrible technology.
    Based on the responses in this post and others, many if not most people on this forum have been affected in a similar way. AI is the black hole swallowing all other considerations about the future.
    Frankly, I feel a good deal of anger about the current state of affairs. All these otherwise thoughtful and nice people are working to build potentially world-ending technology as fast as possible. I'm angry so few people are paying any attention to the danger. I'm angry that invasive thoughts about AGI destroying the world are making it harder for me to focus on work that might have a non-zero chance of saving it. And I'm frustrated we've been so closed-minded in our approach to solving this problem.
    The current situation is almost exactly analogous to the creation of the atomic bomb during World War 2. There's a bunch of very smart people gathered together, supplied with a fuckton of money by powerful groups, working towards the same destructive goal. There is social proof all around them that they are doing the right thing. There are superficially plausible reasons to think that everything will turn out fine. There is token engagement with the concerns raised by people concerned about the implications of the technology. But at the end of the day, the combination of personal incentives, social proof and outside impetus make everyone turn their head and ignore the danger. On the rare occasion that people are convinced to leave, it's almost always the most conscientious, most cautious people, ensuring the remaining team is even less careful.
    There is so much magical thinking going on among otherwise intelligent people. Everyone seems to be operating on the assumption that no technology can destroy us, that everything will magically turn out fine, despite the numerous historical examples of new inventions destroying or nearly destroying the world (see the Cuban Missile Crisis or the great oxidation event when oxygen-producing bacteria extincted themselves and most other life on earth and caused a 300 million year ice age).
    Frankly I've found the response from the EA/rationalist community has been pretty lackluster so far. Every serious solution that has been proposed revolves around solving alignment before we make AGI, yet I know ZERO people who are working on slowing down capabilities progress. Hell, until just a month ago, EA orgs like 80,000 hours were RECOMMENDING people join AI research labs and work on CREATING superintelligence.
    The justifications I've read for this behavior always seem to be along the lines of "we don't want to alienate the people working at top AI orgs because we feel that will be counter-productive to our goals of convincing them that AI alignment is important." Where has this strategy gotten us? Does the current strategy of getting a couple of members of the EA/Rationalist community onto the safety teams at major AI orgs actually have a chance at working? And is it worth foregoing all efforts to slow down progress towards AGI?
    The goals of DeepMind, OpenAI, and all the other top research labs are fundamentally opposed to the goal of alignment. The founding goal of Deepmind is to "solve intelligence, then use that to solve everything else." That mission statement has been operationalized as paying a bunch of extremely smart people ridiculous salaries to create and distribute the blueprints for (potentially) world-ending AGI...
    9 min
  • LW - Brain Efficiency Cannell Prize Contest Award Ceremony by Alexander Gietelink Oldenziel
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Brain Efficiency Cannell Prize Contest Award Ceremony, published by Alexander Gietelink Oldenziel on July 24, 2023 on LessWrong.
    Previously Jacob Cannell wrote the post "Brain Efficiency" which makes several radical claims: that the brain is at the pareto frontier of speed, energy efficiency and memory bandwith, that this represent a fundamental physical frontier.
    Here's an AI-generated summary
    The article "Brain Efficiency: Much More than You Wanted to Know" on LessWrong discusses the efficiency of physical learning machines. The article explains that there are several interconnected key measures of efficiency for physical learning machines: energy efficiency in ops/J, spatial efficiency in ops/mm^2 or ops/mm^3, speed efficiency in time/delay for key learned tasks, circuit/compute efficiency in size and steps for key low-level algorithmic tasks, and learning/data efficiency in samples/observations/bits required to achieve a level of circuit efficiency, or per unit thereof. The article also explains why brain efficiency matters a great deal for AGI timelines and takeoff speeds, as AGI is implicitly/explicitly defined in terms of brain parity.
    The article predicts that AGI will consume compute & data in predictable brain-like ways and suggests that AGI will be far more like human simulations/emulations than you'd otherwise expect and will require training/education/raising vaguely like humans1.
    Jake further has argued that this has implication for FOOM and DOOM.
    Considering the intense technical mastery of nanoelectronics, thermodynamics and neuroscience required to assess the arguments here I concluded that a public debate between experts was called for. This was the start of the Brain Efficiency Prize contest which attracted over a 100 in-depth technically informed comments.
    Now for the winners! Please note that the criteria for winning the contest was based on bringing in novel and substantive technical arguments as assesed by me. In contrast, general arguments about the likelihood of FOOM or DOOM while no doubt interesting did not factor into the judgement.
    And the winners of the Jake Cannell Brain Efficiency Prize contest are
    Ege Erdil
    DaemonicSigil
    spxtr
    ... and Steven Byrnes!
    Each has won $150, provided by Jake Cannell, Eli Tyre and myself.
    I'd like to heartily congratulate the winners and thank everybody who engaged in the debate. The discussion were sometimes heated but always very informed. I was wowed and amazed by the extraordinary erudition and willingness for honest compassionate intellectual debate displayed by the winners.
    So what are the takeaways?
    I will let you be the judge. Again, remember the choice of the winners was made on my (layman) assesment that the participant brought in novel and substantive technical arguments and thereby furthered the debate.
    Steven Byrnes
    The jury was particularly impressed by Byrnes' patient, open-minded and erudite participation in the debate.
    He has kindly written a post detailing his views. Here's his summary
    Some ways that Jacob & I seem to be talking past each other
    I will, however, point to some things that seem to be contributing to Jacob & me talking past each other, in my opinion.
    Jacob likes to talk about detailed properties of the electrons in a metal wire (specifically, their de Broglie wavelength, mean free path, etc.), and I think those things cannot possibly be relevant here. I claim that once you know the resistance/length, capacitance/length, and inductance/length of a wire, you know everything there is to know about that wire's electrical properties. All other information is screened off. For example, a metal wire can have a certain resistance-per-length by having a large number of mobile electrons with low mobility, or it could have the same resistance-per-length by having a smaller number of mobile...
    12 min
  • LW - Cryonics and Regret by MvB
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cryonics and Regret, published by MvB on July 24, 2023 on LessWrong.
    This post is not about arguments in favor of or against cryonics. I would just like to share a particular emotional response of mine as the topic became hot for me after not thinking about it at all for nearly a decade.
    Recently, I have signed up for cryonics, as has my wife, and we have made arrangements for our son to be cryopreserved just in case longevity research does not deliver in time or some unfortunate thing happens.
    Last year, my father died. He was a wonderful man, good-natured, intelligent, funny, caring and, most importantly in this context, loving life to the fullest, even in the light of any hardships. He had a no-bullshit-approach regarding almost any topic, and, being born in the late 1940s in relative poverty and without much formal education, over the course of his life he acquired many unusual attitudes that were not that compatible with his peers (unlike me, he never tried to convince other people of things they could not or did not want to grasp, pragmatism was another of his traits). Much of what he expected from the future in general and technology in particular, I later came to know as transhumanist thinking, though neither was he familiar with the philosophy as such nor was he prone to labelling his worldview. One of his convictions was that age-related death is a bad thing and a tragedy, a problem that should and will eventually be solved by technology.
    I could see his decline, beginning about 15 years ago. He just was not as energetic anymore as I had known him, and I kept telling myself that this is a part of the usual aging process. Some five years ago, when he started to have troubles walking, I proposed to see a neurologist. He was diagnosed with cardiovascular dementia and it was clear where things were going.
    I was alone with him, holding his hand and looking into his eyes when he took his last breath. It was one of the most intense experiences of my life. In the months after his death, I told myself that I did anything I could for him. But now it is clear to me that I did not, and my regrets weigh more heavily than my mourning.
    I know that he would have been all in. A second chance to live, rejuvenation, a future with maybe endless possibilities. And I just was not aware anymore of the process that could have given him that chance. The funds were there, everything could have been arranged in time. I just did not think about the possibility then. And now he is gone without any chance to get him back.
    My wife tells me that his brain was damaged to an extent that vitrification probably would have failed and pushes me to rationalize along the lines of "there was not much left to safe". It does not work for me, I know what was left of him - though his brain looked like swiss cheese, his core personality was still there. The few things he was still able to speak told me enough about what he still knew and understood, his last utterance was a witty joke about me being there with him and not being at work instead.
    I can easily accept most of the mistakes I made, I can live with my past shortcomings. But this I do regret, and I do not think that my regret will ever vanish, that I did not arrange for a second chance for my dad.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Who's right about inputs to the biological anchors model? by rosehadshar
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Who's right about inputs to the biological anchors model?, published by rosehadshar on July 24, 2023 on The Effective Altruism Forum.
    In this post, I compared forecasts from Ajeya Cotra and from forecasters in the Existential Risk Persuasion Tournament (XPT) relating to some of the inputs to Cotra's biological anchors model.
    Here, I give my personal take on which of those forecasts seem more plausible.
    Note that:
    I'm only considering the inputs to the bio anchors model which we have XPT forecasts for. This notably excludes the 2020 training requirements distribution, which is a very important driver of model outputs.
    My take is based on considering the explicit arguments that Cotra and the XPT forecasters gave, rather than on independent research.
    My take is subjective.
    I've been working with the Forecasting Research Institute (who ran the XPT) since November 2022, and this is a potential source of bias.
    I'm publishing this post in a personal capacity and it hasn't gone through FRI's review process.
    I originally wrote this early in 2023. I've tried to update it as new information came out, but I likely haven't done a comprehensive job of this.
    To recap, here are the relevant forecasts:
    See workings here and here. The 'most aggressive' and 'most conservative' forecasts can be considered equivalent to 90% confidence intervals for the median estimate.
    Hardware
    For FLOP/$ in 2025, I think both Cotra and the XPGT forecasters are wrong, but Cotra will prove more right.
    Epoch's current estimate of highest GPU price-performance is 4.2e18 FLOP per $.
    They also find a trend in GPU price-performance of 0.1 OOM/year for state of the art GPUs. So I'll extrapolate 4.2e18 to 5.97E+18.
    For compute price halving time to 2100, I think it depends how likely you think it is that novel technologies like optical computing will reduce compute prices in future.
    This is the main argument Cotra puts forward for expecting such low prices.
    It's an argument made in XPT too, but less weight is put on it.
    Counterarguments given in XPT: fundamental physical limits, progress getting harder, rare materials capping how much prices can drop, catastrophe/extinction, optimisation shifting to memory architectures.
    Cotra mentions some but not all of these (she doesn't mention rare materials or memory architectures).
    Cotra flags that she thinks after 2040 her forecasts on this are pretty unreliable.
    But, because of how wrong their 2024 and 2030 forecasts seem to be, I'm not inclined to put much weight on XPT forecasts here either.
    I'll go with the most aggressive XPT figure, which is close to Cotra's.
    I don't have an inside view on the likelihood of novel technologies causing further price drops.
    Note that the disagreement about compute price halving times drives a lot of the difference in model output.
    Willingness to spend
    On the most expensive training run by 2025, I think Cotra is a bit too aggressive and XPT forecasters are much too conservative.
    In 2022, Cotra updated downwards a bit on the likelihood of a $1bn training run by 2025.
    There isn't much time left for Cotra to be right.
    Cotra was predicting $20m by the end of 2020, and $80m by the end of 2021.
    GPT-3 was $4.6m in 2020. If you buy that unreleased proprietary models are likely to be 2-8x more expensive than public ones (which Cotra argues), that XPT forecasters missed this consideration, and that GPT-3 isn't proprietary and/or unreleased (flagging because I'm unsure what Cotra actually means by proprietary/unreleased), then this could be consistent with Cotra's forecasts.
    Epoch estimates that GPT-4 cost $50m to train at some point in 2022.
    Again, this could be in line with Cotra's predictions.
    More importantly, GPT-4 costs make XPT forecasters look quite wrong already - their 2024 prediction was surpassed in 2022. This is especially striking i...
    12 min
  • EA - What we learned from training EA community members in facilitation skills by AlexanderSaeri
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What we learned from training EA community members in facilitation skills, published by AlexanderSaeri on July 24, 2023 on The Effective Altruism Forum.
    Summary
    We designed and delivered online facilitator training to engaged EAs in Australia and New Zealand. Participants included community leaders and staff at EA organisations.
    The training course covered key elements of facilitation, including asking questions, active listening, planning and preparation, reading the room, and managing participants.
    We conducted an evaluation of the course, which found that participants were very confident the course was very helpful for their facilitation practice, and identified the practical exercises with rapid reflection and feedback as key components that made the course valuable for them.
    We believe that facilitation training, implemented effectively, could assist community building in Effective Altruism by improving learning, group discussions, and successful events.
    We were motivated to develop and deliver facilitation training because it's an important 'soft skill' in community building
    Our motivation
    We are two academics with extensive experience in teaching, training, and facilitation, who also are engaged in the Effective Altruism community in Australia. After working with the team at BlueDot Impact early in 2023 to revise their training for course facilitators, we thought that a similar approach could be useful for community leaders and community-facing staff at EA organisations.
    We think that facilitation in particular is an important "soft skill" in supporting learning, productive group interactions, and successful events. Facilitation is distinct from instructing, managing, or event planning.
    While lecturing is "sage on the stage", facilitation is "guide by the side". We think that a lot of EA community building involves being a 'guide by the side', consistent with valuing autonomy, critical thinking, and supporting the development of intrinsic motivations to act in alignment with EA values.
    Some of the specific skills involved in being this 'guide' include asking good questions, helping people feel like their contributions were heard, navigating group dynamics and energy, adapting to unexpected situations, and managing participant disagreement or conflict. A good facilitator clarifies and advances the shared purpose of the group.
    Participants' motivation
    We sought interest from prospective participants in Australia and New Zealand with an identified need for improving their facilitation practice. 14 people expressed interest and we invited 11 to participate. We asked participants to complete a pre-survey to help us understand their specific needs for facilitation practice. Some of the needs that participants expressed:
    Improving skills in leading engaging discussions. For example, they wanted to be able to draw people into conversations, ask the right questions to encourage reflection (e.g., in a reading group, or in an EA introductory fellowship), and guiding discussions towards topics of interest to participants (e.g., managing vocal or dominant members, or recurring but unproductive topics of discussion).
    Developing empathy and a better understanding of group dynamics. For example, they wanted to be able to pick up on emotional states and discomfort (so it could be addressed), identify and resolve blocks for discussion or decision-making, or encourage participants to be active learners or otherwise promote practical action from attendees.
    Improving enjoyment, inclusion, and engagement at planned events. For example, they held frequent networking events, community meetups, workshops, working bees, or other meetings with community members.
    We developed and delivered an online facilitation training course using Miro, Zoom, and Google Docs
    Development Process
    We pre...
    16 min
  • LW - Rationality !== winning by Raemon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rationality !== winning, published by Raemon on July 24, 2023 on LessWrong.
    I think "Rationality is winning" is a bit of a trap.
    (The original phrase is notably "rationality is systematized winning", which is better, but it tends to slide into the abbreviated form, and both forms aren't that great IMO)
    It was coined to counteract one set of failure modes - there were people who were straw vulcans, who thought rituals-of-logic were important without noticing when they were getting in the way of their real goals. And, also, there outside critics who'd complain about straw-vulcan-ish actions, and treat that as a knockdown argument against "rationality."
    "Rationalist should win" is a countermeme that tells both groups of people "Straw vulcanism is not The Way. If you find yourself overthinking things in counterproductive ways, you are not doing rationality, even if it seems elegant or 'reasonable' in some sense."
    It's true that rationalists should win. But I think it's not correspondingly true that "rationality" is the study of winning, full stop. There are lots of ways to win. Sometimes the way you win is by copying what your neighbors are doing, and working hard. There is rationality involved in sifting through the various practices people suggest to you, and figuring out which ones work best. But, the specific skill of "sifting out the good from the bad" isn't always the best approach. It might take years to become good at it, and it's not obvious that those years of getting good at it will pay off.
    Rationality is the study (and applied skill) of finding cognitive algorithms that form better beliefs and make better decisions. Sometimes this is the appropriate tool for the job, and sometimes it's not.
    The reason this is particularly important is in deciding what feedback loops to focus on when developing rationality. If you set your feedback loops as "am I winning, generally?" (or, "are the students of my rationality curriculum winning, generally?"), well, that's a really noisy feedback loop that's probably swamped by random environmental variables. It's not nothing, but it's really slow.
    There's also a problem where, well, a lot of techniques that help with winning-at-life just aren't especially about rationality in particular. If you're running a rationality program with "win-at-life" as your goal, you may find yourself veering in a direction that's not really capitalizing on the things rationality was actually especially good at, and become a generic self-help program. Maybe that's fine, but the result seems to lose something of the original spirit.
    The domains where rationality matters are domain where information is scarce, and the common wisdom of people around you is inadequate.
    The coronavirus pandemic was a good example where rationality was relevant: a very major change disrupted society, there was not yet a scientific consensus on the subject, there were reasons to doubt some claims by scientific authorities, and your neighbors were probably slow to react. (I think rationalists did well at navigating the early pandemic, but, alas, also stayed in overly-stressful-lockdown-mode longer than was appropriate, and lacked some other key skills)
    Building a startup is a domain where I think rationality is pretty relevant. There is a lot of common wisdom that is relevant. Paul Graham et al have useful advice. But because you need to outperform a lot of competitors, common wisdom isn't enough. You need to continuously model the world, design products people don't know they want yet while soaking in new information so you can continuously update and iterate. You need to stare into the darkness and admit major mistakes.
    Rationality is particularly helpful for solving problems we don't know how to solve.
    Rationality is useful for judges and jurors, who must accurately wei...
    7 min
  • LW - Which rationality posts are begging for further practical development? by LoganStrohl
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Which rationality posts are begging for further practical development?, published by LoganStrohl on July 23, 2023 on LessWrong.
    Which posts seem to you like they have really important practical implications for how we ought to think and act, but don't seem to come with enough information about how to use the skills, or to develop them, or even what exactly the relevant skills are?Context: I'd like to devote many of my hours in the upcoming few months to naturalist study of topics that are critical to fully understanding the most important existing resources in rationality. I mainly have my eye on posts from Eliezer's Sequences, but I'm also open to other essays, books, and even videos. I'll be taking extensive notes, and turning them into companion pieces to complement the original essays. (See "Lies, Damn Lies, and Fabricated Options" and "Investigating Fabrication" for an example of what I'm proposing.)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - QAPR 5: grokking is maybe not that big a deal? by Quintin Pope
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: QAPR 5: grokking is maybe not that big a deal?, published by Quintin Pope on July 23, 2023 on LessWrong.
    [Thanks to support from Cavendish Labs and a Lightspeed grant, .I've been able to restart the Quintin's Alignment Papers Roundup sequence.]
    Introduction
    Grokking refers to an observation by Power et al. (below) that models trained on simple modular arithmetic tasks would first overfit to their training data and achieve nearly perfect training loss, but that training well past the point of overfitting would eventually cause the models to generalize to unseen test data. The rest of this post discusses a number of recent papers on grokking.
    Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
    In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of learning can be studied in great detail. In some situations we show that neural networks learn through a process of "grokking" a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting. We also study generalization as a function of dataset size and find that smaller datasets require increasing amounts of optimization for generalization. We argue that these datasets provide a fertile ground for studying a poorly understood aspect of deep learning: generalization of overparametrized neural networks beyond memorization of the finite training dataset.
    My opinion:
    When I first read this paper, I was very excited. It seemed like a pared-down / "minimal" example that could let us study the underlying mechanism behind neural network generalization. You can read more of my initial opinion on grokking in the post Hypothesis: gradient descent prefers general circuits.
    I now think I was way too excited about this paper, that grokking is probably a not-particularly-important optimization artifact, and that grokking is no more connected to the "core" of deep learning generalization than, say, the fact that it's possible for deep learning to generalize from an MNIST training set to the testing set.
    I also think that using the word "grokking" was anthropomorphizing and potentially misleading (like calling the adaptive information routing component of a transformer model its "attention"). Evocative names risk letting the connotations of the name filter into the analysis of the object being named. E.g.,
    "Grokking" brings connotations of sudden realization, despite the fact that the grokking phase in the above plot starts within the first ~5% - 20% of the training process, though it appears much more abrupt due to the use of a base 10 logarithmic scale on the x-axis.
    "Grokking" also brings connotations of insight, realization or improvement relative to some previously confused baseline. This leads to the impression that things which grok are better than things which don't.
    Humans often use the word "grokking" to mean deeply understanding complex domains that actually matter in the real world. Using the same word in an ML context suggests that ML grokking is relevant to whatever mechanisms might let an ML system deeply understand complex domains that actually matter in the real world.
    I've heard several people say things like:
    Studying grokking could significantly advance ML capabilities, if doing so were to lead to a deeper understanding of the mechanisms underlying generalization in ML.
    Training long enough could eventually result in grokking occurring in ML domains of actual relevance, such as language, and thereby lead to sudden capabilities gains or break alignment properties.
    Grokking is an example of how thinking longer about a...
    17 min