The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - More findings on Memorization and double descent by Marius Hobbhahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More findings on Memorization and double descent, published by Marius Hobbhahn on February 1, 2023 on LessWrong.
    Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.
    I’d like to thank Wes Gurnee, Aryan Bhatt, Eric Purdy and Stefan Heimersheim for discussions and Evan Hubinger, Neel Nanda, Adam Jermyn and Chris Olah for mentorship and feedback.
    The post contains a lot of figures, so the suggested length is deceiving. Code can be found in these three colab notebooks [1][2][3].
    I have split the post into two parts. The first one is concerned with double descent and other general findings in memorization and the second focuses on measuring memorization using the maximum data dimensionality metric. This is the first post in a series of N posts on memorization in transformers.
    Executive summary
    I look at a variety of settings and experiments to better understand memorization in toy models. My primary motivation is to increase our general understanding of NNs but I also suspect that understanding memorization better might increase our ability to detect backdoors/trojans.
    The work heavily builds on two papers by Anthropic, “Toy models of superposition” and “Superposition, Memorization and double descent”. I successfully replicate a subset of their findings.
    I specifically look at three different setups of NNs that I speculate are most relevant to understanding memorization in the non-attention parts of transformers.
    Bottlenecks between layers, i.e. when projecting from high-dimensional spaces (e.g. MLPs) into lower dimensions (e.g. the residual stream). This is similar to the setting in the toy models of superposition paper and its sequel.
    MLP blocks, i.e. when projecting from lower-dimensional spaces (e.g. the residual stream) into higher dimensions with ReLU non-linearities.
    The final layer, i.e. when projecting from the end of the residual stream into the vocab space. The main difference to the previous scenarios is that we use the cross-entropy loss for the experiments which has a different inductive bias than the MSE loss.
    I’m able to find the double descent phenomenon in all three settings. My takeaway from this is that the transition between memorization and learning general features seems to be a very regular and predictable phenomenon (assuming you know the sparsity and number of features of your network). Furthermore, it seems like the network is “confused” (e.g. has much higher test loss) when it is right between memorization and generalization.
    I test the limits of reconstruction in different settings, i.e. the ability of the neural network to reconstruct its inputs given different dataset sizes, hidden sizes, number of features, importance distributions and sparsities. The findings mostly confirm what we would predict, e.g. more sparsity or larger hidden sizes lead to better reconstructions. A speculative claim is that if we had better measures of sparsity and importance in real-world models, we might be able to derive scaling laws that could tell us how many “concepts” a network has learned.
    Interpreting NNs that memorized in the simplest settings is extremely straightforward--the network literally creates a dictionary that you can just read off the weights. However, even small increases in complexity make this dictionary much harder to read and I have not yet found a method to decompile it into a human-readable form (maybe in the next posts).
    Isolated components
    In the following, we isolate three settings that seem like important components of memorization. They are supposed to model the non-attention parts of a transformer (primarily because I speculate that memorization mostly happens in the non-attention parts).
    Bottleneck
    By bottleneck we mean a situation in which a model projects from many into fewer dimensions, e.g. fro...
    31 min
  • AF - Temporally Layered Architecture for Adaptive, Distributed and Continuous Control by Roman Leventov
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Temporally Layered Architecture for Adaptive, Distributed and Continuous Control, published by Roman Leventov on February 2, 2023 on The AI Alignment Forum.
    A preprint is published by Devdhar Patel, Joshua Russell, Francesca Walsh, Tauhidur Rahman, Terrance Sejnowski, and Hava Siegelmann in December 2022.Abstract:
    We present temporally layered architecture (TLA), a biologically inspired system for temporally adaptive distributed control. TLA layers a fast and a slow controller together to achieve temporal abstraction that allows each layer to focus on a different time-scale. Our design is biologically inspired and draws on the architecture of the human brain which executes actions at different timescales depending on the environment's demands. Such distributed control design is widespread across biological systems because it increases survivability and accuracy in certain and uncertain environments. We demonstrate that TLA can provide many advantages over existing approaches, including persistent exploration, adaptive control, explainable temporal behavior, compute efficiency and distributed control.
    We present two different algorithms for training TLA: (a) Closed-loop control, where the fast controller is trained over a pre-trained slow controller, allowing better exploration for the fast controller and closed-loop control where the fast controller decides whether to "act-or-not" at each timestep; and (b) Partially open loop control, where the slow controller is trained over a pre-trained fast controller, allowing for open loop-control where the slow controller picks a temporally extended action or defers the next n-actions to the fast controller. We evaluated our method on a suite of continuous control tasks and demonstrate the advantages of TLA over several strong baselines.
    Conclusion:
    In this work, we presented Temporally Layered Architecture (TLA), a framework for distributed, adaptive response time in reinforcement learning. The framework allows the RL agent to achieve smooth control in a real-time setting using a slow controller while a fast controller monitors and intervenes as required. Additionally, we demonstrated an alternative setting where the slow controller can gate the fast controller, activating it only when required for efficient control. We demonstrate faster convergence and more action repetition in the closed-loop approach and fewer decision and faster convergence in the partially-open loop approach. Additionally, we demonstrate in a real time setting, where processing and actuation delays are taken into account, and show that our approach outperforms the current approaches in the delayed setting while picking fewer actions.
    Our work demonstrates that a temporally adaptive approach has similar benefits for AI as has been demonstrated in biology and is an important direction for future research in artificially intelligent control.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Interviews with 97 AI Researchers: Quantitative Analysis by Maheen Shermohammed
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interviews with 97 AI Researchers: Quantitative Analysis, published by Maheen Shermohammed on February 2, 2023 on The Effective Altruism Forum.
    TLDR: Last year, Vael Gates interviewed 97 AI researchers about their perceptions on the future of AI, focusing on risks from advanced AI. Among other questions, researchers were asked about the alignment problem, the problem of instrumental incentives, and their interest in AI alignment research. Following up after 5-6 months, 51% reported the interview had a lasting effect on their beliefs. Our new report analyzes these interviews in depth. We describe our primary results and some implications for field-building below. Check out the full report (interactive graph version), a complementary writeup describing whether we can predict a researcher’s interest in alignment, and our results below!
    [Link to post on LessWrong]
    Overview
    This report (interactive graph version) is a quantitative analysis of 97 interviews conducted in Feb-March 2022 with machine learning researchers, who were asked about their perceptions of artificial intelligence (AI) now and in the future, with particular focus on risks from advanced AI systems. Of the interviewees, 92 were selected from NeurIPS or ICML 2021 submissions, and 5 were informally recommended experts. For each interview, a transcript was generated, and common responses were identified and tagged to support quantitative analysis. The transcripts, as well as a qualitative walkthrough of common perspectives, are available at Interviews.
    Several core questions were asked in these interviews:
    When advanced AI (~AGI) would be developed (note that this term was imprecisely defined in the interviews)
    A probe about the alignment problem: “What do you think of the argument ‘highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous’?”
    A probe about instrumental incentives: “What do you think about the argument: ‘highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous’?”
    Whether interviewees were interested in working on AI alignment, and why or why not
    Whether interviewees had heard of AI safety or AI alignment
    Findings Summary
    Some key findings from our primary questions of interest:
    Most participants (75%), at some point in the conversation, said that they thought humanity would achieve advanced AI (imprecisely labeled “AGI” for the rest of this summary) eventually, but their timelines to AGI varied. Within this group:
    32% thought it would happen in 0-50 years
    40% thought 50-200 years
    18% thought 200+ years
    and 28% were quite uncertain, reporting a very wide range.
    (These sum to more than 100% because several people endorsed multiple timelines over the course of the conversation.) (Source)
    Among participants who thought humanity would never develop AGI (22%), the most commonly cited reason was that they couldn't see AGI happening based on current progress in AI. (Source)
    Participants were pretty split on whether they thought the alignment problem argument was valid. Some common reasons for disagreement were (Source):
    A set of responses that included the idea that AI alignment problems would be solved over the normal course of AI development (caveat: this was a very heterogeneous tag).
    Pointing out that humans have alignment problems too (so the potential risk of the AI alignment problem is capped in some sense by how bad alignment problems are for humans).
    AI systems will be tested (and humans will catch issues and implement safeguards before systems are rolled out in the real world).
    The objective function will not be designed in a way that causes the alignment problem / dangerous consequences of the alignment problem to arise.
    Perfe...
    12 min
  • LW - Schizophrenia as a deficiency in long-range cortex-to-cortex communication by Steven Byrnes
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Schizophrenia as a deficiency in long-range cortex-to-cortex communication, published by Steven Byrnes on February 1, 2023 on LessWrong.
    (Written in a hurry. I was almost going to title this “My poorly-researched pet theory of schizophrenia”. Hoping for feedback and pointers to relevant prior literature. I am very far from a schizophrenia expert.)
    1. What’s my hypothesis?
    My hypothesis is that the root cause of schizophrenia is (.drumroll.) a deficiency in medium- to long-range cortex-to-cortex connections. Some elaboration:
    When I say “deficiency”, I mean either “the connections aren’t there in their normal numbers” or “the connections are there, but for some reason they’re not accomplishing what they accomplish in neurotypical people”.
    When I say “cortex-to-cortex connections”, I think the main culprit is direct connections between Cortex Region A and Cortex Region B, but it’s also possible that the relevant thing is indirect connections between Cortex Region A and Cortex Region B, e.g. via the thalamus or cerebellum.
    When I say “medium- to long-range”, this definitely includes e.g. connections between different lobes, and it probably also includes connections across a few centimeters of cortex in humans. I haven’t really thought about what would happen if there was a deficiency in all connections of any length, including the very short ones, but I would weakly guess that this would present as schizophrenia as well.
    2. How did I originally come up with that hypothesis?
    I can pinpoint the exact moment: I was reading about visual processing abnormalities in schizophrenia, and more specifically the paper Weak Suppression of Visual Context in Chronic Schizophrenia (Dakin, Carlin, Hemsley 2005). They showed people pictures like this:
    The task was to match the image contrast in the red circle to one of the circles on the left. The eye-popping results were:
    the schizophrenics did better than the control group,
    .with a p-value of 0.0000002!
    Indeed, “12 out of 15 [subjects with schizophrenia] were more accurate than the most-accurate control.”
    .i.e., there were only three subjects with schizophrenia who did not outperform each and every one of the 33 people in the control groups. And one of those three was later re-diagnosed as not schizophrenic after all!
    Back to the task, the control group gets misled by the contrast level outside of the red circle, and the schizophrenic group didn’t. Here’s how I interpreted that:
    Each part of the visual cortex is trying to make sense of (more specifically, predict) the sensory inputs that it’s getting. To do a good job at that task, in normal life, it’s helpful to take account of information happening elsewhere in the visual field. After all, distant parts of the visual field are conveying information about ambient light levels, textures and slopes, and things like that. When a neurotypical person grows up viewing naturalistic images, they form lots of predictive models that are near-optimally incorporating all that distant “context” information, pulled in from all across the visual cortex and maybe elsewhere in the brain too. And that screws them up in this particular task. The goal of this task is to ignore all context and look at the red circle contents in isolation. The visual system does not have a set of predictive models that are tailored for accomplishing this task, so the control groups tend to do poorly.
    But schizophrenics are succeeding at the same task. Why? Well, presumably for them, the distant “context” information is not available in the first place. So their visual cortex is generally unable to leverage it for better predictive models. This makes their sensory predictions generally worse in a naturalistic setting. But it allows them to do better in this unnatural setting, where the contextual information is only there t...
    19 min
  • EA - Focus on the places where you feel shocked everyone's dropping the ball by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Focus on the places where you feel shocked everyone's dropping the ball, published by So8res on February 2, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • AF - Research agenda: Formalizing abstractions of computations by Erik Jenner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Research agenda: Formalizing abstractions of computations, published by Erik Jenner on February 2, 2023 on The AI Alignment Forum.
    Big thanks to Leon Lang, Jérémy Scheurer, Adam Gleave, and Shoshannah Tekofsky for their feedback on a draft of this post, to Euan McLean (via FAR AI) for his feedback and a lot of help with editing, and to everyone else who discussed this agenda with me, in particular Johannes Treutlein for frequent research check-ins!
    Summary
    My current agenda is to develop a formal framework for thinking about abstractions of computations. These abstractions are ways to partially describe the “algorithm” a neural network or other computation is using, while throwing away irrelevant details.
    Ideally, this framework would tell us 1) all possible abstractions of a given computation, and 2) which of these are most useful (for a specific purpose, such as detecting deception). “Useful” doesn’t necessarily mean “easily human-understandable”—I see that as an open question.
    I anticipate the main applications to alignment to be automating interpretability or mechanistic anomaly detection. There are also potential connections to other alignment topics, such as natural abstractions or defining terms like “search process”.
    This agenda is at an early stage (I have been thinking about it for ~2 months, and about related topics for another ~2 months before that). So feedback now could change my future direction.
    I also list a few potential projects that seem self-contained. If you’re interested in working on any of those, or collaborating in some other way, please get in touch!
    I encourage you to skip around and/or only read parts of the post. Here’s an overview:
    Introduction and Q&A mostly talk about motivation and connections to alignment..
    What are abstractions of computations? discusses my current guess as to what the framework should look like.
    There’s a list of Some potential projects.
    Appendix: Related work gives a quick overview of relevant work in academia, and the relation between this agenda and other alignment research.
    This post doesn’t contain any actual theorems or experiments, so if you’re only interested in that, you can stop reading.
    Introduction
    Humans can’t just look at the weights of a neural network and tell what it’s doing. There are at least two reasons for this:
    Neural network weights aren’t a format we’re great at thinking about.
    Neural networks are often huge.
    The second point would likely apply to any system that does well on complicated tasks. For example, neural networks are decision trees, but this doesn’t mean we can look at the decision tree corresponding to a network and understand how it works. To reason about these systems, we will likely have to simplify them, i.e. throw away details that are irrelevant for whatever we want to find out. In other words, we are looking for abstractions of computations (such as neural networks).
    Abstractions are already how we successfully reason about many other complicated systems. For example, if you want to understand the Linux kernel, you wouldn’t start by reading the entire source code top to bottom. Instead, you’d try to get a high-level understanding—what are the different modules, how do they interact? Similarly, we use pseudocode to more easily communicate how an algorithm works, abstracting away low-level details.
    If we could figure out a general way to find useful abstractions of computations, or at least of neural networks, perhaps we could apply this to understand them in a similar way. We could even automate this process and mechanically search for human-understandable abstractions.
    Making things easier to understand for humans isn’t the only application of abstractions. For example, abstractions have been used for more efficient theorem proving and in model checking (e.g. abstract inte...
    53 min
  • EA - Trends in the dollar training cost of machine learning systems by Ben Cottier
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Trends in the dollar training cost of machine learning systems, published by Ben Cottier on February 1, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • LW - Focus on the places where you feel shocked everyone's dropping the ball by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Focus on the places where you feel shocked everyone's dropping the ball, published by So8res on February 2, 2023 on LessWrong.
    Writing down something I’ve found myself repeating in different conversations:
    If you're looking for ways to help with the whole “the world looks pretty doomed” business, here's my advice: look around for places where we're all being total idiots.
    Look for places where everyone's fretting about a problem that some part of you thinks it could obviously just solve.
    Look around for places where something seems incompetently run, or hopelessly inept, and where some part of you thinks you can do better.
    Then do it better.
    For a concrete example, consider Devansh. Devansh came to me last year and said something to the effect of, “Hey, wait, it sounds like you think Eliezer does a sort of alignment-idea-generation that nobody else does, and he's limited here by his unusually low stamina, but I can think of a bunch of medical tests that you haven't run, are you an idiot or something?" And I was like, "Yes, definitely, please run them, do you need money".
    I'm not particularly hopeful there, but hell, it’s worth a shot! And, importantly, this is the sort of attitude that can lead people to actually trying things at all, rather than assuming that we live in a more adequate world where all the (seemingly) dumb obvious ideas have already been tried.
    Or, this is basically my model of how Paul Christiano manages to have a research agenda that seems at least internally coherent to me. From my perspective, he's like, "I dunno, man, I'm not sure I can solve this, but I also think it's not clear I can't, and there's a bunch of obvious stuff to try, that nobody else is even really looking at, so I'm trying it". That's the sort of orientation to the world that I think can be productive.
    Or the shard theory folks. I think their idea is basically unworkable, but I appreciate the mindset they are applying to the alignment problem: something like, "Wait, aren't y'all being idiots, it seems to me like I can just do X and then the thing will be aligned".
    I don't think we'll be saved by the shard theory folk; not everyone audaciously trying to save the world will succeed. But if someone does save us, I think there’s a good chance that they’ll go through similar “What the hell, are you all idiots?” phases, where they autonomously pursue a path that strikes them as obviously egregiously neglected, to see if it bears fruit. (Regardless of what I think.)
    Contrast this with, say, reading a bunch of people's research proposals and explicitly weighing the pros and cons of each approach so that you can work on whichever seems most justified. This has more of a flavor of taking a reasonable-sounding approach based on an argument that sounds vaguely good on paper, and less of a flavor of putting out an obvious fire that for some reason nobody else is reacting to.
    I dunno, maybe activities of the vaguely-good-on-paper character will prove useful as well? But I mostly expect the good stuff to come from people working on stuff where a part of them sees some way that everybody else is just totally dropping the ball.
    In the version of this mental motion I’m proposing here, you keep your eye out for ways that everyone's being totally inept and incompetent, ways that maybe you could just do the job correctly if you reached in there and mucked around yourself.
    That's where I predict the good stuff will come from.
    And if you don't see any such ways?
    Then don't sweat it. Maybe you just can't see something that will help right now. There don't have to be ways you can help in a sizable way right now.
    I don't see ways to really help in a sizable way right now. I'm keeping my eyes open, and I'm churning through a giant backlog of things that might help a nonzero amount—but I think it's importa...
    6 min
  • EA - Eli Lifland on Navigating the AI Alignment Landscape by Ozzie Gooen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Eli Lifland on Navigating the AI Alignment Landscape, published by Ozzie Gooen on February 1, 2023 on The Effective Altruism Forum.
    Recently I had a conversation with Eli Lifland about the AI Alignment landscape. Eli Lifland has been a forecaster at Samotsvety and has been investigating said landscape.
    I’ve known Eli for the last 8 months or so, and have appreciated many of his takes on AI alignment strategy.
    This was my first recorded video, so there were a few issues, but I think most of it is understandable.
    Full (edited) transcript below. I suggest browsing the section titles for a better overview of our discussion.
    Transcript
    Sections
    Samotsvety, a Recent Forecasting Organization
    Reading, “Is Power-Seeking AI an Existential Risk?”
    Categories of AI Failures: Accident, Misuse, and Structural
    Who Is Making Strategic Progress on Alignment?
    Community Building: Arguments For
    Community Building: Fellowships and Mentorship
    Cruxes in the AI Alignment Space
    Crux: How Promising is AI Interpretability?
    Crux: Should We Use Narrow AIs to Help Solve Alignment?
    The Need for AI Alignment Benchmarks
    Crux: Conceptual Insights vs. Empirical Iteration
    Vehicles and Planes as Potential Metaphors
    Samotsvety, a Recent Forecasting Organization
    Ozzie Gooen: So to get started, I want to talk a little bit about Samotsvety.
    Eli Lifland: It's a Russian name. Samotsvety currently has about 15 forecasters. We've been releasing forecasts for the community on topics such as nuclear risk and AI. We’re considering how to create forecasts for different clients and make public forecasts on existential risk, particularly AI.
    Team forecasting has been valuable, and I've encouraged more people to do it. We have a weekly call where we choose questions to discuss in advance. If people have time, they make their forecasts beforehand, and then we discuss the differences and debate. It's beneficial for team bonding, forming friendships, and potential future work collaborations.
    It's also interesting to see which forecasts are correct when they resolve. It's a good activity for different groups, such as AI community groups, to try.
    Ozzie Gooen: How many people are in the group right now?
    Eli Lifland: Right now, it's about 15, but on any given week, probably closer to five to ten can come. Initially, it was just us three. It was just Nuño, Misha, and I, and we would meet each weekend and discuss different questions on either Foretell (now INFER) or Good Judgment Open, but now it's five to ten people per week, from a total pool of 15 people.
    Ozzie Gooen: That makes sense. I know Samotsvety has worked on nuclear risk and a few other posts. What do you forecast when you're not working on those megaprojects?
    Eli Lifland: Yeah. We do a mix of things. Some things we've done for specific clients haven't been released publicly. Some things are still in progress and haven't been released yet. For example, we've been working on forecasting the level of AI existential risk for the Future Fund, now called the Open Philanthropy Worldview Prize, for the past 1-2 months. We meet each week to revise and discuss different ways to decompose the risk, but we haven't finished yet. Hopefully, we will.
    Sometimes we just choose a few interesting questions for discussion, even if we don't publish a write-up on them.
    Ozzie Gooen: So the idea is to have more people do very similar things, just like other teams are three to five, they're pretty independent; do you give them like coaching or anything? If I wanted to start my own group like this, what do I do?
    Eli Lifland: Feel free to reach out to any of us for advice on how we did it. As I mentioned, it was fairly simple—choosing and discussing questions each week.
    In terms of value, I believe it was valuable for all of us and many others who joined us. Some got more interested in effec...
    47 min
  • AF - More findings on Memorization and double descent by Marius Hobbhahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More findings on Memorization and double descent, published by Marius Hobbhahn on February 1, 2023 on The AI Alignment Forum.
    Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.
    I’d like to thank Wes Gurnee, Aryan Bhatt, Eric Purdy and Stefan Heimersheim for discussions and Evan Hubinger, Neel Nanda, Adam Jermyn and Chris Olah for mentorship and feedback.
    The post contains a lot of figures, so the suggested length is deceiving. Code can be found in these three colab notebooks [1][2][3].
    I have split the post into two parts. The first one is concerned with double descent and other general findings in memorization and the second focuses on measuring memorization using the maximum data dimensionality metric. This is the first post in a series of N posts on memorization in transformers.
    Executive summary
    I look at a variety of settings and experiments to better understand memorization in toy models. My primary motivation is to increase our general understanding of NNs but I also suspect that understanding memorization better might increase our ability to detect backdoors/trojans.
    The work heavily builds on two papers by Anthropic, “Toy models of superposition” and “Superposition, Memorization and double descent”. I successfully replicate a subset of their findings.
    I specifically look at three different setups of NNs that I speculate are most relevant to understanding memorization in the non-attention parts of transformers.
    Bottlenecks between layers, i.e. when projecting from high-dimensional spaces (e.g. MLPs) into lower dimensions (e.g. the residual stream). This is similar to the setting in the toy models of superposition paper and its sequel.
    MLP blocks, i.e. when projecting from lower-dimensional spaces (e.g. the residual stream) into higher dimensions with ReLU non-linearities.
    The final layer, i.e. when projecting from the end of the residual stream into the vocab space. The main difference to the previous scenarios is that we use the cross-entropy loss for the experiments which has a different inductive bias than the MSE loss.
    I’m able to find the double descent phenomenon in all three settings. My takeaway from this is that the transition between memorization and learning general features seems to be a very regular and predictable phenomenon (assuming you know the sparsity and number of features of your network). Furthermore, it seems like the network is “confused” (e.g. has much higher test loss) when it is right between memorization and generalization.
    I test the limits of reconstruction in different settings, i.e. the ability of the neural network to reconstruct its inputs given different dataset sizes, hidden sizes, number of features, importance distributions and sparsities. The findings mostly confirm what we would predict, e.g. more sparsity or larger hidden sizes lead to better reconstructions. A speculative claim is that if we had better measures of sparsity and importance in real-world models, we might be able to derive scaling laws that could tell us how many “concepts” a network has learned.
    Interpreting NNs that memorized in the simplest settings is extremely straightforward--the network literally creates a dictionary that you can just read off the weights. However, even small increases in complexity make this dictionary much harder to read and I have not yet found a method to decompile it into a human-readable form (maybe in the next posts).
    Isolated components
    In the following, we isolate three settings that seem like important components of memorization. They are supposed to model the non-attention parts of a transformer (primarily because I speculate that memorization mostly happens in the non-attention parts).
    Bottleneck
    By bottleneck we mean a situation in which a model projects from many into fewer dimensi...
    31 min