The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Advice I found helpful in 2022 by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Advice I found helpful in 2022, published by Akash on January 28, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - EA is going through a bunch of conflict. Here’s some social technologies that may help. by Severin T. Seehrich
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA is going through a bunch of conflict. Here’s some social technologies that may help., published by Severin T. Seehrich on January 29, 2023 on The Effective Altruism Forum.
    There’s two ways I can see EA develop after the latest community drama: The community can have all the necessary debates in good faith, grow past them, and become a more mature movement. Or, neither side feels heard, nothing changes, and way too many competent people lose trust in EA and focus their energies elsewhere. After all, I’ve heard people from either side declare EA’s moral bankruptcy in consequence of the Bostrom email debate.
    In order to shift the balance towards a little bit more of movement maturation and a little bit less of quitting, we need to talk. The conversations to be had here are between you and me, between him and them, and not between a small band of anonymous rebels on the one side and the EA establishment at CEA on the other. The more isolated and unheard people feel in their opinions, the more angry and polarized things get. Starting to pop the bubbles starts right in our own local EA groups.
    For that, I’d like to share some tools that I found useful for resolving conflict in groups. It might be worthwhile to explore them on your own, and it might be even more worthwhile to spread them in your local EA group.
    But how does this fit into our other community building work?
    To answer this question, I’d like to summarize my favorite EAF post on community building of all times: Jan Kulveit’s “Different forms of capital”.
    He claims that there are different forms of capital we can optimize for in community building, and that there are two EA tends to over-optimize for:
    Financial capital: The total amount of donations committed and sent to our charities of choice.
    Human capital: The number of active community members and their level of commitment.
    Further, he argues, there are two other forms of capital EA tends to neglect, probably because they are harder to measure:
    Network capital: The number and closeness of ties between community members.
    Structural capital: The institutions and processes we have in place for doing stuff in a sensibly structured manner.
    And I’d like to add a fifth dimension:
    Memetic capital: The sum total of the useful ideas, psycho- and social technologies we have widely available in the community. An instance of memetic capital without which EA would be unthinkable is the capability to use language. Some other examples for memetic capital include knowledge about priorization, forecasting, or productivity tools, about where and how to apply for grants, how we think about mental health and staying productive in the long-term - and our strategies for addressing and resolving conflict with friends, colleagues, and fellow EAs.
    I think it would be wise for EA community building to start explicitly taking all five of these factors into account. In line with that, this post aims at increasing the community’s memetic capital. Trying these tools one-off to resolve a particular issue is good, but what I’m hoping for is that over time, they just become part of the way we do things. That way, they’d have the strongest and long-lasting positive impact on our network capital through frequent and casual prevention and repair of conflict.
    Social technologies for resolving and preventing conflict
    Repair
    Rule 0 (coined by Seek Healing)
    WHY: The longer we sit on irritations, the bigger they grow, until at some point we write very, very long and elaborate EA Forum posts. Addressing points of conflict sooner rather than later helps create less friction and improve feedback loops in groups of people, no matter how uncomfortable it initially is.
    HOW: Rule 0 is a rule of thumb that goes like this: “If you feel queasy about addressing something with somebody, that’s a sign that y...
    10 min
  • AF - Structure, creativity, and novelty by Tsvi Benson-Tilsen
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Structure, creativity, and novelty, published by Tsvi Benson-Tilsen on January 29, 2023 on The AI Alignment Forum.
    [Metadata: crossposted from. First completed 26 June 2022. I'm likely to not respond to comments promptly.]
    A high-level confusion that I have that seems to be on the way towards understanding alignment, is the relationship between values and understanding. This essay gestures at the idea of structure in general (mainly by listing examples).
    Why do we want AGI at all?
    We want AGI in order to understand stuff that we haven't yet understood.
    (This is not a trivial claim. It might be false. It could be that to secure the future of humane existence, something other than understanding is necessary or sufficient; e.g. it's conceivable that solving some large combinatorial problem, akin to playing Go well or designing a protein by raw search with an explicit criterion, would end the acute risk period. But I don't know how to point at such a thing--plans I know how to point at seem to centrally involve understanding that we don't already have.)
    Elements and structure
    Understanding implies some kind of structure. (This is a trivial claim, or a definition: structure is what a mind is or participates in, when it understands.) Structure is made of elements. "Structure" is the mass noun of, or continuous substance version of, "element". The point of the word "element" is just to abbreviate "any of that pattern-y, structure-y stuff, in a mind or in the world in general".
    Elements. An element (of a mind) is anything that combines to constitute the mind, at any level of organization or description.
    Examples of elements. Any instance within a mind of any of the following categories is an element: features, aspects, properties, parts, components, subagents, pieces, inputs, algorithms, code, processes, concepts, ideas, skills, methods, procedures, values, goals, architecture, modules, thoughts, propositions, beliefs, probabilities, principles, rules, axioms, heuristics, plans, operations, connections, associations, metaphors, abstractions, memories, arguments, reasons, purposes, modes, emotions, tendencies, organs, ingredients, functions, dynamics, structures, data, types, languages, proofs, justifications, motives, images, searches, knowledge, computations, rewards, reinforcement, specifications, information, intuitions, ideologies, protocols, stimuli, responses, domains, gradients, objective functions, optimizers, satisficers, control systems, basins of attraction, tasks, attitudes, stances, dispositions, words, terms, definitions, nexi, drives, perceptions, grammar, criteria,
    possibilities, combinations, categories, inferences, actions.
    Examples of elements.
    Any instance within a mind of any of the following categories is an element: features, aspects, properties, parts, components, subagents, pieces, inputs, algorithms, code, processes, concepts, ideas, skills, methods, procedures, values, goals, architecture, modules, thoughts, propositions, beliefs, probabilities, principles, rules, axioms, heuristics, plans, operations, connections, associations, metaphors, abstractions, memories, arguments, reasons, purposes, modes, emotions, tendencies, organs, ingredients, functions, dynamics, structures, data, types, languages, proofs, justifications, motives, images, searches, knowledge, computations, rewards, reinforcement, specifications, information, intuitions, ideologies, protocols, stimuli, responses, domains, gradients, objective functions, optimizers, satisficers, control systems, basins of attraction, tasks, attitudes, stances, dispositions, words, terms, definitions, nexi, drives, perceptions, grammar, criteria, possibilities, combinations, categories, inferences, actions.
    How elements are. Mental elements overlap, crisscross, lie on spectra, control, use, associate w...
    17 min
  • LW - Spooky action at a distance in the loss landscape by jhoogland
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Spooky action at a distance in the loss landscape, published by jhoogland on January 28, 2023 on LessWrong.
    Produced as part of the SERI ML Alignment Theory Scholars Program Winter 2022 Cohort.
    Not all global minima of the (training) loss landscape are created equal.
    Even if they achieve equal performance on the training set, different solutions can perform very differently on the test set or out-of-distribution. So why is it that we typically find "simple" solutions that generalize well?
    In a previous post, I argued that the answer is "singularities" — minimum loss points with ill-defined tangents. It's the "nastiest" singularities that have the most outsized effect on learning and generalization in the limit of large data. These act as implicit regularizers that lower the effective dimensionality of the model.
    Even after writing this introduction to "singular learning theory", I still find this claim weird and counterintuitive. How is it that the local geometry of a few isolated points determines the global expected behavior over all learning machines on the loss landscape? What explains the "spooky action at a distance" of singularities in the loss landscape?
    Today, I'd like to share my best efforts at the hand-waving physics-y intuition behind this claim.
    It boils down to this: singularities translate random motion at the bottom of loss basins into search for generalization.
    Random walks on the minimum-loss sets
    Let's first look at the limit in which you've trained so long that we can treat the model as restricted to a set of fixed minimum loss points.
    Here's the intuition pump: suppose you are a random walker living on some curve that has singularities (self-intersections, cusps, and the like). Every timestep, you take a step of a uniform length in a random available direction. Then, singularities act as a kind of "trap." If you're close to a singularity, you're more likely to take a step towards (and over) the singularity than to take a step away from the singularity.
    It's not quite an attractor (we're in a stochastic setting, where you can and will still break away every so often), but it's sticky enough that the "biggest" singularity will dominate your stable distribution.
    In the discrete case, this is just the well-known phenomenon of high-degree nodes dominating most of expected behavior of your graph. In business, it's behind the reason that Google exists. In social networks, it's similar to how your average friend has more friends than you do.
    To see this, consider a simple toy example: take two polygons and let them intersect at a single point. Next, let a random walker run loose on this setup. How frequently will the random walker cross each point?
    If you've taken a course in graph theory, you may remember that the equilibrium distribution weights nodes in proportion to their degrees. For two intersecting lines, the intersection is twice as likely as the other points. For three intersecting lines, it's three times as likely, and so on.
    Now just take the limit of infinitely large polygons/step size to zero, and we'll recover the continuous case we were originally interested in.
    Brownian motion near the minimum-loss set
    Well, not quite. You see, restricting ourselves to motion along the minimum-loss points is unrealistic. We're more interested in messy reality, where we're allowed some freedom to bounce around the bottoms of loss basins.
    This time around, the key intuition-pumping assumption is to view the behavior of stochastic gradient descent late in training as a kind of Brownian motion. When we've reached a low training-loss solution, variability between batches is a source of randomness that no longer substantially improves loss but just jiggles us between solutions that are equivalent from the perspective of the training set.
    To understand these dy...
    6 min
  • LW - LW Filter Tags (Rationality/World Modeling now promoted in Latest Posts) by Ruby
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LW Filter Tags (Rationality/World Modeling now promoted in Latest Posts), published by Ruby on January 28, 2023 on LessWrong.
    In case you missed it, you can customize your Latest Posts feed on the frontpage based on tag. That is, you can promote or reduce the prominence of posts based on their tags such AI, Rationality, Practical, etc.
    LessWrong's Core Tags are suggested as filters by default, but any tag can be used to limit or promote the visibility of posts. Core Tags are special in both being high-level and in having at least one applied to almost every post (moderators ensure this), meaning they're especially useful for tailoring your Latest Posts.
    Improved UI
    We just altered the UI for this to be both more visible so people notice the feature, and more intuitive (hopefully) so how to use it is more immediately obvious.
    Below the Latest Posts heading is a row of Tag Filter buttons. By default, they're displayed for the Core Tags but you can use '+' button to add any existing tag. Hovering your mouse on the button brings up a description of the tag (rewritten to be clearer) and options for changing the visibility:
    Hidden: posts with will not display in Latest Posts
    Reduced: posts with will be displayed as though they had 50% of their actual karma
    Default: posts with are displayed normally
    Promoted: posts with are displayed as though they had +25 karma. This has a larger effect on low-karma posts
    Other: select your own modifier. Values less than 1 are multiplicative, greater than 1 are additive
    (I think it'd be better if modifiers were multiplicative both for promoting and reducing visibility but things ended up this way for historical reasons and it's not the right resource allocation to change that right now.)
    Rationality and World Modeling are promoted by default
    We've just made a change so that posts tagged Rationality and World Modeling are by default given a +10 karma boost in the Latest Posts list (e.g. treated as though they have 10 more karma than they actually do).
    This only applies to users who have not already configured their Frontpage Tag Filter settings. If you had already adjusted your filters, they will not be changed.
    If you don't like these changes, you can easily change them to what you'd like.
    Why did we boost Rationality and World Modeling?
    As elaborated in So, geez there's a lot of AI content these days, there's been a real uptick in AI and Alignment related content on LessWrong in the last year. This is good! My impression is much of the "deluge" of AI/Alignment content is valuable progress on Alignment, arguably the most important problem in the world right now. It's pretty cool that LessWrong is host to that discussion, and I think all the time about how the LessWrong mod/dev team can make that discussion go better.
    At the same, it feels important to for new and old users to remember that LessWrong is not defined by its interest in AI. LessWrong's core mission is perfecting the Art of Human Rationality, and making intellectual progress, particularly on the world's most important problems. It happens to be that AI is a very important problem right now, but in worlds where something else was the problem, I hope LessWrong's attention would be focused there.
    It's also the case that I expect (not that I'm not the most qualified to opine) that successful Alignment solutions will come from those who have studied broadly, rather than those who've narrowly focused on "AI", making it more appropriate that LessWrong not limit itself.
    This past year, we've taken multiple actions to reinforce LessWrong's identity as having generalized values of rationality/truth-seeking/growth-mindset/improving-the-world at its core. For example, we made Sequences Highlights as a less intimidating way to start reading The Sequences that intimidating in length. We also...
    8 min
  • EA - Giving Multiplier Paper by James Montavon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Giving Multiplier Paper, published by James Montavon on January 28, 2023 on The Effective Altruism Forum.
    The Joshua Greene and Lucius Caviola article about their givingmultiplier.org work has just been published. Here's the abstract:
    The most effective charities are hundreds of times more impactful than typical charities. However, most donors favor charities with personal/emotional appeal over effectiveness. We gave donors the option to split their donations between their personal favorite charity and an expert-recommended highly effective charity. This bundling technique increased donors’ impact without undermining their altruistic motivation, boosting effective donations by 76%. An additional boost of 55% was achieved by offering matching donations with increasing rates for allocating more to the highly effective charity. We show further that matching funds can be provided by donors focused on effectiveness through a self-sustaining process of micromatching. We applied these techniques in a new online donation platform (GivingMultiplier.org), which fundraised more than $1.5 million in its first 14 months.
    While prior applied research on altruism has focused on the quantity of giving, the present results demonstrate the value of focusing on the effectiveness of altruistic behavior.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - On not getting contaminated by the wrong obesity ideas by Natália Coelho Mendonça
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On not getting contaminated by the wrong obesity ideas, published by Natália Coelho Mendonça on January 28, 2023 on LessWrong.
    A Chemical Hunger (a), a series by the authors of the blog Slime Mold Time Mold (SMTM), argues that the obesity epidemic is entirely caused (a) by environmental contaminants.
    In my last post, I investigated SMTM’s main suspect (lithium). This post collects other observations I have made about SMTM’s work, not narrowly related to lithium, but rather focused on the broader thesis of their blog post series.
    I think that the environmental contamination hypothesis of the obesity epidemic is a priori plausible. After all, we know that chemicals can affect humans, and our exposure to chemicals has plausibly changed a lot over time. However, I found that several of what seem to be SMTM’s strongest arguments in favor of the contamination theory turned out to be dubious, and that nearly all of the interesting things I thought I’d learned from their blog posts turned out to actually be wrong. I’ll explain that in this post.
    Moreover, I express concern about how the SMTM authors have misrepresented their sources and data quite a few times and have refused to remove inaccuracies from their blog posts after being alerted of them.
    Summary of main points
    It’s not clear that either lab animals or wild animals have been getting more obese.
    There wasn’t an abrupt shift in obesity rates in the late 20th century. People in the United States have been pretty much gradually getting heavier since the early 20th century.
    There is good evidence, which the SMTM authors have never addressed, that the lower obesity prevalence at high altitudes is caused by the lower atmospheric pressure at those altitudes (rather than by the relative absence of environmental contaminants, which is their chosen explanation).
    Demographic factors and altitude might adequately explain the disparities in obesity rates across US states. Contaminants don’t seem necessary to explain them.
    Being underweight is less common than it used to be, not more.
    CICO is compatible with body weight regulation and is consistent with the findings of overfeeding studies (though CICO cannot by itself explain the obesity epidemic.)
    It’s not clear that lab animals or wild animals have been getting fatter
    When I was reading A Chemical Hunger, the argument I found most compelling was the following (from Part I: Mysteries (a)):
    Humans aren’t the only ones who are growing more obese — lab animals and even wild animals are becoming more obese as well. Primates and rodents living in research colonies, feral rodents living in our cities, and domestic pets like dogs and cats are all steadily getting fatter and fatter. This can’t be attributed to changes in what they eat, because lab animals live in contained environments with highly controlled diets. They’re being fed the same foods as always, but for some reason, they’re getting fatter.
    The interesting claims here are that lab animals on controlled diets and wild animals have allegedly gotten fatter. So I decided to investigate those claims.
    The hyperlink in that quote leads you to a 2010 study by Klimentidis et al. which, as far as I can tell, is the only study to have investigated the issue. Googling “are lab animals getting fatter” yields a bunch of articles that all seem to cite that one paper as their source (except for a few results that don’t seem related to the question). I combed through the paper’s 179 Google Scholar citations and could not find a replication.
    (By the way, contrary to what SMTM’s writing implies, this paper did not analyze the weight of wild animals, as we’ll discuss below).
    Since Klimentidis et al. (2010)’s results have apparently not been replicated, I decided to attempt to replicate them myself.
    Lab mice
    I first thought this would be extreme...
    55 min
  • EA - OpenBook: New EA Grants Database by Rachel Weinberg
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: OpenBook: New EA Grants Database, published by Rachel Weinberg on January 28, 2023 on The Effective Altruism Forum.
    openbook.fyi is a new website where you can see ~4,000 EA grants from donors including Open Phil, FTX Future Fund, and EA Funds in a single place.
    Why?
    If you're a donor: OpenBook shows you how much orgs have already received, and where other donors you respect have contributed their money.
    If you're a grant applicant: OpenBook shows you what kinds of projects your funders have previously sponsored, and also who funds projects similar to your own.
    If you're neither: browse around to get a sense of how money flows in EA!
    Features
    Right now, you can:
    search through all grants by donor, recipient, and cause area
    go to an organization's page and see all grants they've given and received, plus in some cases organization details (e.g. country, GiveWell review)
    view donation details (e.g. intended use of funds, notes)
    see largest donors and recipients on file
    add a donation that we're missing
    Features I'm thinking of adding soon:
    allow edits to existing donations
    standardize cause areas and add cause area pages for the most common ones where you can view and search all the grants associated with that cause area
    add ACX Grants to database
    allow bulk donation uploads through a CSV
    let users claim a page and create an account where they can add their own donations and disable adding of others
    allow for comments on organizations/grants
    improve the mobile layout
    make color theme changeable (just for fun)
    Please let me know which of these features you'd use, or if there are other features you'd like!
    Credits
    Huge thanks to Vipul Naik and Issa Rice for compiling most of the data for their donations site and making it open source. This meant I mostly just had to build the UI.
    And to Austin Chen for the idea, some code, and the many hours spent debugging with me.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Reflections on Deception & Generality in Scalable Oversight (Another OpenAI Alignment Review) by Shoshannah Tekofsky
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on Deception & Generality in Scalable Oversight (Another OpenAI Alignment Review), published by Shoshannah Tekofsky on January 28, 2023 on LessWrong.
    Just like you can test your skill in experimental design by reviewing existing experiments, you can test your skill in alignment by reviewing existing alignment strategies. Conveniently, Rob Bensinger, in name of Nate Soares and Eliezer Yudkowsky, recently posted a challenge to AI Safety researchers to review the OpenAI alignment plan written by Jan Leike, John Schulman, and Jeffrey Wu. I figured this constituted a test that might net me feedback from both sides of the rationalist-empiricist aisle. Yet, instead of finding ground-breaking arguments for or against scalable oversight to do alignment research, it seems Leike already knows what might go wrong — and goes ahead anyway.
    Thus my mind became split between evaluating the actual alignment plan and modeling the disagreement between prominent clusters of researchers. I wrote up the latter in an informal typology of AI Safety Researchers, and continued my technical review below. The following is a short summary of the OpenAI alignment plan, my views on the main problems, and a final section on recommendations for red lining.
    The Plan
    First, align AI with human feedback, then get AI to assist in giving human feedback to AI, then get AI to assist in giving human feedback to AI that is generating solutions to the alignment problem. Except, the steps are not sequential but run in parallel. This is one form of Scalable Oversight.
    Human feedback is Reinforcement Learning from Human Feedback (RLHF), the assisting AI is Iterated Distillation and Amplification (IDA) and Recursive Reward Modeling (RRM), and the AI that is generating solutions to the alignment problem is. still under construction.
    The target is a narrow AI that will make significant progress on the alignment problem. The MVP is a theorem prover. The full product is AGI utopia.
    Here is a graph.
    OpenAI explains its strategy succinctly and links to detailed background research. This is laudable, and hopefully other labs and organizations will follow suit. My understanding is also that if someone came along with a better plan then OpenAI would pivot in a heart beat. Which is even more laudable. The transparency, accountability, and flexibility they display set a strong example for other organizations working on AI.
    But the show must go on (from their point of view anyway) and so they are going ahead and implementing the most promising strategy that currently exists. Even if there are problems.
    And boy, are there problems.
    The Problems
    Jan Leike discusses almost all objections to the OpenAI alignment plan on his blog. Thus below I will only highlight the two most important problems in the plan, plus two additional concerns that I have not seen discussed so far.
    Alignment research requires general intelligence - If the alignment researcher AI has enough general intelligence to make breakthrough discoveries in alignment, then you can't safely create it without already having solved alignment. Yet, Leike et al. hope that relatively narrow intelligence can already make significant progress on alignment. I think this is extremely unlikely if we reflect on what general intelligence truly is. Though my own thoughts on the nature of intelligence are not entirely coherent yet, I'd argue that having a strong concept of intelligence is key to accurately predicting the outcome of an alignment strategy.
    Specifically in this case, my understanding is that general intelligence is being able to perform a wider set of operations on a wider set of inputs (to achieve a desired set of observations on the world state). For example, I can do addition of 2 apples I see, 2 apples I think about, 2 boats I hear about, 2 functions ...
    13 min
  • LW - Scaling Laws Literature Review by Pablo Villalobos
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Scaling Laws Literature Review, published by Pablo Villalobos on January 27, 2023 on LessWrong.
    Common shape of a scaling law, taken from Hestness et al. (2017)
    Executive summary
    Scaling laws are predictable relations between the scale of a mode and performance or other useful properties.
    I have collected a database of scaling laws for different tasks and architectures, and reviewed dozens of papers in the scaling law literature.
    My main takeaways are:
    Functional forms: a basic power law can effectively model the scaling behavior in the power-law region but not the transitions to the other two regions. For this, either the M4 estimator or the BNSL estimator introduced below seem to be the best options right now.
    Transfer learning: there is not a simple universal scaling law for transfer learning between arbitrary tasks. When the tasks are similar enough, upstream loss and downstream performance are closely related, but when tasks are very different, the details of the architecture and hyperparameters become very relevant.
    See the full table of scaling laws here.
    Introduction
    The term “scaling laws” in deep learning refers to relations between functional properties of interest (usually the test loss or some performance metric for fine-tuning tasks) and properties of the architecture or optimization process (like model size, width, or training compute). These laws can help inform the design and training of deep learning models, as well as provide insights into their underlying principles.
    In this document, I present a detailed table outlining the known scaling laws and a summary of each paper's contributions to the understanding of scaling in deep learning.
    My main goal for this document is to serve as a comprehensive and up-to-date resource for information on scaling laws in deep learning. By presenting a curated list of papers that have explored various scaling laws, and providing a detailed table outlining the known scaling laws and a summary of each paper's contributions, I aim to make it easier to access and understand the current state of knowledge on scaling in deep learning.
    I selected the papers using the following criteria:
    Publication date: I only focus on recent papers, and exclude work prior to 2015
    Contribution: I include papers which contribute in one of these ways:
    Reporting empirical scaling laws
    Proposing better functional forms or fitting methods
    Proposing theoretical models to explain scaling behavior
    Connecting scaling behavior with lower-level properties of models
    Overview
    While the scaling behavior of machine learning models has been studied for a long time (e.g., Statistical Mechanics of Learning from Examples), it was only recently that empirical research into scaling deep learning models became widely known and incorporated into practice.
    Previous theoretical analyses often predicted that the test loss would decrease as a power law of training data, L=BD−b+E, with exponent b=1 or 1/2. However, this clashes with empirical results, in which the scaling exponent is usually smaller than 1/2.
    The modern study of scaling laws arguably started with Hestness et al. (2017), who empirically identified power-law scaling of the test loss with respect to training data size in several different domains. In Hestness et al. (2019) this previous result was used to predict the increases in model and dataset sizes that would be needed to reach important performance milestones.
    Shortly after, Rosenfeld et al. (2020) constructed a joint error function with respect to model and data sizes, given by L=AN−a+BD−b+E and showed that this form could accurately reproduce the empirical error landscape.
    During 2020 and 2021, our understanding of scaling laws was greatly expanded:
    Henighan et al. (2020) found scaling laws for more tasks and architectures.
    Kaplan et al. (2020...
    7 min