The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - What's A "Market"? by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What's A "Market"?, published by johnswentworth on August 9, 2023 on LessWrong.
    Economists have a very mathematically clean class of models of "markets", and spill lots of ink arguing about how well this class of models applies to the markets of the real-world economy. I personally give relatively few shits about how well the mathematical notion of a market applies to real-world economic markets; I'm relatively more interested in applying the same models to systems in biology or ML/AI. They're very generalizable models.
    Unfortunately, the mathematical notion of a "market" tends to be presented in math-heavy econ courses, and the parts I'd consider most central typically see surprisingly little coverage in more conceptual intro courses. So, this post aims to explain what I consider the central concepts of the mathematical notion of a market, without all the associated notation and jargon and proofs, in a way which lends itself to generalization beyond economics.
    The Story About Apples And Bananas
    We've got two people, Alice and Bob. Each of them can produce two goods, apples and bananas. Alice can use her land to produce five tons of apples, or one ton of bananas, or some proportional combination of the two. Bob can use his land to produce five tons of apples or twenty tons of bananas, or some proportional combination of the two. Both want a varied diet of apples and bananas.
    . and you remember from econ 101 roughly how this goes, right? If the two just produce food for themselves separately, then each grows a mix of apples and bananas. But then Alice' opportunity cost for one apple is 1/5 = 0.2 tons of bananas, whereas Bob's opportunity cost for one apple is 20/5 = 4 tons bananas. So, the two could produce a pareto gain of apples and bananas by specializing: Alice can specialize more toward apple production, Bob can specialize more towards banana production. For instance, if Alice shifts production toward 1 more ton of apples, while Bob shifts production toward 1 less ton of apples, then together they produce - 0.21 + 41 = 3.8 tons more bananas with the same number of apples.
    Now the key question for this post: when does this sort of specialization reach equilibrium? Under what conditions do Alice and Bob together decide that they've both specialized the correct amount, and don't need to shift their production around any more?
    In this exact example, they'll only hit equilibrium once one of them is fully specialized - either Bob fully specialized in apples, or Alice fully specialized in bananas. Otherwise, they could always do better by specializing more. But in general, decreasing marginal returns might mean that both should be less-than-fully specialized - e.g. maybe both have some land better suited to apples and some better suited to bananas, so as they shift production their opportunity costs change.
    So when will the two "reach equilibrium"? Well, when their opportunity costs are the same - i.e. when they have the same tradeoff between producing apples vs bananas.
    . and that's a market.
    More generally, we have:
    A bunch of agents, and a bunch of goods.
    Each agent has their own opportunity cost for each good, or marginal trade-off rate between goods.
    At equilibrium, the trade-off rates are the same for all agents (otherwise they can achieve a pareto improvement by specializing more).
    The "market" is the set of agents at equilibrium, and the "market prices" are the (shared) trade-off rates between goods.
    Another Story: Stock Traders
    We have a bunch of stock traders, each with a portfolio of stocks and cash, and a utility function over their portfolio which they seek to maximize. (We'll assume, for simplicity, that the traders are not updating their beliefs over the course of this story, so we can ignore the "expectation" part of their expected utility maximization.)...
    10 min
  • EA - The Hinge of History Hypothesis: Reply to MacAskill (Andreas Mogensen) by Global Priorities Institute
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Hinge of History Hypothesis: Reply to MacAskill (Andreas Mogensen), published by Global Priorities Institute on August 8, 2023 on The Effective Altruism Forum.
    This paper was originally published as a working paper in August 2022 and is forthcoming in Analysis.
    Abstract
    Some believe that the current era is uniquely important with respect to how well the rest of human history goes. Following Parfit, call this the Hinge of History Hypothesis. Recently, MacAskill has argued that our era is actually very unlikely to be especially influential in the way asserted by the Hinge of History Hypothesis. I respond to MacAskill, pointing to important unresolved ambiguities in his proposed definition of what it means for a time to be influential and criticizing the two arguments used to cast doubt on the claim that the current era is a uniquely important moment in human history.
    Introduction
    Some believe that the current era is a uniquely important moment in human history. We are living, they claim, at a time of unprecedented risk, heralded by the advent of nuclear weapons and other world-shaping technologies. Only by responding wisely to the anthropogenic risks we now face can we survive into the future and fulfil our potential as a species (Sagan 1994; Parfit 2011, Bostrom 2014, Ord 2020).
    Following Parfit (2011), call the hypothesis that we live at such a uniquely important time the Hinge of History Hypothesis (3H). Recently, MacAskill (2022) has argued that 3H is "quite unlikely to be true." (332) He interprets 3H as the claim that "[w]e are among the very most influential people ever, out of a truly astronomical number of people who will ever live" (339) and defines a period of time as influential in proportion to "how much expected good one can do with the direct expenditure (rather than investment) of a unit of resources at [that] time" (335), where 'investment' may refer "to both financial investment, and to using one's time to grow the number of people who are also impartial altruists." (335 n.13) MacAskill thus relates the truth or falsity of 3H to the practical question of the optimal time at which to expend resources to achieve morally good outcomes, considered impartially.
    MacAskill presents two arguments against 3H. The first is an argument that the prior probability that we are living at the most influential time in history should be very low, because we should reason as if we represent a random sample from observers in our reference class. The second is an inductive argument that we should expect future people to have more influence over human history because the overall trend throughout human history is for later generations to be more influential.
    In my view, neither of these arguments should convince us. As I argue in section 2, MacAskill's priors argument relies on formulating 3H in a way that does not conform to how this hypothesis is traditionally understood. Moreover, I will argue in section 3 that MacAskill's definition of what it means for a time to be influential leaves too many unresolved ambiguities for his inductive argument to work.
    Read the rest of the paper
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    4 min
  • EA - Getting into an EA-aligned organisation mid-career by Patrick Gruban
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Getting into an EA-aligned organisation mid-career, published by Patrick Gruban on August 8, 2023 on The Effective Altruism Forum.
    Summary
    In this article, I'll share my journey of joining an effective altruism (EA) organisation in the middle of my career. I'll talk about the misconceptions I initially held, how I came to understand the fundamental ideas of EA, increased my involvement in the community, and started connecting with more experienced members. I will also share how I became more active through volunteer work and then intentionally worked towards transitioning my career to have more impact.
    Introduction
    Within the EA community, you will find many young individuals who started engaging with EA ideas during their university years, devoting significant time to podcasts and articles. While the movement has a large youth presence, seasoned professionals are less common. EA-aligned organisations are eager for professionals who embody their core values. However, immersing oneself in these values can be a challenge if you're already occupied with a full-time job and commitments to family and friends.
    I was introduced to EA at the age of 40 and without a university degree. At that point, I couldn't see a clear path to working directly in a high-impact area. Yet, eight years later, I became the Co-Director at EA Germany, guiding individuals on their quest for more impactful career paths. Looking back, I believe I could have made the career transition sooner had I had fewer misunderstandings and focused more on specific areas.
    My Misconceptions about EA
    When I initially began attending my local EA group meetings, I had misconceptions about the requisites for deeper engagement with the EA community. I erroneously believed that it required a complete dedication to veganism, a commitment to donating at least 10% of income, and a lifestyle wholly centred around maximising impact.
    One of my first experiences with the group involved a discussion on animal welfare. As the co-founder of a company that promotes higher animal-welfare yarn from sheep's wool, I felt aligned with their mission. Yet, I noticed that many participants advocated for an entirely vegan lifestyle, excluding all animal products. Their backgrounds, largely consisting of younger students, also seemed to widen the gap of understanding between us.
    Later, I met individuals who committed to donating 10% of their income to impactful charities. I admired their dedication but was uncertain if I could match that contribution level. I also came across people who believed that EA demanded a life entirely devoted to maximising the overall good.
    However, as time passed, I understood these were individual choices rather than a standard or norm within the EA community. I met actively involved EA members who were not vegans, concentrated more on their careers than on donations, and enjoyed leisure activities unrelated to their impact work. I eventually concluded that, for me, prioritising a career change held more significance than modifying my lifestyle.
    Core Values of EA
    While it is true that choices are individual, and people have different values and priorities in their private lives, EA has principles that most people follow. These include prioritisation, impartial altruism, open truth-seeking, and a collaborative spirit. Having individuals who adhere to these values in the community and organisations fosters easier collaboration, as there is a higher level of trust and underlying agreement compared to society at large.
    My previous experiences in business and political activism taught me to present myself in the best light, select beliefs that supported my purpose, and prioritise self-advocacy over collaboration. Engaging more with the EA community required being open to changing my beliefs, practising epistemic hu...
    12 min
  • EA - Fundamentals of Global Priorities Research in Economics Syllabus by poliboni
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fundamentals of Global Priorities Research in Economics Syllabus, published by poliboni on August 8, 2023 on The Effective Altruism Forum.
    This is a 6-9 session syllabus on the fundamentals of global priorities research in economics.
    The purpose is to help economics students and researchers interested in GPR get a big picture view of the field and come up with research ideas.
    Because of this focus on fundamentals, the readings are rather light on economics and heavy on philosophy and empirics of different cause areas.
    Previous versions of this list were used internally at GPI and during GPI's Oxford Global Priorities Fellowship in 2023, where the prompts guided individual reflection and group discussion.
    Many thanks to the following for their help creating and improving this syllabus: Gustav Alexandrie, Loren Fryxell, Arden Koehler, Luis Mota, and Charlotte Siegmann. The readings below don't necessarily represent their views, GPI's, or mine.
    1. Philosophical Foundations
    Topic: Global priorities research is a normative enquiry. It is primarily interested in understanding what we should do in the face of global problems, and only derivatively interested in how those problems work/facts about the world that surround them.
    In this session, we will focus on understanding what ethical theory is, what some of the most important moral theories are, how these theories relate to normative thinking in economics, and what these theories imply about what the most important causes are.
    Literature:
    MacAskill, William. 2019. "The Definition of Effective Altruism" (Section 4 is optional)
    Prompt 1: How aligned with your aims as a researcher is the definition of Effective Altruism proposed in this article (p. 14)?
    Trammell, Philip. 2022. Philosophical foundations (Slides 1-2, 5-9, 12-16, 20-24)
    Prompt 2: What is your best guess theory of welfare? How much do you think it matters to get this right?
    Prompt 3: What is your best guess view in axiology? What are your key uncertainties about it? Do you think axiology is all that matters in determining what one ought to do (excluding empirical uncertainty)?
    Trammell, Philip. 2022. Three sins of economics (Slides 1-24, 27)
    Prompt 4: What are your "normative defaults"? What are views here that you would like to explore more?
    Prompt 5: Do you agree that economics has the normative defaults identified in the reading? Can you give examples of economics work that avoids these?
    Prompt 6: Insofar as economists tend to commit the 3 'sins', what do you think of the strategy of finding problems which are underprovided by those views?
    Extra reading:
    Wilkinson, Hayden. 2022. "Key Lessons From Global Priorities Research" (watch video here - slides are not quite self-contained)
    Which key results are most interesting or surprising to you and why? Do you think any of them are wrong?
    Greaves, Hilary. 2017. "Population axiology"
    Broome, John. 1996. "The Welfare Economics of Population"
    2. Effective altruism: differences in impact and cost-effectiveness estimates
    Topic: In this session we tackle two key issues in cause prioritization. First, how is impact distributed across interventions (or importance across problems). Second, how to compare the cost-effectiveness of interventions which are differentially well-grounded.
    Literature:
    Kokotajlo, Daniel and Oprea, Alexandra. 2020. "Counterproductive Altruism: The Other Heavy Tail" (Skip Sections I and II)
    Prompt 1: Do you think there is a heavy right tail of opportunities to do good? What about a heavy left tail?
    Prompt 2: How do the distributions of impact of interventions aimed at the near-term and long-term compare (specifically, in terms of the heaviness of their tails)?
    Karnofsky, Holden. 2016. "Why we can't take expected value estimates literally (even when they're unbiased)"
    Prompt 3: What, in your view, is ...
    17 min
  • LW - Perpetually Declining Population? by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Perpetually Declining Population?, published by jefftk on August 8, 2023 on LessWrong.
    In With a Whimper: Depopulation and Longtermism, Geruso and Spears give the following argument for why most people who'll ever live may have already died:
    People are generally having children below replacement rate: 1.66 children per woman in the US, and total global annual births peaked in
    2014.
    If you project this forward 300-600 years, annual births fall below ~10M.
    This would leave us with a global population around 560M.
    Only a minor disaster could be enough to wipe out humanity once our population is so low.
    They include a pretty bold chart:
    To be fair, pretty much any continuation of that chart into the future is wild, but the one they've ended up with seems especially so!
    I don't find this argument very convincing for several reasons, but I want to focus on a specific one: even granting all their assumptions I think we'd see evolution for higher fertility long before we got down to 10M annual births.
    The paper says:
    But what, you might ask, about heritability (intergenerational transmission of high-fertility cultural practices)? Won't the Amish or some other high-fertility, perhaps religious, sub-population expand to be as many as we need? For several reasons, no. We have addressed this question at more length in Arenberg (2022). In the very long run (i.e., potentially after the coming few centuries of decline), two facts would have to be true for heritability to be a solution: First, fertility in a high-fertility sub-group would have to be high enough
    (certainly above two, for example). We've already seen above that the
    "high fertility" of high fertility subgroups has been declining over the decades. High fertility used to mean 6 children per woman. Now it means 2.5. Before long, it may mean 1.8. Second, the children of high-fertility parents would have to be very likely to remain in their high-fertility cultural group. Where researchers have studied the empirical magnitude of these intergenerational correlations as they have played out in actual practice, they have found them to be positive, but small - too small, in fact, for the high fertility group to make much of a dent in overall population. It turns out your kids might choose not to inherit your cultural practices and beliefs.
    If cultural evolution isn't enough, what about genetic evolution? They cite their 2022 research note Intergenerational
    Transmission Is Not Sufficient for Positive Long-Term Population
    Growth which does consider both cultural and genetic fertility, but their model of the latter doesn't seem to me to address the main ways I'd expect genetic evolution to reverse global population decline. Let me give a sketch:
    Humans vary a lot in the strength and timing of their desire to reproduce. Some people never have any interest in children, others feel deeply drawn to parenthood from a young age, and for others it varies. One common way that it varies is that many non-parents, as they get older, find themselves suddenly seriously drawn to having children. This suggests, though doesn't demonstrate on its own, that desire to have children can be and sometimes is strongly influenced by our biology.
    If a strong biologically driven desire to reproduce were possible, though, why wouldn't we already have one? Historically, having such a desire probably didn't have much of an effect on how many kids you had. Given the lack of cheap reliable birth control and much lower levels of sexual education, evolution mostly gave us a strong desire to copulate. Combined with cultural taboos on finding alternative outlets for this desire, there wouldn't have been much selection on desire specifically to reproduce. With those conditions changed, however, I'd predict we're already seeing strong evolutionary selection for this d...
    5 min
  • LW - A plea for more funding shortfall transparency by porby
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A plea for more funding shortfall transparency, published by porby on August 8, 2023 on LessWrong.
    [This post is largely from the perspective of AI safety, but most of it should generalize.]
    For recipients, well calibrated estimates about funding probability and quantity are extremely valuable. Funding-dependent individuals and organizations need information to optimize their decisionmaking; incorrect estimates cause waste.
    At the moment, getting that information seems unnecessarily hard.
    To help with this, I would ask organizations up the funding chain to systematically and continuously provide bite-sized updates from their own perspectives on the funding situation when possible.
    This needn't be in the form of a lengthy report or deep-dive (though those are nice too!). For example, for a grantmaking organization with open applications, maybe something like:
    We've received V requests for funding totaling $W in the last month. We anticipate funding up to $X of these requests; we would fund up to about $Y if we had more funding.
    We don't anticipate significant changes in our funding capacity by default.
    Reports like this are already done occasionally. For example, I deeply appreciate reports like this one. My concern is that I'm not aware of any consistent source for this information.
    I worry that this is partially because writing a dense report takes a lot of time, and many of these organizations are massively overwhelmed as it is. To the extent that this is limiting reports, I would suggest that giving a tweet-sized miniupdate as things change (or just every few months) would be a huge improvement over the status quo.
    Even "oof, we're REALLY funding constrained" would be great! If you don't have time for collecting any numbers at all, a vibecheck is still useful!
    There's also no need for each report to attempt to capture the entire field's status, just the perspective of that one organization.
    An anecdote of awareness not propagating like it should
    For the last six-ish months, I've been trying to figure out how to prioritize earning to give and direct alignment research. I've gotten in touch with several people and heard a number of different perspectives.
    None of them were "yeah, the field's pretty funding constrained right now, there are more people than funds, it's a major bottleneck."
    This continued a confusion I had starting with the FTX collapse. While I fully expected the field to be in a crunch immediately following the collapse, the vibe I collected from a number of people was that this was a probably-temporary thing, and this was seemingly supported by other things I heard a few months later.
    A lot of hints - organizations not being able to hire everyone they'd like to hire, not as many grants flowing as I'd expect, and salary targets way too low for a field with tons of cash on hand relative to talent - were inconsistent with the "unconstrained" narrative, but they were too weak in isolation to update me to reality.
    One side effect of this confusion was that I started work on a project before receiving grant funding for it based on an incorrectly high probability of being funded. It fell through; it's not a catastrophe, and I was prepared for that possibility, but I would have chosen differently if I had all the information that had existed privately at the time.
    Then it seems like everything snapped together with the common knowledge created by one post.
    If this is our collective mechanism for creating awareness, something seems broken.
    The future
    While I've felt reasonably productive in my grant-funded research so far, it seems unlikely that my comparative advantage is in full-time alignment research as opposed to earning to give if this kind of funding environment continues.
    In addition to periodic updates about the current funding situation, I've love ...
    5 min
  • AF - Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research by Evan Hubinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research, published by Evan Hubinger on August 8, 2023 on The AI Alignment Forum.
    TL;DR: This document lays out the case for research on "model organisms of misalignment" - in vitro demonstrations of the kinds of failures that might pose existential threats - as a new and important pillar of alignment research.
    If you're interested in working on this agenda with us at Anthropic, we're hiring! Please apply to the research scientist or research engineer position on the Anthropic website and mention that you're interested in working on model organisms of misalignment.
    The Problem
    We don't currently have ~any strong empirical evidence for the most concerning sources of existential risk, most notably stories around dishonest AI systems that actively trick or fool their training processes or human operators:
    Deceptive inner misalignment (a la Hubinger et al. 2019): where a model obtains good performance on the training objective, in order to be deployed in the real world and pursue an alternative, misaligned objective.
    Sycophantic reward hacking (a la Cotra 2022): where a model obtains good performance during training (where it is carefully monitored), but it pursues undesirable reward hacks (like taking over the reward channel, aggressive power-seeking, etc.) during deployment or in domains where it operates with less careful or effective human monitoring.
    Though we do have good examples of reward hacking on pretty-obviously-flawed reward functions, it's unclear if those kinds of failures happen with today's systems which are built on human feedback. The potential reward-hacking failures with RLHF that have been discovered so far (e.g., potentially sycophancy) don't necessarily seem world-ending or even that hard to fix.
    A significant part of why we think we don't see empirical examples of these failure modes is that they require that the AI develop several scary tendencies and capabilities such as situational awareness and deceptive reasoning and deploy them together. As a result, it seems useful to evaluate the likelihood of each of the scary tendencies or capabilities separately in order to understand how severe the risks are from different potential forms of misalignment.
    The Plan
    We need to develop examples of the above alignment failures, in order to:
    Learn more about the possible failures, to understand how likely they are, what causes them to arise, and what techniques may mitigate the failures (discussed here).
    Inform the current conversation about AI risk by providing the best evidence of misalignment risks, if any. We hope this will be helpful for labs, academia, civil society, and policymakers to make better decisions (discussed here). If misalignment issues end up being serious, then it will be critical to form a strong scientific consensus that these issues are real, for which examples of alignment failures are crucial.
    Roadmap
    Stories around AI takeover involve several subcomponents, and the goal of this agenda is to develop models which exhibit each of the subcomponents. Once we've demonstrated each of the subcomponents, then we would string them together in an end-to-end demonstration of what could go wrong. Then, we can use the demonstration as a testbed to study alignment techniques and/or to ring a fire alarm (depending on how serious and unfixable the failure is).
    Possible subcomponents of AI takeover to demonstrate
    Having/developing a misaligned goal: AI takeover stories involve the AI having or developing a misaligned goal, despite having an aligned-looking objective (e.g., human feedback). The misaligned goal (eventually) manifests as a clear, undeniable conflict with human values. See the descriptions of "Deceptive inner misalignment" and "Deceptive reward hacking" earlier for...
    31 min
  • EA - Best Use of 2 Minutes this Month (U.S.) by Rockwell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Best Use of 2 Minutes this Month (U.S.), published by Rockwell on August 7, 2023 on The Effective Altruism Forum.
    Actions that have a large impact sometimes don't feel like much. To counteract that bias, I'm sharing arguably the best use of two minutes this month for those in the U.S.
    Background
    In May, the US Supreme Court upheld the ability of US states to require certain standards for animal products sold within their borders, e.g. California's Prop 12, which banned the sale of animal products that involve certain intensive confinement practices.It was a huge victory! But after their defeat in the Supreme Court, the animal farming industry has turned to Congress, pushing the EATS Act.The proposed legislation would take away state power to regulate the kind of agricultural products that enter their borders. Essentially, if any one state permits the production or sale of a particular agricultural product, every other state could have to do so as well, regardless of how dangerous or unethical the product is and regardless of existing state legislation. Just a small sample of laws that could be subverted by the EATS Act include those governing:
    Chemicals in baby food containers
    Harmful pesticides in communities and applying them directly to crops for human consumption
    Arsenic in feed for animals slaughtered for food and other poison control
    Child labor
    Puppy mills
    Wildlife protection
    Pollutant and emissions standards, e.g., bans on spraying sewage on crops directly before they are sold to people
    Fire hazards
    Drugs that contain opioid properties and alcohol and tobacco sales to minors
    And, of course, legislation like Proposition 12 that improves animal welfare requirements and has been a dominant focus of the pro-animal movement.
    2 Minutes of Action
    10 seconds: Send a written message to your legislators
    ~1.5 minutes: Call your legislators
    Find them
    Call your one U.S. representative and two U.S. senators (found in the federal section), ideally during business hours
    Read a script like this one:
    As your constituent, I urge you to oppose the Exposing Agricultural Trade Suppression' (EATS) Act (S. 2619/HR 4999). This is a dangerous, regressive bill that will undo decades of protections for farmed animals and cause them to endure even more suffering for the profits of animal agricultural interests. It would also have devastating consequences for humans and the environment."Note that there is another bill known as the EATS Act of 2023, which is Enhance Access To SNAP. Please be specific that you are asking them to oppose the Exposing Agricultural Trade Suppression Act.
    For more information, see: (1), (2), (3).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • AF - An interactive introduction to grokking and mechanistic interpretability by Adam Pearce
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An interactive introduction to grokking and mechanistic interpretability, published by Adam Pearce on August 7, 2023 on The AI Alignment Forum.
    Our write up largely agrees with @Quintin Pope's summary, with the addition of training trajectory visualizations and an explanation of the MLP construction that solves modular addition.
    A meta note that didn't make it into the article - with so many people looking into this problem over the last 18 months, I'm surprised this construction took so long to find. The modular addition task with a 1-layer MLP is about as simple as you can get!
    Scaling mechanistic interpretability up to more complex tasks/models seems worth continuing to try, but I'm less sure extracting crisp explanations will be possible. Even if we "solve" superposition, figuring the construction here - where there's no superposition in the generalizing model - wasn't trivial.
    gif/twitter summary
    If we train a MLP to solve modular addition, the generalizing phase has suggestive periodic patterns.
    To figure out why the model generalizes, we first look at task where we know the generalizing solution - sparse parity. You can see the model generalizing as weight decay prunes spurious connections.
    One point from the Omnigrok paper I hadn't internalized before training lots of models: grokking only happens when hyper-parameters are just right.
    We can make other weird things happen too, like AdamW oscillating between low train loss and low weights.
    To understand how a MLP solves modular addition, we train a much smaller model with a circular input embedding baked in.
    Following @Neel Nanda and applying a discrete Fourier transform, we see larger models trained from scratch use the same star trick!
    Finally, we show what the stars are doing and prove that they work:
    Our ReLU activation has a small error, but it's close enough to the exact solution - an x² activation suggested in Grokking modular arithmetic - for the model to patch everything up w/ constructive interference.
    And there are still open question: why are the frequencies with >5 neurons lopsided? Why does factoring Winput not do that same thing as factoring Woutput?
    Also see The Hydra Effect: Emergent Self-repair in Language Model Computations
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Feedbackloop-first Rationality by Raemon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Feedbackloop-first Rationality, published by Raemon on August 7, 2023 on LessWrong.
    I've been workshopping a new rationality training paradigm. (By "rationality training paradigm", I mean an approach to learning/teaching the skill of "noticing what cognitive strategies are useful, and getting better at them.")
    I think the paradigm has promise. I've beta-tested it for a couple weeks. It's too early to tell if it actually works, but one of my primary goals is to figure out if it works relatively quickly, and give up if it isn't not delivering.
    The goal of this post is to:
    Convey the framework
    See if people find it compelling in its current form
    Solicit ideas for improvements, before I decide whether to invest heavily into a larger experiment around it.
    Rationality needs better feedback loops
    Claim: Feedback loops are the most important thing ever. Hard things are hard because they have bad feedback loops. Some of the most important things (e.g. x-risk mitigation research) have the worst feedback loops.
    Bold prediction: You can learn to think better, even about confusing, poor-feedback domains. This requires developing the art of inventing feedback loops. And then, actually putting in a lot of deliberate practice effort.
    I've long been haunted by this Romeo Stevens comment (slightly paraphrased)
    Deliberate practice deliberate practice until you get really good identifying good feedback loops, and working with them.
    People have a really hard time with interventions often because they literally do not have a functioning causal model of the skill in question. People who apply deliberate practice to a working causal model often level up astonishingly quickly. Don't know if you have the appropriate causal model? Well, when you apply deliberate practice do you not get better? You're pulling on fake levers.
    In the past, I've tried to practice thinking. I've done explicit puzzle-solving exercises, and I have a day job that forces me to think about challenging questions on a regular basis. I sometimes have tried to refactor my day-job into something deliberate practice-shaped, but it never gelled.
    I think I've gotten better at thinking in the past 12 years. But I haven't gotten overwhelmingly obviously better at thinking. I recently decided to deliberate practicing "solve confusing problems", until I was demonstrably better at it, and to host some workshops where I tried helping other people practice too.
    I ended up settling into a paradigm of rationality training with five elements:
    Deliberate Practice. Do challenging cognitive exercises, at the edge of your ability, in a variety of domains, where it's obvious how well you're doing (i.e. clear cut answers, or you're making a metric go up).
    Metacognition. After deciding on the final answer for the exercise and finding out if you got it right, reflect on what you could have done better. Try to extract as much insight/wisdom/tools as you can from each exercise.
    Improve your practice feedback loop. Then, find or design better exercises, that cut more closely to your ultimate goals. Optimize exercises both for being concrete (i.e. you can tell if you succeeded), and for extracting as much insight/tools as possible during the metacognition step (i.e. they are a good difficulty in a domain I haven't already exhausted for insight)
    Improve your real-life feedback loop. Think about what sort of cognitive challenges you run into your day-job or main project, where you're bottlenecked in your ability to reason. How can you do better meta-reflection in those fuzzier, longer-timescale domains?
    Illegible goodness. In addition to the formal structure implied by the previous four bullets, also try random stuff that feels vaguely relevant and helpful, even if it you can't explain why. (I think some previous rationality training approaches leaned...
    14 min