The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - You are better at math (and alignment) than you think by Trevor1
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: You are better at math (and alignment) than you think, published by Trevor1 on October 13, 2022 on LessWrong.
    I was absolutely dazzled by the Viliam-Valentine Math-Education debate, which was in the comments section of the Seeking PCK chapter in the Center for Applied Rationality's Rationality Handbook.
    The debate gives an autopsy of why education systems inflicts math on children during their most formative years, resulting in the vast majority of the population falsely believing that they cannot enjoy math. In reality, you can probably get very good at math and have a great time doing it too; and, in fact, you even have a very serious chance of becoming one of the 300 AI safety researchers on earth. Odds are also good that you have a big advantage in terms of "superior-at-actually-using-math-in-real life" genes, which have a surprisingly weak correlation with the "inferior at learning math in a classroom at age 7" genes (such as being praise-motivated, obedient, and comfortable doing the same thing hundreds of times without asking why).
    I strongly recommend reading this debate yourself if you currently don't see yourself as quantitatively skilled or quantitatively employed, and also showing it to other people who might have had strong potential for quantitative skill all along. The phenomena described below seems to be the main reason why such a small proportion of people are willing to do the quantitative work necessary for technical AI alignment, and therefore they are a major alignment bottleneck that is worth tackling.
    Viliam:
    I think that in Slovakia an d Czechia, this style of teaching [PCK, aka paying attention to what it's like to learn something while you are teaching] is referred to as "constructivist education".
    On the other hand, for English-speaking audience, the word "constructivism" seems to refer to quite different things (1, 2, 3). And when I try to explain concepts like this in English, I sometimes get surprising responses when people seem to automatically run along the chain of associations: "trying to understand the student's model" = "constructivism" = "you should never explain math" = "math wars" = "total failure".
    I tried to figure out how this could have happened, and my current best guess is that in the past some people in USA promoted some really stupid and harmful ideas under the banner of "constructivism", which made people associate the word "constructivism" with those stupid ideas. Meanwhile, some of the original good ideas are still taught, but carefully under different labels. (Longer version here.) So perhaps the people who want to learn how to teach well, could find something useful in the writings of Piaget and Vygotsky. However, a Google search for "constructivism" might just return a list of horror stories.
    By the way, I would expect pedagogical content knowledge of STEM topics to be super rare, because it requires an intersection of being good at psychology and math. And, at least in my experience, psychologists are often quite math- and tech-phobic. On the other hand, people good at math often fail to empathize with the beginners, and just keep writing complex equations, preferably without explanation of what the symbols mean.
    Valentine:
    On the other hand, for English-speaking audience, the word "constructivism" seems to refer to quite different things (1, 2, 3). And when I try to explain concepts like this in English, I sometimes get surprising responses when people seem to automatically run along the chain of associations: "trying to understand the student's model" = "constructivism" = "you should never explain math" = "math wars" = "total failure".
    IIRC, this was the result of trying to implement the good version of constructivism in the USA. It wasn't just that some people had bad ideas and called those "constructivism" to...
    36 min
  • EA - A call for EA Hubs to post public guides by Elika
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A call for EA Hubs to post public guides, published by Elika on October 13, 2022 on The Effective Altruism Forum.
    Write a guide to your city if you’re hosting a conference, consider yourself a major EA hub, are having a seasonal push, or expect a constant influx of EAs.
    You can use our template and time-cap yourself at 2 hours.
    Cities we'd love to see guides from (in no particular order):
    New York
    London
    Prague
    Berlin
    DC
    Boston
    Oxford
    Cambridge
    Singapore
    Mexico
    Israel
    Manila
    Bonus: consider doing some outcomes research on how this helps build EA connections and is impactful
    Posts can be short.
    Why is this useful? Building EA connections, building a local and welcoming EA community, it's helpful for new EAs moving to your city, etc. We think it's helpful to make hubs accessible to the community, especially for newer, less well-connected folks.
    You can also save yourself some time by adding in FAQs and forwarding people to the guide!
    Got more time? Share it with 5 friends to add their takes, and share it with a few non-locals to make sure it's useful!
    and countries? Singapore is a city-state. Mexico is mostly Mexico City but maybe a country one works?
    We're currently researching the 'Visiting the Bay Summer 2022' push. If you're interested in helping out - comment below or reach out to Elika.
    We wrote this in 10 minutes
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Pineapple Operations is expanding to include all operations talent (Oct '22 Update) by Vaidehi Agarwalla
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pineapple Operations is expanding to include all operations talent (Oct '22 Update), published by Vaidehi Agarwalla on October 13, 2022 on The Effective Altruism Forum.
    Pineapple Operations aims to streamline the operations talent search and save both employers’ and candidates’ time. We provide a public directory of people interested in operations work in the EA ecosystem.
    Changes to our “operations”
    Our database now includes all operations talent
    We’re excited to announce that we’ve expanded our database to include all operations talent, with 100+ candidates listed so far! Our goal is for the database to be a central, easy-to-use and up-to-date database. Let us know in the comments or here how we could do that better! We are trying out a minimum viable product, so anyone can list themselves on the database (we remove spam).
    We previously only listed PA/ExA talent.
    List yourself | Search the database
    We are not offering any other services as of now
    Due to capacity constraints we are not offering any personalised services or candidate vetting at this time. We think they are likely to be valuable, but unfortunately cannot do them ourselves.
    We think these services are potentially really valuable, especially for organisations with limited capacity and would experiment more in this space if we had capacity.
    If you’re interested and have prior experience in recruiting and/or operations please reach out. We have several ideas we’d like to test (and several potential clients who would like them!).
    Others experimenting on their own or through other projects with outsourcing operations recruiting should also get in touch to see if we can work together
    Note: We will be posting two short guides on hiring / getting hired for PA/ExAs & operations staff, which we hope will help in the meantime.
    New Team
    Vaidehi Agarwalla and Alexandra Malikova run Pineapple Operations. We are grateful to Holly Morgan and previous team members for founding and working on earlier versions of the project.
    How are we doing so far? What can we do better?
    If you’ve found a position or hired someone you found on our database, please let us know!
    Since we are in the beta testing phase, we would also love to know how we can improve. You can contact us at [email protected], share in the comments below or leave feedback anonymously via our feedback form.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - Greed Is the Root of This Evil by Thane Ruthenis
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Greed Is the Root of This Evil, published by Thane Ruthenis on October 13, 2022 on The AI Alignment Forum.
    The SGD's greed, to be specific.
    Consider a ML model being trained end-to-end from initialization to zero loss. Every individual update to its parameters is calculated to move it in the direction of maximal local improvement to its performance. It doesn't take the shortest path from where it starts to the ridge of optimality; it takes the locally steepest path.
    1. What does that mean mechanically?
    Roughly speaking, every feature in NNs could likely be put into one of two categories:
    Statistical correlations across training data, aka "the world model".
    The policy: heuristics/shards, mesa-objectives, and inner optimization.
    The world-model can only be learned gradually, because higher-level features/statistical correlations build upon lower-level ones, and therefore the gradients towards learning them only appear after the lower-level ones are learned.
    Heuristics, in turn, can only attach to the things that are already present in the world-model (same for values). They're functions of abstractions in the world-model, and they fire in response to certain WM-variables assuming certain values. For example, if the world-model is nonexistent, the only available heuristics are rudimentary instincts along the lines of "if bright light, close eyes". Once higher-level features are learned (like "a cat"), heuristics can become functions of said features too ("do X if see a cat", and later, "do Y if expect the social group to assume state S within N time-steps").
    The base objective the SGD is using to train the ML model is, likewise, a function of some feature/abstraction in the training data, like "the English name of the animal depicted in this image" or "the correct action to take in this situation to maximize the number of your descendants in the next generation". However, that feature is likely a fairly high-level one relative to the sense-data the ML model gets, one that wouldn't be loaded into the ML model's WM until it's been training for a while (the way "genes" are very, very conceptually far from Stone Age humans' understanding of reality).
    So, what's the logical path through the parameter-space from initialization to zero loss? Gradually improve the world-model step by step, then, once the abstraction the base objective cares about is represented in the world-model, put in heuristics that are functions of said abstraction, optimized for controlling that abstraction's value.
    But that wouldn't do for the SGD. That entire initial phase, where the world-model is learned, would be parsed as "zero improvement" by it. No, the SGD wants results, and fast. Every update must instantly improve performance!
    The SGD lives by messy hacks. If the world-model doesn't yet represent the target abstraction, the SGD will attach heuristics to upstream correlates/proxies of that abstraction. And it will spin up a boatload of such messy hacks on the way to zero loss.
    A natural side-effect of that is gradient starvation/friction. Once there's enough messy hacks, the SGD won't bother attaching heuristics to the target abstraction even after it's represented in the world-model — because if the extant messy hacks approximate the target abstraction well enough, there's very little performance-improvement to be gained by marginally improving the accuracy so. Especially since the new heuristics will have to be developed from scratch. The gradients just aren't there: better improve on what's already built.
    2. How does that lead to inner misalignment?
    It seems plausible that general intelligence is binary. A system is either generally intelligent, or not; it either implements general-purpose search, or it doesn't; it's either an agent/optimizer, or not. There's no continuum here, the difference...
    16 min
  • EA - Sixty years after the Cuban Missile Crisis, a new era of global catastrophic risks by christian.r
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sixty years after the Cuban Missile Crisis, a new era of global catastrophic risks, published by christian.r on October 13, 2022 on The Effective Altruism Forum.
    Linkpost for a short op-ed I wrote in the Bulletin of the Atomic Scientists in light of the upcoming 60th anniversary of the Cuban Missile Crisis and President Biden's recent comments that "For the first time since the Cuban Missile Crisis, we have a direct threat to the use of nuclear weapons, if in fact things continue down the path they’d been going."
    Was asked to keep it mostly nuclear (i.e., only some narrow AI and no bio, which was in my first draft), but managed to keep in some broader points about technology development and deployment, like "artificial intelligence and other new technologies, if thoughtlessly deployed, could increase the risk of accidents and miscalculation even further."
    First couple of paragraphs:
    This month marks the 60th anniversary of the Cuban Missile Crisis. For two tense weeks from October 16 to October 29, 1962, the United States and the Soviet Union teetered on the brink of nuclear war. Sixty years later, tensions between the world’s major militaries are uncomfortably high once again.
    In recent weeks, Russian President Vladimir Putin’s nuclear-charged threats to use “all available means” in the Russo-Ukrainian war have again raised the prospect of nuclear war. And on October 6, US President Joe Biden reportedly told a group of Democratic donors: “For the first time since the Cuban Missile Crisis, we have a direct threat to the use of nuclear weapons, if in fact things continue down the path they’d been going.”
    Any uncontrolled escalation of these existing conflicts could end in global catastrophe, and the history of the Cuban Missile Crisis suggests that such escalation may be more likely to happen through miscalculation and accidents. Lists of nuclear close calls show the variety of pathways that could have led to disaster during the Cuban crisis. Famously, Soviet naval officer Vasili Arkhipov vetoed the captain of a nuclear submarine who wanted to launch a nuclear-armed torpedo in response to what turned out to be non-lethal depth charges fired by US forces; had Arkhipov not been on this particular vessel, the captain might have had the two other votes he needed to order a launch.
    Today, artificial intelligence and other new technologies, if thoughtlessly deployed, could increase the risk of accidents and miscalculation even further [...]
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Weekly Non-Covid News #1 (10/13/22) by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Weekly Non-Covid News #1 (10/13/22), published by Zvi on October 13, 2022 on LessWrong.
    This is the non-Covid part of what would previously have been the weekly Covid post. About half of the content written for the post this week is being withheld for topic-level future posts, both with longer time horizons (e.g. policy roundups don’t need to be weekly and should benefit from more integration over time) and elevating worthy sub-topics to their own posts, which this week seems likely to include the colonoscopy study.
    We will retain the broad categories of Bad and Good news for various short notes.
    Bad News
    Sysco Teamsters are on strike in three cities, and there are a lot of people cheering the strikers on mood affectation grounds because Sysco is buying up rivals, slashing staff and service and treating everyone like garbage. Is it a monopoly worth investigating? It sure sounds like it has monopoly power and is actively seeking to assemble more of it. If you keep hiking prices (no you cannot simply say ‘inflation!’) and cutting service and everyone hates you and they live in fear of you, you might be a monopoly. If it was my job to investigate such things, I would investigate.
    Signal to stop letting one send SMS messages on Android phones.
    Paper (via MR) claims that there is a causal effect on call center workers where they are ‘less productive’ when they report they are happier. This effect might replicate with respect to call centers in particular, I would be shocked if it replicated to workers in general (and thus my answer to Tyler’s generalized ‘are happy workers less productive?’ is ‘no, quite the opposite, and happy to bet on this.’) If this effect is real and causal, my mechanism is that call center ‘productivity’ is about getting through calls quickly whether or not that is good for the business or customer (or victim/target), and also being willing to engage in essentially hostile interactions with the target to get the desired result. There are particular professions and times and places where one wants to Drive Angry. On the other hand I would have been more productive happier at any job I have ever had.
    Kamil Galeev has an interesting thread, more explicitly than usual going beyond Russia, talking about the difference between violent and non-violent entrepreneurs, and how Musk’s instinct to ‘make a deal’ with China and Russia is exactly how the non-violent creators have gotten owned and curb-stomped for almost all of history. That in Kamil’s model of the world, where someone being allowed to create value non-violently without being wiped out is weird. His version proves too much, but the lesson he is giving here is vital, that we do not know how good we and the world have it right now in this sense. We live in a world where Musk is the rogue who doesn’t play by the rules and abuses and doesn’t respect the law. He’s the bully, not the victim. While it would be even better if he didn’t do that, we should be thrilled about this level of happy problem.
    I want it to be the way where they don’t gaslight us. Alas, it is not that way.
    Market pricing in three years of rough winters for Europe.
    The far forward contracts might be a good buy here given the uncertainty. There are ways to have access to bountiful energy and Europe and its ‘environmentalists’ seem to be strongly against them.
    The rich get higher returns than others on their real estate holdings. Presumably some of this is that rich people prefer areas that are getting richer and more expensive, whereas poorer people do not. Alternatively, rich people are buying real estate as more investment relative to consumption, and can afford to buy ‘too much house’ in order to get an area with better future prospects, and thus this effect is somewhat moderated by lower effective consumption. This still seems to s...
    26 min
  • EA - Let me blind myself to Forum post authors by Will Payne
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Let me blind myself to Forum post authors, published by Will Payne on October 13, 2022 on The Effective Altruism Forum.
    TLDR: I'd like to have the ability to hide the author(s) of all the posts on my news feed so that I see can read the content without knowing about the source. I have some intuitions for why this might be good but mainly I think adding the feature is easy and it lets us check whether or not it's good.
    Is this the right place to post this? I can't remember if there's an obvious place to suggest forum features (I think there might have been a survey a while back but I might have imagined it). Since the bulk of this post is closer to "Why you might want this" than "Why you should add this to the forum you maintain" I think it's better posted publically. Also it seems likely this can be quickly made with a chrome extension.
    The feature
    I'd like to be able to blur out author names wherever they appear like this...
    (It could also be black boxes of regular size or random strings of nonsense characters like this ⏁⊑⟟⌇ ⟟⌇ ⋏⍜⋏⌇⟒⋏⌇⟒)
    I'd like to be able to click on names to reveal them but would be ok with an easy toggle for the feature.
    A larger project would be for folks at CEA to run an AB test where post authors are hidden for one group of users and not hidden for another and publish click through rates and karma from different groups. I think there will be a difference I'm not sure which setting I'd advocate for as a norm though (I'll go into why below).
    EDIT: Since a lot of people are suggesting extensions / greaterwrong style solutions. One benefit of an integrated into the forum solution is the ability to separate blinded Karma from unblinded Karma (even if this is only on the back end). I'm mostly interested in what the frontpage looks like when karma is driven only by post content and not by authorship.
    Why I might (or might not) want this
    I don't think I say anything super surprising in this section, you're welcome to skip it.
    It seems pretty obvious that the authorship of a post affects my click through rate. There are good reasons for this. If I recognise a name as someone who I've read content from before and found that content useful I think it's more likely I'll find their new content useful too. This is the same logic that led me to watch the new Game of Thrones spin off, buy a second pair of Levi jeans, and listen to the latest Dodie album.
    However this policy makes me less great at exploring new sources of insight. Historically I'm significantly more likely to re-read Harry Potter and the Methods of Rationality than pick up any specific book on my to-read list.
    This actually doesn't just affect me but also, through the karma system, everyone else. FYI this is the factor which made me decide to post this. I can think of a few examples off hand where I opened a post I might not have otherwise because I know the author personally and want to know what they're up to, I then end up upvoting the post. If I extrapolate my policy to everyone else on the forum I'd expect posts from authors who have a lot of friends in the community to do better than the exact same post posted anonymously. I could train myself to stop doing this (and suggest others do so) but it would be a lot easier to just anonymise posts.
    Separately once I open a post I'd guess there are a bunch of associations happening at a level of my thinking I'm not aware of when I read a name I recognise at the top of a post (ala Halo/Horn effect). I mostly guessing about what these would be but I expect people I like / have agreed with before get more of the benefit of the doubt and are engaged with less critically. This would mean I'd be more open to novel or unintuitive seeming proposals from people I know who are already established in the community. I think this effect still exists when I ...
    6 min
  • EA - Scout Mindset Poster by Anthony Fleming
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Scout Mindset Poster, published by Anthony Fleming on October 13, 2022 on The Effective Altruism Forum.
    (Edits: shortened the text on the poster according to AllAmericanBreakfast's recommendations, and added an editable link.)
    I made a thing!
    I recently finished reading the Scout Mindset by Julia Galef, and I particularly loved the section on thought experiments that can help wrestle you out of the soldier mindset. I remember thinking, "man, I wish I had a poster with all these thought experiments, so I could put it above my desk and really hone those skills." So, I decided to go on Canva and whip something up (keep in mind, I have absolutely no graphic design background, so if someone wants to remake this and make it look better, that would be great!)
    If anyone has any constructive criticism, feel free to let me know. Otherwise, you can order a print using this Canva link:
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Growth Theory Reading List by LuisMota
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Growth Theory Reading List, published by LuisMota on October 12, 2022 on The Effective Altruism Forum.
    The reading list below is based on a reading list originally used for an internal GPI reading group in spring 2021, organized by Phil Trammell and Leopold Aschenbrenner, and on the ETGP 2022 syllabus, also organized by Phil. I also added a few resources I deemed relevant on some of the topics below, but the credit for almost all of these resources goes to Phil and Leopold.
    My suggested usage for this reading list is in conjunction with, and as a complement to, the slides for the first week of ETGP 2022 (available here), as the course was specifically designed to teach economic theory in a global prioritization context. As such, the structure of the reading list matches that of the course.
    Disclaimer: The views presented in the readings suggested below do not necessarily represent views held by the reading group organizers, me, GPI, or any other GPI staff member.
    Motivation
    There are two main reasons why growth theory is particularly relevant for EAs. First, growth models allow for a lot of flexibility regarding which factors are included in the model, and which timescale one uses to evaluate the effects of changes in these factors. These characteristics make growth models useful even if one is ultimately only interested in factors other than economic growth, like AI or climate change. Second, economic growth plays a central role in the contemporary world, in a way that understanding its dynamics is likely key for global prioritization.
    1. Introduction and basic models
    Jones, Charles I., and Dietrich Vollrath. 2013. Introduction to Economic Growth.
    Chapter 2, up to section 2.1.4
    Trammell, Philip. 2020. “Economic Growth under Transformative AI.”
    Section 2
    2. Accumulation-based models and growth versus level effects
    Jones, Charles I., and Dietrich Vollrath. 2013. Introduction to Economic Growth.
    Chapter 9
    Piketty, Thomas. 2014. Capital in the Twenty-First Century.
    Section “Too Much Capital Kills the Return on Capital” up to and including “Capital-Labor Substitution in the Twenty-First Century: An Elasticity Greater Than One”
    Jones, Charles I. 2003. “Growth, Capital Shares, and a New Perspective on Production Functions.”
    3. Scale effects in researcher-based models
    Jones, Charles I. 2005. “Growth and Ideas.”
    Section 1
    Section 3
    Section 5 (skip 5.4)
    Sections 6.1 and 6.2
    Bloom, Nicholas, Charles I. Jones, John Van Reenen, and Michael Webb. 2020. “Are Ideas Getting Harder to Find?”
    Introduction
    See figures in the slides
    Bond-Smith, Steven. 2019. “The Decades-Long Dispute Over Scale Effects in the Theory of Economic Growth.”
    4. Existential risk and growth
    Aschenbrenner, Leopold. 2020. “Existential Risk and Growth.”
    Trammell, Philip. 2021. “Existential Risk and Exogenous Growth.”
    Hilton, Benjamin. 2021. “Existential Risk Mitigation as a Public Good.”
    Hilary Greaves’s lecture on longtermism and economic growth
    5. Long-run historical growth
    Kremer, Michael. 1993. “Population Growth and Technological Change: One Million B.C. to 1990.”
    Jones, Charles I. “Economic Growth over the Very Long Run.”
    Roodman, David. 2021. “On the Probability Distribution of Long-Term Changes in the Growth Rate of the Global Economy: An Outside View.”
    6. The mechanics of the industrial revolution “phase change”
    Galor, Oded, and David N. Weil. 2000. “Population, Technology, and Growth: From Malthusian Stagnation to the Demographic Transition and Beyond.”
    Crafts, Nicholas. 2011. “Explaining the First Industrial Revolution: Two Views.”
    7. AI and growth
    Trammell, Philip. 2020. “Economic Growth under Transformative AI.”
    Section 1
    Sections 3.1-3.3
    Section 4.1-4.2
    Section 5.2
    Section 6
    Section 7
    Aghion, Philippe, Benjamin F Jones, and Charles I Jones. 2019. “Artificial Intelligence and Economic Growth.”
    GovAI P...
    7 min
  • AF - Cataloguing Priors in Theory and Practice by Paul Bricman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cataloguing Priors in Theory and Practice, published by Paul Bricman on October 13, 2022 on The AI Alignment Forum.
    This post is part of my hypothesis subspace sequence, a living collection of proposals I'm exploring at Refine. Preceded by an exploration of Boolean primitives in the context of coupled optimizers.
    Thanks Alexander Oldenziel and Paul Colognese for discussions which inspired this post.
    Intro
    Simplicity prior, speed prior, and stability prior — what do they all have in common? They are all means of tilting an optimization surface towards solutions with certain properties. In other words, they are all heuristics informing the navigation of model space, trainer space, etc. However, what brings them together is also a systematic divide between their theoretical/conceptual/abstract framings and their practical/engineering implementations. All those priors appear to have been used in contemporary ML in one form or another, yet conceptual-heavy researchers are often unaware of those interesting data points, while ML engineers often treat those implementations as mere tricks to improve performance (e.g. generalization).
    In the language of coupled optimizers I've explored over the few past posts in the sequence, such heuristics are artifacts of meta-optimization (e.g. a human crafting a trainer by building in certain such tendencies), and often tend to be direct human-made artifacts, rather than the result of downstream optimization. Though this need not be the case, as the simplicity prior happens to itself be quite simple... It might emerge naturally from a trainer which itself is trained to be simple. Similarly, the speed prior happens to itself be quite fast, as penalizing a duration is trivial. It might emerge naturally from a trainer, should it be trained to itself be fast.
    Anyway, let's briefly catalog a few popular priors, describe the rationale for employing them, list instances of their use in ML, and finally document possible failure modes associated with blindly following them.
    Priors
    Some members of the list can be better described as heuristics or biases than priors. However, there are some basic connections between those in that they all cause an optimization process to yield certain outcomes more than others. If you start with a prior of possible ML model parametrizations and use training data to update towards your final distribution, your choice of prior will naturally influence the posterior. This prior can be informed by heuristics, such as "we're more interested in simple models than complex ones from the get-go." Bias as in structural bias, inductive bias, and bias-variance trade-off describes a similar process of tailoring the ML model to broadly yield certain types of results efficiently.
    Simplicity
    Informally known as Occam's razor, and extremely formally known as Solomonoff prior, the simplicity prior biases optimization towards solutions which are simple. "Simple" here is often operationalized using the minimum description length: what's the shortest description of an algorithm/model/concept/world/etc. required to accurately specify it? Simple candidates are then the ones with a particularly short such shortest length.
    The rationale behind employing the simplicity prior in an optimization process is that it systematically reduces the variance of the solution. This means that it increases the odds that the solution will behave in a similar way in different situations, as opposed to growing too reliant on the idiosyncrasies of your finite/limited/bound optimization process. When training ML models, simplicity tends to yield strong generalization performance. When building world models, simplicity tends to yield theories which hold better against new empirical data.
    In ML, the use of simplicity is most associated with the bias-variance tr...
    13 min