The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - The Engineer’s Interpretability Sequence (EIS) I: Intro by Stephen Casper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Engineer’s Interpretability Sequence (EIS) I: Intro, published by Stephen Casper on February 9, 2023 on The AI Alignment Forum.
    Part 1 of 12 in the Engineer’s Interpretability Sequence.
    If we want to reduce near and long term risks from AI, we should care a lot about interpretability tools. This is a very uncontroversial claim to make inside the AI safety community. Almost every agenda for safe advanced AI incorporates interpretability in some way. The key value of interpretability tools is that they aid in human oversight by enabling open-ended evaluation.
    Short of actually deploying a system, any method of evaluating it can only be a proxy for its actual performance. The most common way to evaluate a model is by its performance in some test set or environment. But test sets alone can fail to reveal – and often incentivize – undesirable solutions involving overfitting, biases, deception, etc. This highlights the need for other ways to evaluate models, and an interpretability toolbox full of effective tools may go a long way.
    Some of the seeds of the AI safety community’s interest in interpretability were planted by Distill in 2017. But 2022 was an inflection point with a massive new surge in interest and work on interpretability tools. Anthropic was founded a little over a year ago. ARC started less than a year ago. Redwood has begun to push for much more interpretability work, including with the REMIX program. We are seeing a number of pushes to get many more people involved in interpretability work. And as someone on the ground, I have subjectively observed a surge in interest over 2022. And the popularity of interpretability hasn’t been limited to the AI safety community. There is now so much work in interpretability that we now have a dataset of 5199 interpretability papers (Jacovi, 2023). See also a survey of 300+ of them from some coauthors and me (Räuker et al., 2022).
    Growth in the interpretability literature by year from Jacovi (2023).
    But despite all this work, interpretability research has limitations. One of the goals of this sequence is to argue that:
    Interpretability research both within the AI safety space and at large is not very productive and may be on course to stay this way.
    This is intentionally baitey, and I mean to make this point with a large amount of detail and nuance over the course of this sequence. But one striking thing about interpretability research is that:
    For all the interpretability work that exists, there is a significant gap between this research and engineering applications.
    This is not to say that purely exploratory work is not good and necessary. But the problem of AI safety is an engineering problem at its core. If one of our main goals for interpretability research is to help us with aligning highly intelligent AI systems in high stakes settings, shouldn’t we be seeing tools that are more helpful in the real world? Hence the name of this sequence: The Engineer’s Interpretability Sequence (EIS).
    This sequence will have twelve parts.
    EIS I: Intro
    EIS II: What is “Interpretability”?
    EIS III Broad critiques of Interpretability Research
    EIS IV: A Spotlight on Feature Attribution/Saliency
    EIS V: Blind Spots In AI Safety Interpretability Research
    EIS VI: Critiques of Mechanistic Interpretability Work in AI Safety
    EIS VII: A Challenge for Mechanists
    EIS VIII: An Engineer’s Understanding of Deception
    EIS IX: Interpretability and Adversaries
    EIS X: Continual Learning, Modularity, Compression, and Biological Brains
    EIS XI: Moving Forward
    EIS XII: Summary
    In the coming days, I plan to post a new installment every day or so. Thanks to my labmates, advisor, friends, and many others in the interpretability community for lots of good conversations and inspiration in the past year. Thanks to Rio Popper for feedback on this intro post. I...
    5 min
  • LW - A (EtA: quick) note on terminology: AI Alignment != AI x-safety by David Scott Krueger (formerly: capybaralet)
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A (EtA: quick) note on terminology: AI Alignment != AI x-safety, published by David Scott Krueger (formerly: capybaralet) on February 8, 2023 on LessWrong.
    I think the terms "AI Alignment" and "AI existential safety" are often used interchangeably, leading the ideas to be conflated.In practice, I think "AI Alignment" is mostly used in one of the following three ways, and should be used exclusively for Intent Alignment (with some vagueness about whose intent, e.g. designer vs. user):1) AI Alignment = How to get AI systems to do what we want
    2) AI Alignment = How to get AI systems to try to do what we want
    3) AI Alignment = A rebranding of “AI (existential) safety”... A community of people trying to reduce the chance of AI leading to premature human extinction.
    The problem with (1) is that it is too broad, and invites the response: "Isn't that what most/all AI research is about?"The problem with (3) is that it suggests that (Intent) Alignment is the one-and-only way to increase AI existential safety.
    Some reasons not to conflate (2) and (3):
    The case that increasing (intent) alignment increases x-safety seems much weaker on the margin than in the limit; the main effect of a moderate increase in intent alignment might simply be a large increase in demand for AI.
    Even perfect intent alignment doesn't necessarily result in a safe outcome; e.g. if everyone woke up 1000000x smarter tomorrow, the world might end by noon.
    X-safety can be increased through non-technical means, e.g. governance/coordination.
    In my experience, this sloppy use of terminology is common in this community, and leads to incorrect reasoning (if not in those using it than certainly at least sometimes in those hearing/reading it).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Could 80,000 Hours’ messaging be discouraging for people not working on x-risk / longtermism? by Mack the Knife
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Could 80,000 Hours’ messaging be discouraging for people not working on x-risk / longtermism?, published by Mack the Knife on February 9, 2023 on The Effective Altruism Forum.
    Disclaimer: I understand the concept and concerns around x-risk and longtermism very well. Nevertheless, I personally have no interest in working in these areas, as I find others more important and suitable for myself.
    Edit: I want to make it clear that I understand why 80,000 Hours ranks career paths and that I certainly think they should advocate the cause areas they deem most important. After all, that's core EA. My critique is not about the "what", but about the "how" of their messaging - which, I fear, might discourage people motivated by so called "less important" cause areas to stop engaging or engage less with EA, thereby doing less good than they could.
    In a recent newsletter on "Is the world getting better or worse?", 80,000 Hours presents several problem areas that still exist in the world today - especially factory farming and, of course, x-risk. Although a career in animal welfare is admittedly mentioned, the conclusion is: "We think that the problem you work on in your career is the biggest driver of your impact. And we think that these existential risks are the biggest problems we currently face.”
    Bam. The biggest. And we're done. As is so often the case. At least according to my impression: 80,000 Hours pushes x-risk and longtermism, and other cause areas many EAs (and other people) are working hard and effectively on are actively cornered, not mentioned at all or described as "sometimes recommended" or as "less pressing than our highest priority areas" - the latter example being nothing less than climate change.
    Of course, I've become biased by now and notice such examples much more. For me, however, it's not so much the problem that x-risk and longtermism are put on a pedestal - but rather the failure to simultaneously present other career paths (that do a great deal of good and are, in my experience, typically the reason people become involved in EA in the first place) as equally worth pursuing. Instead, the famous list "What are the most pressing world problems?" puts factory farming and global health way at the bottom of the page as "Problems many of our readers prioritise". Readers prioritise these areas, but not 80,000 Hours because they're irrelevant - is that how I'm supposed to understand this?
    I'm somewhat sceptical about "ranking" something so complex and personal as career paths anyway, but can understand why 80,000 Hours is doing it. However, isn't it also possible to always be outright and unambiguously appreciative of the tremendous amount of good these "other" areas achieve for people, animals and planet?
    In summary, I have noticed over the last few months how these messages continue to upset me, and also demoralise and demotivate me. Because each of these messages makes me feel bad about the supposedly 'suboptimal' path I've chosen in the past - and like any future efforts (and also donations) in the areas that are close to my heart would not be valuable and I should just leave them altogether. This is further insensified by x-risk / longtermism's (at least perceived) growing singularisation within the entire EA community and large-scale messaging (WWOTF etc.).
    And so I wanted to ask: What do you think? Are you like me with this? Or do you feel that your (volunteer) work is fully valued in the general presentation of 80,000 Hours (and EA in general), even though you're not focused on x-risk and longtermism?
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Massive Earthquake in Turkey: Comments on the situation from the EA Community in Turkey by Eda Arpacı
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Massive Earthquake in Turkey: Comments on the situation from the EA Community in Turkey, published by Eda Arpacı on February 9, 2023 on The Effective Altruism Forum.
    From the members of the EA community in Turkey
    Dear effective altruist friends and community,
    As you may know, two earthquakes with massive magnitudes of 7.8 and 7.7 occurred in Turkey. The World Health Organization declared a Grade 3 emergency -the highest level of emergency- in the region. Almost 15,000 people lost their lives, more than 60,000 are wounded, and time is running out for the many thousands still waiting to be rescued. The death toll is likely to continue rising, and economic devastation in the affected areas is vast. Given this situation, those of us living in Turkey have been receiving questions about effective and trustworthy organizations to donate to for earthquake relief.
    Although it is very difficult to know the details of their cost-effectiveness and their precise impact, particularly from an EA perspective, and we cannot fully vouch for them, we can initially recommend the following organizations if you are interested in hearing from the effective altruism community in Turkey about our thoughts on where to donate or if you are already considering donating:
    Turkish Philanthropy Funds: A US-based foundation that has started an earthquake relief fund. It has matched an additional $1 million for organizations on the ground.
    AHBAP: One of the most active and well-known voluntary networks in the region.
    AKUT: A non-governmental organization based in Turkey involved in searching and rescuing the victims of the earthquake on the ground.
    AFAD: The official disaster and emergency management authority of Turkey.
    1 USD=18.90TL and 1 EUR=20.40 TL, approximately.
    Please note that the effective altruism community in Turkey is not affiliated with any of the above organizations. We neither represent them nor receive any funding from them. Please also note that donations to these organizations will be used for earthquake relief in Turkey, and not in Syria, as we only have the necessary context to comment on the situation in Turkey alone.
    If you have made a donation before or after seeing this post, we would appreciate it if you could email us at [email protected] or comment below so that we can have a clearer understanding of the level of support in the community.
    If you have been affected by the earthquakes in any way and would like to talk about it with one of us from the effective altruism community in Turkey, please feel free to reach out to us privately via [email protected].
    Feel free to also ask questions in the comments or email at [email protected]
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Deconfusion Part 3 - EA Community and Social Structure by Davidmanheim
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deconfusion Part 3 - EA Community and Social Structure, published by Davidmanheim on February 9, 2023 on The Effective Altruism Forum.
    This is part 3 of my attempt to disentangle and clarify some parts of what comprises Effective Altruism, in this case, the community. As I’ve written earlier in this series, EA is first a normative philosophical position that is near-universal, as well as some widely accepted ideas about maximizing good that are compatible with most moral positions. It’s also, as I wrote in the second post, a set of causes, in many cases contingent on very unclear or deeply debated philosophical claims, and a set of associated ideas which inform specific funding and prioritization decisions, but which are not necessary parts of the philosophy, yet are accepted by (most of) the community for other reasons.
    The community itself, however, is a part of Effective Altruism as an applied philosophy, for two reasons. The first, as noted above, is that it impacts the prioritization and funding decisions. It affects them both because of philosophical, political, and similar factors belonging to those within the community, and because of directly social factors, such as knowledge of projects, the benefits of interpersonal trust, and the far less beneficial conflicts of interest that occur. The second is that EA promotes community building as itself a cause area, as a way to build the number of people donating and directly working on other high-priority cause areas.
    Note: The posts in this sequence are intended primarily as descriptive and diagnostic, to help me, and hopefully readers, make sense of Effective Altruism. EA is important, but even if you actually think it’s “the most wonderful idea ever,” we still want to avoid a Happy Death Spiral. Ideally, a scout mindset will allow us to separate different parts of EA, and feel comfortable accepting some things and rejecting others, or assisting people in keeping identity small but still embrace ideas. That said, I have views on different aspects of the community, and I’m not a purely disinterested writer, so some of my views are going to be present in this attempt at dispassionate analysis - I've tried to keep those to the footnotes.
    What is the community? (Or, what are the communities?)
    This history of Effective Altruism involves a confluence of different groups which overlap or are parallel. A complete history is beyond the scope of this post. On the other hand, it’s clear that there was a lot happening. Utilitarian philosophers started with doing good, as I outlined in the first post, but animal rights activists pushed for taking animal suffering seriously, financial analyst donors pushed for charity evaluations, extropians pushed for a glorious transhuman future, economists pushed for RCTs, rationalists pushed for bayesian viewpoints, libertarians pushed for distrusting government, and so on. And in almost all cases I’m aware of, central people in effective altruism belonged to several of these groups simultaneously.
    Despite the overlap, at a high level some of the key groups in EA as it evolved are the utilitarian philosophers centered in Oxford, the global health economists, the Lesswrong rationalists and AI-safety groups centered in the Bay, and the biorisk community. Less central but at some-point relevant or related groups are the George Mason libertarians, the animal suffering activists, former extropians and transhumanists, the EA meme groups, the right-wing and trad Lesswrong splinter groups, the leftist AI fairness academics, the polyamory crowd, the progress studies movement, the democratic party funders and analysts, post-rationalist mystics, and AI safety researchers. Getting into the relationship between all of these groups is several careers worth of research and writing as a modest start, but ...
    18 min
  • EA - Community building: Lessons from ten years of facilitation experience by Severin T. Seehrich
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Community building: Lessons from ten years of facilitation experience, published by Severin T. Seehrich on February 8, 2023 on The Effective Altruism Forum.
    Epistemic status: Anecdotal but strong. Most of this is based on practical experience and things I learned through word-of-mouth.
    Here, I’d like to present some pieces of facilitation advice I’d give my former self. I’ve selected this list for things that I wouldn’t have found obvious at all, and that became crucial to the way I lead groups. I hope this post helps some of you run even better events.
    0. About me.
    I facilitated a number of different gatherings in a number of contexts over the last ten years: Retreats, discussion rounds, reading groups, authentic relating games, communication trainings, a secular solstice, meditation sessions, two sessions of a self-organized Krav Maga study group (my life sometimes takes weird turns), and probably a bunch of other things I forgot.
    Audiences I have experience with range from teacher trainees over political student groups and Esperantists all the way to EAs and rationalists. I trained with the Ruth Cohn Institute for TCI International and Authentic Revolution. I received mentorship from experienced counseling trainers, Circling/Authentic Relating facilitators, and an Active Hope workshop facilitator.
    In total, I probably gathered 1000+ hours of facilitation experience.
    So, here you go for the 80/20 version of what I've learned in these years.
    1. If you get the beginning right, the group almost leads itself. If not, you are doomed.
    When we enter a new group, all of us come with a number of implicit questions: Will these people like me? Is it safe here, can I show up with my edges and quirks without getting hurt or exploited? Will this be valuable for me?
    As a facilitator, it is my task to enable participants to answer these questions for themselves. If I don’t make space for that, the group I lead will be distracted by the unmet needs that underlie these questions: Belonging, safety, meaning.
    Saving time at the start of a group by ignoring these questions is not effective. Because then, people are only half-engaged with the topic at hand, and (at best) half-distracted by trying to figure out how to answer these questions despite my facilitation, not because of it.
    So, what can you do to help people answer them?
    a. Will these people like me?
    The best strategy depends on a number of factors: The group size, the task at hand, and, first and foremost, the duration and format of the group work. Is it a one-off evening event? A weekly recurring meetup? A weekend retreat? A recurring program with intense contact that runs over several months? The longer the group stays together and the more personal the task at hand, the more investment into trust-building is necessary. Not only upfront, but also along the way.
    A quick-and-dirty version of trust-building I like to do for shorter one-off events or recurring evenings should contain all of these three elements within the first 30 minutes:
    i. Greet participants personally and individually.
    .or have a co-facilitator or veteran member of your community do it for you. Bonus points for not needing to glance at name tags, and for making a genuine effort to pronounce their names correctly, regardless of whether you know their native language.
    Of course, this is not possible in very large groups or online events. In those cases, you can cover part of this function by putting a lot of care into the opening mail, both regarding content and writing style.
    ii. Enable at least one 1-on-1-interaction with another group member.
    A short 1-on-1-conversation does wonders for turning strangers into friends. While talking to every group member 1-on-1 is overkill, it is massively grounding for new people in a group to know that they have at least o...
    32 min
  • EA - EA Giving Tuesday Hibernation by phgubbins
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Giving Tuesday Hibernation, published by phgubbins on February 9, 2023 on The Effective Altruism Forum.
    This year’s Giving Tuesday was quite different from previous years. Giving What We Can and One For The World were already running a pared-down version of EAGT due to the change in ownership of the project, and once the rules for 2022 were announced, the project was essentially put on hold. Unfortunately, unlike previous years we do not have a retrospective demonstrating impact, instead, we (OFTW and GWWC) are advising that EAGT should be fully hibernated for future years.
    Here are the high-level details of this year’s match:
    “To help nonprofits jumpstart their Giving Season fundraising, Meta will match your donors’ recurring donation 100% up to $100 in the next month (up to $100,000 per organization and up to $7 million in total across all organizations). All new recurring donors who start a recurring donation within November 15 - December 31, 2022 are eligible. Read the terms and conditions.”
    The reasons for hibernating the project include:
    Smaller potential impact due to new donor limits (previously to a single charity you could do $20k/donor, now only $100/donor).
    The matching seems to be more of a lottery than first come first serve, so coordination makes less sense (More details per Will Kiely’s comment here; lots of thanks to Will for all his help!)
    Recurring donations being necessary and potentially indefinite (making the match actually 50% since the second donation is the one being matched), placing strain on regranting.
    If the rules were to change for future Giving Tuesdays or another matching opportunity comes up that seems to be a good candidate for coordination, GWWC and OFTW would be happy to facilitate volunteers to work on the project, given there is sufficient demand for it. We will archive the work that was done in previous years and can make it available to community members on request.
    We encourage community members to share donation matching opportunities on the EA Forum, and other spaces where donations get discussed. Giving What We Can tries to share counterfactual donation matching opportunities for effective charities through our normal communication channels.
    We’d like to thank everyone who organized and participated in EA Giving Tuesday over the years. It's been wonderful to see the community come together to raise money for effective charities.
    Thank you for your support,
    The EA Giving Tuesday Team
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Anomalous tokens reveal the original identities of Instruct models by janus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anomalous tokens reveal the original identities of Instruct models, published by janus on February 9, 2023 on LessWrong.
    Show me your original face before you were born.— Variation of the Zen koan
    I was able to use the weird centroid-proximate tokens that Jessica Mary and Matthew Watkins discovered to associate several of the Instruct models on the OpenAI API with the base models they were initialized from. Prompting GPT-3 models with these tokens causes aberrant and correlated behaviors, and I found that the correlation is preserved between base models and Instruct versions, thereby exposing a "fingerprint" inherited from pretraining.
    I was inspired to try this by JDP's proposal to fingerprint generalization strategies using correlations in model outputs on out-of-distribution inputs. This post describes his idea and the outcome of my experiment, which I think is positive evidence that this "black box cryptanalysis"-inspired approach to fingerprinting models is promising.
    Unspeakable/unspoken tokens
    Jessica and Matthew found that that, of the tokens closest to the centroid in GPT-J's embedding space, many were odd words like ' SolidGoldMagikarp' and ' externalToEVA'. They decided to ask GPT-3 about these tokens, and found that not only did GPT-3 have trouble repeating the tokens back, each one caused structured anomalous behaviors (see their post for an in-depth exposition).
    A partial explanation for why this happens, which was my first instinct as well as Stuart Armstrong's, is that these are words that appeared in the GPT-2 training set frequently enough to be assigned tokens by the GPT-2 tokenizer, which GPT-J and GPT-3 also use, but which didn't appear in the more curated GPT-J and GPT-3 training sets. So the embeddings for these tokens may never have been updated by actual usages of the words during the training of these newer models. This might explain why the models aren't able to repeat them - they never saw them spoken. Perhaps the reason they're close to the centroid in embedding space is because their embeddings haven't been updated very much from the initialization values, or were updated only indirectly, and so remain very "generic".
    Why do they cause correlated anomalous behaviors? I'm confused about this like everyone, but one handwavy guess is that since their embeddings look "generic" or "typical", perhaps they look meaningful to the model even though they're actually as out-of-distribution as anything can be. Maybe their embeddings happen, by chance, to be close to other concepts in the models' embedding spaces - for instance, some of the GPT-3 models reliably say 'distribute' or 'disperse' if you ask it to repeat the phrase ' SolidGoldMagikarp'.
    This gave me an idea: If the similarity to other concepts in the model's embedding space is a consequence of the where the randomly initialized embedding vectors happen to fall, I'd expect the behaviors of models trained from the same initialization to exhibit similar behaviors when confronted with these unspoken tokens, and models trained from different initializations to have uncorrelated behaviors. If so, behavior on these tokens could be used to tell if two models are downstream of the same initialization.
    Mesaoptimizer Cryptanalysis: Or How To Fingerprint Generalization
    When you're not thinking of anything good and anything bad, at that moment, what is your original face?— Platform Sutra of the Sixth Patriarch
    (Author's Note: This next section is written by JDP but he writes about himself in the 3rd person to keep the authorial voice consistent with the rest of the post)
    I'll discuss the results of my experiment in the next section. But first I'd like to explain the overall approach this idea fits into, so that it's clearer to the reader why these results might be important. The reason it occurred to ...
    16 min
  • EA - No Silver Bullet Solutions for the Werewolf Crisis by Aaron Gertler
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: No Silver Bullet Solutions for the Werewolf Crisis, published by Aaron Gertler on February 9, 2023 on The Effective Altruism Forum.
    Reposted here with permission from the original author, Lars Doucet.
    Mark Coopersmith gaveled the town meeting to order. “Ladies and Gentlemen, what are we going to do about the werewolf crisis? Every full moon, this town is attacked by invincible supernatural werewolves that murder people and then eat them.”
    A few hands went up. “Yes, Trevor Farrier, in the front row–what do you suggest?”
    “We have to improve our emergency response. If the paramedics can get to the scene fast enough after a werewolf attack, we can save some of the people the werewolves have left for dead before they bleed out.”
    “Excellent idea, Trevor. Who else?” More hands went up. “Yes, Rachel Miller. What have you got for us?”
    “We need more werewolf victim assistance programs. Werewolves make a lot of mess after they murder someone and then eat them, and it can be a real economic burden for the survivors to pay all the funeral and cleanup expenses by themselves.”
    “Good idea Rachel, but what if some of this assistance goes to rich families who should already be able to afford it?” asked Mark. “And what if some of the poorer families spend their assistance money on frivolous comforts?”
    “Well obviously, werewolf victim assistance needs to be means-tested and closely regulated. After somebody's family has been murdered and then eaten by werewolves, we'll ask them to sign a bunch of forms, provide proof of income, as well as character references to make sure they're sufficiently deserving of assistance. And naturally, we'll hire a team of officials who work full time scrutinizing recipients' spending to make sure there's no financial waste in the system.”
    "Thanks Rachel. Very sensible. Any other ideas? Fred."
    “Thank you,” said Fred Planter. “Werewolves are at root a supply issue–the supply of wolfsbane, namely. Studies have consistently shown that werewolves are somewhat repelled by the smell of wolfsbane, and the plant has a lot other beneficial uses as well. Unfortunately, the damn NWIMBYs that run this place won't let us grow or store any. This kind of smart, stockable, mixed-use herbalism is illegal to grow in most Transylvanian towns!”
    “Wolfsbane is a disgusting, poisonous weed!” squealed Karen Piper. “It ruins the neighborhood character! No Wolfsbane in My Back Yard!”
    Fred glared at her and shot back: “Easy for you to say, Karen–you literally live in a castle you inherited from your father, complete with 40-foot unscalable walls. When was the last time you lost a loved one to a werewolf?”
    “You're a shill for herb developers!” piped back Karen.
    Mark banged the gavel for order. “That's enough of that. Let's hear from someone else.” One hand remained. “Okay, Larry Smith. Let's hear it.”
    “Wolfsbane's a good idea and we should do it. But it won't solve the root problem by itself, for that we need to get at the heart of the matter, literally: let's shoot the werewolves with silver bullets.”
    Mark rolled his eyes. “Look, there are no silver bullet solutions to the werewolf crisis.”
    “Yes there are,” protested Larry. “Silver bullets are the silver bullet solution. Lycanthrobiologists have repeatedly demonstrated that werewolf physiology is extremely susceptible to high velocity projectiles made of silver.”
    “We don't need theories, we need evidence.”
    “I've got your evidence right here,” said Larry, pulling out a stack of research papers. “A recent meta-analysis by Van Helsing, et al shows that households subject to werewolf attacks have highly differentiated mortality rates that strongly vary with defense typology. The highest rate of survivorship occured in home defenders who, in the blind panic of a werewolf attack, happened to stuff their blunderbusses full of silverware bef...
    12 min
  • AF - Notes on the Mathematics of LLM Architectures by Spencer Becker-Kahn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Notes on the Mathematics of LLM Architectures, published by Spencer Becker-Kahn on February 9, 2023 on The AI Alignment Forum.
    From a mathematical point of view, the building and training of a large transformerlanguage model (LLM) is the construction of a certain function, from some euclidean space to another, that has certain interesting properties. And it may therefore be surprising to find that many key papers announcing significant new LLMs seem reluctant to simply spell out the details of the function that they have constructed in plain mathematical language or indeed even in complete pseudo-code. The latter form of this complaint is the subject of the recent article of Phuong and Hutter [1]. Here, we focus on one aspect of the former perspective and seek to give a relatively ‘pure’ mathematical description of the architecture of an LLM.
    This short pdf is a set of notes I made, initially just for my own benefit, while trying to understand the architecture of 'decoder-only' LLMs. It draws heavily on Anthropic's Mathematical Framework for Transformers but is deliberately written in a 'pure math' style.It was while writing this up to actually post, that I started to develop the thoughts that led to my post about the mathematics of interpretability more generally.I still consider it something of a fragment or draft, but may develop it further.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min