The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Results from the interpretability hackathon by Esben Kran
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Results from the interpretability hackathon, published by Esben Kran on November 17, 2022 on LessWrong.
    We ran a mechanistic interpretability hackathon (original post) with 25 projects submitted by ~70 participants. Here we share the winning projects but many of the others were also incredibly interesting. In summary:
    An algorithm to automatically make the activations of a neuron in a Transformer much more interpretable.
    Backup name mover heads from “Interpretability in the Wild” have backup heads and all of these are robust to the ablation distribution.
    The specificity benchmark in the ROME and MEMIT memory editing papers does not represent specificity well. A simple modulation shows that factual association editing bleeds into related texts, representing "loud facts".
    TCAV used on an RL agent for a connect four game can have its neural activation compared to the provably best solution as a pilot for comparing learned activations more generally to human-made solutions.
    Thank you to Sabrina Zaki, Fazl Barez, Thomas Steinthal, Joe Hardie, Erin Robertson, Richard Annilo, Itay Yona, other jam site organizers and all the participants for making it all possible.
    Investigating Neuron Behaviour via Dataset Example Pruning and Local Search
    By Alex Foote
    Abstract: This report presents methods for pruning and diversifying dataset examples that strongly activate neurons in a language model, to facilitate research into understanding the behaviour of these neurons. The pruning algorithm takes a dataset example that strongly activates a specific neuron and extracts the core sentence before iteratively removing words, to find the shortest substring that preserves a similar pattern and magnitude of neuron activation.
    This removes extraneous information, providing a much more concise input that is easier to reason about. The extracted substring, referred to as a Minimal Activating Example (MAE), is then used as a seed for local search in the input space. Using BERT, each word in the MAE is replaced by its most probable substitutes, and neuron activation is re-assessed. This creates positive and negative inputs that shed much more light on neuron behaviour than dataset examples alone.
    In two case studies we identify neuron behaviours that were not obvious from the raw dataset examples using this combination of pruning and local search. These methods could facilitate and significantly speed up research into neuron behaviour in language models, which is a key aspect of model interpretability.
    An example of the technique in action can be seen below where it is much more interpretable what the neuron activates for compared to looking through the dataset examples. The example is neuron 1794 in layer 3 of the 8-layer SoLU model.
    PromptPrompt TypeActivation.](bjc201156f1){#fig1Positive2.90.](bjc201256f1)fig1Positive2.90.](bjc201256f1]fig1 Positive2.90 .](bjc201256f1){#;Positive2.90 .](bjc201256f1){#} Positive 2.90 (bjc201256f1){#fig1Negative0.04#bjc201256f1){#fig1Negative0.05 .](\\){#fig1Negative0.03.](thumb){#fig1Negative0.02
    Neel’s comment: This is a really awesome project! I hadn't thought of this idea, and it seems like an intuitive and valuable augmentation to max activating dataset examples. And I really love the use of BERT and the fact that it's automated. I'd love to chat about developing this into a more robust + usable tool, or eg integrating it into EasyTransformer. My main feedback is that this is an autoregressive, GPT-2 style model. This means that neuron activations on e.g. position 5 are ONLY a function of tokens 0 to 5, NOT of token 6. So pruning from the end of the word or augmenting by messing with words after the max act is totally meaningless.
    See the code and research here.
    Backup Transformer Heads are Robust to Ablation Distribution
    By Lucas Jun Koba Sato, Gabe Mukobi...
    12 min
  • EA - AI Forecasting Research Ideas by Jaime Sevilla
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Forecasting Research Ideas, published by Jaime Sevilla on November 17, 2022 on The Effective Altruism Forum.
    Overview
    The linked document contains a collection of AI Forecasting research ideas, prepared by some Epoch employees in a personal capacity
    We think that these are interesting and valuable projects that research interns or students could look into, though they may vary in difficulty (depending on your background/experience)
    This is the result of a quick brainstorming and curation, rather than a thorough deliberative process. We encourage a critical outlook when reading them
    You may also be interested in these other forecasting research ideas suggested by Jaime Sevilla
    You can use Epoch’s database as a resource for finding notable machine learning papers with parameter, compute, and dataset sizes. On Epoch website you can also find our past research, a tool for visualising the dataset, and some other tools like this compute calculator.
    Please feel free to contact us for clarification about these questions!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Media attention on EA (again) by Julia Wise
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Media attention on EA (again), published by Julia Wise on November 17, 2022 on The Effective Altruism Forum.
    Six months ago I wrote that EA would likely get more attention soon.
    Well, that’s certainly true now (and not for the reason I expected). Here’s where I think things stand now in this regard:
    EA has more communications expertise than it did six months ago. My colleague Shakeel Hashim at CEA is focused on communications for all of EA, not CEA in particular.
    Journalists might be contacting organizations, groups, and individuals affiliated with EA. CEA’s usual advice about talking to journalists still stands. As someone who’s put my foot in my mouth more than once while talking to a journalist, I expect this is an especially hard time to do interviews. It’s particularly tricky because any comment you make publicly about FTX could have legal implications. Feel free to ask Shakeel for advice if you receive enquiries: [email protected]
    There will likely be a bunch of negative media pieces about EA. There’s probably not much for a typical EA to do about that.
    As Shakeel wrote here, the leaders of EA organizations can’t say a lot right now, and we know that’s really frustrating.
    For people working on projects that are able to continue, keeping up the heartbeat of EA’s work toward a better world is so valuable. Thank you.
    Doomscrolling is not that good for most of us.
    I don’t mean any of this as “stop discussing community problems and how to solve them.” It’s important work to reckon with whatever the hell just happened, what it means, and what changes we should make as a community.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Sadly, FTX by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sadly, FTX, published by Zvi on November 17, 2022 on The Effective Altruism Forum.
    It has been quite a past two weeks, with different spheres in deeply divided narratives. In addition to the liberation of Kherson City, the midterm elections that of course took forever to resolve and the ongoing hijinks of Elon Musk taking over Twitter, there was the complete implosion of Sam Bankman-Fried (hereafter ‘SBF’), his crypto exchange FTX and his crypto trading firm Alameda Research.
    The situation somehow kept getting worse several times a day, even relative to my updated expectations, as new events happened and new revelations came out.
    In the wake of those events, there are not only many questions but many categories of questions. Here are some of them.
    What just happened?
    What happened in the lead-up to this happening?
    Why did all of this happen?
    What is going to happen to those involved going forward?
    What is going to happen to crypto in general?
    Why didn’t we see this coming, or those who did see it speak louder?
    What does this mean for FTX’s charitable efforts and those getting funding?
    What does this mean for Effective Altruism? Who knew what when?
    What if anything does this say about utilitarianism?
    How are we casting and framing the movie Michael Lewis is selling, in which he was previously (it seems) planning on portraying Sam Bankman-Fried as the Luke Skywalker to CZ’s Darth Vader? Presumably that will change a bit.
    This post is my attempt to take my best stab at as much of this as possible, give my model of what happened and is likely to happen from here, what implications we can and should draw, and to compile as many sources as possible that I have found useful.
    This is a fast-moving complicated situation that involves a lot of lying and fraud. Anyone attempting to sort it all out, especially quickly, is going to make mistakes. I decided that this was the time when the value of synthesis exceeded the cost of such errors. Still, doubtless there will be mistakes here. I will correct them as they are discovered, either by myself or others. I apologize for them in advance.
    Thus I highly encourage you to read this post on the web rather than via an email or RSS version, in case there have been substantial revisions. There have as of yet not been any substantive revisions since publication, which was the morning of 11/17/22. The original version (and the one I will try hardest to keep updated) is here.
    Also, yes, long post is long. By all means read only the sections you care about.
    What The Hell Happened?
    Background: FTX is a crypto exchange largely owned and run by SBF. Alameda Research is a prop trading firm owned by SBF. FTT is FTX’s exchange token, where they commit a portion of profits to buying back FTT. Thus, FTT is a non-registered security, functionally similar to junior non-voting stock in FTX.
    The events of the past week, and the proximate cause of them, seem to have gone as follows, with some events likely slightly out of order or happening simultaneously.
    Alameda Research, SBF’s crypto trading firm, has its balance sheet leaked.
    The balance sheet contains a lot of FTT, such that if FTT loses its value it is not clear that Alameda would remain solvent.
    This raises questions. Some people notice. Caroline, CEO of Alameda, says this is incomplete and that Alameda is fine.
    CZ, head of Binance, who had been in various battles with SBF, notices. He announces the intention to sell >$500mm of FTT, but does not actually sell.
    Alameda offers to buy all his FTT at $22/coin, almost full market price at the time but a deal cannot be reached.
    Other people sell a lot of FTT. There are not other buyers. Alameda spends capital trying to defend the $22 price, and ultimately fails. They continue to sell everything they can to support FTT and to pay FTX depositors, but ulti...
    1 hr 12 min
  • LW - Sadly, FTX by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sadly, FTX, published by Zvi on November 17, 2022 on LessWrong.
    It has been quite a past two weeks, with different spheres in deeply divided narratives. In addition to the liberation of Kherson City, the midterm elections that of course took forever to resolve and the ongoing hijinks of Elon Musk taking over Twitter, there was the complete implosion of Sam Bankman-Fried (hereafter ‘SBF’), his crypto exchange FTX and his crypto trading firm Alameda Research.
    The situation somehow kept getting worse several times a day, even relative to my updated expectations, as new events happened and new revelations came out.
    In the wake of those events, there are not only many questions but many categories of questions. Here are some of them.
    What just happened?
    What happened in the lead-up to this happening?
    Why did all of this happen?
    What is going to happen to those involved going forward?
    What is going to happen to crypto in general?
    Why didn’t we see this coming, or those who did see it speak louder?
    What does this mean for FTX’s charitable efforts and those getting funding?
    What does this mean for Effective Altruism? Who knew what when?
    What if anything does this say about utilitarianism?
    How are we casting and framing the movie Michael Lewis is selling, in which he was previously (it seems) planning on portraying Sam Bankman-Fried as the Luke Skywalker to CZ’s Darth Vader? Presumably that will change a bit.
    This post is my attempt to take my best stab at as much of this as possible, give my model of what happened and is likely to happen from here, what implications we can and should draw, and to compile as many sources as possible that I have found useful.
    This is a fast-moving complicated situation that involves a lot of lying and fraud. Anyone attempting to sort it all out, especially quickly, is going to make mistakes. I decided that this was the time when the value of synthesis exceeded the cost of such errors. Still, doubtless there will be mistakes here. I will correct them as they are discovered, either by myself or others. I apologize for them in advance.
    Thus I highly encourage you to read this post on the web rather than via an email or RSS version, in case there have been substantial revisions. I will attempt to update others but the Substack version is canonical. There have as of yet not been any substantive revisions since publication, which was on the morning of 11/17/22.
    Also, yes, long post is long. By all means read only the sections you care about.
    What The Hell Happened?
    Background: FTX is a crypto exchange largely owned and run by SBF. Alameda Research is a prop trading firm owned by SBF. FTT is FTX’s exchange token, where they commit a portion of profits to buying back FTT. Thus, FTT is a non-registered security, functionally similar to junior non-voting stock in FTX.
    The events of the past week, and the proximate cause of them, seem to have gone as follows, with some events likely slightly out of order or happening simultaneously.
    Alameda Research, SBF’s crypto trading firm, has its balance sheet leaked.
    The balance sheet contains a lot of FTT, such that if FTT loses its value it is not clear that Alameda would remain solvent.
    This raises questions. Some people notice. Caroline, CEO of Alameda, says this is incomplete and that Alameda is fine.
    CZ, head of Binance, who had been in various battles with SBF, notices. He announces the intention to sell >$500mm of FTT, but does not actually sell.
    Alameda offers to buy all his FTT at $22/coin, almost full market price at the time but a deal cannot be reached.
    Other people sell a lot of FTT. There are not other buyers. Alameda spends capital trying to defend the $22 price, and ultimately fails. They continue to sell everything they can to support FTT and to pay FTX depositors, but ultimately cannot keep pace....
    1 hr 12 min
  • EA - Samo Burja: What the collapse of FTX means for effective altruism by peterhartree
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Samo Burja: What the collapse of FTX means for effective altruism, published by peterhartree on November 17, 2022 on The Effective Altruism Forum.
    Samo Burja has the best analysis I've seen so far.
    CONTENT WARNING: Samo's analysis may be upsetting and demoralising.
    If you're feeling low, anxious, or otherwise in a bad way, I strongly recommend you bookmark this post and come back when you're on better form.
    If you're ready for a calm attempt to understand what is happening, and what this all means, read on.
    This post is written in a personal capacity
    I'm sharing this in a personal capacity.
    I am not speaking on behalf of any current or past employer.
    I asked a couple of people for their thoughts before I posted, but the decision to post is mine alone.
    Disclosure
    I personally received ~$60K funding from FTX senior staff (Nishad & Claire) in 2020, to pursue a year of independent study. (We had no pre-existing social or professional relationship before they made the grant.)
    Two projects I run, Radio Bostrom and Parfit Archive, were funded via the FTX Future Fund Regrantors Programme.
    80,000 Hours, a previous employer for whom I currently serve as a freelance advisor, also received significant donations from SBF and Alameda.
    I don't follow crypto closely, but I did put about $10K into FTX wallets over the past couple of years. I've not logged into these accounts for months, so I've no idea what they are/were worth.
    Personal comment
    I don't follow crypto closely. My understanding of things over at FTX and Alameda is entirely based on what is being reported on Twitter and the newspapers.
    I can't verify all of the factual claims that I've excerpted below. The ones I actually know about all seem correct, to the best of my knowledge.
    I am personally feeling calm and fine about things. Part of me is devasted, another part is angry. But I am good at compartmentalising. My capability for questionable gallows humor remains in evidence.
    I agree with (put high credence on) roughly all of Samo's takes that are excerpted below.
    I semi-independently arrived at most of these takes between Wednesday 9 - and Friday 11 November, and they've been fairly confident and robust since then. My thinking was supported by many conversations over those days (mostly not with staff from Effective Ventures, CEA or 80K—who are all extremely busy).
    For me, this analysis is a big positive update on Samo and his team's general capability for insight and analysis. I have followed Samo for a while, read his Great Founder Theory, and listened to at least 10 interviews. I always have a difficult time assessing claims about history and historical forces, though—especially when they have fluent, charismatic and media-savvy proponents.
    Selected excerpts
    The moral authority and intellectual legitimacy of EA will be reduced
    Most importantly, the collapse of FTX will reduce the moral authority and intellectual legitimacy of the “Effective Altruism” (EA) movement, which has become a rising cultural force in Silicon Valley and a rising financial force in global philanthropy. Sam Bankman-Fried was a highly visible example of an “Effective Altruist.” He was a major donor both to the movement and to causes favored by the movement, for example committing $160 million in grants through the FTX Future Fund. Bankman-Fried also frequently boosted its ideology in press interviews and reportedly only earned money so that he could donate more of it to the cause, an ethos known as “earn to give” that was invented by the EA movement.
    Good summary of SBF political activity
    Bankman-Fried has also shown a serious interest in engaging with mainstream U.S. politics. He has funded political candidates aligned with his own views in congressional primary races and ballot initiatives in California. In 2020, he became President Joseph Bide...
    12 min
  • EA - Introducing School of Thinking by Luca Parodi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing School of Thinking, published by Luca Parodi on November 17, 2022 on The Effective Altruism Forum.
    (Cross-posted on LessWrong)
    Introduction
    School of Thinking (SoT) is a media startup.
    Our purpose is to spread Effective Altruist, longtermist, and rationalist values and ideas as much as possible to the general public by leveraging new media. We aim to reach our goal through the creation of high-quality material posted on an ecosystem of YouTube channels, profiles on social media platforms, podcasts, and SoT's website.
    Our priority is to produce content in English and Italian, but we will cover more languages down the line. We have been funded by the Effective Altruism Infrastructure Fund (EAIF) and the FTX Future Fund.
    Summary
    School of Thinking was launched on Instagram in 2020 by Luca Parodi. For a year and a half, Luca's goal has been to do rationalist outreach on Instagram in Italian. The profile reached 18.000 followers in March 2022. In January 2022, Luca received his first grant (from EAIF) to leave his job, work full-time on the project, and expand it on other platforms in Italian. The goal at that moment was to do EA community building in Italy.
    In March 2022, Eloisa Margherita Calafiore joined the project as a co-founder and Chief Operating Officer, and in April 2022, School of Thinking received a grant from FTX Future Fund to expand globally. Being able to upscale the project, the goal shifted towards doing outreach more globally (whilst continuing our local work in Italian).
    Our current strategy is to focus on creating weekly long-format videos for YouTube and daily short-format videos and other content (e.g. carousels and IG stories) on TikTok, YouTube, and Instagram. In the last few weeks, we had several viral videos on Instagram Italy (+630.000 views in the last 30 days). We have a total of ~25.000 followers across six platforms and two languages and are growing steadily.
    We create educational content about several important concepts related to rationalism, longtermism, and effective altruism. We aim to explain them to the general public in an easy, engaging, and practical yet accurate way.
    We have two paths to impact, listed here in descending order of importance:
    Attracting new people in EA
    Help people new to the Effective Altruist, longtermist, and rationalist communities become more engaged.
    Spreading Good Values and Improving Reflective Processes
    Raise awareness about specific topics in untouched but influential portions of the general population
    Increase epistemic hygiene
    Spreading important ideas
    Our History
    The First Few Years
    The general idea for School of Thinking (SoT) was formulated for the first time by me (Luca Parodi) in 2017 when I discovered LessWrong. Nonetheless, I worked on it informally for years before launching the project. The official launch happened in July 2020, with an Instagram profile with the bold goal of becoming the Italian LessWrong's voice on social media. For a year and a half, SoT was just a side activity while I worked as a management consultant. I dedicated an average of 15 extra work hours per week of my time to this project.
    In September 2021, the Instagram profile reached 10.000 followers, and in November 2021, I applied for the Effective Altruism Infrastructure Fund to leave my job and work full time on School of Thinking. The goal was to move to other platforms - like YouTube or a podcast - ‌and leverage School of Thinking's audience to grow the Italian EA community. EAIF accepted my grant request, so I left my job and started working on SoT full-time in January 2022.
    Rapidly, I realized that I was one of the few full-time content creators with a strong track record (18k followers on IG in 17 months) within the global EA community, so I started thinking globally. With Eloisa Margherita Calafiore, n...
    18 min
  • EA - late entry for the EA criticism contest by eigenrobot
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: late entry for the EA criticism contest, published by eigenrobot on November 17, 2022 on The Effective Altruism Forum.
    I recently wrote a post digesting my impression of what the SBF meltdown means for EA. I hadn't intended to post it here (I don't identify as an EA and this feels a bit like barging into a conversation) but an EA mutual suggested that I ought to, and that it would be welcome. I'm still not sure the tone is quite right for this forum, and my epistemic confidence isn't especially high, but on the off chance any of it's useful: here you are.
    Quick summary: I think EA as a movement is extremely vulnerable to capture by individuals and organizations with very different value systems, but there are some things that might be done to at least mitigate this danger.
    Good luck guys.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - Some research ideas in forecasting by Jsevillamol
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some research ideas in forecasting, published by Jsevillamol on November 15, 2022 on LessWrong.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - Stop Overreacting re: FTX / SBF by MattBall
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Stop Overreacting re: FTX / SBF, published by MattBall on November 16, 2022 on The Effective Altruism Forum.
    Written as tomorrow's public blog post in part as a reaction to this, but given what I see on this forum and what I hear from otherwise level-headed individuals, I'm sharing it here now:
    As anyone who has read my blog or Losing My Religions knows, I have my disagreements with some effective altruists (EAs). I also have some fundamental philosophical differences. But after this FTX fiasco, some non-EAs are taking one person's failures as an excuse to mock and condemn the entire idea of effective altruism. (And some EAs are losing their minds and questioning their very identity and "purpose" in life.)Think about it. Fine, SBF seemingly had lots of money. And he said he was pursuing money for EA causes. But he is just one guy! His money doesn’t make him any more important than anyone else; even less so because his thoughts and ideas are not central to EA.Just because one Effective Altruist did things we don’t like doesn’t mean we throw out utilitarianism or toss Peter Singer under the bus, just as we don’t become Republicans because Bill Clinton was sleazy or give up on meditation because some teachers were predators.The EA community is a lot of people.
    Some of them are bad. Some of them are mentally unstable. Some of them are crazy. We don’t judge all of EA on one person – that is lunacy.Let’s think about this from first principles.These people are altruists. In short, they care about others, not just themselves. They want others to not suffer but instead thrive. But they also know:
    Some individuals are suffering more than others. My hands and back hurt and my tinnitus is screaming, but I can say with absolute certainty that many people out there are suffering worse, such that they wish to die. So if you are a rational altruist, you recognize differences in need, regardless of any other factors.
    Efforts and money have different impacts in different situations. An additional million dollars to Harvard has a different impact than a million dollars in Polio eradication. (And both have a different sign compared to a million dollars to the homophobic Church of Jesus Christ and Latter-day Saints.)
    Now there is a lot more to say about effective altruism, and no one knows this more than EAs. But this is the bottom line: They want to be as effective as possible in helping others.Is this true of the critics? Or do the critics simply not want to feel guilty? Guilty about buying a fancy car, owning several houses, taking lavish vacations? Giving to their church, their alma mater, their kid's soccer team? Caring infinitely more for their kids than every non-human animal on the planet?Before you pile on along with those attacking effective altruists, ask yourself if doing so is the best way to help the less fortunate.
    And note: If you are sure that you know what EAs should do and how they should do it, I am willing to bet all my savings that you are wrong. These questions are simply too complex to have certain questions.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min