The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Infohazards vs Fork Hazards by jimrandomh
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Infohazards vs Fork Hazards, published by jimrandomh on January 5, 2023 on LessWrong.
    I think actual infohazardous information is fairly rare. Far more common is a fork: you have some idea or statement, you don't know whether it's true or false (typically leaning false), and you kow that either it's false or it's infohazardous. Examples include unvalidated insights about how to build dangerous technologies, and most acausal trade/acausal blackmail scenarios. Phrased slightly differently: "infohazardous if true".
    If something is wrong/false, it's at least mildly bad to spread/talk about it. (With some exceptions; wrong ideas can sometimes inspire better ones, maybe you want fake nuclear weapon designs to trip up would-be designers, etc). And if something is infohazardous, it's bad to spread/talk about it, for an entirely different reason. Taken together, these form a disjunctive argument for not spreading the information.
    I think this trips people up when they see how others relate to things that are infohazardous-if-true. When something is infohazardous-if-true (but probably false), peopple bias towards treating it as actually-infohazardous; after all,if it's false, there's not much upside in spreading bullshit. Other people seeing this get confused, and think it's actually infohazardous, or think it isn't but that the first person thinks it is (and therefore thinks the first person is foolish).
    I think this is pretty easily fixed with a slight terminology tweak: simply call thinks "infohazardous if true" rather than "infohazardous" (adjective form), and call them "fork hazards" rather that "infohazards" (noun form). This clarifies that you only believe the conditional, and not the underlying statement.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Prioritization Research Careers - Probably Good by Probably Good
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prioritization Research Careers - Probably Good, published by Probably Good on January 5, 2023 on The Effective Altruism Forum.
    We’re really happy to start 2023 with a new career path profile! This time, we look at prioritization researchers, a type of researcher who uses tools from a range of disciplines – spanning economics, philosophy, and mathematics – to help make decisions about how we can best utilize our resources to do good.
    Our overall impression is that this path is likely to be a high impact option for those who are a good fit, particularly if you stand a reasonable chance at getting into one of the most promising organizations that conduct prioritization research. We think this is a career path that readers of the forum might be particularly interested in exploring.
    Read the full profile here!
    Test tasks
    One of the things we’re most excited about in this profile are the inclusion of two test tasks created in collaboration with GiveWell. These tasks involve creating an intervention report and a cost-effectiveness analysis of a drug to reduce child mortality, and we think they’re a great way to test your fit for more quantitative, details-oriented prioritization research (though we also discuss other types of prioritization research in the profile).
    We link to instructions in the full path profile, or you can access them separately here.
    We estimate the tasks will take a combined total of 15-20 hours to complete, though this can be made shorter by just completing one or taking a less thorough approach.
    Feedback
    As always, we’d love to hear from you. We have exciting plans for this year, including an expansion of our team and lots of new content. As such, we’re eager to hear what types of content would be useful to the broader community and/or to specific groups.
    If you have any thoughts, let us know by leaving a comment below or emailing [email protected].
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • LW - How to slow down scientific progress, according to Leo Szilard by jasoncrawford
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to slow down scientific progress, according to Leo Szilard, published by jasoncrawford on January 5, 2023 on LessWrong.
    Leo Szilard—the physicist who first conceived of the nuclear chain reaction and who urged the US to undertake the Manhattan Project—also wrote fiction. His book of short stories, The Voice of the Dolphins, contains a story “The Mark Gable Foundation,” dated 1948, from which I will present to you an excerpt, without comment:
    “I’m thinking of setting up a trust fund. I want to do something that will really contribute to the happiness of mankind; but it’s very difficult to know what to do with money. When Mr. Rosenblatt told me that you’d be here tonight I asked the mayor to invite me. I certainly would value your advice.”
    “Would you intend to do anything for the advancement of science?” I asked.
    “No,” Mark Gable said. “I believe scientific progress is too fast as it is.”
    “I share your feeling about this point,” I said with the fervor of conviction, “but then why not do something about the retardation of scientific progress?”
    “That I would very much like to do,” Mark Gable said, “but how do I go about it?”
    “Well,” I said, “I think that shouldn’t be very difficult. As a matter of fact, I think it would be quite easy. You could set up a foundation, with an annual endowment of thirty million dollars. Research workers in need of funds could apply for grants, if they could make out a convincing case. Have ten committees, each composed of twelve scientists, appointed to pass on these applications. Take the most active scientists out of the laboratory and make them members of these committees. And the very best men in the field should be appointed as chairmen at salaries of fifty thousand dollars each. Also have about twenty prizes of one hundred thousand dollars each for the best scientific papers of the year. This is just about all you would have to do. Your lawyers could easily prepare a charter for the foundation. As a matter of fact, any of the National Science Foundation bills which were introduced in the Seventy-ninth and Eightieth Congresses could perfectly well serve as a model.”
    “I think you had better explain to Mr. Gable why this foundation would in fact retard the progress of science,” said a bespectacled young man sitting at the far end of the table, whose name I didn’t get at the time of introduction.
    “It should be obvious,” I said. “First of all, the best scientists would be removed from their laboratories and kept busy on committees passing on applications for funds. Secondly, the scientific workers in need of funds would concentrate on problems which were considered promising and were pretty certain to lead to publishable results. For a few years there might be a great increase in scientific output; but by going after the obvious, pretty soon science would dry out. Science would become something like a parlor game. Some things would be considered interesting, others not. There would be fashions. Those who followed the fashion would get grants. Those who wouldn’t would not, and pretty soon they would learn to follow the fashion, too.”
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Conversational canyons by Henrik Karlsson
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conversational canyons, published by Henrik Karlsson on January 4, 2023 on LessWrong.
    Conversations are streams: they pass by without leaving much behind. But if you add a notebook—if you write down short summaries of what you talk about so you can return to it in later conversations and expand—then you have a landscape for the conversation to flow through, and this changes it in interesting ways.
    When I started writing summaries, I thought it would be a small change. I already kept a detailed journal—I could just as well keep one for conversations, too. I could jot down a few notes about the key ideas, turning a two-hour conversation into 400 words or so. After a few months, however, it became clear that these notes were in fact not a small change. The notebook had entirely altered the shape of my conversations.
    Because I had the notes, I would review them if I was to speak with the same person again, adding comments about thoughts I’d had in the interim, rank ordering which topics I wanted to revisit, and so on. This way good ideas and topics would not be dropped but revisited and expanded. They would be connected to new insights, merged, and refined. Through this, the conversations became richer. They became more useful and interesting.
    The conversational stream was flowing through a landscape. On its banks ideas would wash up. Gradually, the banks would move, too—the landscape is shaped by the flow of words into a canyon. A canyon that shapes the stream of words in turn.
    I don’t write down the conversations I have with most people. But there are a few who I keep coming back to—friends with minds that are forever churning, minds like mysterious machines I can never fully grasp but am continually delighted by. When we talk they all take me to different places; when I add a notebook they all go deep.
    The first conversation I wrote down was with Torbjörn, a software engineer who has an uncanny ability to guess my thoughts. When he gets excited he SPEAKS IN ALL CAPS, and he gets excited about everything.
    At the time, we talked on the phone once a week for about two hours. I would set off from my house toward the ocean and cross into the nature reserve. Walking along the ravine and the cliffs for a few hours, I would talk to Torbjörn about open research questions and software projects we were involved with. He would ask me for strategic advice about his consultancy. We also talked about our relationships, aliens, the ravings of the world around us, mopeds.
    While talking, we would take turns jotting down keywords to aid our memory, and then, when I got home, I would add what we had said to my note-taking system. I revised old notes in light of how our thinking had evolved, adding new links between different ideas, and so on. (For this particular conversation, I used Obsidian. But the tools are not important. And I roughly follow the note-taking strategy that Andy Matuschak outlines here. But that is not important either.) If Torbjörn wanted some of the notes, I would email edited excerpts to him, so I didn’t have to format my notebook in a way that makes sense for him to read.
    First, I mixed the conversational notes in with my other thoughts. But I’ve since found that keeping the conversational notes separate from other notes is better—it creates a stronger sense of place. Now, I enter the Torbjörn notes, and all past conversations flow up. Mixed in with the other notes, they were diluted. (I do interlink it heavily with other notes, though.)
    During the week or so until we talked again, I would occasionally revisit the notes and add a thought or two. It kept the conversation alive in my mind. I’d update a list of topics I wanted to cover next time, shuffling their relative importance.
    This priority list, which was added to deal with the mess the notes became after a few...
    11 min
  • AF - Paper: Superposition, Memorization, and Double Descent (Anthropic) by Lawrence Chan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: Superposition, Memorization, and Double Descent (Anthropic), published by Lawrence Chan on January 5, 2023 on The AI Alignment Forum.
    (This is a follow up to Anthropic's prior work on Toy Models of Superposition.)
    The authors study how neural networks interpolate between memorization and generalization in the "ReLU Output" toy model from the first toy model paper:
    They train models to perform a synthetic regression task with T training points, for models with m hidden dimensions.
    First, they find that for small training sets, while the features are messy, the training set hidden vectors hi=WXi (the projection of the input datapoints into the hidden space) often show clean structures:
    They then extend their old definition of feature dimensionality to measure the dimensionalities allocated to each of the training examples:
    DXi = hi2∑j(^hihj)2and plot this against the data set size (and also test loss):
    This shows that as you increase the amount of data, you go from a regime with high dimensionality allocated to training vectors and low dimensionality allocated to features, to one where the opposite is true. In between the two, both feature and hidden vector dimensionalities receive low dimensionality, which coincides with an increase in test loss, which they compare to the phenomena of "data double descent" (where as you increase data on overparameterized models with small amounts of regularization, test loss can go up before it goes down).
    Finally, they visualize how varying T and m affects test loss, and find double descent along both dimensions:
    They also included some (imo very interesting) experiments from Adam Jermyn, 1) replicating the results, 2) exploring how weight decay interacts with this double descent-like phenomenon, and 3) studying what happens if you repeat particular datapoints.
    Some limitations of the work, based on my first read through:
    The authors note that the results seem quite sensitive to hyperparameters, especially for low hidden dimension m∈{2,3}. For example, Adam Jermyn's results differ from the Anthropic Interp team results (though the figures still look qualitatively similar).
    I'm still not super convinced how much results from the superposition work apply in practice. I'd be interested in seeing more work along the lines of the MNIST preliminary experiment done by Chris Olah at the bottom.
    (I'll probably have more thoughts as I think for longer.)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Running by Default by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Running by Default, published by jefftk on January 5, 2023 on LessWrong.
    In video games it's common for characters to
    run by default: it's faster, why wouldn't you? I think this is also a good approach to apply in real life, at least over short distances. It saves you time and it's a bit of exercise. Over pretty much any distance short enough that I'm not going to break a sweat I'll run.
    I've done some of this most of my life (I can still hear my middle school principal shouting to stop running in the hallway—Jeeeeff-reeeey!) though over the last five years or so I've pushed myself to do it more consistently. The more I do it the easier it gets, and the shorter commutes become.
    Note that this is another example of describing something I do that works for me, and that I expect would likely work for many more people than I see applying it. But probably not more than 30% of people, though?
    (Prompted by a coworker talking about the inconvenience of an 8min distance between two buildings at work that I think of as a 2-3min distance.)
    Comment via: facebook, mastodon
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - On being compromised by Gavin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On being compromised, published by Gavin on January 5, 2023 on The Effective Altruism Forum.
    Only the young and the saints are uncompromised. Everyone else has tried to do something in the world and eventually slipped up (or just been associated with someone else who slipped up).
    Say that you are compromised if it is easy for someone to shame you. This takes lots of forms:
    "We are all sinners", say the Christians.
    "We are all privileged", say the identitarians.
    "We all have some self-serving motives", says everyone sensible.
    "Even just living quietly we destroy things", say the environmentalists.
    "Even our noblest actions fall horribly short of the mark", say the EAs.
    Lots of people on this forum have struggled with the feeling of being compromised. Since FTX. Or Leverage. Or Guzey. Or Thiel. Or Singer. Or Mill or whatever.
    But this is the normal course of a life, including highly moral lives. (Part of this normality comes from shame usually being a common sense matter - and common sense morals correlate with actual harm, but are often wrong in the precise ways this movement is devoted to countering!)
    But the greater part of it being normal is that all action incurs risk, including moral risk. We do our best to avoid them (and in my experience grantmakers are vigilant about negative EV things), but you can't avoid it entirely. (Again: total inaction also does not avoid it.) Empirically, this risk level is high enough that nearly everyone eventually bites it.
    e.g.
    The EU is a Nobel peace prize winning organisation you might have heard of. But their Common Agricultural Policy causes billions of dollars of damage to poor-world farmers, and has been called a "crime against humanity".
    Mother Theresa's well-resourced clinics and hospices were remarkably incompetent and rarely prescribed pain medication, apparently under the belief that suffering brings us closer to God.
    Gandhi's (and Nehru's) economic policies perpetuated poverty to the tune of millions of dead children equivalents.
    The American labour hero Cesar Chavez sold out undocumented Mexicans and opposed immigration in a classic protectionist scheme.
    The Vatican.
    and so on.
    Despite appearances, this isn't a tu quoque defence of FTX! The point is to set the occasionally appropriate recriminations of the last month in context. You will make mistakes, and people will rightly hold you to them. It will feel terrible. If you join a movement it will embarrass you eventually. Sorry.
    (Someone could use the above argument to licence risky behaviour - "in for a penny". But of course, like anything, being compromised is a matter of degree. Higher degrees are to be avoided fervently, insofar as they are downstream of actual harm, which they probably are.)
    You might think that the idle (like the chattering classes) aren't compromised, but they are. They stood by while the millions suffered, despite their remarkable power to help.
    Quite true, since we are all living now rather than say under feudalism.
    Maybe this sounds like a strawman to you, but consider our disdain for Mackenzie Scott giving her wealth to poor and artsy Americans.
    Bentham is perhaps the second most-demonised consequentialist - and yet he strikes me as nearly uncompromised. His much-mooted imperialism is not one, for instance. The most you can say is that he was a bit naive about state power, privacy, legibility.
    What's a prior then, if the incidence is 99%?
    Say 70 years in which to disgrace yourself. How many actions per year? Well, one tweet can do it, so potentially thousands. Call it 300.
    99% / 21000 = a 0.005% risk of compromise per-action. Clearly a very fragile estimate.
    He was also very racist, but this isn't the sort of thing that can plausibly fall under understandable moral risk.
    also sometimes wrongly
    I await a quantification of compromise, so that ...
    4 min
  • EA - Misleading phrase in a GiveWell Youtube ad by Thomas Kwa
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Misleading phrase in a GiveWell Youtube ad, published by Thomas Kwa on January 5, 2023 on The Effective Altruism Forum.
    In a sponsored segment for GiveWell on a video by the channel Half as Interesting, the narrator Sam Denby says:
    [...] Personally, I'd give to the Helen Keller Foundation, which I found through GiveWell, because they help save thousands of lives through distributing Vitamin A supplements to children. Vitamin A supplements can help save the lives of children suffering from vitamin A deficiencies and only cost one dollar to deliver a supplement and save a child.
    This seems to reinforce the misconception that saving lives in the developing world is incredibly cheap. GiveWell's cost effectiveness estimates actually range from ~$1500 to ~$27,000 for Helen Keller's various regional programs, so this is off by 3 OOMs.
    I'm not sure if this quote was under GiveWell's editorial control, but to the extent it was I'm disappointed. Surely GiveWell should try to prevent this kind of thing from happening in the future, even if the sponsoree is speaking for themself, the misleading statement is brief, the misinformation looks favorable to GiveWell, or other charities' ads are also misleading.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • EA - Results from the AI testing hackathon by Esben Kran
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Results from the AI testing hackathon, published by Esben Kran on January 2, 2023 on The Effective Altruism Forum.
    We (Apart Research) ran a hackathon for AI testing research projects with 11 projects submitted by 34 participants between the 16th and 18th December. Here we share the winning projects. See them all here. In summary:
    Found that unsupervised latent knowledge representation is generalizable and takes the first steps towards a benchmark using the ETHICS ambiguous / unambiguous examples with latent knowledge evaluation.
    Created a new way to use token loss trajectories as a marker for targeting our interpretability methods towards a focus area.
    Investigated three potential inverse scaling phenomena: Counting letters, chaining premises and solving equations. Found incidental inverse scaling on one of them and U-shaped scaling on another.
    Implemented Trojans into Transformer models and used a gradient arithmetic technique to combine multiple Trojan triggers into one Transformer model.
    (honorable mention) Invented a way to test how quickly models become misaligned by negative example fine-tuning.
    Thank you to Zaki, Fazl, Rauno, Charbel, Nguyen, more jam site organizers, and the participants for making it all possible.
    Discovering Latent Knowledge in Language Models Without Supervision - extensions and testing
    By Agatha Duzan, Matthieu David, Jonathan Claybrough
    Abstract: Based on the paper "Discovering Latent Knowledge in Language Models without Supervision" this project discusses how well the proposed method applies to the concept of ambiguity.
    To do that, we tested the Contrast Consistent Search method on a dataset which contained both clear cut (0-1) and ambiguous (0,5) examples: We chose the ETHICS-commonsense dataset.
    The global conclusion is that the CCS approach seems to generalize well in ambiguous situations, and could potentially be used to determine a model’s latent knowledge about other concepts.
    These figures show how the CCS results for last layer activations splits into two groups for the non-ambiguous training samples while the ambiguous test samples on the ETHICS dataset reveals the same ambiguity of latent knowledge by the flattened Gaussian inference probability distribution.
    Haydn & Esben’s judging comment: This project is very good in investigating the generality of unsupervised latent knowledge learning. It also seems quite useful as a direct test of how easy it is to extract latent knowledge and provides an avenue towards a benchmark using the ETHICS unambiguous/ambiguous examples dataset. Excited to see this work continue!
    Read the report and the code (needs updating).
    Investigating Training Dynamics via Token Loss Trajectories
    By Alex Foote
    Abstract: Evaluations of ML systems typically focus on average statistical performance on a dataset measured at the end of training. However, this type of evaluation is relatively coarse, and does not provide insight into the training dynamics of the model.
    We present tools for stratifying tokens into groups based on arbitrary functions and measuring the loss on these token groups throughout the training process of a Language Model. By evaluating the loss trajectory of meaningful groups of tokens throughout the training process, we can gain more insight into how the model develops during training, and make interesting observations that could be investigated further using interpretability tools to gain insight into the development of specific mechanisms within a model.
    We use this lens to look at the training dynamics of the region in which induction heads develop. We also zoom in on a specific region of training where there is a spike in loss and find that within this region the majority of tokens follow the loss trajectory of a spike, but a small set follow the inverse trajectory.
    Haydn & Esben’s ju...
    9 min
  • AF - Why I'm joining Anthropic by Evan Hubinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I'm joining Anthropic, published by Evan Hubinger on January 5, 2023 on The AI Alignment Forum.
    Personal blogpost. Previously: “What I'll be doing at MIRI”
    For the last three years (since I left OpenAI), I've been a Research Fellow at MIRI. Starting next week, however, I'm going to be stepping down from that position and joining Anthropic as a researcher instead.[1]
    To start with, some things that this does not mean:
    That I have anything in particular against MIRI.
    That I endorse everything Anthropic does.
    That I'm going to stop doing theoretical AI safety research.
    Any particular substantive change in my beliefs about AI safety.
    So why do I think this is a good idea? The most basic reason is just that I think things are heating up and it'll be valuable for me to be closer to the action. I think current large language models are getting quite scary but that there's a lot of work to be done in understanding exactly how and what to do about it—see e.g. the recent paper I collaborated with Anthropic on. I'll be splitting my time between theoretical and empirical work—both of which I'll continue to do—with the idea that being closer to current models should improve my ability to do both.
    I expect that a lot of my time will be spent on the Conditioning Predictive Models agenda I've been working on for the past ~6 months, which I expect will be published in the next month or so. Until then, this post and this post probably contain the best current public writeups of some of the basic ideas. That being said, I won't be particularly tied down to it and might end up deciding to work on something completely different (e.g. what happened the last time I wrote up a big agenda).
    Since I'm sure I'm going to be asked about it a bunch now, some of my thoughts on Anthropic as an organization (obviously all thoughts are my own):
    I think it's really impressive and a very positive sign (and costly signal) that Anthropic has been able to do a bunch of capability work without publishing it.
    I think Anthropic is doing some of the best AI safety work of any organization right now—e.g. Transformer Circuits.
    Nevertheless, Anthropic is undoubtedly working on improving the capabilities of their models, and that absolutely has negative externalities, e.g. in increasing overall competition between labs.
    That being said, as someone whose overall estimates of AI existential risk are on the pessimistic side, I think high-variance bets—e.g. build cutting edge models so we can do cutting-edge AI safety work, have more leverage to influence other AI labs, etc.—can often make a lot of sense, especially when combined with strategies for mitigating potential downsides (e.g. not publishing capabilities advances).
    Overall, I think Anthropic's current strategy seems pretty reasonable to me, but I don't have a very strong take here.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min