The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Citability of Lesswrong and the Alignment Forum by Leon Lang
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Citability of Lesswrong and the Alignment Forum, published by Leon Lang on January 8, 2023 on LessWrong.
    This post is short: as far as I know, there is no easy support to cite lesswrong or the alignment forum in academic papers. I think this should exist. This idea is related to the one on putting Alignment Forum/Lesswrong posts on arxiv.
    Such support already exists elsewhere, e.g. on Mathoverflow:
    I didn't try for long to figure out if this idea was already proposed before or whether there is already support for this which I missed.
    Thanks to Nadia Montazeri and Magdalena Wache for discussing this idea with me.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - Announcing the Launch of the NYU Wild Animal Welfare Program by Sofia Fogel
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Launch of the NYU Wild Animal Welfare Program, published by Sofia Fogel on January 9, 2023 on The Effective Altruism Forum.
    We are thrilled to announce the launch of the NYU Wild Animal Welfare Program later this month!
    The NYU Wild Animal Welfare (WAW) program aims to advance understanding about what wild animals are like, how humans and wild animals interact, and how humans can improve our interactions with wild animals at scale. We pursue this goal through foundational research in the humanities, social sciences, and natural sciences, as well as through outreach to academics, advocates, policymakers, and the general public. The team includes Becca Franks and Jeff Sebo as co-directors, me (Sofia) as coordinator, and Arthur Caplan, Lucius Caviola, Kyle Ferguson, Jennifer Jacquet, Dale Jamieson, Colin Jerolmack, Sonali McDermid, Danielle Spiegel-Feld, Christine Webb, and others as faculty affiliates.
    The program will launch on January 27, 2023 with a roundtable discussion titled "How can humans improve our interactions with wild animals at scale?" The panel will include program directors Becca Franks and Jeff Sebo and program affiliates Christine Webb, Colin Jerolmack, and Dale Jamieson. The discussion will cover an array of topics including:
    Why does wild animal welfare matter more than ever?
    What are the most urgent and actionable issues confronting wild animals?
    How does wild animal welfare relate to conservation biology and other fields?
    We will also have plenty of time for discussion with the audience. We welcome you to join us in person or online.
    We will soon be announcing additional spring events as well as opportunities for early-career researchers. If you are interested in receiving occasional updates about our work and offerings, we encourage you to sign up for our email list.
    Please also feel free to contact us with other inquiries.
    This launch follows on the heels of our October 2022 launch of the NYU Mind, Ethics, and Policy Program, which may also be of interest to readers of this post.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - A Study of EA Orgs’ Social Media by Stan Pinsent
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Study of EA Orgs’ Social Media, published by Stan Pinsent on January 9, 2023 on The Effective Altruism Forum.
    Summary
    I collected data on the social media accounts of 79 EA-related organisations. Key findings:
    Orgs have far more followers on Facebook and Twitter than on Instagram. Facebook accounts typically have more followers than Twitter accounts, but this varies between cause areas
    A number of Longtermism- and Infrastructure-focused orgs have stepped away from social media since 2021
    On Facebook, posting regularly correlates weakly with having a larger following
    Overall, it appears that EA-aligned organisations should consider their cause area within EA when forming a social media strategy
    Methodology
    I checked the Facebook, Twitter and Instagram accounts of each of the orgs on this list of EA-related organisations and collected data on 1) the number of days since last posting and 2) the number of followers. See the footnotes for details on methodology[1] and limitations[2] [Footnotes are included in the main text because the forum is being glitchy].
    You can find the full dataset here.
    Data Overview
    87% of the 79 organisations surveyed have an account on either Facebook, Twitter or Instagram
    Of the organisations on social media, only 58% had been active in the past week
    Most of the organisations use Facebook and Twitter and less than half of them use Instagram:
    There was a good spread of cause areas in the survey:
    Cause areaNumber of organisationsAnimal Advocacy12Longtermism15Global Health & Poverty18Infrastructure24Other10Total79
    Orgs to Watch
    A number of the organisations stood out for their aptitude on social media. Below are “ten organisations to watch”, with links to their respective accounts:
    OrgCause areaLinksReasons to watchThe Humane LeagueAnimal AdvocacyRegular posts and diverse content. Strong on all platformsAnimal EthicsFacebook powerhouse. Diverse Instagram content, cross-channel promotionFuture of Life InstituteLongtermismHarnessing clips and quotes from their podcast, highly active on all platformsSightsaversGlobal Health & Poverty40,000 tweets and countingGiveDirectlyPutting a human face on their work on InstagramAbdul Latif Jameel Poverty Action Lab Regular, diverse Twitter contentEvidence ActionEngaging visuals with every post80,000 HoursInfrastructureLonger, regular but infrequent Facebook postsHigh Impact AthletesEyeball-grabbing visuals on InstagramOur World in DataOtherMaking data sing on Instagram
    FB
    T
    IG
    FB
    T
    IG
    FB
    T
    IG
    FB
    T
    IG
    FB
    T
    IG
    FB
    T
    FB
    T
    IG
    FB
    T
    IG
    FB
    T
    IG
    FB
    T
    IG
    Follower Data
    Facebook or Twitter? It depends on the cause area.
    65% of organisations had more followers on Facebook than on Twitter. Comparing the median value of Facebook followers per Twitter follower within each cause area, we see significant variation (with the caveat that sample sizes are small).
    It appears that cause area has a strong correlation with which platform an organisation does best in: all Animal Advocacy orgs had more Facebook followers than Twitter followers, while 5 of 8 Longtermist orgs had the opposite trend.
    Why such a difference between the cause areas? I have a few suggestions:
    Anecdotally, Facebook is better than Twitter for harnessing broad audiences with shared interests. Animal Advocacy orgs may have been able to snare the audiences of big groups like Greenpeace and the WWF (3.1M and 3.6M Facebook followers respectively). The other cause areas don’t have such big peers to poach from (Oxfam has 1.1M followers, and Longtermism barely exists beyond EA)
    It seems plausible that Twitter is simply more popular among Effective Altruists. This would explain why the Animal Advocacy movement, within which EAs are a minority, is an exception here (although EAs are also a minority in Global Health & Poverty)
    Some Longtermist and Infrastructure orgs have...
    14 min
  • LW - How to Bounded Distrust by Zvi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Bounded Distrust, published by Zvi on January 9, 2023 on LessWrong.
    Scott Alexander points out that the media, from The New York Times to Infowars, very rarely lies explicitly and directly.
    Alas, the media often misleads. It implies and insinuates that which is not. It abuses the language. It selectively omits. It is highly motivated by partisanship and ideology and its own interests. It does not do or understand the research. It is terrible at interpreting science. It confuses cause and effect. It purports to use technically accurate data to show, even prove, conclusions known to be false, in ways that are designed to mislead and obviously in bad faith.
    Nor does it much care.
    They cite someone else, and claim this excuses them from all responsibility. All they said was ‘police say this guy is guilty’ or ‘this ‘expert’ irregularities he says shows fraud.’
    The rules are not what they used to be.
    Then there are the op-ed pages and headlines, which are far worse.
    This leads to a situation of Bounded Distrust, which I analyze at length here. I then work through some examples here. If you want to think about the problem in detail, start at these links.
    A shorter, more practical version was needed.
    This attempts to offer that. It leaves a lot out. Consider reading the long version.
    What are the Rules?
    Some special rules about the headline. They also apply to op-eds. Headlines are:
    Not chosen by the author.
    Allowed to lie.
    Allowed to blatantly contradict the article’s content.
    Laying out a Narrative the source wants you to believe.
    The body of a news article is more reliable. The rules are simple. The article:
    Has a Narrative, likely revealed by the headline.
    Is not allowed to lie, in a way that could count as being physically falsified.
    Is not allowed to assert facts without reliable sources.
    Is allowed to do almost anything else.
    Is often part of an implicit conspiracy to suppress true information or spread false information, without explicit denial of the true info or explicit claims of the false info.
    Is allowed to repeat any claim if it attributes that claim to its source.
    Can call anyone an expert. Expert consensus means three people. ‘Some investors’ and similar phrases mean two (as does ‘surrounded by.’)
    Is allowed to withhold or not seek relevant information, selectively quote, frame, insinuate, imply, condemn via association, misconstrue. This includes calling true things lies or ‘misinformation’ if they imply disliked things.
    Is allowed to change meanings of words in different contexts or over time.
    Will draw conclusions in ways that defy logic, or that would be obvious errors to anyone with ordinary skill in the art. This is allowed.
    Can and will find an ‘expert’ to support anything they want.
    Will maximallyshape the story to fit the Narrative and reality tunnel.
    Will face potential negative reputational and other consequences for breaking these rules, and sometimes choose to break all of them.
    Won’t face consequences for breaking these rules if everyone else is also breaking them to the same degree in similar spots.
    Will face other negative consequences for insufficient Narrative support.
    When the expected consequences of rule breaking exceed any plausible benefits from breaking the rules, you can mostly trust that the rules above are followed.
    When the stakes are so high that the consequences could be seen as worth paying for either the reporter or the outlet, they might do that, which can be called Using the One Time. You must be extra careful.
    The reporter is allowed to lie in order to get the story, the way a cop can lie during their investigation. Both often do so.
    Consider the Source
    There is no getting around the need to consider and examine the original source.
    For each source at all levels, and each class of source, one must maintain a Translati...
    8 min
  • EA - Dangers of deference by TsviBT
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dangers of deference, published by TsviBT on January 8, 2023 on The Effective Altruism Forum.
    [Written September 02, 2022. Note: I'm likely to not respond to comments promptly.]
    Sometimes people defer to other people, e.g. by believing what they say, by following orders, or by adopting intents or stances. In many cases it makes sense to defer, since other people know more than you about many things, and it's useful to share eyes and ears, and coordination and specialization are valuable, and one can "inquisitively defer" to opinions by taking them as challenges to investigate further by trying them out for oneself. But there are major issues with deferring, among which are:
    Deferral-based opinions don't contain the detailed content that generated the opinions, and therefore can't direct action effectively or update on new evidence correctly. Acting based on deferral-based opinions is discouraging because it's especially not the case that the whole of you can see why the action is good. Acting based on deferral-based opinions to some extent removes the "meaning" of learning new information; if you're just going to defer anyway, it's sort of irrelevant to gain information, and your brain can kind of tell that, so you don't seek information as much. Deference therefore constricts the influx of new information to individuals and groups. A group with many people deferring to others will amplify [information cascades]() by double-triple-quadruple-counting non-deferral-based evidence. A group with many people deferring to others will have mistakenly correlated beliefs and actions, and so will fail to explore many worthwhile possibilities.
    The deferrer will copy beliefs mistakenly imputed to the deferred-to that would have explained the deferred-to's externally visible behavior. This pushes in the direction opposite to science because science is the way of making beliefs come apart from their pre-theoretical pragmatic implications. Sometimes the deferrer, instead of imputing beliefs to the deferred-to and adopting those beliefs, will adopt the same model-free behavioral stance that the deferred-to has adopted to perform to onlookers, such as pretending to believe something while acting towards no coherent purpose other than to maintain the pretense.
    If the deferred-to takes actions for PR reasons, e.g. attempting to appear from the outside to hold some belief or intent that they don't actually hold, then the PR might work on the deferrer so that the deferrer systematically adopts the false beliefs and non-held intents performed by the deferred-to (rather than adopting beliefs and intents that would actually explain the deferred-to's actions as part of a coherent worldview and strategy). Allocating resources based on deferral-based opinions potentially opens up niches for non-epistemic processes, such as hype, fraud, and power-grabbing. These dynamics will be amplified when people choose who to defer to according to how much the person is already being deferred to. To the extent that these dynamics increase the general orientation of deference itself, deference recursively amplifies itself.
    Together, these dynamics make it so that deferral-based opinions are under strong pressure to not function as actual beliefs that can be used to make successful plans and can be ongoingly updated to track reality. So I recommend that people
    keep these dynamics in mind when deferring, track the difference between believing someone's testimony vs. deferring to beliefs imputed to someone based on their actions vs. adopting non-belief performative stances, and give substantial parliamentary decision-weight to the recommendations made by their expectations about facts-on-the-ground that they can see with their own eyes.
    Not to throw away arguments or information from other people, or to avoid...
    4 min
  • LW - Wolf Incident Postmortem by jefftk
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wolf Incident Postmortem, published by jefftk on January 9, 2023 on LessWrong.
    Incident #210
    Status
    Summary
    Impact
    Root causes
    Sentinel generated noisy alerts due to premature deployment, incomplete training, and overly monotonous task. Oncalls failed to respond to true positive due to alert fatigue.
    Trigger
    Wolf.
    Resolution
    Gathered flock. Deployed replacement sentinel.
    Detection
    Sentinel did not report at end of shift.
    Action Items
    Priority Action Item Type Status P0 Gather flock mitigate complete P0 Deploy replacement sentinel mitigate complete P1 Update playbook for wolf alerts prevent complete P2 Update remaining sentinels prevent complete P2 Revise sentinel training program prevent complete P2 Investigate equipping sentinels with flutes or slings prevent in progress
    Lessons Learned
    What went well
    Flock gathering proceeded without issues.
    No flock injuries or losses.
    Replacement sentinel did not exhibit false positive alerts.
    What went wrong
    Noisy alerts not addressed.
    Alerts silenced contrary to playbook.
    Loss of sentinel.
    Where we got lucky
    Only one wolf.
    Wolf sated after sentinel consumption.
    Replacement sentinel available.
    Timeline
    All times local
    March 3rd:
    16:32 Oncalls paged "wolf".
    16:34 First oncall arrives at sentinel location.
    16:34 Alert diagnosed as false positive. No corrective action performed.
    March 4th:
    14:15 Oncalls paged "wolf".
    14:19 First oncall arrives at sentinel location.
    14:19 Alert diagnosed as false positive. No corrective action performed.
    March 5th:
    17:03 (Reconstructed) Outage begins, sentinel notices wolf.
    17:03 Oncalls paged "wolf".
    17:04 Oncalls paged "wolf".
    17:04 Oncalls paged "real wolf".
    17:05 (Reconstructed) Wolf consumes sentinel.
    18:45 Sentinel does not report at end of shift.
    19:05 Primary oncall dispatched to field.
    19:10 Oncall diagnoses issue.
    19:10 Incident begins, secondary and tertiary oncalls paged.
    19:15 First sheep located.
    19:52 Last sheep located.
    20:05 Flock safe in pens.
    20:05 Outage ends, flock protection fully restored.
    20:45 Replacement sentinel identified.
    March 6th:
    07:38 Replacement sentinel deployed
    18:45 Replacement sentinel reports at end of shift
    18:45 Incident ends, 24hr without wolf alerts or activity (exit criterion).
    Comment via: facebook, mastodon
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - What specific thing would you do with AI Alignment Research Assistant GPT? by quetzal rainbow
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What specific thing would you do with AI Alignment Research Assistant GPT?, published by quetzal rainbow on January 8, 2023 on LessWrong.
    Why I think this question is important: I asked myself, "What would my AGI timelines be if some AI could summarize Yudkowsky-Ngo debates on alignment difficulty in a way that both participants agree with this summary, everyone who reads this summary understands both positions and participants can check understanding in conversation?" My semi-intuitive answer: "Five years tops and two years as modal prediction". Debate Summarizer is not a very useful Alignment Assistant, it can't boost research by 10x. If someone told me that Alignment Assistant suggested idea that sparked optimism in MIRI, I would think that we have exact amount of time it takes for someone to turn every tools needed to build such an Alignment Assistant to the creation of AGI (conditional on "this Alignment Assistant is not AGI itself").
    I.e., if you bet on assistance of narrow AI in alignment research, you should also bet on finding solution quickly. Quick search for a solution requires an already existing plan. On the other hand, we are talking about a narrow AI, you can't just ask "solve alignment problem for me". You should ask specific questions, test pre-selected hypotheses, prove well-defined statements. Therefore, I think that those who want to use Alignment Assistants should outline this set of specific things as soon as possible.
    UPD: Thanks janus for the link, it helped me to clarify what I would like to see as a perfect answer.
    Let's suppose that your immediate answer is "brainstorming". Then the perfect specific answer is something like that:
    "In my opinion, the most narrow bottleneck in AI alignment is the lack of ideas about X, so I will brainstorm about it with Alignment Assistant."
    Extremely unrealistic example:
    "I have The Grand Theory of Alignment, but it critically depends on Goldbach conjecture, so I will try to prove it."
    My very (very) simplified model of Paul Cristiano's answer:
    "80% of alignment can be solved with ELK strategy, so we can make builder-breaker debate on (counter)examples for ELK between Assistant and ARC until we figure out the solution."
    Yet another possible answer:
    "I don't know! We are still in early exploratory mode, I can't imagine a specific thing. I just want to become as effectively smart as possible and see where it gets us."
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - Simulacra are Things by janus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simulacra are Things, published by janus on January 8, 2023 on The AI Alignment Forum.
    This is a note I wrote about a year ago. It's fairly self-contained, so I decided to make a post out of it after Vladimir_Nesov's comment caused me to dig up this text and TsviBT's The Thingness of Things reminded me of it again.
    "Simulacra" refer to things simulated by simulators such as GPT, in the ontology introduced in Simulators.
    What are simulacra?
    “Physically”, they’re strings of text output by a language model. But when we talk about simulacra, we often mean a particular character, e.g. simulated Yudkowsky. Yudkowsky manifests through the vehicle of text outputted by GPT, but we might say that the Yudkowsky simulacrum terminates if the scene changes and he’s not in the next scene, even though the text continues. So simulacra are also used to carve the output text into salient objects.
    Essentially, simulacra are to a simulator as “things” are to physics in the real world. “Things” are a superposable type – the entire universe is a thing, a person is a thing, a component of a person is a thing, and two people are a thing. And likewise, “simulacra” are superposable in the simulator, Things are made of things. Technically, a random collection of atoms sampled randomly from the universe is a thing, but there’s usually no reason to pay attention to such a collection over any other. Some things (like a person) are meaningful partitions of the world (e.g. in the sense of having explanatory/predictive power as an object in an ontology). We assign names to meaningful partitions (individuals and categories).
    Like things, simulacra are probabilistically generated by the laws of physics (the simulator), but have properties that are arbitrary with respect to it, contingent on the initial prompt and random sampling (splitting of the timeline). They are not necessary but contingent truths; they are particular realizations of the potential of the simulator, a branch of the implicit multiverse. In a GPT simulation and in reality, the fact that there are three (and not four or two) people in a room at time is not necessitated by the laws of physics, but contingent on the probabilistic evolution of the previous state that is contingent on (.) an initial seed(prompt) generated by an unknown source that may itself have arbitrary properties.
    We experience all action (intelligence, agency, etc) contained in the potential of the simulator through particular simulacra, just like we never experience the laws of physics directly, only through things generated by the laws of physics. We are liable to accidentally ascribe properties of contingent things to the underlying laws of the universe, leading us to conclude that light is made of particles that deflect like macroscopic objects, or that rivers and celestial bodies are agents like people.
    Just as it is wrong to conclude after meeting a single person who is bad at math that the laws of physics only allow people who are bad at math, it is wrong to conclude things about GPT’s global/potential capabilities from the capabilities demonstrated by a simulacrum conditioned on a single prompt. Individual simulacra may be stupid (the simulator simulates them as stupid), lying (the simulator simulates them as deceptive), sarcastic, not trying, or defective (the prompt fails to induce capable behavior for reasons other than the simulator “intentionally” nerfing the simulacrum – e.g. a prompt with a contrived style that GPT doesn’t “intuit”, a few-shot prompt with irrelevant correlations). A different prompt without these shortcomings may induce a much more capable simulacrum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - Moral Weights according to EA Orgs by Simon M
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Moral Weights according to EA Orgs, published by Simon M on January 8, 2023 on The Effective Altruism Forum.
    This post was motivated by SoGive's moral weights being (to a first check) quite different to Founders Pledge (FP) and Happier Lives Institute (HLI). Upon checking in more detail, this appears to be the largest discrepency across any organisation. (Although we are still waiting to find out many missing values in the grid as HLI's research is ongoing).
    Summary
    GiveWellFPHLISoGiveWELLBY-0.530.55-1 income doubling for 1 year111 11 year of severe depression~1.51 ()1.28 0.71-1.42 41 additional year of life2.301.95-2.8 to 2.91 ()-1 death under 5117.7123.2-1001 death over 583.683.7-100
    Broadly all organisations (with the exception of SoGive's view on depression) are very much aligned.
    () means I expect the organisation would not endorse the figures used here. In the case of GiveWell my best guess is this is roughly inline with what they would use. For Happier Lives Institute it is an upper bound I expect they will be far below when they finish their research.
    Details
    Open Philanthropy
    Open Phil's summary of their moral weights is very clear and interesting, but:
    For now, in order to be more consistent in our practices, we’re going to defer to GiveWell and start to use the number of DALYs that would be implied by extrapolating their moral weights.
    I have left them off here, as I would just be duplicating the GiveWell numbers.
    GiveWell
    GiveWell's weights are sourced from here. I have made a few small calculations to align these numbers with the other orgs.
    Founders Pledge
    Founders Pledge's moral weights are avaiable here.
    Happier Lives Institute
    Unfortunately, their moral weights are still in the process of being generated. You can determine the range of weights they will use in future in their article The elephant in the bednet.
    SoGive
    SoGive's weights can be found here. I have used them verbatim
    This is a calculation. A 100% increase in income/consumption is worth 1.27 / 0.69 = 1.86 WELLBYs (in HLI terms). (See inputs tab C25) We want this to be 1-unit, so we take 1/1.86 = 0.55 to be a WELLBY and other numbers are calculated from this.
    GiveWell has a strong aversion to disability weights used blindly, so take this number with a grain of salt.
    Founders Pledge don't explicitly include depression in their data. I have used the disability weights they used in their public CEA of StrongMinds. I am under the impression they are working to move towards HLI's model for this.
    This is also a calculation. HLI are inconsistent in how they calculate the impact of depression in WELLBYs. Here they say depression is worth 1.3 WELLBYs. (So 1.3 0.55 = 0.71) in units of income doubling. One potential explanation is that "depression" is less severe than "severe depression" so potentially this number could be doubled - they estimate the effect of StrongMinds to be ~1.8 WELLBYs)
    GiveWell uses a metric "Years lived with disease/disability" which as far as I can tell is equivalent to "value of averting 1 year of death".
    As mentioned above HLI are still in the process of deciding what their moral weights are. I am taking the upperbound of their deprevationist model, the highest number it could be. The highest number is a deprevationist model of losing 4.95 WELLBY. (4.95 - 0). The lowest number is the same model using a neutral point of 10. "would seem unintuitive to most, but relates to tranquilism and minimalist axiologies" (See inputs tab C18)
    I have taken the average of "death averted from malaria" and "death averted from vitamin A". The numbers are similar and I don't think material to the analysis here.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    5 min
  • AF - 200 COP in MI: Image Model Interpretability by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Image Model Interpretability, published by Neel Nanda on January 8, 2023 on The AI Alignment Forum.
    This is the eighth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started. Look up jargon in my Mechanistic Interpretability Explainer
    I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!
    Motivating papers: Thread: Circuits, Multimodal Neurons in Artificial Neural Networks
    Disclaimer: My area of expertise is language model interpretability, not image models - it would completely not surprise me if this section contains errors, or if there are a lot of great open problems that I’ve missed!
    Background
    A lot of the early work in mechanistic interpretability was focused on reverse engineering image classification models, especially Inceptionv1 (GoogLeNet). This work was largely (but not entirely!) led by Chris Olah and the OpenAI interpretability team. They got a lot of fascinating results, most notably (in my opinion):
    Finding a technique called Feature Visualization to visualize what neurons are “looking at”, essentially creating a picture that represents what most activates a given neuron
    Intuitively, this technique exploits the fact that each neuron is basically a function that maps an image to a scalar (the neuron activation). Images live in a continuous space (we can vary a pixel by an infinitesimal amount to slightly change the image), so we can do gradient descent on the image to find a max activating image
    Curve Circuits - they reverse engineered a 50,000 ish parameter circuit used to form curve detecting neurons, and understood it well enough that they could hand-code the weights to the neurons, insert in these hand-coded neurons, and (mostly) recover the original performance.
    Multimodal Neurons - they looked at CLIP (a model that takes in an image and a caption and outputs a score for how well they match) and found a bunch of fascinating abstract neurons that seemed to activate on concepts and on things related to that concept - eg a Donald Trump neuron that actives on his picture and on MAGA hats, or neurons corresponding to Halloween, or anime, or Catholicism
    Motivation
    I think the image interpretability results are awesome, and one of the main things that convinced me that reverse engineering neural networks was even possible! But also, very few people worked on these questions! There was enough work done to give a good base to build off of and to expose a lot of dangling threads, but also a lot of open questions left.
    My personal goal with mech interp is to get good enough at understanding systems that we can eventually understand what’s going on in a human-level frontier model, and use this to help align it. From this perspective, is it worth continuing image circuits work? This is not obvious to me! I think language models (and to a lesser degree transformers) are far more likely to be a core part of how we get transformative AI (though I do expect transformative AI to have significant multimodal components), and most of the mech interp field is now focused on LLMs as a result.
    But I also think that just any progress on reverse engineering neural networks is good. And at least some insights transfer. Though there are obviously a lot of differences - Inception has a continuous rather than discrete input space, doesn’t have attention heads or a residual stream, and is doing classification rather than generation. I’m personally most excited about image circuits work driving towards fundamental questions about reverse engineering networks:
    How architecture or model specific is reverse eng...
    10 min