The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - The special nature of special relativity by adamShimi
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The special nature of special relativity, published by adamShimi on January 9, 2023 on LessWrong.
    I honestly thought that both theories of relativity were theories of gravitation. Special and general relativity, the latter being the generalized version of the latter. That makes sense, right?
    Then I started actually digging into special relativity. I studied the original annus mirabilis paper, entitled “On the Electrodynamics of Moving Bodies”. Yes, the title was the first clue, although I didn’t see it at the time. The second clue lied in a conspicuous absence: neither gravity nor gravitation is mentioned in the paper. Not even once. No, instead, Einstein spends 5 sections on kinematics (which is basically about geometrically deducing the value of some parameters of the mechanical system from others given as initial conditions); then he applies his kinematics to electromagnetism.
    This is when I finally realized that special relativity had nothing to do with gravity. General relativity, that’s a theory of gravity, one you can compare with Newtonian Mechanics in a variety of ways. But special relativity is not a theory in this sense; that’s the wrong type signature. No, special relativity looks far more like a meta conservation law.
    Because instead of conserving a quantity, special relativity assert that the laws of physics themselves are invariant by specific transformations called Lorentz transformations. This is stated by Einstein itself in his popular account of his theories (Relativity: The Special and General Theory, Chapter 14):
    Once in possession of the Lorentz transformation, however, we can combine this with the principle of relativity, and sum up the theory thus:
    [...] Or in brief: General laws of nature are co-variant with respect to Lorentz transformations.
    This is a definite mathematical condition that the theory of relativity demands of a natural law, and in virtue of this, the theory becomes a valuable heuristic aid in the search for general laws of nature. If a general law of nature were to be found which did not satisfy this condition, then at least one of the two fundamental assumptions of the theory would have been disproved.
    Although the transformations had been discovered independently earlier (by Lorentz notably), Einstein rederived this constraint on laws of nature from his two principles:
    That the laws of physics are invariant by uniform (so not accelerated) motion of the reference frame (technically that they’re the same in all inertial reference frames)
    That the constancy of the speed of light in the void is such a law of physics
    In classical mechanics, 2 contradicted 1 because the addition of velocities meant that light coming out of a moving source would have to move faster or slower than the constant speed of light. But since Einstein assumed both principles as strong phenomenological laws, he looked for a way to reconcile them. And Lorentz transformations give just that: they transform coordinates from a stationary reference frame to a uniformly moving one while maintaining the constancy of the speed of light in the new frame. Which they do by contracting the lengths and dilating time, two ideas familiar to anyone who has read pop physics books.
    But more than the precise mechanics of the Lorentz transformation, what I find fascinating here is that what I thought was an imperfect theory of gravitation actually proved to be an (imperfect) conservation law, a constraint on the laws of nature.
    What can you do with such a constraint? I see two main applications, both of which were pursued by Einstein himself and other physicists at the time.
    Test on existing theories
    As explained in the paper, Maxwell’s theory of electromagnetism passed this test: it was already a relativistic theory. Classical mechanics on the other hand contradicted it direc...
    6 min
  • LW - Wentworth and Larsen on buying time by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wentworth and Larsen on buying time, published by Akash on January 9, 2023 on LessWrong.
    This post summarizes a discussion between Thomas Larsen and John Wentworth. They discuss efforts to buy time, outreach to AGI labs, models of institutional change, and other ideas relating to a post that Thomas, Olivia Jimenez, and I released: Instead of technical research, more people should focus on buying time.
    I moderated the discussion and prepared a transcript with some of the key parts of the conversation.
    On the value of specific outreach to lab decision-makers vs. general outreach to lab employees
    John:
    When thinking about "social" strategies in general, a key question/heuristic is "which specific people do you want to take which specific actions?"
    For instance, imagine I have a start-up building software for hospitals, and I'm thinking about marketing/sales. Then an example of what not to do (under this heuristic) would be "Buy google ads, banner ads in medical industry magazines, etc". That strategy would not identify the specific people who I want to take a specific action (i.e. make the decision to purchase my software for a hospital).
    An example of what to do instead would be "Make a list of hospitals, then go to the website of each hospital one-by-one and look at their staff to figure out who would probably make the decision to purchase/not purchase my software. Then, go market to those people specifically - cold call/email them, track them down at conferences, set up a folding chair on the sidewalk outside their house and talk to them when they get home, etc.
    Applying that to e.g. efforts to regulate AI development, the heuristic would say to not try to start some giant political movement. Instead, identify which specific people you need to write a law (e.g. judges or congressional staff), which specific people will decide whether it goes into effect (e.g. judges or swing votes in congress), which specific people will implement/enforce it (e.g. specific bureaucrats, who may be sufficient on their own if we don't need to create new law).
    Applying it to e.g. efforts to convince AI labs to stop publishing advances or shift their research projects, the heuristic would say to not just go convincing random ML researchers. Instead, identify which specific people at the major labs are de-facto responsible for the decisions you're interested in (like e.g. decisions about whether to publish some advancement), and then go talk to those people specifically. Also, make sure to walk them through the reasoning enough that they can see why the decisions you want them to make are right; a vague acknowledgement that AI X-risk is a thing doesn't cut it.
    Thomas:
    It seems like we’re on a similar page here.
    I do think that on current margins, if typical OpenAI employees become more concerned about x-risk, this will likely have positive follow through effects on general OpenAI epistemics. And I expect this will likely lead to improved decisions.
    Perhaps you disagree with that?
    John: Making random OpenAI (or Deepmind, or Anthropic, or ...) employees more concerned about X-risk is plausibly net-positive value in expectation; I'm unsure about that. But more importantly, it is not plausibly high value in expectation.
    When I read your post on time-buying, the main picture I end up with is a bunch of people running around randomly spreading the gospel of AI X-risk. More generally, that seems-to-me to be the sort of thing most people jump to when they think about "time-buying".
    In my mind, 80% of the value is in identifying which specific people we want to make which specific decisions, and then getting in contact with those specific people. And I usually don't see people thinking about that very much, when they talk about "time-buying" interventions.
    Thomas:
    Fully agree with this [the last tw...
    20 min
  • EA - My personal takeaways from EAGxLatAm by CristinaSchmidtIbáñez
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My personal takeaways from EAGxLatAm, published by CristinaSchmidtIbáñez on January 10, 2023 on The Effective Altruism Forum.
    This is a personal post and does not necessarily represent the views of Rethink Priorities.
    This year I attended EAGxLatAm (6-8 January) in Mexico City.
    I thought it would be a very valuable exercise to share my own takeaways from it as well as to encourage other attendees’ to share theirs. My takeaways are based on the 22 1-on-1s I had (not counting casual 1-on-1s that happened during the conference). I talked mostly (ca. 70%) with Latin American students at different levels of their undergraduate degrees.
    So without further ado, these are my takeaways:
    Things I was rather surprised about
    Latin American students don’t seem to
    be aware of the concept of option value and more specifically, they don’t seem to be explicitly considering this a criteria for their career decision-making
    be considering the “ladder of tests” e.g. when thinking about pursuing a specialization (like a master’s or PhD) often they had not yet considered “cheaper test” to see if that degree would be the right move for them
    more broadly, they don’t seem to be aware of the different criteria they are comparing different options with
    Latin American students seem (internally) to feel really pressured to do a master’s and/or PhD abroad (e.g. US).
    When I dug deeper for the reasons for this, they mentioned
    Job/financial security
    A pathway for them “to be taken seriously”
    (this one really saddened me): That they feel they are “worth very little in EA” if they don’t get these degrees
    Latin American students seem to have a hard time thinking ambitiously, probably more so than the “average EA”.
    When these students come from low/middle income backgrounds then the issue seems to be that they are carrying a lot of “baggage” from their past that makes it hard for them to “think big” about the impact they can have in the future
    If they come from more high income backgrounds then the issue seems to rather be imposter syndrome.
    Things I was more or less aware of but were (mentally) strongly highlighted
    Founders/directors of projects usually don’t plan for the changes in the type of work they do as their projects/organizations grow and therefore find themselves after a couple of years doing a lot of tasks they don’t enjoy doing
    Operations management (particularly HR) isn’t well planned (if planned at all) when nonprofit entrepreneurs are drafting project plans and fundraising. As a result, project budgets don’t account properly for these costs.
    When operations management is planned for a new project it looks something like “we’ll budget an ops person (and that'll solve our issues)”. The result of this is that by the time that (ops) person is hired, it is expected that one person solves all operational issues for the project
    Often by the time projects receive funding they don’t know “what to do with the money” and start looking fast into fiscal sponsors or other ways to receive the funds
    Some things I suggest (or even suggested during my 1-on-1s)
    For community builders to talk with their community members more about how they are comparing different career options. I suggested to one community builder experimenting with a workshop for doing career weighted factor models
    For students in particular to seek (career) mentoring opportunities
    Having a public list of project ideas (for the EA space) for non-programmatic things like operations, HR, management etc. so people that would like to work on these things have a better sense on what ideas to prioritize
    For a service to exist that offers founders “career check-ins” once a year where they have to take inventory of how their list of responsibilities has changed and consider alternative paths either within their own organizations or outside of the...
    5 min
  • EA - There is now an EA Managers Slack by Ben Kuhn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There is now an EA Managers Slack, published by Ben Kuhn on January 9, 2023 on The Effective Altruism Forum.
    tl;dr: fill out this ~2-min form for access!
    In Some notes on common challenges building EA orgs, I noted that a lot of managers at EA orgs face a set of similar challenges related to being new to their role and not having peers or mentors within their org to learn from.
    I thought it could help to have a shared, cross-organization discussion forum for management issues, so that if you're dealing with some kind of management problem for the first time—say, hiring, or coaching someone who's underperforming, or reorganizing a team—you can get advice from people who have done it before, even if none of your direct coworkers have. I intend to be pretty active answering questions, seeding discussion, etc. for the first while.
    For now, the group is open to paid employees of EA organizations (borrowing the EA Ops slack criteria) with at least one paid direct report. If that's you, fill out this form for access!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    2 min
  • AF - Review AI Alignment posts to help figure out how to make a proper AI Alignment review by Oliver Habryka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Review AI Alignment posts to help figure out how to make a proper AI Alignment review, published by Oliver Habryka on January 10, 2023 on The AI Alignment Forum.
    I've had many conversations over the last few years about the health of the AI Alignment field and one of the things that has come up most frequently (including in conversations with Rohin, Buck and various Open Phil people) is that many people wish there was more of a review process in the AI Alignment field.
    I also think there is a bunch of value in better review processes, but have felt hesitant to create something very official and central, since AI Alignment is a quite preparadigmatic field, which makes creating shared standards of quality hard, and because I haven't had the time to really commit to maintain something great here.
    Separately, I am also quite proud of the LessWrong review, and am very happy about the overall institution that we've created there, and I realized that the LessWrong review might just be a good test bed and bandaid for having a better AI Alignment review process. I think the UI we built for it is quite good, and I think the vote does have real stakes and a lot of the people voting are also people quite active in AI Alignment.
    So this year, I would like to encourage many of the people who expressed a need for better review processes in AI Alignment to try reviewing some AI Alignment posts from 2021 as part of the LessWrong review. I personally got quite a bit of personal value out of doing that, and e.g. found that my review of the MIRI dialogues helped crystallize some helpful new directions for me to work towards, and I am also hoping to write a longer review of Eliciting Latent Knowledge that I also think will help clarify some things for me, and is something that I will feel comfortable linking to later when people ask me about my takes on ELK-adjacent AI Alignment research.
    I am also interested in comments on this post with takes for better review-processes in AI Alignment. I am currently going through a period where I feel quite confused how to relate to the field at large, so it might be a good time to also have a conversation about what kind of standards we even want to have in the field.
    Current AI Alignment post frontrunners in the review
    We've had an initial round of preliminary voting, in which people cast non-binding votes that help prioritize posts during the Review Phase. Among Alignment Forum voters, the top Alignment Forum posts were:
    ARC's first technical report: Eliciting Latent Knowledge
    What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs)
    Another (outer) alignment failure story
    Finite Factored Sets
    Ngo and Yudkowsky on alignment difficulty
    My research methodology
    Fun with +12 OOMs of Compute
    The Plan
    Comments on Carlsmith's “Is power-seeking AI an existential risk?”
    Ngo and Yudkowsky on AI capability gains
    There are also a lot of other great alignment posts in the review (a total of 88 posts were nominated), and I do expect things to shift around a bit, but I do think all 10 of these top essays deserve some serious engagement and a relatively in-depth review, since I expect most of them will get read by people for many years to come, and people might be basing new research approaches and directions on them.
    To review a post, you can navigate to the post page, and click the "Review" button at the top of the page (just under the post title). It looks like this:
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Evidence under Adversarial Conditions by PeterMcCluskey
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Evidence under Adversarial Conditions, published by PeterMcCluskey on January 9, 2023 on LessWrong.
    I recently noticed similarities between how I decide what stock market evidence to look at, and how the legal system decides what lawyers are allowed to tell juries.
    This post will elaborate on Eliezer's Scientific Evidence, Legal
    Evidence, Rational
    Evidence. In particular, I'll try to generalize about why there's a large class of information that I actively avoid treating as Bayesian evidence.
    Here's an example of a briefly popular meme that helped teach me to ignore certain types of information:
    Companies need to be forgetting organizations. Enron Corp., which has repeatedly been tagged as the nation's most innovative corporation, is exhibit A as a world-class forgetting organization. It's not wedded to what it did yesterday. Enron chiefs Kenneth Lay and Jeff Skilling have figured out how to operate like a band of pirates. Got an idea? Don't dally. Go for it while it's an original!
    Everything here seems at least technically true (The Media Very Rarely
    Lies). I suspect it was mostly intended to be honest and helpful. But as a reader, I found it discouraged me from focusing on some key questions, such as: who was tagging Enron as innovative? What were the goals of those taggers? What was their track record at helping readers identify good investments? How much of a connection is there between innovation and good investments? What are some examples of Enron's innovation, and how do they resemble the kind of innovations that have caused other companies to succeed?
    Enron seems to have done a few things which qualify as fraud. But most of the Enron stock price bubble was generated through misdirecting investor's attention (and maybe the attention of Lay and Skilling) away from the risks that Enron was taking.
    I was in fact tempted to invest in Enron when I listened to that kind of meme. Whenever I felt that temptation, I looked at its financial statements and charts of its stock price, and decided I wouldn't miss much if I waited for better evidence.
    The standard Bayesian worldview asks me to update on all relevant information. Why do both I as an investor and the legal system exclude large classes of information from that rule?
    Both contexts involve some zero sum or negative sum conflicts. That makes it likely that someone is packaging information in order to mislead.
    If I, or a jury, only notice a datum because someone thought it would mislead me or the jury, odds are that updating on that information would reduce the accuracy of my (or the jury's) beliefs. We reject that information in order to ensure that its expected value will be zero, because the alternative tends to be that its expected value is negative.
    In both contexts, rules have been developed to admit evidence only if it's hard to manipulate or if it comes from a source that's unlikely to manipulate it.
    Accounting rules strongly limit how many ways key information can be presented. They also ensure that someone can in principle be punished if the information misleads.
    Similar rules ensure that company press releases are fairly safe for investors to read. The company will often be sued if the press release contains questionable claims that subsequently become linked to investor losses. That deters a wide variety of harmful propaganda (and presumably a fair amount of helpful evidence). It still leaves some gray areas for vaguely misleading memes, such as talk of Enron's innovation. Companies have mostly shifted to getting other respected sources to promote those ideas for them, rather than putting them in their own press releases.
    Still, I consider it an important skill to read those press releases in somewhat the same way that courts admit evidence: carefully distinguishing between "facts" and everything else....
    6 min
  • EA - GWWC Should Require Public Charity Evaluations by Jeff Kaufman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GWWC Should Require Public Charity Evaluations, published by Jeff Kaufman on January 9, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - Forecasting extreme outcomes by AidanGoth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Forecasting extreme outcomes, published by AidanGoth on January 9, 2023 on The Effective Altruism Forum.
    This document explores and develops methods for forecasting extreme outcomes, such as the maximum of a sample of n independent and identically distributed random variables. I was inspired to write this by Jaime Sevilla’s recent post with research ideas in forecasting and, in particular, his suggestion to write an accessible introduction to the Fisher–Tippett–Gnedenko Theorem.
    I’m very grateful to Jaime Sevilla for proposing this idea and for providing great feedback on a draft of this document.
    Summary
    The Fisher–Tippett–Gnedenko Theorem is similar to a central limit theorem, but for the maximum of random variables. Whereas central limit theorems tell us about what happens on average, the Fisher–Tippett–Gnedenko Theorem tells us what happens in extreme cases. This makes it especially useful in risk management, when we need to pay particular attention to worst case outcomes. It could be a useful tool for forecasting tail events.
    This document introduces the theorem, describes the limiting probability distribution and provides a couple of examples to illustrate the use (and misuse!) of the Fisher–Tippett–Gnedenko Theorem for forecasting. In the process, I introduce a tool that computes the distribution of the maximum n iid random variables that follow a normal distribution centrally but with an (optional) right Pareto tail.
    Summary:
    The Fisher–Tippett–Gnedenko Theorem says (roughly) that if the maximum of n iid random variables—which is itself a random variable—converges as n grows to infinity, then it must converge to a generalised extreme value (GEV) distribution
    Use cases:
    When we have lots of data, we should try to fit our data to a GEV distribution since this is the distribution that the maximum should converge to (if it converges)
    When we have subjective judgements about the distribution of the maximum (e.g. a 90% credible interval and median forecast), we can use these to determine parameters of a GEV distribution that fits these judgements
    When we know or have subjective judgements about the distribution of the random variables we’re maximising over, the theorem can help us determine the distribution of the maximum of n such random variables for large n – but this can give very bad results when our assumptions / judgements are wrong
    Limitations:
    To get accurate forecasts about the maximum of n random variables based on the distribution of the underlying random variables, we need accurate judgements about the right tail of the underlying random variables because the maximum will very likely be drawn from the tail, especially as n gets large
    Even for data that is very well described by a normal distribution for typical values, normality can break down at the tails and this can greatly affect the resulting forecasts
    I use the example of human height: naively assuming normality underestimates how extreme the tallest and shortest humans are because height is “only” normally distributed up to 2-3 standard deviations around the mean
    Modelling the tail separately (even with quite a crude model) can improve forecasts
    This simple tool might be good enough for forecasting purposes in many cases
    It assumes that the underlying r.v.s are iid and normally distributed up to k standard deviations above the mean and that there is a Pareto tail beyond this point
    Inputs:
    90% CI for the underlying r.v.s
    n (the number of samples of the underlying random variables)
    k (the number of SDs above the mean at which the Pareto tail starts); set this high if you don’t want a Pareto tail
    Output: cumulative distribution function, approximate probability density function and approximate expectation of the maximum of n samples of the underlying random variables
    Request for feedback: I’m not a...
    5 min
  • AF - [MLSN #7]: an example of an emergent internal optimizer by Josh Clymer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [MLSN #7]: an example of an emergent internal optimizer, published by Josh Clymer on January 9, 2023 on The AI Alignment Forum.
    As part of a larger community building effort, CAIS is writing a safety newsletter that is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here or follow the newsletter on twitter here.
    Welcome to the 7th issue of the ML Safety Newsletter! In this edition, we cover:
    ‘Lie detection’ for language models
    A step towards objectives that incorporate wellbeing
    Evidence that in-context learning invokes behavior similar to gradient descent
    What’s going on with grokking?
    Trojans that are harder to detect
    Adversarial defenses for text classifiers
    And much more.
    Alignment
    Discovering Latent Knowledge in Language Models Without Supervision
    Is it possible to design ‘lie detectors’ for language models? The author of this paper proposes a method that tracks internal concepts that may track truth. It works by finding a direction in feature space that satisfies the property that a statement and its negation must have opposite truth values. This has similarities to the seminal paper “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings” (2016), which captures latent neural concepts like gender with PCA, but this method is unsupervised and about truth instead of gender.
    The method outperforms zero-shot accuracy by 4% on average, which suggests something interesting: language models encode more information about what is true and false than their output indicates. Why would a language model lie? A common reason is that models are pre-trained to imitate misconceptions like “If you crack your knuckles a lot, you may develop arthritis.”
    This paper is an exciting step toward making models honest, but it also has limitations. The method does not necessarily serve as a `lie detector’; it is unclear how to ensure that it reliably converges to the model’s latent knowledge rather than lies that the model may output. Secondly, advanced future models could adapt to this specific method if they are aware of it.
    This may be a useful baseline for analyzing models that are designed to deceive humans, like models trained to play games including Diplomacy and Werewolf.
    [Link]
    How Would the Viewer Feel? Estimating Wellbeing From Video Scenarios
    Many AI systems optimize user choices. For example, a recommender system might be trained to promote content the user will spend lots of time watching. But choices, preferences, and wellbeing are not the same! Choices are easy to measure but are only a proxy for preferences. For example, a person might explicitly prefer not to have certain videos in their feed but watch them anyway because they are addictive. Also, preferences don’t always correspond to wellbeing; people can want things that are not good for them. Users might request polarizing political content even if it routinely agitates them.
    Predicting human emotional reactions to video content is a step towards designing objectives that take wellbeing into account. This NeurIPS oral paper introduces datasets containing 80,000+ videos labeled by the emotions they induce. The paper also explores “emodiversity”---the variety of experienced emotions---so that systems can recommend a variety of positive emotions, rather than pushing one type of experience. The paper includes analysis of how it bears on advanced AI risks in the appendix.
    [Link]
    Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers
    Especially since the rise of large language models, in-context learning has become increasingly important. In some cases, few-shot learning can outperform fine tuning. This preprint proposes a dual view between gradients induced by fine tunin...
    11 min
  • AF - Trying to isolate objectives: approaches toward high-level interpretability by Arun Jose
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Trying to isolate objectives: approaches toward high-level interpretability, published by Arun Jose on January 9, 2023 on The AI Alignment Forum.
    Epistemic status: There’s a lot in this post and my general approach while working on it that, in retrospect, wasn’t thought out well enough. I’m posting it because I figure sharing flawed ideas is better than letting this languish in a doc forever, while I work on improving them.
    Thanks to Paul Colognese, Tassilo Neubauer, John Wentworth, and Fabien Roger for useful conversations, Kyle McDonell and Laria Reynolds for suggesting something that led me to some of the ideas I mention here, and Shoshannah Tekofsky for feedback on a draft.
    I spent some time trying to find the high-level structure in a neural net corresponding to a deep reinforcement learning model’s objective. In this post, I describe some of the stuff I tried, thoughts from working on this, and mistakes I made.
    Context
    Many approaches in current interpretability work as I understand them involve understanding low-level components of a model (Circuits and subsequent work, for example), as a way of building up to more complex components. I think we can make progress from the opposite frontier at the same time, and try to identify the presence or nature of certain high-level mechanistic structures or properties in a model. Examples of this include verifying whether a model is doing optimization, whether it’s myopic, isolating the objective of an optimizer, and essentially the entire class of properties with a singular answer for the entire model.
    I prefer to distinguish these directions as low-level and high-level interpretability respectively for descriptive clarity. The framing of best-case and worst-case transparency as laid out in the transparency tech tree also points at the same concept. I expect that both directions are aiming at the same goal, but working top-down seems pretty tractable at least for some high-level targets that don’t require lots of deconfusion to even know what we’re looking for.
    Ideally, an approach to high-level interpretability would be robust to optimization pressure and deception (in other words, to gradient descent and adversarial mesa optimizers), but I think there’s a lot we can learn toward that ideal from more naive approaches. In this post, I describe my thoughts on trying to identify the high-level structure in a network corresponding to a deep reinforcement learning model’s objective.
    So I didn’t set out on this particular approach expecting to succeed at the overall goal (I don’t expect interpretability to be that easy), instead hoping that trying out a bunch of naive directions will give us insight into future high-level interpretability work. To that end, I try to lay out my reasoning at various points, riddled with gaps at points, while thinking about this.
    I think there are plenty of plausible avenues to try out here, and for the most part will only be describing the problems with ones I thought of. Further, the directions I tried don’t seem very non-obvious and there are likely much better methods to be tried given more thought here.
    A naive approach and patching
    Take the case of a small RL model that can optimize for a reward in a given environment. What we want is information about how the model’s objective is represented in the network’s weights. Concretely, we might think about what kind of tests or processes disproportionately affect these weights more than any other, and try to isolate the information we want from there.
    One approach we might try is:
    Train the initialized model on reward RA for an extended period of time. Let’s call the model at the end of this step MA. We may think of MA as having learned a pretty sophisticated world model at this point, such that further training wouldn’t update it strongly.
    Tra...
    14 min