The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Self-driving car bets by paulfchristiano
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Self-driving car bets, published by paulfchristiano on July 29, 2023 on LessWrong.
    This month I lost a bunch of bets.
    Back in early 2016 I bet at even odds that self-driving ride sharing would be available in 10 US cities by July 2023. Then I made similar bets a dozen times because everyone disagreed with me.
    The first deployment to potentially meet our bar was Phoenix in 2022. I think Waymo is close to offering public rides in SF, and there are a few more cities being tested, but it looks like it will be at least a couple of years before we get 10 cities even if everything goes well.
    Back in 2016 it looked plausible to me that the technology would be ready in 7 years. People I talked to in tech, in academia, and in the self-driving car industry were very skeptical. After talking with them it felt to me like they were overconfident. So I was happy to bet at even odds as a test of the general principle that 7 years is a long time and people are unjustifiably confident in extrapolating from current limitations.
    In April of 2016 I gave a 60% probability to 10 cities. The main point of making the bets was to stake out my position and maximize volume, I was obviously not trying to extract profit given that I was giving myself very little edge. In mid 2017 I said my probability was 50-60%, and by 2018 I was under 50%.
    If 34-year-old Paul was looking at the same evidence that 26-year-old Paul had in 2016 I think I would have given it a 30-40% chance instead of a 60% chance. I had only 10-20 hours of information about the field, and while it's true that 7 years is a long time it's also true that things take longer than you'd think, 10 cities is a lot, and expert consensus really does reflect a lot of information about barriers that aren't easy to articulate clearly. 30% still would have made me more optimistic than a large majority of people I talked to, and so I still would have lost plenty of bets, but I would have made fewer bets and gotten better odds.
    But I think 10% would have been about as unreasonable a prediction as 60%. The technology and regulation are mature enough to make deployment possible, so exactly when we get to 10 cities looks very contingent. If the technology was better then deployment would be significantly faster, and I think we should all have wide error bars about 7 years of tech progress. And the pandemic seems to have been a major setback for ride hailing. I'm not saying I got unlucky on any of these - my default guess is that the world we are in right now is the median - but all of these events are contingent enough that we should have had big error bars.
    Lessons
    People draw a lot of lessons from our collective experience with self-driving cars:
    Some people claim that there was wild overoptimism, but this does not match up with my experience. Investors were optimistic enough to make a bet on a speculative technology, but it seems like most experts and people in tech thought the technology was pretty unlikely to be ready by 2023. Almost everyone I talked to thought 50% was too high, and the three people I talked to who actually worked on self-driving cars went further and said it seemed crazy. The evidence I see for attributing wild optimism seems to be valuations (which could be justified even by a modest probability of success), vague headlines (which make no attempt to communicate calibrated predictions), and Elon Musk saying things.
    Relatedly, people sometimes treat self-driving as if it's an easy AI problem that should be solved many years before e.g. automated software engineering. But I think we really don't know. Perceiving and quickly reacting to the world is one of the tasks humans have evolved to be excellent at, and driving could easily be as hard (or harder) than being an engineer or scientist. This isn't some post hoc rationalization...
    8 min
  • EA - Why isn't there a charity evaluator for longtermist projects? by BrownHairedEevee
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why isn't there a charity evaluator for longtermist projects?, published by BrownHairedEevee on July 29, 2023 on The Effective Altruism Forum.
    i.e., Why isn't there an org analogous to GiveWell and Animal Charity Evaluators that evaluates and recommends charities according to how much impact they can have on the long-term future, e.g. by reducing existential risk? As opposed to only making grants directly and saying "just trust us" like the EA Funds.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    1 min
  • EA - Announcing the ITAM AI Futures Fellowship by AmAristizabal
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the ITAM AI Futures Fellowship, published by AmAristizabal on July 29, 2023 on The Effective Altruism Forum.
    We are thrilled to introduce the AI Futures Fellowship, an eight-week program in Mexico City (January and February 2024) designed to support exceptional individuals in understanding and mitigating catastrophic and existential risks from advanced AI.
    TLDR
    Apply now to become one of our fellows (deadline: end of August 24, AOE)!
    Share this opportunity with potential candidates from your networks.
    Apply via this form if you would like to mentor one or more of our fellows.
    See this post and fill out this form if you would like to visit Mexico City at some point or throughout January and February to interact with the AI Futures Fellowship and other AI researchers.
    About the Program
    For eight weeks, fellows will join the Instituto Tecnológico Autónomo de México (ITAM) in Mexico City and have the opportunity to interact with a rich intellectual community of AI researchers visiting Mexico City in January and February 2024.
    Fellows will pursue a project agreed upon with a mentor at the beginning of the program. We generally expect fellows to produce a research report on a specific problem in various AI subfields, but we are open to different outputs. For example, fellows may also focus on learning more about a particular topic in AI, such as interpretability research, global race dynamics, or compute governance, and summarize their findings. Fellows may also collaborate with other participants or experts of the program.
    There will be weekly meetings, seminars, and Q&As with leading experts in the field.
    Mexico City Office
    The fellowship will take place in an office space located in La Condesa, a very green and calm area of Mexico City. The office faces Amsterdam street and Parque Mexico, which offer a fantastic mix of nature, cafes and restaurants.
    Who Should Apply?
    We are looking for early-career individuals and students from all over the world. We expect most participants to be at the late undergraduate, Master, Ph.D., and postdoctoral levels, but other exceptional candidates are also welcome. While we expect candidates to apply their research skills and knowledge to issues of advanced AI, they are not required to have previous technical experience or expertise in machine learning or AI.
    For Potential Mentors
    We are actively seeking mentors for our program, so fill out this form if you think this could be you! The role will be tailored to the fellows' needs, but it might involve the following responsibilities:
    Help a fellow define a project and support them throughout its execution (This may be a project you've been interested in doing but haven't found the time to work on!)
    Availability to commit ~1 hour weekly for 8 weeks to support the fellow (this can include reviewing drafts and having regular calls).
    Assist the fellow in expanding their professional network, identifying additional resources, and valuable opportunities.
    Offer career guidance and motivation whenever needed. However, our staff will meet regularly with mentees to follow up on their time management, accountability, wellbeing and intrinsic motivation. You are primarily expected to provide object-level guidance.
    There is a compensation of $1000 USD for the valuable time and effort invested by mentors.
    Mentors can be remote: you don't need to come to Mexico to be eligible. That said, we want to encourage in person collaboration and synergies, so if you are interested in spending some time in Mexico City during the winter, consider applying as a mentor interested in visiting Mexico in the form.
    For Potential Visitors
    If you want to visit Mexico City in January & February to interact with the AI Futures Fellowship, see this post and fill out our form as a visitor.
    Feel free to email any questi...
    5 min
  • LW - Introductory Textbook to Vision Models Interpretability by jeanne
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introductory Textbook to Vision Models Interpretability, published by jeanne on July 29, 2023 on LessWrong.
    Last year, we taught an AGISF course to Master's students at a French university (ENS Ulm). The course consisted of 10 sessions, each lasting 2 hours, and drew inspiration from the official AGISF curriculum. Currently, we are writing the corresponding textbook, which aims to provide a succinct overview of the essential aspects covered during the sessions. Here, we are sharing the section dedicated to vision interpretability to obtain feedback and because we thought it might be valuable outside of the course's context for anyone trying to learn more about interpretability. Other topics such as reward misspecification, goal misgeneralization, scalable oversight, transformer interpretability, and governance are currently a work in progress, and we plan to share them in the future.
    We welcome any comments, corrections, or suggestions you may have to improve this work. This is our first attempt at creating pedagogical content, so there is undoubtedly room for improvement. Please don't hesitate to contact us at [email protected] and [email protected].
    The pdf can be found here.
    It introduces the following papers:
    Feature Visualization (Olah, et al., 2017)
    Activation Atlas (Carter, et al., 2019)
    Zoom In: An Introduction to Circuits (Olah, et al., 2020)
    An Overview of Early Vision in InceptionV1 (Olah, et al., 2020)
    Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (Selvaraju, et al., 2017)
    Understanding RL Vision (Hilton, et al., 2020)
    Network Dissection: Quantifying Interpretability of Deep Visual Representations (Bau, et al., 2017)
    Compositional Explanations of Neurons (Mu & Andreas, 2020)
    Natural Language Description of Deep Visual Features (Hernandez, et al., 2022)
    Other papers briefly mentioned:
    The Building Blocks of Interpretability (Olah, et al., 2018)
    Sanity Checks for Saliency Maps (Adebayo, et al., 2018)
    Multimodal Neurons in Artificial Neural Networks (Goh, et al., 2021)
    The associated deck of slides can be found here.
    Thanks to Agatha Duzan for her useful feedback.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • LW - Why You Should Never Update Your Beliefs by Arjun Panickssery
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why You Should Never Update Your Beliefs, published by Arjun Panickssery on July 29, 2023 on LessWrong.
    Epistemic status: Invincible
    Since Cavalry scouts are often in direct contact with the enemy, their job can be considered one of the most dangerous jobs the Army has to offer.
    something called "Operation Military Kids"
    There's some irony that Julia Galef's rationalist self-help book The Scout Mindset compares favorably the scout, who hunts for new and reliable evidence, to the soldier, who fights off threats. But scouts have one of the most dangerous military occupations. To quote a random website, "cavalry scouts and recon units tread uncharted ground when it comes to conflict zones. They are usually at the tip of any advance and, therefore, meet the brunt of whatever resistance is lying in wait for them."
    Uncharted epistemic territory is dangerous because it's awash with incorrect arguments which might convince you of their false conclusions. Many of these arguments are designed to be persuasive regardless of their accuracy. Scott Alexander describes succumbing to an "epistemic learned helplessness" after his failure to refute crackpots whose arguments are too carefully crafted to refute in any reasonable length of time:
    What finally broke me out wasn't so much the lucidity of the consensus view so much as starting to sample different crackpots. Some were almost as bright and rhetorically gifted as Velikovsky, all presented insurmountable evidence for their theories, and all had mutually exclusive ideas. After all, Noah's Flood couldn't have been a cultural memory both of the fall of Atlantis and of a change in the Earth's orbit, let alone of a lost Ice Age civilization or of megatsunamis from a meteor strike. So given that at least some of those arguments are wrong and all seemed practically proven, I am obviously just gullible in the field of ancient history.
    Given a total lack of independent intellectual steering power and no desire to spend thirty years building an independent knowledge base of Near Eastern history, I choose to just accept the ideas of the prestigious people with professorships in Archaeology, rather than those of the universally reviled crackpots who write books about Venus being a comet.
    You could consider this a form of epistemic learned helplessness, where I know any attempt to evaluate the arguments is just going to be a bad idea so I don't even try. If you have a good argument that the Early Bronze Age worked completely differently from the way mainstream historians believe, I just don't want to hear about it. If you insist on telling me anyway, I will nod, say that your argument makes complete sense, and then totally refuse to change my mind or admit even the slightest possibility that you might be right.
    (This is the correct Bayesian action: if I know that a false argument sounds just as convincing as a true argument, argument convincingness provides no evidence either way. I should ignore it and stick with my prior.)
    The solution is to ignore most evidence that would change your views. This strategy is well-supported by epistemology and psychology:
    Critical thinking is altogether on dubious footing. See Michael Huemer's "Is Critical Thinking Epistemically Responsible?" (the link goes to his blog post summary; the full text is available at his website in Papers Epistemology). He discusses the rationality of three strategies for forming a view on a "publicly-discussed issue":"Credulity: You canvass the opinions of a number of experts, and adopt the belief held by most of them. In the best case, you find a poll of the experts; failing that, you may look through several books and articles and identify their overall conclusions.Skepticism: You give upon finding the answer, i.e., immediately suspend judgement.Critical Thinking: You gather ...
    6 min
  • LW - When can we trust model evaluations? by evhub
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When can we trust model evaluations?, published by evhub on July 28, 2023 on LessWrong.
    Thanks to Joe Carlsmith, Paul Christiano, Richard Ngo, Kate Woolverton, and Ansh Radhakrishnan for helpful conversations, comments, and/or feedback.
    In "Towards understanding-based safety evaluations," I discussed why I think evaluating specifically the alignment of models is likely to require mechanistic, understanding-based evaluations rather than solely behavioral evaluations. However, I also mentioned in a footnote why I thought behavioral evaluations would likely be fine in the case of evaluating capabilities rather than evaluating alignment:
    However, while I like the sorts of behavioral evaluations discussed in the GPT-4 System Card (e.g. ARC's autonomous replication evaluation) as a way of assessing model capabilities, I have a pretty fundamental concern with these sorts of techniques as a mechanism for eventually assessing alignment.
    That's because while I think it would be quite tricky for a deceptively aligned AI to sandbag its capabilities when explicitly fine-tuned on some capabilities task (that probably requires pretty advanced gradient hacking), it should be quite easy for such a model to pretend to look aligned.
    In this post, I want to try to expand a bit on this point and explain exactly what assumptions I think are necessary for various different evaluations to be reliable and trustworthy. For that purpose, I'm going to talk about four different categories of evaluations and what assumptions I think are needed to make each one go through.
    Before I do that, however, I want to go over the distinction between capabilities evaluations and alignment evaluations, as it'll be an important one throughout the post. Specifically:
    A capabilities evaluation is a model evaluation designed to test whether a model could do some task if it were trying to. For example: if the model were actively trying to autonomously replicate, would it be capable of doing so?
    An alignment evaluation is a model evaluation designed to test under what circumstances a model would actually try to do some task. For example: would a model ever try to convince humans not to shut it down?
    In my opinion, if you want to craft a good governance scheme around model evaluations, you're going to need both capabilities and alignment evaluations. For example, a very simplified scheme here could be something like:
    Do a bunch of capabilities evaluations for various risks:
    If we believe the scaling laws for our capabilities evaluations are such that the next model to be trained won't be capable of causing a catastrophe even if it were trying to, then it's fine to train.
    If we believe that the next model might be capable of causing a catastrophe if it were trying to, then do a bunch of alignment evals for whether the model would try to cause a catastrophe:
    If we believe that the scaling laws for our alignment evaluations are such that we're confident that the next model will be aligned and won't try to cause a catastrophe, then it's fine to train.
    Otherwise don't train any larger models.
    That's not to say that the above scheme is the best one (or even a good one) - I only provide it as an example of what the interplay between capabilities evaluations and alignment evaluations might look like.
    Now we'll look at different evaluations and see:
    What assumptions are necessary for them to be trustworthy.
    How they can help us in evaluating capabilities and/or alignment.
    1. Behavioral Non-Fine-Tuning Evaluations
    Key assumption: no evaluation gaming.
    We'll start with the most straightforward type of evaluation, behavioral non-fine-tuning evaluations - that is, just directly evaluating the model on some specific dataset, without any task-specific fine-tuning. Most current evaluations fall into this category, including e.g. o...
    16 min
  • EA - Are there diseconomies of scale in the reputation of communities? by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are there diseconomies of scale in the reputation of communities?, published by Lizka on July 29, 2023 on The Effective Altruism Forum.
    Summary: in what we think is a mostly reasonable model, the amount of impact a group has increases as the group gets larger, but so do the risks of reputational harm. Unless we believe that, as a group grows, the likelihood of scandals grows slowly (at most as quickly as a logarithmic function), this model implies that groups have an optimal size beyond which further growth is actively counterproductive - although this size is highly sensitive to uncertain parameters. Our best guesses for the model's parameters suggest that it's unlikely that EA has hit or passed this optimal size, so we reject this argument for limiting EA's growth. (And our prior, setting the model aside, is that growth for EA continues to be good.)
    You can play with the model (insert parameters that you think are reasonable) here.
    Epistemic status: reasonable-seeming but highly simplified model built by non-professionals. We expect that there are errors and missed considerations, and would be excited for comments pointing these out.
    Overview of the model
    Any group engaged in social change is likely to face reputational issues from wrongdoing by members, even if it avoids actively promoting harmful practices, simply because its members will commit wrongdoing at rates in the ballpark of the broader population.
    Wrongdoing becomes a scandal for the group if the wrongdoing becomes prominently known by people inside and outside the group, for instance if it's covered in the news (this is more likely if the person committing the wrongdoing is prominent themselves).
    Let's pretend that "scandals" are all alike (and that this is the primary way by which a group accrues reputational harm).
    Reputational harm from scandals diminishes the group's overall effectiveness (via things like it being harder to raise money).
    Conclusion of the model: If the reputational harm accrued by the group grows more quickly than the benefits (impact not accounting for reputational harm), then at some point, growth of the group would be counterproductive. If that's the case, the exact point past which growth is counterproductive would depend on things like how likely and how harmful scandals are, and how big coordination benefits are.
    To understand whether a point like this exists, we should compare the rates at which reputational harm and impact grow with the size of the group. Both might grow greater than linearly.
    Reputational harm accrued by the group in a given period of time might grow greater than linearly with the size of the group, because:
    The total reputational harm done by each scandal probably grows with the size of the group (because more people are harmed).
    The number of scandals per year probably grows roughly linearly with the size of the group, because there are simply more people who each might do something wrong.
    These things add up to greater-than-linear growth in expected reputational damage per year as the number of people involved grows.
    The impact accomplished by the group (not accounting for reputational damage) might also grow greater than linearly with the size of the group (because more people are doing what the group thinks is impactful, and because something like network effects might help larger groups more).
    Implications for EA
    If costs grow more quickly than benefits, then at some point, EA should stop growing (or should shrink); additional people in the community will decrease EA's positive impact.
    The answer to the question "when should EA stop growing?" is very sensitive to parameters in the model; you get pretty different answers based on plausible parameters (even if you buy the setup of the model).
    However, it seems hard to choose parameters that imply that ...
    30 min
  • LW - Mech Interp Puzzle 2: Word2Vec Style Embeddings by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mech Interp Puzzle 2: Word2Vec Style Embeddings, published by Neel Nanda on July 28, 2023 on LessWrong.
    Code can be found here. No prior knowledge of mech interp or language models is required to engage with this.
    Language model embeddings are basically a massive lookup table. The model "knows" a vocabulary of 50,000 tokens, and each one has a separate learned embedding vector.
    But these embeddings turn out to contain a shocking amount of structure! Notably, it's often linear structure, aka word2vec style structure. Word2Vec is a famous result (in old school language models, back in 2013!), that 'man - woman == king - queen'. Rather than being a black box lookup table, the embedded words were broken down into independent variables, "gender" and "royalty". Each variable gets its own direction, and the embedded word is seemingly the sum of its variables.
    One of the more striking examples of this I've found is a "number of characters per token" direction - if you do a simple linear regression mapping each token to the number of characters in it, this can be very cleanly recovered! (If you filter out ridiculous tokens, like 19979: 512 spaces).
    Notably, this is a numerical feature not a categorical feature - to go from 3 tokens to four, or four to five, you just add this direction! This is in contrast to the model just learning to cluster tokens of length 3, of length 4, etc.
    Question 2.1: Why do you think the model cares about the "number of characters" feature? And why is it useful to store it as a single linear direction?
    There's tons more features to be uncovered! There's all kinds of fundamental syntax-level binary features that are represented strongly, such as "begins with a space".
    Question 2.2: Why is "begins with a space" an incredibly important feature for a language model to represent? (Playing around a tokenizer may be useful for building intuition here)
    You can even find some real word2vec style relationships between pairs of tokens! This is hard to properly search for, because most interesting entities are multiple tokens. One nice example of meaningful single token entities is common countries and capitals (idea borrowed from Merullo et al). If you take the average embedding difference for single token countries and capitals, this explains 18.58% of the variance of unseen countries! (0.25% is what I get for a randomly chosen vector).
    Caveats: This isn't quite the level we'd expect for real word2vec (which should be closer to 100%), and cosine sim only tracks that the direction matters, not what its magnitude is (while word2vec should be constant magnitude, as it's additive). My intuition is that models think more in terms of meaningful directions though, and that the exact magnitude isn't super important for a binary variable.
    Question 2.3: A practical challenge: What other features can you find in the embedding? Here's the colab notebook I generated the above graphs from, it should be pretty plug and play. The three sections should give examples for looking for numerical variables (number of chars), categorical variables (begins with space) and relationships (country to capital). Here's some ideas - I encourage you to spend time brainstorming your own!
    Is a number
    How frequent is it? (Use pile-10k to get frequency data for the pile)
    Is all caps
    Is the first token of a common multi-token word
    Is a first name
    Is a function word (the, a, of, etc)
    Is a punctuation character
    Is unusually common in German (or language of your choice)
    The indentation level in code
    Relationships between common English words and their French translations
    Relationships between the male and female version of a word
    Please share your thoughts and findings in the comments! (Please wrap them in spoiler tags)
    Thanks for listening. To help us out with The Nonlinear Library or to learn mo...
    4 min
  • LW - Yes, It's Subjective, But Why All The Crabs? by johnswentworth
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Yes, It's Subjective, But Why All The Crabs?, published by johnswentworth on July 28, 2023 on LessWrong.
    Crabs
    Nature really loves to evolve crabs.
    Some early biologist, equipped with knowledge of evolution but not much else, might see all these crabs and expect a common ancestral lineage. That's the obvious explanation of the similarity, after all: if the crabs descended from a common ancestor, then of course we'd expect them to be pretty similar.
    . but then our hypothetical biologist might start to notice surprisingly deep differences between all these crabs. The smoking gun, of course, would come with genetic sequencing: if the crabs' physiological similarity is achieved by totally different genetic means, or if functionally-irrelevant mutations differ across crab-species by more than mutational noise would induce over the hypothesized evolutionary timescale, then we'd have to conclude that the crabs had different lineages. (In fact, historically, people apparently figured out that crabs have different lineages long before sequencing came along.)
    Now, having accepted that the crabs have very different lineages, the differences are basically explained. If the crabs all descended from very different lineages, then of course we'd expect them to be very different.
    . but then our hypothetical biologist returns to the original empirical fact: all these crabs sure are very similar in form. If the crabs all descended from totally different lineages, then the convergent form is a huge empirical surprise! The differences between the crab have ceased to be an interesting puzzle - they're explained - but now the similarities are the interesting puzzle. What caused the convergence?
    To summarize: if we imagine that the crabs are all closely related, then any deep differences are a surprising empirical fact, and are the main remaining thing our model needs to explain. But once we accept that the crabs are not closely related, then any convergence/similarity is a surprising empirical fact, and is the main remaining thing our model needs to explain.
    Agents
    A common starting point for thinking about "What are agents?" is Dennett's intentional stance:
    Here is how it works: first you decide to treat the object whose behavior is to be predicted as a rational agent; then you figure out what beliefs that agent ought to have, given its place in the world and its purpose. Then you figure out what desires it ought to have, on the same considerations, and finally you predict that this rational agent will act to further its goals in the light of its beliefs. A little practical reasoning from the chosen set of beliefs and desires will in most instances yield a decision about what the agent ought to do; that is what you predict the agent will do.
    Daniel Dennett, The Intentional Stance, p. 17
    One of the main interesting features of the intentional stance is that it hypothesizes subjective agency: I model a system as agentic, and you and I might model different systems as agentic.
    Compared to a starting point which treats agency as objective, the intentional stance neatly explains many empirical facts - e.g. different people model different things as agents at different times. Sometimes I model other people as planning to achieve goals in the world, sometimes I model them as following set scripts, and you and I might differ in which way we're modeling any given person at any given time. If agency is subjective, then the differences are basically explained.
    . but then we're faced with a surprising empirical fact: there's a remarkable degree of convergence among which things people do-or-don't model as agentic at which times. Humans yes, rocks no. Even among cases where people disagree, there are certain kinds of arguments/evidence which people generally agree update in a certain direction - e.g. ...
    11 min
  • AF - When can we trust model evaluations? by Evan Hubinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When can we trust model evaluations?, published by Evan Hubinger on July 28, 2023 on The AI Alignment Forum.
    Thanks to Joe Carlsmith, Paul Christiano, Richard Ngo, Kate Woolverton, and Ansh Radhakrishnan for helpful conversations, comments, and/or feedback.
    In "Towards understanding-based safety evaluations," I discussed why I think evaluating specifically the alignment of models is likely to require mechanistic, understanding-based evaluations rather than solely behavioral evaluations. However, I also mentioned in a footnote why I thought behavioral evaluations would likely be fine in the case of evaluating capabilities rather than evaluating alignment:
    However, while I like the sorts of behavioral evaluations discussed in the GPT-4 System Card (e.g. ARC's autonomous replication evaluation) as a way of assessing model capabilities, I have a pretty fundamental concern with these sorts of techniques as a mechanism for eventually assessing alignment.
    That's because while I think it would be quite tricky for a deceptively aligned AI to sandbag its capabilities when explicitly fine-tuned on some capabilities task (that probably requires pretty advanced gradient hacking), it should be quite easy for such a model to pretend to look aligned.
    In this post, I want to try to expand a bit on this point and explain exactly what assumptions I think are necessary for various different evaluations to be reliable and trustworthy. For that purpose, I'm going to talk about four different categories of evaluations and what assumptions I think are needed to make each one go through.
    Before I do that, however, I want to go over the distinction between capabilities evaluations and alignment evaluations, as it'll be an important one throughout the post. Specifically:
    A capabilities evaluation is a model evaluation designed to test whether a model could do some task if it were trying to. For example: if the model were actively trying to autonomously replicate, would it be capable of doing so?
    An alignment evaluation is a model evaluation designed to test under what circumstances a model would actually try to do some task. For example: would a model ever try to convince humans not to shut it down?
    In my opinion, if you want to craft a good governance scheme around model evaluations, you're going to need both capabilities and alignment evaluations. For example, a very simplified scheme here could be something like:
    Do a bunch of capabilities evaluations for various risks:
    If we believe the scaling laws for our capabilities evaluations are such that the next model to be trained won't be capable of causing a catastrophe even if it were trying to, then it's fine to train.
    If we believe that the next model might be capable of causing a catastrophe if it were trying to, then do a bunch of alignment evals for whether the model would try to cause a catastrophe:
    If we believe that the scaling laws for our alignment evaluations are such that we're confident that the next model will be aligned and won't try to cause a catastrophe, then it's fine to train.
    Otherwise don't train any larger models.
    That's not to say that the above scheme is the best one (or even a good one) - I only provide it as an example of what the interplay between capabilities evaluations and alignment evaluations might look like.
    Now we'll look at different evaluations and see:
    What assumptions are necessary for them to be trustworthy.
    How they can help us in evaluating capabilities and/or alignment.
    1. Behavioral Non-Fine-Tuning Evaluations
    Key assumption: no evaluation gaming.
    We'll start with the most straightforward type of evaluation, behavioral non-fine-tuning evaluations - that is, just directly evaluating the model on some specific dataset, without any task-specific fine-tuning. Most current evaluations fall into this categ...
    16 min