The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Meta announces Llama 2; "open sources" it for commercial use by LawrenceC
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meta announces Llama 2; "open sources" it for commercial use, published by LawrenceC on July 18, 2023 on LessWrong.
    See also their Llama 2 website here:, and their research paper here:/
    From their blog post:
    Takeaways
    Today, we're introducing the availability of Llama 2, the next generation of our open source large language model.
    Llama 2 is free for research and commercial use.
    Microsoft and Meta are expanding their longstanding partnership, with Microsoft as the preferred partner for Llama 2.
    We're opening access to Llama 2 with the support of a broad set of companies and people across tech, academia, and policy who also believe in an open innovation approach to today's AI technologies.
    Compared to the first Llama, LLama 2 is trained for 2T tokens instead of 1.4T, has 2x the context length (4096 instead of 2048), uses Grouped Query Attention, and performs better across the board, with performance generally exceeding code-davinci-002 on benchmarks:
    They also release both a normal base model (Llama 2) and a RLHF'ed chat model (Llama 2-chat). Interestingly, they're only releasing the 7B/13B/70B models, and not the 34B model, "due to a lack of time to sufficiently red team".
    More importantly, they're releasing it both on Microsoft Azure and also making it available for commercial use. The form for requesting access is very straightforward and does not require stating what you're using it for: (EDIT: they gave me access ~20 minutes after submitting the form, seems pretty straightforward.)
    Note that their license is not technically free for commercial use always; it contains the following clauses:
    [1.] v. You will not use the Llama Materials or any output or results of the Llama Materials to improve any other large language model (excluding Llama 2 or derivative works thereof).
    2. Additional Commercial Terms. If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee's affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Meta otherwise expressly grants you such rights.
    See also the Llama 2 Acceptable Use Policy (which seems pretty standard).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - I'm interviewing Jan Leike, co-lead of OpenAI's new Superalignment project. What should I ask him? by Robert Wiblin
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I'm interviewing Jan Leike, co-lead of OpenAI's new Superalignment project. What should I ask him?, published by Robert Wiblin on July 18, 2023 on The Effective Altruism Forum.
    I'm interviewing Jan Leike for The 80,000 Hours Podcast (personal website, Twitter).
    He's been Head of Alignment at OpenAI and is now leading their Superalignment team which will aim to figure out "how to steer & control AI systems much smarter than us" - and do it in under 4 years!
    They've been given 20% of the compute OpenAI has secured so far in order to work on it. Read the official announcement about it or Jan Leike's Twitter thread.
    What should I ask him?
    (P.S. Here's Jan's first appearance on the show back in 2018.)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    1 min
  • AF - Tiny Mech Interp Projects: Emergent Positional Embeddings of Words by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tiny Mech Interp Projects: Emergent Positional Embeddings of Words, published by Neel Nanda on July 18, 2023 on The AI Alignment Forum.
    This post was written in a rush and represents a few hours of research on a thing I was curious about, and is an exercise in being less of a perfectionist. I'd love to see someone build on this work! Thanks a lot to Wes Gurnee for pairing with me on this
    Introduction
    A particularly notable observation in interpretability in the wild is that part of the studied circuit moves around information about whether the indirect object of the sentence is the first or second name in the sentence. The natural guess is that heads are moving around the absolute position of the correct name. But even in prompt formats where the first and second names are in the different absolute positions, they find that the informations conveyed by these heads are exactly the same, and can be patched between prompt templates! (credit to Alexandre Variengien for making this point to me).
    This raises the possibility that the model has learned what I call emergent positional embeddings - rather than representing "this is the token in position 5" it may represent "this token is the second name in the sentence" or "this token is the fourth word in the sentence" or "this is the third sentence in the paragraph" etc. Intuitively, models will often want to do things like attend to the previous word, or the corresponding word in the previous sentence, etc - there are lots of things it will plausibly want to do that are natural in some emergent coordinate scheme that are unnatural in the actual token coordinate scheme.
    I was curious about this, and spent an afternoon poking around with Wes Gurnee at whether I could convince myself that these emergent positional embeddings were a thing. This post is an experiment: I'm speedrunning a rough write-up on a few hours of hacky experiments, because this seemed more interesting to write-up than not to, and I was never going to do the high-effort version. Please take all this with a mountain of salt, and I'd love to see anyone build on my incredibly rough results - code here.
    Experiments
    You can see some terrible code for these experiments here. See the Appendix for technical details
    I wanted to come up with the dumbest experiment I could that could shed light on whether this was a thing. One thing that models should really care about is the ability to attend to tokens in the previous word. Words can commonly range from 1 to 3 tokens (and maybe much longer for rare or mispelt words) so this is naturally done with an emergent scheme saying which word a token is part of.
    My experiment: I took prompts with a fixed prefix of 19 tokens and then seven random lowercase English words of varying token length, like token|izer| help| apple| dram|at|isation| architecture| sick| al|p|aca. I ran GPT-2 Small on this, look the residual stream after layer 3 (33% of the way through the model) and then trained a logistic regression probe on the residual stream of the token at the end of each word to predict which word it was in.
    This is the key plot, though it takes a bit of time to get your head around. The x axis is the absolute position of the token in the prompt and the row is the ground truth of the word index. The bar for each absolute position and row shows the distribution of guesses given on the probe validation set. The colours correspond to the seven possible indices (note that the legend is not in numerical order, sigh).
    For example: take the third bar in the second row (index=1, abs_pos=22). This is mostly red (index = 1, correct!), with a bit of blue at the bottom (index = 0, incorrect) and a bit of green at the top (index = 2, incorrect). In contrast, the bar in the row below (second bar in the third row, index=2, abs_pos=23) is mostly g...
    14 min
  • AF - Still no Lie Detector for LLMs by Daniel Herrmann
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Still no Lie Detector for LLMs, published by Daniel Herrmann on July 18, 2023 on The AI Alignment Forum.
    Background
    This post is a short version of a paper we wrote that you can find here. You can read this post to get the core ideas. You can read the paper to go a little deeper.
    The paper is about probing decoder-only LLMs for their beliefs, using either unsupervised methods (like CCS from Burns) or supervised methods. We give both philosophical/conceptual reasons we are pessimistic and demonstrate some empirical failings using LLaMA 30b. By way of background, we're both philosophers, not ML people, but the paper is aimed at both audiences.
    Introduction
    One child says to the other "Wow! After reading some text, the AI understands what water is!". The second child says "All it understands is relationships between words. None of the words connect to reality. It doesn't have any internal concept of what water looks like or how it feels to be wet. ." .
    Two angels are watching [some] chemists argue with each other. The first angel says "Wow! After seeing the relationship between the sensory and atomic-scale worlds, these chemists have realized that there are levels of understanding humans are incapable of accessing." The second angel says "They haven't truly realized it. They're just abstracting over levels of relationship between the physical world and their internal thought-forms in a mechanical way. They have no concept of [$!&&!@] or [#@&#]. You can't even express it in their language!"
    Scott Alexander, Meaningful
    Do large language models (LLMs) have beliefs? And, if they do, how might we measure them?
    These questions are relevant as one important problem that plagues current LLMs is their tendency to generate falsehoods with great conviction. This is sometimes called lying and sometimes called hallucinating. One strategy for addressing this problem is to find a way to read the beliefs of an LLM directly off its internal state. Such a strategy falls under the broad umbrella of model interpretability, but we can think of it as a form of mind-reading. Detecting lies in LLMs has many obvious applications, and is especially relevant for things like ELK.
    We tackle the question about the status of beliefs in LLMs head-on. We proceed in two stages. First, we assume that LLMs do have beliefs, and consider two current approaches for how we might measure them, due to Azaria and Mitchell and to Burns et al. We provide empirical results from LLaMA 30b that show that these methods fail to generalize in very basic ways. We then argue that, even if LLMs have beliefs, these methods are unlikely to be successful for conceptual reasons. Thus, there is still no lie-detector for LLMs.
    After describing our empirical results we take a step back and consider whether or not we should expect LLMs to have something like beliefs in the first place. We consider some recent arguments aiming to show that LLMs cannot have beliefs. We show that these arguments are misguided and rely on a philosophical mistake. We provide a more productive framing of questions surrounding the status of beliefs in LLMs. Our analysis reveals both that there are many contexts in which we should expect systems to track the truth in order to accomplish other goals but that the question of whether or not LLMs have beliefs is largely an empirical matter. We provide code at.
    Challenge in Deciphering the Beliefs of Language Models
    For now, let's assume that in order to generate human-like text, LLMs (like humans) have beliefs about the world. We might then ask how we can measure and discover their beliefs. This question immediately leads to a number of problems:
    Unreliable Self-Reporting
    Asking an LLM directly about its beliefs is insufficient. As we've already discussed, models have a tendency to hallucinate or even lie. So...
    38 min
  • EA - EAGxCambridge 2023 Retrospective by David Mears
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EAGxCambridge 2023 Retrospective, published by David Mears on July 18, 2023 on The Effective Altruism Forum.
    Written by the core organising team for EAGxCambridge, this retrospective evaluates the conference, gives feedback for the CEA events team, and makes recommendations for future organisers. It's also a general update for the community about the event, and an exercise in transparency. We welcome your feedback and comments about how we can improve EA conferences in the future.
    You can watch 19 of the talks on CEA's YouTube channel, here.
    Attendees' photos are here, and professional photos are in a subfolder.
    Summary
    We think EAGxCambridge went well.
    The main metric CEA uses to evaluate their events is 'number of connections'. We estimate around 4200 new connections resulted, at an all-told cost of around £53 per connection (=$67 at time of writing), which is a better cost-per-connection than many previous conferences. The low cost-per-connection is partly driven by the fact that the event was on the large side compared to the historical average (enabling economies of scale to kick in) and encompassed 3 days; it was also kept low by limiting travel grants.
    Of these 4200 new connections, around 1700 were potentially 'impactful' as rated by attendees. (Pinch of salt: as a rule, people don't know how impactful any given connection is.)
    The likelihood-to-recommend scores were on a par with other EA conferences, which are usually very highly rated. (The average answer was 8.7 on a 1-to-10 scale.)
    Besides making connections, we also wanted to encourage and inspire attendees to take action.
    82% of survey respondents said they planned to take at least one of a list of actions (e.g. 'change degree') as a result of the conference, including 14.5% resolving to found an EA organisation and 30% resolving to work full-time for such an organisation or in a primary cause area. After applying a pinch of salt, those numbers suggest the conference inspired people to take significant action. We heard of several anecdotal cases where the conference triggered people to apply for particular jobs or funding, or resulted in internships or research collaborations.
    We're very thankful to everyone who made this happen: volunteers, attendees, session hosts, and many others.
    Contents
    For more in-depth commentary, click through to the relevant part of the Google Doc using the links below.
    Basic stats
    Strategy
    Why Cambridge?
    Focussing on the UK and Ireland
    Core team
    Compressed timelines
    Budget
    Admissions
    Some admissions statistics
    Stewardship
    Venues
    Main venue: Guildhall first floor
    Secondary venue: ground floor
    Tertiary venue: Lola Lo
    Acoustics
    Coordinating with venue staff on the day
    Overall view
    Volunteers
    Numbers
    Attendee experience
    More snacks
    Better acoustics
    Faster wifi
    Food was "incredible" / "amazing" / "extremely good" / "really excellent"
    Attendee Slack
    Content
    Attendee favourites
    "Were any sessions you attended particularly good? Which ones?"
    Merch
    Satellite events
    Communications
    Design
    Comms strategy
    Comms tactics
    Closer to the conference itself
    Feedback survey results
    Net Promoter Score
    Demographics
    Resulting actions
    Welcomingness
    For the sake of the search index, those videos are:Testing Your Suitability For AI Alignment Research | Esben KranGood News in Effective Altruism | Shakeel HashimCombating Catastrophic Biorisk With Indoor Air Safety | Jam KraprayoonShould Effective Altruists Focus On Air Pollution? | Tom BarnesAlternative Proteins and How I Got There | Friederike Grosse-HolzHow Local Groups Can Have Global Impact | Cambridge Alternative Protein ProjectGlobal Food System Failure And What We Can Do About It | Noah WescombeInfecting Humans For Vaccine Development | Jasmin KaurExistential Risk Pessimism And The Time Of Perils | David ThorstadHow To Make The Most of EAGx | Oll...
    7 min
  • AF - Meta announces Llama 2; "open sources" it for commercial use by Lawrence Chan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meta announces Llama 2; "open sources" it for commercial use, published by Lawrence Chan on July 18, 2023 on The AI Alignment Forum.
    See also their Llama 2 website here:, and their research paper here:/
    From their blog post:
    Takeaways
    Today, we're introducing the availability of Llama 2, the next generation of our open source large language model.
    Llama 2 is free for research and commercial use.
    Microsoft and Meta are expanding their longstanding partnership, with Microsoft as the preferred partner for Llama 2.
    We're opening access to Llama 2 with the support of a broad set of companies and people across tech, academia, and policy who also believe in an open innovation approach to today's AI technologies.
    Compared to the first Llama, LLama 2 is trained for 2T tokens instead of 1.4T, has 2x the context length (4096 instead of 2048), uses Grouped Query Attention, and performs better across the board, with performance generally exceeding code-davinci-002 on benchmarks:
    More importantly, they're releasing it both on Microsoft Azure and also making it available for commercial use. The form for requesting access is very straightforward and does not require stating what you're using it for:
    Note that their license is not technically free for commercial use always; it contains the following clauses:
    [1.] v. You will not use the Llama Materials or any output or results of the Llama Materials to improve any other large language model (excluding Llama 2 or derivative works thereof).
    2. Additional Commercial Terms. If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee's affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Meta otherwise expressly grants you such rights.
    See also the Llama 2 Acceptable Use Policy (which seems pretty standard).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Measuring and Improving the Faithfulness of Model-Generated Reasoning by Ansh Radhakrishnan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Measuring and Improving the Faithfulness of Model-Generated Reasoning, published by Ansh Radhakrishnan on July 18, 2023 on LessWrong.
    TL;DR: In two new papers from Anthropic, we propose metrics for evaluating how faithful chain-of-thought reasoning is to a language model's actual process for answering a question. Our metrics show that language models sometimes ignore their generated reasoning and other times don't, depending on the particular task + model size combination. Larger language models tend to ignore the generated reasoning more often than smaller models, a case of inverse scaling. We then show that an alternative to chain-of-thought prompting - answering questions by breaking them into subquestions - improves faithfulness while maintaining good task performance.
    Paper Abstracts
    Measuring Faithfulness in Chain-of-Thought Reasoning
    Large language models (LLMs) perform better when they produce step-by-step, "Chain-of -Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT(e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT's performance boost does not seem to come from CoT's added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study.
    Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.
    Question Decomposition Improves the Faithfulness of Model-Generated Reasoning
    As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior. One approach to help with this issue is to prompt LLMs to externalize their reasoning, e.g., by having them generate step-by-step reasoning as they answer a question (Chain-of-Thought; CoT). The reasoning may enable us to check the process that models use to perform tasks. However, this approach relies on the stated reasoning faithfully reflecting the model's actual reasoning, which is not always the case. To improve over the faithfulness of CoT reasoning, we have models generate reasoning by decomposing questions into subquestions. Decomposition-based methods achieve strong performance on question-answering tasks, sometimes approaching that of CoT while improving the faithfulness of the model's stated reasoning on several recently-proposed metrics.
    By forcing the model to answer simpler subquestions in separate contexts, we greatly increase the faithfulness of model-generated reasoning over CoT, while still achieving some of the performance gains of CoT. Our results show it is possible to improve the faithfulness of model-generated reasoning; continued improvements may lead to reasoning that enables us to verify the correctness and safety of LLM behavior.
    Externalized Reasoning Oversight Relies on Faithful Reasoning
    Large language models (LLMs) are operating in increasingly challenging domains, ranging from programming assistance (Chen et al., 2021) to open-ended internet research (Nakano et al., 2021) and scientific writing (Taylor et al., 2022). However, verifying model behavior for safety and correctness becomes increasingly difficult as the difficulty of tasks increases. To make model behavior easier to check, one promising approach is to prompt LLMs to produce step-by-step "Chain-of...
    11 min
  • EA - Five Years of Rethink Priorities: Impact, Future Plans, Funding Needs (July 2023) by Rethink Priorities
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Five Years of Rethink Priorities: Impact, Future Plans, Funding Needs (July 2023), published by Rethink Priorities on July 18, 2023 on The Effective Altruism Forum.
    Overview
    This piece highlights Rethink Priorities' accomplishments, mistakes, and changes since its establishment in 2018. We discuss RP's future plans as well as potential constraints to our impact. Finally, we call for donations and invite people to engage in an Ask Me Anything (AMA) discussion with Co-CEO Peter Wildeford that will be announced by July 19 in our newsletter and social media.
    You can also read this post as a PDF with visualizations.
    Executive summary
    Key accomplishments (2018-2023)
    In five years, RP has published over 125 pieces of research, completed more than another 100 research projects, provided various grantmakers with consultation, influenced tens of millions of dollars in funding, fiscally sponsored nine projects, and drove forward the promising field of invertebrate welfare. Specific accomplishments include:
    Collaborating with dozens of European Union (EU) animal advocacy organizations to work on setting medium-term policy strategies for farmed animal welfare.
    Providing expert consultation to the Chilean government as they considered a bill (which has advanced to the next legislative stage) to recognize animals as sentient.
    Contributing significantly to burgeoning fields, such as invertebrate welfare - including work related to shrimps and insects (see more here).
    Completing the Moral Weight Project to try to help funders decide how to best allocate resources across species.
    Producing 23 reports commissioned by Open Philanthropy answering their questions about global health and development issues and interventions.
    Conducting 205 tailored surveys and data analysis to help several organizations that build communities of people working on global priorities.
    Launching projects such as Condor Camp and fiscally sponsoring organizations like Epoch and Apollo Research via our Special Projects team, which provides operational support.
    Setting up an Artificial Intelligence (AI) Governance and Strategy team.
    Growing from a two-person operation in 2018 to a team that will soon include 75 RP employees, 30 contractors, and 25 staff of fiscally sponsored projects.
    Mistakes and challenges
    We believe that some of RP's past projects failed because we did not adequately consider the project's probability of success, its potential value, and the resources required. For example, our 2018 PriorityWiki project would have involved a large volunteer coordination effort we weren't well-placed to execute, and it is unclear how valuable it would have been even if successful.
    Our biggest early mistake was not building a plan for each project's path to influence and not putting enough resources into measuring our impact. We initially relied too much on producing research and hoping that it would be impactful just by existing.
    Thus far, we have not publicly shared as much of our existing internal impact tracking as we had initially intended due to time constraints.
    While we did some preparation prior to scaling in 2022, we would have liked to have established more robust project management systems beforehand.
    Our project timelines are not always predictable, as it is difficult to determine when you should stop researching due to diminishing returns. We think there are several cases where we spent too long working on a piece of research and would've had more impact by releasing the work earlier.
    Changes
    We now have better project management systems and spend much more time thinking through how to communicate our research and ensure each piece has a higher chance for impact.
    We are focusing more on measuring our impact, and hired a Chief Strategy Analyst last year.
    In addition to researching neglected areas (e....
    29 min
  • LW - Proof of posteriority: a defense against AI-generated misinformation by jchan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Proof of posteriority: a defense against AI-generated misinformation, published by jchan on July 17, 2023 on LessWrong.
    Summary
    A proof of priority shows that a piece of data must have been created before a certain time.
    A proof of posteriority shows that a piece of data must have been created after a certain time.
    By combining the two, you can prove that the data must have been created within a certain time interval. If it would be computationally infeasible for AI to generate the data within that time, then this proves that the data is authentic (i.e. depicting real events) and not AI-generated misinformation.
    Proof of priority is easy; proof of posteriority is tricky. Therefore the success of the whole scheme hinges upon the latter.
    Illustration of the two kinds of proofs: car rental case
    Let's say you're renting from ACME Rent-a-Car. You know that ACME is maximally sleazy and will try to charge you a fee for every dent and scratch, even if it was already like that beforehand. Therefore, before taking possession of the car, you take out your phone and record a video where you walk around the car, pointing out all the existing damage. You expect to use this video as evidence if/when a dispute arises.
    But this isn't enough - ACME can accuse you of having made the video at the end of your rental, after you've already caused all the damage. To guard against this accusation, you immediately embed a hash of your video into a Bitcoin transaction, which is incorporated into the public blockchain. Then you can point to this transaction as proof that the video existed before the time you are known to have driven the car off the lot. This is a proof of priority.
    When you return the car, you're faced with the dual problem. The car is in the same condition as when you picked it up, but you're worried that ACME will come after you for damage that occurs later (due to ACME's own recklessness, or some subsequent renter). So you make another walk-around video, pointing out all the undamaged areas. But again this isn't good enough, since ACME might accuse you of having made that video at the beginning of your rental, before you (supposedly) busted up the car. To guard against this accusation, you do the reverse of what you did before - you write the hash of the latest Bitcoin block on a piece of paper, and make sure that this paper is continually in view as you make the video. Then you can point to the block with that hash as proof that the video was made after you returned the car. This is a proof of posteriority.
    General principle
    Both proofs make use of a public, timestamped source of randomness (e.g. the Bitcoin blockchain) that (a) cannot be predicted in advance, and (b) can be modified so as to become "causally downstream" of whatever data you create. You prove priority by incorporating your data into the randomness, and you prove posteriority by incorporating the randomness into your data.
    Application: proving that data wasn't generated by AI
    We are already at the stage (or soon will be) when AI can generate convincing videos showing arbitrary things happening. This fatally undermines the entire information ecosystem to which we've grown accustomed in the modern era, since we can no longer accept the mere existence of a certain string of bits as proof that something happened. If you like living in a society where truth matters and not just power relations, then you'll think this is a bad thing.
    But we might be able to salvage some of the "trustless attestation" quality of video recordings if we can combine a proof of priority and posteriority to constrain the production of the video to such a short period of time that it would be computationally infeasible for AI to generate a similarly-realistic video within that time. By "computationally infeasible" I mean: it would requir...
    8 min
  • AF - Measuring and Improving the Faithfulness of Model-Generated Reasoning by Ansh Radhakrishnan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Measuring and Improving the Faithfulness of Model-Generated Reasoning, published by Ansh Radhakrishnan on July 18, 2023 on The AI Alignment Forum.
    TL;DR: In two new papers from Anthropic, we propose metrics for evaluating how faithful chain-of-thought reasoning is to a language model's actual process for answering a question. Our metrics show that language models sometimes ignore their generated reasoning and other times don't, depending on the particular task + model size combination. Larger language models tend to ignore the generated reasoning more often than smaller models, a case of inverse scaling. We then show that an alternative to chain-of-thought prompting - answering questions by breaking them into subquestions - improves faithfulness while maintaining good task performance.
    Paper Abstracts
    Measuring Faithfulness in Chain-of-Thought Reasoning
    Large language models (LLMs) perform better when they produce step-by-step, "Chain-of -Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT(e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT's performance boost does not seem to come from CoT's added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study.
    Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.
    Question Decomposition Improves the Faithfulness of Model-Generated Reasoning
    As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior. One approach to help with this issue is to prompt LLMs to externalize their reasoning, e.g., by having them generate step-by-step reasoning as they answer a question (Chain-of-Thought; CoT). The reasoning may enable us to check the process that models use to perform tasks. However, this approach relies on the stated reasoning faithfully reflecting the model's actual reasoning, which is not always the case. To improve over the faithfulness of CoT reasoning, we have models generate reasoning by decomposing questions into subquestions. Decomposition-based methods achieve strong performance on question-answering tasks, sometimes approaching that of CoT while improving the faithfulness of the model's stated reasoning on several recently-proposed metrics.
    By forcing the model to answer simpler subquestions in separate contexts, we greatly increase the faithfulness of model-generated reasoning over CoT, while still achieving some of the performance gains of CoT. Our results show it is possible to improve the faithfulness of model-generated reasoning; continued improvements may lead to reasoning that enables us to verify the correctness and safety of LLM behavior.
    Externalized Reasoning Oversight Relies on Faithful Reasoning
    Large language models (LLMs) are operating in increasingly challenging domains, ranging from programming assistance (Chen et al., 2021) to open-ended internet research (Nakano et al., 2021) and scientific writing (Taylor et al., 2022). However, verifying model behavior for safety and correctness becomes increasingly difficult as the difficulty of tasks increases. To make model behavior easier to check, one promising approach is to prompt LLMs to produce step-by-s...
    11 min