The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • AF - When discussing AI risks, talk about capabilities, not intelligence by Victoria Krakovna
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When discussing AI risks, talk about capabilities, not intelligence, published by Victoria Krakovna on August 11, 2023 on The AI Alignment Forum.
    Public discussions about catastrophic risks from general AI systems are often derailed by using the word "intelligence". People often have different definitions of intelligence, or associate it with concepts like consciousness that are not relevant to AI risks, or dismiss the risks because intelligence is not well-defined. I would advocate for using the term "capabilities" or "competence" instead of "intelligence" when discussing catastrophic risks from AI, because this is what the concerns are really about. For example, instead of "superintelligence" we can refer to "super-competence" or "superhuman capabilities".
    When we talk about general AI systems posing catastrophic risks, the concern is about losing control of highly capable AI systems. Definitions of general AI that are commonly used by people working to address these risks are about general capabilities of the AI systems:
    PASTA definition: "AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement".
    Legg-Hutter definition: "An agent's ability to achieve goals in a wide range of environments".
    We expect that AI systems that satisfy these definitions would have general capabilities including long-term planning, modeling the world, scientific research, manipulation, deception, etc. While these capabilities can be attained separately, we expect that their development is correlated, e.g. all of them likely increase with scale.
    There are various issues with the word "intelligence" that make it less suitable than "capabilities" for discussing risks from general AI systems:
    Anthropomorphism: people often specifically associate "intelligence" with being human, being conscious, being alive, or having human-like emotions (none of which are relevant to or a prerequisite for risks posed by general AI systems).
    Associations with harmful beliefs and ideologies.
    Moving goalposts: impressive achievements in AI are often dismissed as not indicating "true intelligence" or "real understanding" (e.g. see the "stochastic parrots" argument). Catastrophic risk concerns are based on what the AI system can do, not whether it has "real understanding" of language or the world.
    Stronger associations with less risky capabilities: people are more likely to associate "intelligence" with being really good at math than being really good at politics, while the latter may be more representative of capabilities that make general AI systems pose a risk (e.g. manipulation and deception capabilities that could enable the system to overpower humans).
    High level of abstraction: "intelligence" can take on the quality of a mythical ideal that can't be met by an actual AI system, while "competence" is more conducive to being specific about the capability level in question.
    It's worth noting that I am not suggesting to always avoid the term "intelligence" when discussing advanced AI systems. Those who are trying to build advanced AI systems often want to capture different aspects of intelligence or endow the system with real understanding of the world, and it's useful to investigate and discuss to what extent an AI system has (or could have) these properties. I am specifically advocating to avoid the term "intelligence" when discussing catastrophic risks, because AI systems can pose these risks without possessing real understanding or some particular aspects of intelligence.
    The basic argument for catastrophic risk from general AI has two parts: 1) the world is on track to develop generally capable AI systems in the next few decades, and 2) generally capable AI systems are likely to outcompete or overpower humans. Both of these arguments ar...
    6 min
  • AF - Linkpost: We need another Expert Survey on Progress in AI, urgently by David Mears
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Linkpost: We need another Expert Survey on Progress in AI, urgently, published by David Mears on August 11, 2023 on The AI Alignment Forum.
    Philosophy Bear writes:
    One way to get a sense of when new technologies might exist is to ask people working in the area. With high stakes, but hard to predict, technologies like AI, this serves a special role. These surveys are often used to give existential risk researchers and policymakers a sense of how quickly things may happen. This last function is very important right now, as high level policy discussions about AI- and even existential risk- are happening around the world.
    The last survey of machine learning experts on the future of AI happened in June-August 2022. On average, researchers predicted the arrival of artificial general intelligence (which the survey called HLMI- Human level machine intelligence) in 2059.
    I think that, since then, there has been a bit of a sea change, and there are very strong reasons to commission another survey:
    Palm540B, ChatGPT and GPT-4 have all came out since the survey.
    There has seemingly been a shift in public views. Plausibly, this has affected experts as well for many reasons- most notably, a good chunk of the new information might have been new to experts as well (e.g. GPT-4). I don't think we would have got these results from the public back in '22.
    There has been an apparent sudden rise in expert concern over AI risk- in many cases due to concern over existential risk from AGI. See, for example, this statement on existential risk from AI signed by numerous prominent researchers in the area. It is very plausible this rise in expert concern is linked to shorter estimated timelines. Even if AI timelines haven't shifted, assessing what proportion of experts broadly agree with the statement on existential risk would be valuable.
    At the time of the last survey, Metaculus's aggregate prediction for AGI (open to everyone) had just fallen from about 2054 to 2040 about a month before the survey, most likely as an update due to the launch of PALM540B- which was, at the time, a tremendous step forward. It's plausible experts had not fully internalized this information by the time of the survey, unlike prediction markets which internalize new information quickly (sometimes too quickly). Since the survey finished Metaculus's aggregate prediction has fallen another eight years, from 2040 to 2032- almost half the 'remaining' time on this model- most likely due to the launch of Chat-GPT and GPT-4. In total, since the start of 2022, the aggregated forecast on Metaculus fell from about 30 years to about 10 years.
    The old estimates may, at this point, not only be not as good as the could be, but might actively mislead policy makers into relative complacency. 2059 is a long time away- for one thing, it's outside the life expectancy of many policy makers. If, as I suspect, expert timelines are now significantly shorter, we need to be able to testify to that ASAP.
    Given how quickly events are moving, the survey should probably be annual anyway. I won't say now is the most crucial time in AI policymaking, but there's an important combination: the stakes are high and the situation is relatively malleable.
    Especially relative to the benefits, the costs are not that high. It's just a survey after all.
    I'm currently reading through some materials prepared for policymakers. Updated aggregate expert estimates are absolutely likely to move the needle on policy discussion around this.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • EA - The cost of (not) taking marketing seriously by James Odene [User-Friendly]
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The cost of (not) taking marketing seriously, published by James Odene [User-Friendly] on August 11, 2023 on The Effective Altruism Forum.
    TLDR: EA organisations are forfeiting impact if they don't take marketing seriously. User-friendly are offering a free and friendly 'sense-check' consultancy call to all EA-aligned organisations. Email [email protected] to enquire.
    Introduction
    By not properly addressing the role of marketing in the effective altruism movement, there is substantial impact being forfeited; potentially ~20% impact loss, though in some cases, likely even more. Our efforts, no matter how rational or impact-seeking, can only create the desired change if they are effectively communicated and disseminated to our intended audience. Marketing provides us with the tools to bridge the gap between intention and action, between knowledge and support, and between ideas and real-world impact. If we continue to communicate without adequate marketing consideration, we are consistently short-changing our impact.
    The effective altruist approach is highly analytical. Due to this, we often assume that our intended audience won't be impacted by marketing, however, everyone is (yes, even you are) impacted by aesthetic, by language choice, by an interface and experience quality. You may not think of your organisation as a 'brand' with a 'customer-base' trying to 'sell', but you are still beholden to all the same rules that consumer facing brands are as these are derived from human-nature and our shared (biassed) psychology.
    Below I will highlight some key research and how EA organisations can take advantage of the findings.
    5 key pieces of research on why marketing matters;
    "The Long and Short of It" by Les Binet and Peter Field:
    This influential research study analysed over 1,400 advertising campaigns and demonstrated the importance of long-term brand building and balancing it with short-term activation. The findings indicate that organisations that invest in long-term brand building experience significant increases in brand-related key performance indicators, market share growth, and profitability.
    How can EA organisations benefit from this:
    Firstly, it's important to note that most organisations don't have a marketing budget - so this is step one. What this should be will vary for each organisation, but I would expect a rough gauge of 10-30% of your annual budget to be on marketing.
    For companies that allocate 60% or more of their marketing budget to long-term brand building activities achieve a 140% increase in brand-related metrics. Additionally, organisations that strike the right balance between long-term and short-term campaigns experience 2.5x higher market share growth and 3x higher profitability growth compared to those with an unbalanced approach. For EA organisations, this could translate into higher donation volumes, fellowship applications or related organisational objectives. Even less directly public-facing organisations will rely on long-term brand building as without being known or being salient, the organisation objectives will suffer no matter how effective your core work is.
    "How Brands Grow" by Byron Sharp and the Ehrenberg Bass Institute:
    Sharp's research challenges traditional marketing assumptions and provides evidence-based insights into brand growth and audience behaviour. The study emphasises the significance of reaching a broad audience and creating strong mental availability through consistent and distinctive branding. The research findings indicate that organisations focusing on broadening their audience base can achieve a 60% to 80% increase in brand penetration and a 25% to 50% increase in market share. Moreover, improving mental availability through consistent and distinctive branding contributes to a 10% to 25% increase in m...
    14 min
  • EA - Predicting Virus Relative Abundance in Wastewater by Jeff Kaufman
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Predicting Virus Relative Abundance in Wastewater, published by Jeff Kaufman on August 11, 2023 on The Effective Altruism Forum.
    At the Nucleic Acid Observatory (NAO) we're evaluating pathogen-agnostic surveillance. A key question is whether metagenomic sequencing of wastewater can be a cost-effective method to detect and mitigate future pandemics. In this report we investigate one piece of this question: at a given stage of a viral pandemic, what fraction of wastewater metagenomic sequencing reads would that virus represent?
    To make this concrete, we define RA(1%). If 1% of people are infected with some virus (prevalence) or have become infected with it during a given week (incidence), RA(1%) is the fraction of sequencing reads (relative abundance) generated by a given method that would match that virus. To estimate RA(1%) we collected public health data on sixteen human-infecting viruses, re-analyzed sequencing data from four municipal wastewater metagenomic studies, and linked them with a hierarchical Bayesian model.
    Three of the viruses were not present in the sequencing data, and we could only generate an upper bound on RA(1%). Four viruses had a handful of reads, for which we were able to generate rough estimates. For the remaining nine viruses we were able to narrow down RA(1%) for a specific virus-method combination to approximately an order of magnitude. We found RA(1%) for these nine viruses varied dramatically, over approximately six orders of magnitude. It also varied by study, with some viruses seeing an RA(1%) three orders of magnitude higher in one study than another.
    The NAO plans to use the estimates from this study as inputs into a modeling framework to assess the cost effectiveness of wastewater MGS detection under different pandemic scenarios, and we include an outline of such a framework with some rough estimates of the costs of different monitoring approaches.
    Read the full report: Predicting Virus Relative Abundance in Wastewater.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    3 min
  • EA - What Does a Marginal Grant at LTFF Look Like? Funding Priorities and Grantmaking Thresholds at the Long-Term Future Fund by Linch
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Does a Marginal Grant at LTFF Look Like? Funding Priorities and Grantmaking Thresholds at the Long-Term Future Fund, published by Linch on August 10, 2023 on The Effective Altruism Forum.
    The Long-Term Future Fund (LTFF) makes small, targeted grants with the aim of improving the long-term trajectory of humanity. We are currently fundraising to cover our grantmaking budget for the next 6 months. We would like to give donors more insight into how we prioritize different projects, so they have a better sense of how we plan to spend their marginal dollar. Below, we've compiled fictional but representative grants to illustrate what sort of projects we might fund depending on how much we raise for the next 6 months, assuming we receive grant applications at a similar rate and quality to the recent past.
    Our motivations for presenting this information are a) to provide transparency about how the LTFF works, and b) to move the EA and longtermist donor communities towards a more accurate understanding of what their donations are used for. Sometimes, when people donate to charities (EA or otherwise), they may wrongly assume that their donations go towards funding the average, or even more optimistically, the best work of those charities. However, it is usually more useful to consider the marginal impact for the world that additional dollars would buy. By offering illustrative examples of the sort of projects we might fund at different levels of funding, we hope to give potential donors a better sense of what their donations might buy, depending on how much funding has already been committed. We hope that this post will help improve the quality of thinking and discussions about charities in the EA and longtermist communities.
    For donors who believe that the current marginal LTFF grants are better than marginal funding of all other organizations, please consider donating! Compared to the last 3 years, we now have both a) unusually high quality and quantity of applications and b) unusually low amount of donations, which means we'll have to raise our bar substantially if we do not receive additional donations. This is an especially good time to donate, as donations are matched 2:1 by Open Philanthropy (OP donates $2 for every $1 you donate). That said, if you instead believe that marginal funding of another organization is (between 1x and 3x, depending on how you view marginal OP money) better than current marginal LTFF grants, then please do not donate to us, and instead donate to them and/or save the money for later.
    Background on the LTFF
    We are committed to improving the long-term trajectory of civilization, with a particular focus on reducing global catastrophic risks.
    We specialize in funding early stage projects rather than established organizations.
    From March 2022 to March 2023, we received 878 applications and funded 263 as grants, worth ~$9.1M dollars total (average $34.6k/grant). To our knowledge, we have made more small grants in this time period than any other longtermist- or EA- motivated funder.
    Other funders in this space include Open Philanthropy, Survival and Flourishing Fund, and recently Lightspeed Grants and Manifund.
    Historically, ~40% of our funding has come from Open Phil. However, we are trying to become more independent of Open Phil. As a temporary stopgap measure, Open Phil is matching donations to LTFF 2:1 instead of granting to us directly.
    100% of money we fundraise for LTFF qua LTFF goes to grantees; we fundraise separately and privately for operational costs.
    We try to be very willing to fund weird things that the grantmakers' inside views believe are really impactful for the long-term future.
    You can read more about our work at our website here, or in our accompanying payout report here.
    Methodology for this analysis
    At the LTFF, we assign ea...
    18 min
  • EA - Manifolio: The tool for making Kelly optimal bets on Manifold Markets by Will Howard
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Manifolio: The tool for making Kelly optimal bets on Manifold Markets, published by Will Howard on August 10, 2023 on The Effective Altruism Forum.
    I've made a calculator that makes it easy to make correctly sized bets on Manifold. You just put in the market and your estimate of the true probability, and it tells you the right amount to bet according to the Kelly criterion.
    "The right amount to bet according to the Kelly criterion" means maximising the expected logarithm of your wealth.
    There is a simple formula for this in the case of bets with fixed odds, but this doesn't work well on prediction markets in general because the market moves in response to your bet. Manifolio accounts for this, plus some other things like the risk from other bets in your portfolio. I've aimed to make it simple and robust so you can focus on estimating the probability and trust that you are betting the right amount based on this.
    You can use it here (with a market prefilled as an example), or read a more detailed guide in the github readme. It's also available as a chrome extension... which currently has to be installed in a slightly roundabout way (instructions also in the readme). I'll update here when it's approved in the chrome web store.
    Why bet Kelly (redux)?
    Much ink has been spilled about why maximising the logarithm of your wealth is a good thing to do. I'll just give a brief pitch for why it is probably the best strategy, both for you, and for "the good of the epistemic environment".
    For you
    Given a specific wealth goal, it minimises the expected time to reach that goal compared to any other strategy.
    It maximises wealth in the median (50th percentile) outcome.
    Furthermore, for any particular percentile it gets arbitrarily close to being the best strategy as the number of bets gets very large. So if you are about to participate in 100 coin flip bets in a row, even if you know you are going to get the 90th percentile luckiest outcome, the optimal amount to bet is still close to the Kelly optimal amount (just marginally higher). In my opinion this is the most compelling self-interested reason, even if you get very lucky or unlucky it's never far off the best strategy.
    (the above are all in the limit of a large number of iterated bets)
    There are also some horror stories of how people do when using a more intuition based approach... it's surprisingly easy to lose (fake) money even when you have favourable odds.
    For the good of the epistemic environment
    A marketplace consisting of Kelly bettors learns at the optimal rate, in the following sense:
    Special property 1: the market will produce an equilibrium probability that is the wealth weighted average of each participant's individual probability estimate. In other words it behaves as if the relative wealth of each participant is the prior on them being correct.
    Special property 2: When the market resolves one way or the other, the relative wealth distribution ends up being updated in a perfectly Bayesian manner. When it comes time to bet on the next market, the new wealth distribution is the correctly updated prior on each participant being right, as if you had gone through and calculated Bayes' rule for each of them.
    Together these mean that, if everyone bets according to the Kelly criterion, then after many iterations the relative wealth of each participant ends up being the best possible indicator of their predictive ability. And the equilibrium probability of each market is the best possible estimate of the probability, given the track record of each participant. This is a pretty strong result!
    I'd love to hear any feedback people have on this. You can leave a comment here or contact me by email.
    Thanks to the people who funded this project on Manifund, and everyone who has given feedback and helped me test it out
    This is shown...
    4 min
  • AF - Could We Automate AI Alignment Research? by Stephen McAleese
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Could We Automate AI Alignment Research?, published by Stephen McAleese on August 10, 2023 on The AI Alignment Forum.
    Summary
    Creating an AI that can do AI alignment research (an automated alignment researcher) is one part of OpenAI's three-part alignment plan and a core goal of their Superalignment team.
    The automated alignment researcher will probably involve an advanced language model more capable than GPT-4 and possibly with human-level intelligence. Current alignment approaches such as recursive reward modeling could be used to align the automated alignment researcher.
    A key problem with creating an automated alignment researcher is a chicken-and-egg problem where alignment is needed before the AI can be safely created and the AI is needed to create the alignment solution. One solution to this problem is bootstrapping involving an initial alignment solution and iterative improvements to the AI's alignment and capabilities.
    I argue that the success of the plan depends on aligning the automated alignment researcher being easier than aligning AGI.
    Automating alignment research could be a form of differential technological development and could be a Pivotal Act.
    Creating a human-level automated alignment researcher seems risky because it is a form of AGI and therefore has similar risks.
    Related ideas include Cyborgism and creating whole-brain emulations of AI alignment researchers before creating AGI.
    Key cruxes: the level of capabilities needed to solve alignment and the difficulty of aligning an AI with the goal of solving alignment compared to other more open-ended goals.
    Introduction
    OpenAI's alignment plan has three pillars: improving methods for aligning models using human feedback (e.g. RLHF), training models to assist human evaluation (e.g. recursive reward modeling), and training AI systems to do alignment research.
    I recently listened to the AXRP podcast episode on Jan Leike and the OpenAI superalignment team which inspired me to write this post. My goal is to explore and evaluate OpenAI's third approach to solving alignment: creating an AI model that can do AI alignment research or automated AI alignment researcher to solve the alignment problem. I'm interested in exploring the approach because it seems to be the newest and most speculative part of their plan and my goal in this post is to try and explain how it could work and the key challenges that would need to be solved to make it work.
    At first, the idea may sound unreasonable because there's an obvious chicken-and-egg problem: what's the point in creating an automated alignment researcher to solve alignment if we need to solve alignment first to create the automated alignment researcher? It's easy to dismiss the whole proposal given this problem but I'll explain how there is a way around it. I'll also go through the risks associated with the plan.
    Why would an automated alignment researcher be valuable?
    Scalable AI alignment is a hard unsolved problem
    In their alignment plan, OpenAI acknowledges that creating an indefinitely scalable solution to AI alignment is currently an unsolved problem and will probably be difficult to solve.
    Reinforcement learning with human feedback (RLHF) is OpenAI's current alignment approach for recent systems such as ChatGPT and GPT-4 but it probably wouldn't scale to aligning superintelligence so new ideas are needed. One major problem is that RLHF depends on human evaluators understanding and evaluating the outputs of AIs. But once AI's are superintelligent, human evaluators probably won't be able to understand and evaluate their outputs. RLHF also has other problems such as reward hacking and can incentivize deception. See this paper for a more detailed description of problems with RLHF.
    Recursive reward modeling (RRM) improves RLHF by using models to help humans evalu...
    29 min
  • EA - Two Years Community Building, Ten Lessons (Re)Learned by Rockwell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Two Years Community Building, Ten Lessons (Re)Learned, published by Rockwell on August 10, 2023 on The Effective Altruism Forum.
    Today is my two-year anniversary with EA NYC, where I serve as director. To say I've learned a lot would be a tremendous understatement. The more visible parts of that learning are often technical: who's doing what in which organization, why [niche thing I'd never heard of] is actually really important, what mentorship opportunities exist for people with [these very specific qualities], how to wrangle 100+ people into a group photo and still have everyone visible. (I've gotten super good at that last part!) But I've also learned - or relearned - some much larger life lessons.None of the below lessons are wholly novel, but I think they're worth stating for the broader community. Regardless of where I personally am in five or ten years, I think this is a list I'll return to. The contents are simple but so, so easy to forget. Without further ado, here are ten major lessons I've from my tenure thus far:
    1. Many people who want to do good in the world feel alone as a result
    Through my role with EA NYC, I'm often among people's first touchpoints with the EA community. I constantly have conversations with people who feel deeply unfulfilled in their life and their life's work, with an underlying knowledge they're not currently doing much of meaning. Often, they are in their 30s or 40s. Often, they are - or think they are - the only one in their circles who wants to do something that matters. To them, even just learning the EA community exists feels like a door into a new world, one where they no longer need to feel deeply alone in their desire to do good. There are many people who want to do good and are willing to devote their lives to it, it's just a matter of finding them and helping them believe it's possible. Which leads me to:
    2. Extremely impressive people have imposter syndrome
    I routinely speak with extremely accomplished individuals who I'm so excited to have join the community and have multiple promising routes to impact lying before them. Yet, they profoundly question their abilities. More than once, I've felt quietly intimidated while speaking to someone, only to have them then ask me, "Is there a place for me in EA? Do you think someone like me has anything to bring to the table?" People you would least expect are plagued by self-doubt. People with a range of backgrounds and skill sets are nearly universally poorly calibrated when it comes to their own potential. As a result:
    3. People often want "permission" to do good
    Many people are waiting for an external green light to signal "you can do something that matters." Others, like the SuccessIf advising team, have explained this concept more in-depth than I will here. In a word, people often want an external party to say, "Yes, this is okay to do," before they do it. That external validation or assurance can be a deciding factor in whether or not they take an action. And as community builders and community members, it's our job to provide that permission slip, reassuring others that their contributions matter, their thinking is rational, and they're not alone in this journey.Tangentially, I worry that EA's focus on professionalized channels toward improving the world further fosters "permission culture" in comparison to, for example, grassroots social movements.
    A job offer should not be a prerequisite to making an impact, and a rejection letter certainly shouldn't stop you from picking up your phone and calling your elected official or volunteering for a human challenge trial. Being able to keep going and keep believing in your ability to have an impact can mean recognizing that:
    4. Tempering rejection is a learned skill
    A lot of my observations above come back to a fear of rejection....
    8 min
  • EA - Utilitarianism.net Audio by Richard Y Chappell
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Utilitarianism.net Audio, published by Richard Y Chappell on August 10, 2023 on The Effective Altruism Forum.
    Hi all, a quick announcement that Utilitarianism.net now has AI-narrated audio (courtesy of the good folks at Type III Audio)! Click the audio icon on almost any page of the website, or subscribe to the podcast to listen to it all in your preferred podcast app.
    Enjoy!
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org
    1 min
  • LW - LLMs are (mostly) not helped by filler tokens by Kshitij Sachan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LLMs are (mostly) not helped by filler tokens, published by Kshitij Sachan on August 10, 2023 on LessWrong.
    Thanks to Ryan Greenblatt, Fabien Roger, and Jenny Nitishinskaya for running some of the initial experiments and to Gabe Wu and Max Nadeau for revising this post. This work was done at Redwood Research. The views expressed are my own and do not necessarily reflect the views of the organization.
    I conducted experiments to see if language models could use 'filler tokens' - unrelated text output before the final answer - for additional computation. For instance, I tested whether asking a model to produce "a b c d e" before solving a math problem improves performance.
    This mini-study was inspired by Tamera Lanham, who ran similar experiments on Claude and found negative results. Similarly, I find that GPT-3, GPT-3.5, and Claude 2 don't benefit from filler tokens. However, GPT-4 (which Tamera didn't study) shows mixed results with strong improvements on some tasks and no improvement on others. These results are not very polished, but I've been sitting on them for a while and I thought it was better to publish than not.
    Motivation
    Language models (LMs) perform better when they produce step-by-step reasoning, known as "chain of thought", before answering. A nice property of chain of thought is that it externalizes the LM's thought process, making it legible to us humans. Some researchers hope that if a LM makes decisions based on undesirable factors, e.g. the political leanings of its supervisors (Perez 2022) or whether or not its outputs will be closely evaluated by a human, then this will be visible in its chain of thought, allowing supervisors to catch or penalize such behavior.
    If, however, we find that filler tokens alone improve performance, this would suggest that LMs are deriving performance benefits from chain-of-thought prompting via mechanisms other than the human-understandable reasoning steps displayed in the words they output. In particular, filler tokens may still provide performance benefits by giving the model more forward passes to operate on the details provided in the question, akin to having "more time to think" before outputting an answer.
    Why Might Filler Tokens Be Useful?
    This is an abstracted drawing of a unidirectional (i.e. decoder-only) transformer. Each circle represents the residual stream at a given token position after a given layer. The arrows depict how attention passes information between token positions. Note that as we add more tokens (i.e. columns), the circuit size increases but the circuit depth (defined to be the maximum length of a path in the computational graph) remains capped at the number of layers. Therefore, adding pre-determined filler tokens provides parallel computation but not serial computation. This is contrasted with normal chain-of-thought, which provides extra serial computation because the tokens themselves are determined by the state of the model in the previous last layer.
    An example of a task that can benefit from parallel computation is computing the greatest common divisor of two numbers. The typical approach is Euclid's algorithm, which is bottlenecked on serial compute. However, given enough parallel computation, the model could also implement a more naive strategy to calculate the GCD of two numbers a and b:
    An early layer delegates a potential divisor to each filler token position
    An intermediate layer, in parallel at each filler token position, computes if a and b are divisible by the potential divisor
    A later layer aggregates the results and returns the largest found divisor
    More generally, there are likely tasks on which language models have rough heuristics that operate independently and parallelizing these heuristics could improve performance.
    On the other hand, many tasks are bottlenecked by seri...
    12 min