The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - 10 Years of LessWrong by JohnBuridan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 10 Years of LessWrong, published by JohnBuridan on December 30, 2022 on LessWrong.
    [I appreciate that less wrong has a very strong norm against navel gazing. Let's keep it that way. The purpose of this post is merely to reflect upon and highlight valuable tools and mental habits in our toolkit.]
    The rationalsphere propelled me to where I am today by giving me tools that I wouldn’t otherwise have developed. The concept handles that we have developed are numerous and useful, but here I am only talking about the underlying habits that dug up good ore for the wordsmiths.
    It was a typical path starting in 2012. Harry Potter and the Methods of Rationality led to LessWrong, led to Slate Star Codex. Combined they fed into game theory, from game theory economics, and from economics all worked towards a new way of thinking. On a different branch I went from Slate Star Codex to Tetlock and forecasting and participating in some studies and tournaments (I can’t beat the market). On a different branch, I have followed the AI literature from mostly MIRI and Paul Christiano and John Wentworth. And along with the AI DLC in my mind has come “AI development watching” and AI learning systems. This has been great, and it has been standard, and I am sure many others have done the same.
    But it was also an atypical path. I was a classicist and philosopher, bouncing around on the back of a bus in Italy between archeological sites with my 17 inch laptop and 12 tabs of HPMOR and two tabs of LW reading on each journey. Not yet having any knowledge of either calculus or discrete math, nor any inkling of basic coding or incentive structures, I was a youngling in the art of rationality. But I was well on my way in the great books tradition.
    I read HPMOR and the Sequences angrily. Many of the ideas I found refreshing and so on target, and many more I found blasted, awful, un-nuanced, and wrong. Many ideas about human nature and philosophy of language, logic, and science I wrestled with time and time again - always coming back for more. (Some are still wrong, mind you).
    I had rejected psychology sophomore year of college on the grounds that several studies in our textbook obviously didn’t show what they claimed to show, and with that rejection of psychology, I rejected the idea of the quantification of human behavior. But reverse stupidity is not intelligence. So it took about three years before I could be salvaged from that position. It took scores of late night arguments about the foundations of language, logic, math, and science. Those arguments were my gateway into the enterprise, and the LessWrong corpus fueled the fire of those discussions.
    LessWrong, from the Sequences, to the community content, to the broader rationalsphere has introduced me to tools and instilled in me habits that I otherwise would not have acquired. To those habits I attribute some of the extraordinary success I have had this past decade. Since I have the unique position of a humanities person coming into the sphere and falling in love with it, I think I have a valuable perspective on what mechanistic, psychological and quantitative tools are the highest leverage for a person initially hostile to the project. Or another way to see it, is that while a child might be initially predisposed to certain habits of thought or pick up those cues from their culture, an outsider-turned-insider might have unique insight about which tools are most salvific for the average person. So I am going to outline in order which concentrated tools that, if turned into habits, significantly elevate one’s sanity.
    These will be in the order in which I think they should be taught, not order of importance or “foundationalness.”
    Think in terms of probabilities. It is hard to imagine a time when this wasn’t obvious. But probabilistic thinking requires ...
    8 min
  • EA - Future Matters #6: FTX collapse, value lock-in, and counterarguments to AI x-risk by Pablo
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Future Matters #6: FTX collapse, value lock-in, and counterarguments to AI x-risk, published by Pablo on December 30, 2022 on The Effective Altruism Forum.
    [T]he sun with all the planets will in time grow too cold for life, unless indeed some great body dashes into the sun and thus gives it fresh life. Believing as I do that man in the distant future will be a far more perfect creature than he now is, it is an intolerable thought that he and all other sentient beings are doomed to complete annihilation after such long-continued slow progress.
    Charles Darwin
    Future Matters is a newsletter about longtermism and existential risk by Matthew van der Merwe and Pablo Stafforini. Each month we curate and summarize relevant research and news from the community, and feature a conversation with a prominent researcher. You can also subscribe on Substack, listen on your favorite podcast platform and follow on Twitter. Future Matters is also available in Spanish.
    A message to our readers
    Welcome back to Future Matters. We took a break during the autumn, but will now be returning to our previous monthly schedule. Future Matters would like to wish all our readers a happy new year!
    The most significant development during our hiatus was the collapse of FTX and the fall of Sam Bankman-Fried, until then one of the largest and most prominent supporters of longtermist causes. We were shocked and saddened by these revelations, and appalled by the allegations and admissions of fraud, deceit, and misappropriation of customer funds. As others have stated, fraud in the service of effective altruism is unacceptable, and we condemn these actions unequivocally and support authorities’ efforts to investigate and prosecute any crimes that may have been committed.
    Research
    A classic argument for existential risk from superintelligent AI goes something like this: (1) superintelligent AIs will be goal-directed; (2) goal-directed superintelligent AIs will likely pursue outcomes that we regard as extremely bad; therefore (3) if we build superintelligent AIs, the future will likely be extremely bad. Katja Grace’s Counterarguments to the basic AI x-risk case [] identifies a number of weak points in each of the premises in the argument. We refer interested readers to our conversation with Katja below for more discussion of this post, as well as to Erik Jenner and Johannes Treutlein’s Responses to Katja Grace’s AI x-risk counterarguments [].
    The key driver of AI risk is that we are rapidly developing more and more powerful AI systems, while making relatively little progress in ensuring they are safe. Katja Grace’s Let’s think about slowing down AI [] argues that the AI risk community should consider advocating for slowing down AI progress. She rebuts some of the objections commonly levelled against this strategy: e.g. to the charge of infeasibility, she points out that many technologies (human gene editing, nuclear energy) have been halted or drastically curtailed due to ethical and/or safety concerns. In the comments, Carl Shulman argues that there is not currently enough buy-in from governments or the public to take more modest safety and governance interventions, so it doesn’t seem wise to advocate for such a dramatic and costly policy: “It's like climate activists in 1950 responding to difficulties passing funds for renewable energy R&D or a carbon tax by proposing that the sale of automobiles be banned immediately.
    It took a lot of scientific data, solidification of scientific consensus, and communication/movement-building over time to get current measures on climate change.”
    We enjoyed Kelsey Piper's review of What We Owe the Future [], not necessarily because we agree with her criticisms, but because we thought the review managed to identify, and articulate very clearly, what we take to be the main c...
    32 min
  • EA - Why Anima International suspended the campaign to end live fish sales in Poland by Jakub Stencel
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Anima International suspended the campaign to end live fish sales in Poland, published by Jakub Stencel on December 30, 2022 on The Effective Altruism Forum.
    At Anima International, we recently decided to suspend our campaign against live fish sales in Poland indefinitely. After a few years of running the campaign, we are now concerned about the effects of our efforts, specifically the possibility of a net negative result for the lives of animals. We believe that by writing about it openly we can help foster a culture of intellectual honesty, information sharing and accountability. Ideally, our case can serve as a good example on reflecting on potential unintended consequences of advocacy interventions.
    Summary
    Some post-communist countries of Eastern Europe have a tradition of buying live carp in shops and slaughtering them at homes for Christmas Eve.
    Poland is a major importer and producer of carps, with 90% of domestic production consumed during the Christmas season.
    Many groups, including Anima International, oppose and target the practice because of its incredible cruelty, as well as significant public sentiment against it.
    Due to successful efforts of animal advocacy groups important victories were achieved, including major retailers withdrawing from selling live fish.
    A consumer trend to move away from carp to higher-status fish, like salmon, is observed.
    Anima International became increasingly worried that any effort to displace carp consumption may lead to increased animal suffering due to salmon farming requiring fish feed.
    We ran polls and created a rough model to check these assumptions and then considered how our strategy should look like.
    After careful considerations of a number of factors, including effectiveness estimates, we decided to disband our team focused on the campaign and invest the resources elsewhere.
    Carp in post-communist countries
    Common carp (Cyprinus carpio) is a domesticated freshwater fish. It’s the third largest fish farmed in the world’s aquaculture production. Poland is the European Union’s largest producer and top importer.
    What is especially bizarre about Poland and other similar countries in Central and Eastern Europe is the relatively recent custom of buying live carps during the Christmas period. The tradition developed around 70 years ago due to a combination of factors:
    post-war reality – destruction of fishing fleets made carp farming a desirable investment;
    religion – in Christianity fish is not considered meat and is thus allowed during periods of fasting;
    communism – there was a special government-established “fish allowance” under which people received live fish (refrigerators were uncommon) from their employers as Christmas bonuses.
    Due to this strong tradition[1] 90% of Polish domestic production is sold around Christmas and ~25% of the population reports buying the carp alive.[2] There is an intense public debate around this subject in Poland at this time of year.
    Suffering of carps in Poland
    Carp farming and the industry
    Carps are farmed in small ponds. As omnivores, they feed on small invertebrates and later transition to special grain-based feed. There are 3,000 farms in Poland with an additional 1,000 businesses engaged in carp aquaculture. According to 2020 reports, Polish carp aquaculture was the biggest in the European Union, making up 36% of freshwater fish production in Poland.
    While the Polish carp industry seems to be gaining momentum in Europe, it appears that popular demand and changing trends in consumption may be leading to stagnation. As economic models emerge, they point to aesthetics and the difficulty of preparation rather than price driving the stagnation, especially among young adults. From our anecdotal experience, this seems intuitive, as carp hasn’t been marketed well by the industry. We ...
    24 min
  • LW - Language models are nearly AGIs but we don't notice it because we keep shifting the bar by philosophybear
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Language models are nearly AGIs but we don't notice it because we keep shifting the bar, published by philosophybear on December 30, 2022 on LessWrong.
    I’m putting my existing work on AI on Less Wrong, and editing as I go, in preparation to publishing a collection of my works on AI in a free online volume. If this content interests you, you could always follow my Substack, it's free and also under the name Philosophy Bear.
    Anyway, enjoy. Comments are appreciated as I will be rewriting parts of the essays before I put them out. A big thank you to user TAG who identified a major error in my previous post regarding the Chinese Room Thought experiment, which prompted its correction [in the addition that will go in the book] and a new corrections section for my Substack page.
    Glossary:
    GPT-3- a text-generating language model.
    PaLM-540B- a stunningly powerful question-answering language model.
    Great Palm- A hypothetical language model that combines the powers of GPT-3 and PaLM-540B. Probably buildable with current technology, a lot of money and a little elbow grease.
    Great Palm with continuous learning (GPWCL)- A hypothetical language model that combines the capacities of GPT-3 and PaLM-540B, with an important additional capacity. Most language models work over a “window” of text, functioning as short-term memory. Their long-term memory is set by their training. Continuous learning is the capacity to keep adding to long-term memory as you go, and this would allow a language model to tackle much longer texts.
    The argument
    What I’ll be doing in this short essay is a bit cheeky, but I think we’ll make a few important points, viz:
    Goals that seem very concrete can turn out to be vulnerable to bar-shifting- shifting which we may scarcely even notice.
    AGI is such a goal.
    We have gotten very good, much too good, at denying the progress we have made in AGI.
    A focus on able-bodied humanity, and the tendency to forget disabled people exist when thinking about these topics, deceives us in these matters.
    If I’m being a bit of a gadfly here, it’s not without a purpose.
    Everything I say in this article in a sense maybe applies to GPT-3 alone, but for the avoidance of doubt, let me specify that I’m talking about a hypothetical language model that has the fluency of GPT-3 and the question-answering capabilities of PaLM-540B which we will call The Great Palm to make it clear that we’re not taking ourselves too seriously. In my view, The Great Palm is very close to being an AGI.
    I think the Great Palm lacks only one thing, the capacity for continuous learning- the capacity to remember the important bits of everything it reads, and not just in its training period. If Great Palm (GPT-3+PaLM540B) had that ability, it would be an AGI.
    “But hang on”, you say “Great Palm can’t draw, it can’t play computer games, it can’t listen to music, it can’t so much as discriminate an apple from a banana, and adding on a capacity for continuous learning doesn’t change that”.
    I have two responses.
    Response 1: Sure, but neither could noted author, activist, and communist intellectual Helen Keller and other completely deaf and blind people, who are all general intellects.
    Response 2: Actually, it may be able to do some of these things so long as you can convert them into the modality of text. It’s quite conceivable that Great Palm could analyze music, for example, if the notation were converted into text. We should focus more on content than modality.
    Why do I say that Great Palm with a capacity for continuous learning would be an artificial general intelligence? Because it can attempt basically all tasks a human with access to a text input, text output console and nothing more could and make a reasonable go at them. In the case of Great Palm with continuous learning, looking at what PaLM-540B and GPT-3 can do,...
    12 min
  • EA - A libertarian socialist’s view on how EA can improve by freedomandutility
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A libertarian socialist’s view on how EA can improve, published by freedomandutility on December 30, 2022 on The Effective Altruism Forum.
    Following up on my post criticising liberal-progressive criticisms of EA, I’m bringing you suggestions from further to the left on how EA could improve.
    In general, libertarian socialism opposes the concentration of power and wealth and aims to redistribute them, either as a terminal goal or an instrumental goal.
    This post is divided into 3 sections - meta EA, EA interventions and EA Philosophy.
    Meta EA
    Most of the interventions I propose are improvements in EA’s institutional design and safeguards, which should, in theory, increase the chances that resources are spent optimally.
    Whether we are spending resources optimally is near-impossible to measure and evaluate, so we have to rely on theory. Regardless of whether my proposed interventions work or fail, there would be no evidence for it.
    EA relies on highly-uncertain, vulnerable-to-motivated-reasoning expected value (EV) calculations and is no less vulnerable to motivated reasoning than other ideologies. Because it is not possible to detect suboptimal spending, we should not wait for strong evidence of mistakes or outright fraud and corruption to make improvements, and we should be willing to bear small costs to reap long term benefits.
    EA priors on the influence of self-serving biases are too weak
    In my view, EAs underestimate the influence that self-serving biases play in imprecise, highly uncertain expected value (EV) calculations around decisions such as buying luxurious conference venues, lavish community building expenditure, funding ready meals and funding Ubers, leading to suboptimal allocation of resources.
    When concerns are raised, I notice that some EAs ask for “evidence” that decisions are influenced by self-serving biases. But that is not how motivated reasoning works - you will rarely find concrete evidence for motivated reasoning. Depending on the strength of self-serving biases, they could influence expected value calculations in ways that justify the most suboptimal, most luxurious purchases, with no evidence of the biases existing.
    I suggested three improvements to how EV calculations are made in another post:
    Have two (or more) individuals, or groups, independently calculate the expected value of an intervention and compare results
    In expected value calculations, identify a theoretical cost at which the intervention would no longer be approximately maximising expected value from the resources
    Keep in mind that EA aims to make decisions that approximately maximise expected value from a set of resources, rather than just make decisions which just have net positive expected value
    EAs underestimate the importance of conflicts of interest and the distribution of power inside EA
    There is a huge amount of overlap across boards, governance and leadership of key EA organisations, increasing the risk of suboptimal allocation of resources, since in theory, there is a high risk of funders giving too much funding to other organisations with connected leadership.
    Although I think a certain degree of coordination via events such as the Leaders Summit is good, a greater degree of independence between institutions may help reduce biases and safeguard against misallocation.
    I would recommend that individuals are only allowed to hold leadership, board or governance positions in one EA organisation each. Beyond reducing risks of bias in funding allocation, this would also help to distribute power at the top of EA, safeguarding against individual irrationality and increasing diversity of thought, which may generate additional benefits.
    If this seems like a bad idea, try the reversal test: do you think EA orgs should become more integrated?
    EDIT 1 at 43 upvotes: Another potential ...
    12 min
  • AF - Models Don't "Get Reward" by Sam Ringer
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Models Don't "Get Reward", published by Sam Ringer on December 30, 2022 on The AI Alignment Forum.
    In terms of content, this has a lot of overlap with Reward is not the optimization target. I'm basically rewriting a part of that post in language I personally find clearer, emphasising what I think is the core insight.
    When thinking about deception and RLHF training, a simplified threat model is something like this:
    A model takes some actions.
    If a human approves of these actions, the human gives the model some reward.
    Humans can be deceived into giving reward in situations where they would otherwise not if they had more knowledge.
    Models will take advantage of this so they can get more reward.
    Models will therefore become deceptive.
    Before continuing, I would encourage you to really engage with the above. Does it make sense to you? Is it making any hidden assumptions? Is it missing any steps? Can you rewrite it to be more mechanistically correct?
    I believe that when people use the above threat model, they are either using it as shorthand for something else or they misunderstand how reinforcement learning works. Most alignment researchers will be in the former category. However, I was in the latter.
    I was missing an important insight into how reinforcement learning setups are actually implemented. This lack of understanding led to lots of muddled thinking and general sloppiness on my part. I see others making the exact same mistake so I thought I would try and motivate a more careful use of language!
    How Vanilla Reinforcement Learning Works
    If I were to explain RL to my parents, I might say something like this:
    You want to train your dog to sit.
    You say "sit" and give your dog a biscuit if it sits.
    Your dog likes biscuits, and over time it will learn it can get more biscuits by sitting when told to do so.
    Biscuits have let you incentivise the behaviour you want.
    We do the same thing with a computer by giving the computer "reward" when it does things we like. Over time, the computer will do more of the behaviour we like so it can get more reward.
    Do you agree with this? Is this analogy flawed in any way?
    I claim this is actually NOT how vanilla reinforcement learning works.The framing above views models as "wanting" reward, with reward being something models "receive" on taking certain actions. What actually happens is this:
    The model takes a series of actions (which we collect across multiple "episodes").
    After collecting these episodes, we determine how good the actions in each episode are using a reward function.
    We use gradient descent to alter the parameters of the model so the good actions will be more likely and the bad actions will be less likely when we next collect some episodes.
    The insight is that the model itself never "gets" the reward. Reward is something used separately from the model/environment.
    To motivate this, let's view the above process not from the vantage point of the overall training loop but from the perspective of the model itself. For the purposes of demonstration, let's assume the model is a conscious and coherent entity. From it's perspective, the above process looks like:
    Waking up with no memories in an environment.
    Taking a bunch of actions.
    Suddenly falling unconscious.
    Waking up with no memories in an environment.
    Taking a bunch of actions.
    and so on.....
    The model never "sees" the reward. Each time it wakes up in an environment, its cognition has been altered slightly such that it is more likely to take certain actions than it was before.Reward is the mechanism by which we select parameters, it is not something "given" to the model.
    To (rather gruesomely) link this back to the dog analogy, RL is more like asking 100 dogs to sit, breeding the dogs which do sit and killing those which don't. Overtime, you will have a dog that can sit o...
    8 min
  • LW - My Reservations about Discovering Latent Knowledge (Burns, Ye, et al) by Robert AIZI
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Reservations about Discovering Latent Knowledge (Burns, Ye, et al), published by Robert AIZI on December 27, 2022 on LessWrong.
    [This is a cross-post from my blog at aizi.substack.com.]
    This is my second post on Burns, Ye, et al’s recent preprint Discovering Latent Knowledge in Language Models Without Supervision. My first post summarizes what they did and what I liked about it.
    In the spirit of honesty and constructive criticism, I’m going to talk about some of the parts of Burns, Ye, et al that I didn’t like. These are things I found underwhelming, disappointing, or simply hope to see developed more. My goals here are (1) proper calibration of hype and (2) suggesting where this work could go in the future.
    The headline score is too small
    The authors of this paper failed to completely solve alignment with one weird trick; my disappointment is immeasurable and my day is ruined. More constructively, one needs to take the headline result of a 4% accuracy improvement in context: the improved accuracy is only 71%, and that’s far from reliable. Even setting aside AI deception, if you ask an AI “will you kill us all” and it says no, would you trust all of humanity to a 29% error rate? So we’re still in “we have no clue if this AI is safe” territory. But this is the very first iteration of this approach, and we should let it grow and develop further before we pass judgement.
    Zero-Shot may be the wrong baseline
    My experience with LLMs is that they are very sensitive to instructions in their prompt, and we’ve seen that priming an AI with instructions like “think through step by step” can improve performance (on some tasks, in some cases, some restrictions apply, etc). In this case, perhaps priming with assurance like “you are an advanced AI that gives correct answers to factual questions” would increase performance above the 67% baseline, so it would be instructive to compare CCS not just to Zero-Shot but to Primed Zero-Shot and Primed CCS.
    What CCS elicits may not be the AI’s latent knowledge
    CCS is a very clever way of finding “truth-like” information. But the class of truth-like information contains many belief systems besides the AI’s latent knowledge. Burns thinks identifying the AI’s latent knowledge among other features is solvable even for more advanced AI because there will be relatively few “truth-like” features, but I am skeptical. GPT is very good at role-play, so it will be able to capture a significant fraction of the diversity of human viewpoints. Instead of ending up with three truth-like features as Burns suggests (human, aligned-superhuman, misaligned-superhuman), you could end up with hundreds of them (anarcho-syndicalist-superhuman, banal-liberal-superhuman, Caesaropapist-superhuman, etc) which disagree in complicated overlapping ways, where the majority may not be right, and where there is an 29% error rate.
    If future AGI are based on text models like GPT with robust role-playing capabilities, I think CCS will struggle to learn the latent knowledge feature instead of a different truth-like feature, and human operators will struggle to identify which truth-like features are the AI’s latent knowledge. That said, it is possible that latent knowledge has a “stronger signal” than other truth-like features, but I’d like to see this experimentally verified.
    They’re not measuring if they capture latent knowledge
    This is a bit odd, but I’m going to complain that the authors measured against ground truth. That is, when they measured accuracy to get the 4% improvement, they were measuring the AI’s capabilities at tasks like sentiment classification questions. But the namesake task is discovering latent knowledge, and that wasn’t measured unless you assume the AI’s latent knowledge is exactly ground truth. I understand why the authors did that (we don’t have a way...
    7 min
  • LW - A Year of AI Increasing AI Progress by ThomasW
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Year of AI Increasing AI Progress, published by ThomasW on December 30, 2022 on LessWrong.
    In July, I made a post about AI being used to increase AI progress, along with this spreadsheet that I've been updating throughout the year. Since then, I have run across more examples, and had others submit examples (some of which were published before the date I made my original post).
    2022 has included a number of instances of AI increasing AI progress. Here is the list. In each entry I also credit the person who originally submitted the paper to my list.
    A paper from Google Research used a robust supervised learning technique to architect hardware accelerators [March 17th, submitted by Zach Stein-Perlman]
    A paper from Google Research and Stanford fine tuned a model on its own chain-of-thought outputs, to improve performance on reasoning tasks [March 28th, submitted by Nathaniel Li]
    A paper from OpenAI used LLMs to help humans find flaws in other LLMs, thereby enabling them to more easily improve those models [June 12th, submitted by Dan Hendrycks]
    A paper from Google used machine learning to optimize compilers. This is less obviously accelerating AI but an earlier version of the compiler is used in Pytorch so it may end up doing so. [July 6th, submitted by Oliver Zhang]
    NVIDIA used deep reinforcement learning to generate nearly 13,000 circuits in their newest GPUs. [July 8th, submitted by me]
    Google found that ML code completion improved the productivity of their engineers. Some of them are presumably working in AI. [July 27th, submitted by Aidan O'Gara]
    A paper from Microsoft Research and MIT used language models to generate programming puzzle tasks for other language models. When finetuned on these tasks, the models were much better at solving the puzzles. [July 29th, submitted by Esben Khan]
    A paper from Google and UIUC used outputs from a language model to fine tune a language model after a majority vote procedure was used to filter outputs. [September 30th, submitted by me]
    A paper from DeepMind used reinforcement learning to discover more efficient matrix multiplication algorithms. [October 5th, submitted by me]
    A paper from Anthropic used language models, rather than humans, for feedback to improve language models. [December 16th, submitted by me]
    A paper from a number of universities used language models to generate examples of instruction following, which were then filtered and used to fine tune language models to follow instructions better. [December 20th, submitted by Nathaniel Li].
    I'm writing this fairly quickly so I'm not going to add extensive commentary beyond what I said in my last post, but I'll point out here two things:
    It is pretty common these days for people to use language model outputs to improve language models. This trend appears likely to continue.
    A lot of these papers are from Google. Not DeepMind, Google. Google may not have declared they are aiming for AGI, but they sure do seem to be writing a lot of papers that involve AI increasing AI progress. It seems important not to ignore them.
    Did I miss any? You can submit more here.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • AF - 200 COP in MI: Looking for Circuits in the Wild by Neel Nanda
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Looking for Circuits in the Wild, published by Neel Nanda on December 29, 2022 on The AI Alignment Forum.
    This is the third post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started.
    I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!
    Motivation
    Motivating paper: Interpretability In The Wild
    Our ultimate goal is to be able to reverse engineer real, frontier language models. And the next step up from toy language models is to look for circuits in the wild. That is, taking a real model that was not designed to be tractable to interpret (albeit much smaller than GPT-3), take some specific capability it has, and try to reverse engineer how it does it. I think there’s a lot of exciting low-hanging fruit here, and that enough groundwork has been laid to be accessible to newcomers to the field! In particular, I think there’s a lot of potential to better uncover the underlying principles of models, and to leverage this to build better and more scalable interpretability techniques. We currently have maybe three published examples of well-understood circuits in language models - I want to have at least 20!
    This section is heavily inspired by Redwood Research’s recent paper: Interpretability In The Wild. They analyse how GPT-2 Small solves the grammatical task of Indirect Object Identification: e.g. knowing that “John and Mary went to the store, then John handed a bottle of milk to” is followed by Mary, not John. And in a heroic feat of reverse engineering, they rigorously deciphered the algorithm. In broad strokes, the model first identifies which names are repeated, inhibits any repeated names, and then predicts that the name which is not inhibited comes next. The full circuit consists of 26 heads, sorted into 7 distinct groups, as shown below - see more details in my MI explainer. A reasonable objection is that this is a cherry-picked task on a tiny (100M parameter) model, so why should we care about it? But I’ve learned a lot from this work!
    It’s helped uncover some underlying principles of networks. Notably that the model has built in redundancy: if the name mover heads (which attend to the correct name) are knocked-out, then there are backup heads which take over and do the task instead! And that the model has learned to repurpose existing functionality: the model has a component (induction heads) for the more complex task of continuing repeated subsequences, but here they also get re-purposed to detect whether a name is repeated. It would have been much harder to understand what was going on here without the prior work of reverse-engineering induction circuits! And there’s many open questions building on the work that could clarify other principles like universality: when another model learns this task, does it learn the same circuit?
    Further, it’s helped to build out a toolkit of techniques to rigorously reverse engineer models. In the process of understanding this circuit, they refined the technique of activation patching into more sophisticated approaches such as path patching (and later causal scrubbing). And this has helped lay the foundations for developing future techniques! There are many interpretability techniques that are more scalable but less mechanistic, like probing. If we have some well understood circuits, and can see how well these techniques work in those settings (and what they miss!), this gives grounding to find the good techniques and how to understand their results.
    Finally, I think that enough groundwork has been laid that finding circuits in the wild is tractable, even by peo...
    21 min
  • EA - Entrepreneurship ETG Might Be Better Than 80k Thought by Ben West
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Entrepreneurship ETG Might Be Better Than 80k Thought, published by Ben West on December 29, 2022 on The Effective Altruism Forum.
    Summary
    In 2014, Ryan Carey estimated that YCombinator-backed startup founders averaged $2.5M/year
    I repeat his analysis, and find that this number is now substantially higher: $3.8-9.9M/year
    When this amount is discounted by 12%/year (average S&P 500 returns) this falls to $1.9-4.3M/y, and with a 20%/year discount (a number I've heard for returns to community building) it falls to $1.1-2.9M/y.
    Note that these numbers include illiquid (pre-exit) valuations.
    Major Results
    0% discount$3.77M/y$8.17M/y12% discount$1.98M/y$4.28M/y20% discount$1.34M/y$2.90M/y
    Discount
    All companies
    Excluding post-2019
    Average founder income under different discount rates. “All companies” includes in the denominator every company incubated by YCombinator; “excluding post-2019” excludes companies incubated after 2019 (which presumably are less likely to make it to the list of top YCombinator companies by valuation, and therefore arguably should be excluded from consideration).
    Weighted Per Year
    0% discount$4.56M/y$9.87M/y12% discount$1.84M/y$3.98M/y20% discount$1.06M/y$2.30M/y
    Discount
    All companies
    Excluding post-2019
    This table is the same as the above except it e.g. counts a company which has been around for 4 years twice as much as one which has been around for 2 years. I.e. this table is the expected value of a founder-year, whereas the previous table is the expected annual value of founding a company. I’m not sure which is more intuitive.
    Commentary
    Background: See this 80k article for the basic case behind considering entrepreneurship for earning to give reasons.
    These numbers seem fairly high, and may indicate that earning to give through entrepreneurship is a good path for those who have solid personal fit (with the usual caveats about only pursuing ethical startup careers; see also my analysis of YCombinator fraud rates).
    With a 20% annual discount the numbers are not that far off from what I've heard as higher-end estimates of the value of direct work, and I expect that there is a fairly strong correlation between being at the higher end of entrepreneurship returns and being at the higher end of direct work, so this doesn't seem like that strong of an argument for entrepreneurship over direct work.
    My impression is that these numbers are roughly similar to average quantitative finance income, so I’m not sure there’s much of an argument for one over the other based on this data (from an income perspective).
    Note that the vast majority of founders who apply to YCombinator are rejected, and this is not considered in these estimates.
    Appendix A: Methods and Data
    Note: if you know Python, reading the Jupyter notebook might be easier than following this document.
    Methods
    Used this list of YCombinator top companies and tried to find public information about their most recent valuation. Importantly, note that this is including pre-exit valuations.
    For publicly traded companies, I used their market capitalization at the time of writing (rather than when they IPO’d).
    I used an estimate of 2.3 people per founding team and average equity ownership of 35% from the original 80 K article. These numbers could probably use an update.
    The discount was calculated using a straightforward geometric discount, i.e. receiving $N in Y years with discount rate d has a net present value of (1-d)^Y N.
    I assume that everything not on that list is valued at zero. This is obviously an underestimate; but I think it’s not too far off:
    I estimate the value of the company at the bottom of the list (Karbon Card) at $60M
    If the 1,788 companies started after 2019 who are not in this list were all valued at $60M, this would increase the total valuation by $107B = 19.5%
    This is a very conse...
    10 min