The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • EA - Some data on the stock of EA™ funding by NunoSempere
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some data on the stock of EA™ funding, published by NunoSempere on November 20, 2022 on The Effective Altruism Forum.
    Overall Open Philanthropy funding
    Open Philanthropy’s allocation of funding through time looks as follows:
    Dustin Moskovitz’s wealth looks, per Bloomberg, like this:
    If we plot the two together, we don’t see that much of a correlation:
    Holden Karnofsky, head of Open Philanthropy, writes that the Blomberg estimates might not be all that accurate:
    Our available capital has fallen over the last year for these reasons. That said, as of now, public reports of Dustin Moskovitz and Cari Tuna’s net worth give a substantially understated picture of our available resources. That’s because, among other issues, they don’t include resources that are already in foundations. (I also note that META stock is not as large a part of their portfolio as some seem to assume)
    In mid 2022, Forbes put Sam Bankman-Fried’s wealth at $24B. So in some sense, the amount of money allocated to or according to Effective Altruism™ peaked somewhere close to $50B.
    Funding flow restricted to longtermism & global catatrophic risks (GCRs)
    The analysis becomes a bit more interesting if we look only at longtermism and GCRs:
    In contrast, per Forbes,[^1] the FTX Foundation had given out $160M by September 2022. My sense is that most (say, maybe 50% to 80%) of those grants went to “longtermist” cause areas, broadly defined. In addition, SBF and other FTX employees led a $580M funding round for Anthropic
    Further analysis
    It’s unclear what would have to happen for Open Philanthropy to pick up the slack here. In practical terms, I’m not sure whether their team has enough evaluation capacity for an additional $100M/year, or whether they will choose to expand that.
    Two somewhat informative posts from Open Philanthropy on this are here and here
    I’d be curious about both interpretative analysis and forecasting on these numbers. I am up for supporting the later by e.g., committing to rerunning this analysis in a year.
    Appendix: Code
    The code to produce these plots can be found here; lines 42 to 48 make the division into categories fairly apparent. To execute this code you will need a working R installation and a document named grants.csv, which can be downloaded from Open Philanthropy’s website.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - On EA messaging - being a doctor in a poorer country by Luke Eure
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On EA messaging - being a doctor in a poorer country, published by Luke Eure on November 20, 2022 on The Effective Altruism Forum.
    EA has growing communities in low and middle-income countries. Off the top of my head: Philippines, Nigeria, South Africa, Kenya.
    This is very good. It also means that EA orgs and communicators should move away from "western-by-default" messaging and thinking.
    Here's an instance of how assuming readers are all western can lead to wrong, or at least incompletely thought-through, conclusions.
    EA thinking tends to lead people in the US/UK away from being doctors. I've heard "don't be a doctor if you care about people" used as a shorthand for the at-first-counterintuitive recommendations EA can sometimes give.
    But, in poorer countries, it might be reasonable on EA grounds to be a doctor:
    There is a shortage of doctors in countries like Kenya -> higher direct impact
    Earning opportunities are much lower -> lower impact through counterfactual options
    It may be that it still nets out that being a doctor is not the best career choice for people in poorer countries. Seems pretty uncertain, because I don’t think anyone has thought about it in detail. In the 80k hours post, the top-level recommendation "people likely to succeed at medical school admission could have a greater impact outside medicine" is only backed up with evidence from US/UK.
    Takeaway: As EA attracts people from all over the world, we need to move away from “western-by-default” communications to ensure people get the correct, and correctly-reasoned, advice.
    In many circumstances it will still make sense to focus on a western audience for different types of communication. But this should be a decision based on the specifics of what you're communicating, not an unthinking default based on "all EAs are western"
    Quick back of the envelope on this based on stats from an 80k interview with Gregory Lewis:
    Assume that over a UK career, a doctor saves saves 6 lives. You could do the same by donating ~$30k (assuming $5k per life saved)
    He says it could be 10x in a developing country (here he says it could be more like 40-50x). This would require donating $300k to offset
    A very very good job in Kenya would be earning $50k / year. Someone making that much and donating 10% would not donate $300k over their lifetime (unless they worked for 60 years)
    > being a doctor looks pretty reasonable for someone in Kenya, relative to earning to give
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • LW - Elastic Productivity Tools by Simon Berens
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Elastic Productivity Tools, published by Simon Berens on November 19, 2022 on LessWrong.
    Like most tech bros, I’m a little too interested in productivity and optimizing my life. I’ve even made a few of my own tools to help me stay focused and efficient. In the process of trying to find and build the best productivity tools possible, I discovered common elements among all the productivity tools I found most effective.
    I call such tools elastic, because of their similarities to rubber bands. And so, elastic tools
    stretch, meaning that they’re designed to allow for some leeway
    snap back, i.e. after stretching they automatically go back to their intended form
    don’t break, in the sense that you can never stretch them too far
    Elastic tools are a marked improvement over standard tools, which typically
    are brittle, allowing no leeway
    don’t help you get back on track
    end up getting ditched
    Inelastic Tool Examples
    Freedom: gives no way of getting out of a focus session, so when I really needed to access a blocked app I would disable it, and it would take me days to re-enable it.
    Blocklist: when I needed to go to a blocked site, it gave me no choice but to disable it.
    I had 3 tasks in the windows task scheduler that would shut my computer down at 9, 10, and 11pm respectively in an attempt to enforce a consistent bedtime. However, when I knew I needed to get work done I would disable them, and then go to bed at 2am the days following that, until I would turn them back on.
    Elastic Tool Examples
    After Freedom didn’t work out for me, I switched to LockMeOut, where I could pay $2 to unlock it for the day. Unlocking for the day is a little too much for my taste, but much better than disabling it altogether.
    As for Blocklist, I made my own alternative, where if you really want to go on a blocked website, you have to effectively stare at your screen for ~1 minute. After that, you can go on previously blocked websites for ~1 hour, after which it blocks the websites again. I found that a little bit of boredom is the perfect amount of deterrent.
    To replace my shutdown tasks, I made my own tool to shut down my computer every 30 minutes after 9pm. If I need to do work, I can skip the next shutdown. This effectively gives me a maximum of 30 minutes to slack off after I finish working before my computer shuts down.
    Things That Should Be Made Elastic
    All my lights are connected to smart plugs, which I schedule to turn on at 7am (and off at 9pm) with Alexa. However, sometimes I really need to sleep in, so I adjust the schedule to turn on at 10am, but then I usually forget/am too lazy to set it back to 7am, resulting in sleeping/lying in bed too much. An elastic solution would allow me to change my wakeup time for a day, and automatically revert back to my normal schedule.
    “States” on my phone, like grayscale and do not disturb would ideally be elastic too. If I needed to look at a photo or be open to phone calls for a few minutes, ideally I would turn them off, and after a set amount of time they would turn back on.
    The general problem of maintaining habits is elastic in nature: you want to push people towards maintaining habits, allow them to break those habits when truly necessary, and make it easy for them to pick the habit back up.
    More Elasticity, Less Breaking
    If you build or have thought of building productivity tools, please make them elastic! For everyone else, hopefully now you have a framework to assess the effectiveness of potential tools.
    More
    I tried to use various things to block distracting websites, but I always ended up turning them off “for a little bit” to look at something they were blocking and, uh, “forgetting” to turn them back on.
    Shape the default- the ideal situation is for doing the right action to feel like the default, so it takes no willpower
    Thanks for listening....
    4 min
  • LW - Choosing the right dish by Adam Zerner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Choosing the right dish, published by Adam Zerner on November 19, 2022 on LessWrong.
    1
    From what I understand, despite its simplicity, there aren't many dishes held in higher esteem by the culinary community than the French omelette.
    To all of you uncultured numskulls out there who don't spend hours upon hours watching YouTube videos about cooking like I do, a French omelette looks like this:
    See how it's underdone and creamy inside? It's not the same thing that you cook for breakfast in the morning with ham and cheese. Adding cheese to a proper French omelette would be like adding ketchup to a good steak. A good steak doesn't need ketchup, and a good French omelette doesn't need cheese. It's perfect just the way it is and you'd be covering up that perfection if you added something to it.
    But getting a French omelette right requires a certain level of mastery. Let's hear what Dan Gritzer of Serious Eats has to say:
    The French omelette looms large in kitchen legend, and the story you'll most often hear is that it was the dish chefs would use to test prospective cooks. They chose an omelette of all things because, in a matter of minutes, it could show a chef everything he needed to know about the cook. Did he make an egg-splattered mess or keep things clean? Was he wasteful, or did he scrape every last bit of egg into the pan? Did he handle the pan correctly, seasoning the traditional carbon steel to give it a perfect nonstick surface? Was he quick, deft, efficient? And, after everything, did he produce that textbook almond-shaped package? Was it baveuse?
    I'd been making French omelettes for years at that point, but the depth of detail he gave in each step made it all seem new to me. The heat had to be high the whole time, with temperature controlled by moving the pan on and off of it. The pan had to be just hot enough that you couldn't press the back of your fingers on it for more than half a second. The butter had to foam but not brown. The eggs had to be beaten just until the last trace of whites vanished—but no more than that—and salted lightly at the last second. The pepper had to be white, to avoid little black flecks....Then Soltner looked at me and said, "Now you try." I'm not sure I've ever been as nervous as when I cooked my omelette under his solitary gaze. It didn't matter that I'd made hundreds, if not thousands, of French omelettes during my restaurant career. My hands shook and my fingers trembled; my heart thumped up, pressing its way into my neck.
    All my moves became timid and uncertain as I realized I'd just been cast into that fabled omelette test. I had one shot to earn the approval of a godlike chef."It's not bad," Soltner said, inspecting my omelette after I'd plated it. Not exactly high praise, but I took it.
    Wow. That's a lot.
    Forgive me if I'm being naive, but does it really matter if there are little black flecks of pepper on it? If it is a perfect almond shape? If it is baveuse?
    Kenji-Lopez Alt doesn't think so. He sees a lot of merit to a Thai-style omelette, and even prefers it to the French one.
    Instead of being perfectly smooth, careful not to add any browning like the French omelette, the Thai one is unapologetically fried and crispy. Yum! It's served over rice for some balance and contrast, and often times people will throw in something like ground pork or aromatic herbs in there. It's also common to have it with a dipping sauce. Kenji used palm sugar, garlic, chilies, fish sauce, and lime for his, although it's traditional to just squeeze some fish sauce or sriracha on top. And unlike the French omelette, it doesn't take years to get past the point of "it's not bad".
    I'm with Kenji. I think the Thai-style omelette is better. And I think this says something about ideas vs execution.
    Even if you train for years and years and master the F...
    16 min
  • AF - ARC paper: Formalizing the presumption of independence by Erik Jenner
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ARC paper: Formalizing the presumption of independence, published by Erik Jenner on November 20, 2022 on The AI Alignment Forum.
    (I did not have anything to do with this paper and these are just my own takes.)
    The Alignment Research Center recently published their second report, Formalizing the presumption of independence. While it's not explicitly about AI alignment, it's probably still interesting for some people here.
    Summary
    The paper is about "heuristic arguments". These are similar to proofs, except that their conclusions are not guaranteed to be correct and can be overturned by counterarguments. Mathematicians often use these kinds of arguments, but in contrast to proofs, they haven't been formalized. The paper mainly describes the open problem of finding a good formalization of heuristic arguments. They do describe one attempt, "cumulant propagation", in Appendix D, but point out it can behave pathologically.
    So what's the "presumption of independence" from the title? Lots of heuristic arguments work by assuming that some quantities are independent to simplify things, and that's what the paper focuses on. Such an argument can be overturned by showing that there’s actually some correlation we initially ignored, which should then lead to a more sophisticated heuristic argument with a potentially different conclusion.
    What does this have to do with alignment?
    The paper only very briefly mentions alignment (in Appendix F), more detailed discussion is planned for the future. But roughly:
    Avoiding catastrophic failures. Heuristic arguments can let us better estimate the probability of rare failures, or failures which occur only on novel distributions where we cannot easily draw samples. This can be used during validation to estimate risk, or potentially during training to further reduce risk.
    Eliciting latent knowledge. Heuristic arguments may let us see “why” a model makes its predictions. We could potentially use them to distinguish cases where similar behaviors are produced by very different mechanisms—for example distinguishing cases where a model predicts that a smiling human face will show up on camera because it predicts there will actually be a smiling human in the room, from cases where it makes the same prediction because it predicts that the camera will be tampered with. [...]
    Neither of these applications is straightforward, and it should not be obvious that heuristic arguments would allow us to achieve either goal. [...]
    Heuristic arguments can be seen as somewhere between interpretability and formal verification: unlike interpretability, heuristic arguments are meant to be machine-checkable and don't have to be human-understandable. But unlike formal proofs, they don't require perfect certainty and might be much easier to find.
    Readers here might also be reminded of Logical Induction. This paper is trying to do something somewhat different though:
    [Approaches to logical uncertainty] have primarily focused on establishing coherence conditions and on capturing inductive reasoning, i.e. ensuring that a reasoner eventually successfully predicts φ(n) given observations of φ(1), φ(2), . . . φ(n − 1). These systems would not automatically recognize intuitively valid heuristic arguments [...], although they would eventually learn to trust these arguments after observing them producing good predictions in practice.
    Indeed, we can view ourselves as reasoners in exactly this situation, trying to understand and formalize a type of reasoning that appears to often make good predictions in practice. Formalizations of inductive reasoning may help clarify the standards we should use for evaluating a proposed heuristic estimator, but do not constitute a good heuristic estimator themselves.
    So should you read the paper?
    Given it's a 60-page report (though most of that's...
    5 min
  • AF - Update to Mysteries of mode collapse: text-davinci-002 not RLHF by janus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Update to Mysteries of mode collapse: text-davinci-002 not RLHF, published by janus on November 19, 2022 on The AI Alignment Forum.
    I (and many others) did not realize this before, but: text-davinci-002 and text-davinci-001, the InstructGPT models on the OpenAI API, were not trained with RLHF (reinforcement learning from human feedback) as described in the InstructGPT paper, but a "similar but slightly different" method that uses the same human feedback data. Apparently, this other method is not technically RLHF.
    Since this update has potentially nontrivial implications for interpreting the phenomena exhibited by text-davinci-002 described in Mysteries of mode collapse (formerly titled "Mysteries of mode collapse due to RLHF"), I'm making this separate post for a signal boost.I have not corrected the original text of "Mysteries of mode collapse due to RLHF", but I've added a section at the beginning with further details on this update, copied here:
    I have received evidence from multiple credible sources that text-davinci-002 was not trained with RLHF.
    The rest of this post has not been corrected to reflect this update. Not much besides the title (formerly "Mysteries of mode collapse due to RLHF") is affected: just mentally substitute "mystery method" every time "RLHF" is invoked as the training method of text-davinci-002. The observations of its behavior otherwise stand alone.
    This is kind of fascinating from an epistemological standpoint. I was quite surprised to learn that text-davinci-002 was probably not trained with RLHF. I don't remember exactly how "text-davinci-002 is RLHF" got elevated to an unquestioned assumption in my mind. I might have mistook not being contradicted by people who I assumed were in the know as confirmation. I certainly did not expect to talk for months to dozens of people about odd behaviors I've observed in a well-known model "due to RLHF" without being contradicted in a world where the model in question wasn't trained with RLHF, but that's what happened. It wasn't just me either: the assumption that text-davinci-002(/text-davinci-001) is InstructGPT is RLHF seems ambient (e.g. search "text-davinci-002 rlhf" on Twitter, this LW post, this article, and many others). I contributed to perpetuating this misinformation cascade, and for that I apologize.
    text-davinci-002's behaviors described in this post also contributed to my confidence because RLHF seemed to be a likely and potentially satisfying explanation. Its apparently unsubstantiated confidence in very specific outcomes seems antithetical to the outer objective of self-supervised learning, which is optimized by epistemic calibration, meaning the model's entropy should be as high as possible while fitting the data. In contrast, as several comments have pointed out, it makes sense that RL kills entropy. The presence of "attractors" made me additionally suspect that optimization from non-myopic outcome-supervision was formative to text-davinci-002's psyche.
    Mode collapse and attractors do seem to also be caused by RLHF (see Dumbass policy pls halp and Inescapable wedding parties). So the update is that some other training method also gives rise to these phenomena, as they are manifested by text-davinci-002.
    Whether and how speculations concerning the causes of mode collapse/attractors should be affected depends on how text-davinci-002's training method differs from RLHF.
    What is known about text-davinci-002's training method
    Publicly available information suggests that the mystery method may not be so different from RLHF. Just today I discovered this sidenote in OpenAI's blog post Aligning Language Models to Follow Instructions:
    The InstructGPT models deployed in the API are updated versions trained using the same human feedback data. They use a similar but slightly different training me...
    5 min
  • EA - EA Organization Updates: November 2022 by Lizka
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Organization Updates: November 2022, published by Lizka on November 19, 2022 on The Effective Altruism Forum.
    These monthly posts originated as the "Updates" section of the EA Newsletter. Organizations submit their own updates, which we edit for clarity.
    Job listings that these organizations highlighted (as well as a couple of other impactful jobs) are at the top of this post. Some of the jobs have extremely pressing deadlines.
    You can see previous updates on the "EA Organization Updates (monthly series)" topic page, or in our repository of past newsletters. Notice that there’s also an “org update” tag, where you can find more news and updates that are not part of this consolidated series.
    The organizations are in alphabetical order.
    Note also: Why you’re not hearing as much from EA orgs as you’d like.
    Announcements
    The EA survey is out! Consider taking 10 minutes to complete it (regardless of your level of involvement with effective altruism).
    Applications are open for EAGxBerkeley (2-4 December 2022). The conference will be bringing together people interested in effective altruism from across the United States, particularly those who are looking to build their network and engage more deeply with EA concepts.
    Job listings
    Consider also exploring jobs listed on “Job listing (open).”
    Animal Advocacy Careers:
    Researcher and M&E Manager (12-month contract, Remote, apply by 20 November)
    Clinton Health Access Initiative:
    Technical Advisor, Maximum Impact Incubator (Remote)
    Program Manager, Maximum Impact Incubator (Remote)
    Program Associate, Maximum Impact Incubator (Remote)
    Effective Institutions Project:
    Spring and Summer Research Fellows (Remote, apply by December 4)
    Effective Ventures Operations:
    Office Manager at the Harvard Square EA Office (Cambridge, MA)
    Epoch
    Research Data Analyst (12-month contract, remote, apply by 14 December)
    Fish Welfare Initiative:
    International Generalist (Partially in India and partially remote)
    Project Design & Research Manager (India preferred / remote possible)
    Program Coordinator (India)
    Family Empowerment Media
    Head of Research (Remote)
    Director of Development (Remote)
    Founders Pledge:
    Senior Researcher/Grantmaker (Climate) (Remote)
    Senior Advisor (Remote)
    Accountant (Remote
    GiveDirectly
    Country Directors (Liberia, Mozambique, & Nigeria)
    Data Scientist (Remote)
    Senior UX/UI Designer (Remote)
    GiveWell:
    Research Analyst (Remote / Oakland, CA; apply by 20 November)
    Senior Researcher (Remote / Oakland, CA)
    Other jobs on our research team or content editing team (Remote / Oakland, CA)
    Global Priorities Institute:
    Research Fellows/Senior Research Fellows in Philosophy (Oxford, apply by 6 December)
    Research Fellows/Senior Research Fellows in Economics (Oxford, apply by 6 December)
    Predoctoral Research Fellows in Economics (Oxford, apply by 6 January)
    IDinsight:
    India Regional Director (India)
    Associate Director/Director (Philippines)
    Junior Data Engineer Francophone (Morocco)
    Associates and Senior Associates (Multiple locations)
    Longview Philanthropy:
    Head of Events Production (London, apply by 18 December)
    Sol Head of Office Operations (London, apply by 11 December)
    Sol Office Operations Associate (London, apply by 11 December)
    Open Philanthropy:
    Lead Researcher (Remote)
    Program Operations Assistant, Global Health & Wellbeing (Remote, US working hours)
    Probably Good:
    Multiple positions in operations, growth, and community management (Remote)
    Organizational updates
    These are in alphabetical order.
    80,000 Hours
    This month, 80,000 Hours launched their updated job board, which now has a search function, improved job filtering, and an email alert system.
    They also released Anonymous advice: If you want to reduce AI risk, should you take roles that advance AI capabilities? and shared seven new problem profiles:
    Immigration restrictions
    Preven...
    20 min
  • AF - A Short Dialogue on the Meaning of Reward Functions by Leon Lang
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Short Dialogue on the Meaning of Reward Functions, published by Leon Lang on November 19, 2022 on The AI Alignment Forum.
    The following is a short slack dialogue between Leon Lang, Quintin Pope, and Peli Grietzer that emerged as part of the SERI-MATS stream on shard theory. Alex Turner encouraged us to share it.
    To follow the dialogue, it seems beneficial to have read Reward is not the optimization target before.
    Leon:
    I was thinking about what the "limits" of the idea "reward is not the optimization target" is. Playing devils advocate: Maybe the reward doesn't matter at all! We spend all this time thinking about how to design reward functions that somehow capture our values. But maybe the only thing we need to do is have some kind-of-dumb reward, and an environment where the agent can only reach this reward by acting in accordance to our values.E.g.: We might reward the agent for winning some board game which is (partially) cooperative. But then in the environment, real people only cooperate with the agent if it displays cooperative/friendly/nice behavior, and other behaviors are a losing strategy.
    Never did the reward itself say anything about being cooperative, but this still emerged in accordance with the environment, and so the agent ends up pretty nice.Someone might argue against this view: Being nice is, in this environment, only instrumentally valuable for achieving reward, and a smart agent knows this, plays nice to win games and, once it has enough power, will overthrow humanity. This is a version of the problem of deceptive alignment.However, I would argue (still playing devil's advocate): The agent's learning process doesn't care about whether something is an instrumental value or the actual reward! The behavior to be friendly will be reinforced just as much, and the trained agent doesn't necessarily reason about whether something is instrumentally or finally valuable, it just wants to be nice if this was reinforced.Not playing devil's advocate anymore, I'm a bit skeptical of this type of reasoning.
    Human's are generally pretty social, but we also know that powerful humans often lose quite a lot of their morality.
    Peli:
    I like this a lot! I think there's a novel and true insight here, which I'd maybe paraphrase as 'the reinforcement schedule is a function of the reward function and environment together, and it's the reinforcement schedule that matters.'
    Quintin:
    The way I'd put it is not "Maybe the reward doesn't matter at all!", but "the labels you assign your reward function do not matter at all". You say "Never did the reward itself say anything about being cooperative", but what physical fact does this correspond to? You have a physical system (the one that implements the reward function) whose actual behavior is to reinforce cognition that leads to cooperative behavior.If you look at the implementation of the reward function, there isn't any sort of classifier that checks if the agent's behavior seems cooperative and only lets the reward event occur if the agent seems to have been sufficiently cooperative. You can take this fact and say that the reward isn't "about" being cooperative.
    But the property of the reward being "about" something is a hanging reference without basis in the actual computational behavior of the reward implementation.E.g., imagine we were to substitute the "winning board game" reward with a "win the game AND be nice" reward. However, by assumption, this doesn't change the distribution of game trajectories X rewards at all, since winning always implies being nice. We've switched what the reward is "about", without actually changing anything about its functional behavior.Another way of putting it: there is no Cartesian boundary between the thing you call the reward function and the thing you call the environment. If your enviro...
    5 min
  • AF - By Default, GPTs Think In Plain Sight by Fabien Roger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: By Default, GPTs Think In Plain Sight, published by Fabien Roger on November 19, 2022 on The AI Alignment Forum.
    Epistemic status: Speculation with some factual claims in areas I’m not an expert in.
    Thanks to Jean-Stanislas Denain, Charbel-Raphael Segerie, Alexandre Variengien, and Arun Jose for helpful feedback on drafts, and thanks to janus, who shared related ideas.
    Main claims
    GPTs’ next-token-prediction process roughly matches System 1 (aka human intuition) and is not easily accessible, but GPTs can also exhibit more complicated behavior through chains of thought, which roughly matches System 2 (aka human conscious thinking process).
    Human will be able understand how a human-level GPTs (trained to do next-token-prediction) complete complicated tasks by reading the chains of thought.
    GPTs trained with RLHF will bypass this supervision.
    System 2 and GPTs’ chains of thought are similar
    A sensible model of the human thinking process
    Here is what I feel like I’m doing when I’m thinking:
    Repeat
    Sample my next thought from my intuition
    Broadcast this thought to the whole brain
    When you ask me what is my favorite food, it feels like some thoughts “pop” into consciousness, and the following thoughts deal with previous thoughts. This is also what happens when I try to prove a statement: ideas and intuitions come to my mind, then new thoughts about these intuitions appear.
    This roughly matches the model described in Consciousness and the Brain by Stanislas Dehaene, and I believe it’s a common model of the brain within neuroscience.
    How GPTs “think”
    Autoregressive text models are performing the same kind of process when they generate text. Sampling text is using the following algorithm:
    Repeat:
    Do a forward pass, and sample the next token from the output distribution
    Add the generated token to the input. It makes it part of the input for the next forward pass, which means it can be used by lots of different attention heads, including specialized heads at earlier layers.
    This looks similar the human thinking process, and the rest of the post will be exploring this similarity and draw some conclusion we might draw from this.
    System 2 and GPTs’ chains of thought have similar strengths
    A theoretical reason
    The sample + broadcast algorithm enables the execution of many serial steps. Neurons take about 1ms to fire, which means that if a thought takes 200ms to be generated, the brain can only do 200 serial operations. This is similar to the hundredths of serial matrix multiplications GPT-3 does in its 96 layers. But by sampling and thought and broadcasting it, both GPT-3 using chains of thought and the brain using System 2 can in principle implement algorithms which require much more serial steps.
    More precisely, in a Transformer, the number of serial steps used to generate one token is #layers (no matter the prompt length), whereas the number of serial steps used to generate N tokens is N×#layers (where one step is one path trough a block made of an attention layer, an MLP, and a residual connection).
    As you can see in the figure below, each path the information can take to generate the first token is exactly #layers long, but paths to generate the last token can be up to N×#layers long.
    (Connections skipping more than 2 positions were omitted to increase readability.)
    The actual number of FLOPs used is the same when generating N tokens from an M token prompt and when generating 1 tokens from and N+M−1 token prompt because a large prompt enables more parallel computations to take place. But parallel computations are not always the bottleneck, and in particular, I think that planning to deceive humans and taking control is the kind of task which requires a large number of serial steps.
    Empirical evidence
    Chains of thought helped complete tasks which human need System 2 for: for ex...
    18 min
  • EA - Trying to keep my head on straight by ChanaMessinger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Trying to keep my head on straight, published by ChanaMessinger on November 19, 2022 on The Effective Altruism Forum.
    I’m a little confused who this is for: I think it’s for anyone who might want thoughts on orienting to the FTX situation in ways they’d most endorse later, especially if they are in a position of leadership or have people relying on them for guidance. It might not be coherent, it's just some thoughts, in the spirit of Scattered Takes and Unsolicited Advice.
    This is written in my personal capacity, not as an employee of CEA
    Something I’m thinking about a lot right now is how rationality, values, and judgment can be hardest to use when you need them most. My vision of a community is one that makes you most likely to be your best self in those times. I think I'm seeing a lot of this already, and I hope to see even more.
    So for anyone thinking about FTX things, or talking about them with others, or planning to write things in that strange dialect known as “comms”, here’s my set of things I don’t want to forget. Please feel encouraged to add your own in the comments.
    Integrity
    There is no party line (I say by fiat) - I want EA to be ok after this, and it’s sure true that there are things people could say that would make that less likely, but I just really really don’t want EA to be a place where people can’t think and say things
    I want to give explicit okness to questions and wondering around judgment, decision quality or integrity of EA, EAs and EA leaders, and I don’t want to have a missing mood about people’s understandable curiosity and concern
    That said, obviously people in sensitive situations may not answer all questions you’re curious about, and in fact many of them have legal reasons not to.
    It is not your job by dint of being an EA to protect “EA the brand”. You may decide that the brand is valuable in service of its goals; you may also have your own opinions on what the brand worth protecting is.
    Sometimes I have opinions about what makes sense to share based on confidentiality or other things, but at a broad stroke, I tend to be into people saying the truth out loud (or if my system 1 says different, I want to be into it).
    Soldier-iness (here meaning the feeling of wanting to defend “your tribe”) is normal and some of it is tracking real and important things about the value of what we’ve built here. Integrity doesn’t mean highlighting every bad faith criticism. (But also don’t let the desire to protect what is valuable warp your own beliefs about the world)
    There are going to be a lot of incentives to pile on, especially if any particular narrative starts emerging, and I also want EA to be a place where you can say “this thing that looks bad doesn’t seem actually object level bad to me for these reasons”, or “Utilitarianism is good, actually” or “EA is/isn’t worse than the reference class on this” or “I think the ways in which EA is different from other movements was a worthwhile bet, even if it added risk” or “I don’t think I know enough about this to have a take.”
    Updating your views makes sense, but probably you for the moment have most of the same views you had two weeks ago, and overupdating also lands you in the wrong place
    I would be sad if people jumped too quickly to repudiate their system of ethics, or all the unusual features of it that have let us aim at doing an unusual amount of good. I would also be sad if the vibe of our response felt disingenuous - aiming to appear less consequentialist than is the case (whatever that true case is), less willing to think about tradeoffs, etc.
    You don’t even need to have one take - you can just say a lot of things that seem true to you
    I want to say things here, on twitter, out loud, etc, that are filtered by “is this helping me and others think better and more clearly”. I might not always be ma...
    10 min