The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - Dan Luu on "You can only communicate one top priority" by Raemon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Dan Luu on "You can only communicate one top priority", published by Raemon on March 18, 2023 on LessWrong.
    h/t to rpglover64 who pointed me towards this twitter thread in this comment.
    Here's Dan Luu's take on what happens when orgs try to communicate nuanced priorities. (Related to my You Get About Five Words post)
    One thing it took me quite a while to understand is how few bits of information it's possible to reliably convey to a large number of people. When I was at MS, I remember initially being surprised at how unnuanced their communication was, but it really makes sense in hindsight.
    For example, when I joined Azure, I asked people what the biggest risk to Azure was and the dominant answer was that if we had more global outages, major customers would lose trust in us and we'd lose them forever, permanently crippling the business.
    Meanwhile, the only message VPs communicated was the need for high velocity. When I asked why there was no communication about the thing considered the highest risk to the business, the answer was if they sent out a mixed message that included reliability, nothing would get done.
    The fear was that if they said that they needed to ship fast and improve reliability, reliability would be used as an excuse to not ship quickly and needing to ship quickly would be used as an excuse for poor reliability and they'd achieve none of their goals.
    When I first heard this, I thought it was odd, but having since paid attention to what happens when VPs and directors attempt to communicate information downwards, I have to concede that it seems like the MS VPs were right and nuanced communication usually doesn't work at scale.
    I've seen quite a few people in upper management attempt to convey a mixed/nuanced message since my time at MS and I have yet to observe a case of this working in a major org at a large company (I have seen this work at a startup, but that's a very different environment).
    I've noticed this problem with my blog as well. E.g., I have some posts saying BigCo $ is better than startup $ for p50 and maybe even p90 outcomes and that you should work at startups for reasons other than pay. People often read those posts as "you shouldn't work at startups".
    I see this for every post, e.g., when I talked about how latency hadn't improved, one of the most common responses I got was about how I don't understand the good reasons for complexity. I literally said there are good reasons for complexity in the post!
    As noted previously, most internet commenters can't follow constructions as simple as an AND, and I don't want to be in the business of trying to convey what I'd like to convey to people who won't bother to understand an AND since I'd rather convey nuance
    But that's because, if I write a blog post and 5% of HN readers get it and 95% miss the point, I view that as a good outcome since was useful for 5% of people and, if you want to convey nuanced information to everyone, I think that's impossible and I don't want to lose the nuance
    If people won't read a simple AND, there's no way to simplify a nuanced position, which will be much more complex, enough that people in general will follow it, so it's a choice between conveying nuance to people who will read and avoiding nuance since most people don't read
    But it's different if you run a large org. If you send out a nuanced message and 5% of people get it and 95% of people do contradictory things because they understood different parts of the message, that's a disaster. I see this all the time when VPs try to convey nuance.
    BTW, this is why, despite being widely mocked, "move fast & break things" can be a good value. It coneys which side of the trade-off people should choose. A number of companies I know of have put velocity & reliability/safety/etc. into their values and it's failed every t...
    5 min
  • LW - "Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities) by David Scott Krueger (formerly: capybaralet)
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities), published by David Scott Krueger (formerly: capybaralet) on March 18, 2023 on LessWrong.
    This is a brief, stylized recounting of a few conversations I had at some point last year with people from the non-academic AI safety community:
    Me: you guys should write up your work properly and try to publish it in ML venues.
    Them: well that seems like a lot of work and we don't need to do that because we can just talk to each other and all the people I want to talk to are already working with me.
    Me: What about the people who you don't know who could contribute to this area and might even have valuable expertise? You could have way more leverage if you can reach those people. Also, there is increasing interest from the machine learning community in safety and alignment... because of progress in capabilities people are really starting to consider these topics and risks much more seriously.
    Them: okay, fair point, but we don't know how to write ML papers.
    Me: well, it seems like maybe you should learn or hire people to help you with that then, because it seems like a really big priority and you're leaving lots of value on the table.
    Them: hmm, maybe... but the fact is, none of us have the time and energy and bandwidth and motivation to do that; we are all too busy with other things and nobody wants to.
    Me: ah, I see! It's an incentive problem! So I guess your funding needs to be conditional on you producing legible outputs.
    Me, reflecting afterwards: hmm... Cynically, not publishing is a really good way to create a moat around your research... People who want to work on that area have to come talk to you, and you can be a gatekeeper. And you don't have to worry about somebody with more skills and experience coming along and trashing your work or out-competing you and rendering it obsolete...
    There are sort of 3 AI safety communities in my account:1) people in academia2) people at industry labs who are building big models3) the rest (alignment forum/less wrong and EA being big components). I'm not sure where to classify new orgs like Conjecture and Redwood, but for the moment I put them here.
    I'm referring to the last of these in this case.
    I'm not accusing anyone of having bad motivations; I think it is almost always valuable to consider both people's concious motivations and their incentives (which may be subconscious drivers of their behavior).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - "Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities) by David Scott Krueger
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities), published by David Scott Krueger on March 18, 2023 on The AI Alignment Forum.
    This is a brief, stylized recounting of a few conversations I had at some point last year with people from the non-academic AI safety community:
    Me: you guys should write up your work properly and try to publish it in ML venues.
    Them: well that seems like a lot of work and we don't need to do that because we can just talk to each other and all the people I want to talk to are already working with me.
    Me: What about the people who you don't know who could contribute to this area and might even have valuable expertise? You could have way more leverage if you can reach those people. Also, there is increasing interest from the machine learning community in safety and alignment... because of progress in capabilities people are really starting to consider these topics and risks much more seriously.
    Them: okay, fair point, but we don't know how to write ML papers.
    Me: well, it seems like maybe you should learn or hire people to help you with that then, because it seems like a really big priority and you're leaving lots of value on the table.
    Them: hmm, maybe... but the fact is, none of us have the time and energy and bandwidth and motivation to do that; we are all too busy with other things and nobody wants to.
    Me: ah, I see! It's an incentive problem! So I guess your funding needs to be conditional on you producing legible outputs.
    Me, reflecting afterwards: hmm... Cynically, not publishing is a really good way to create a moat around your research... People who want to work on that area have to come talk to you, and you can be a gatekeeper. And you don't have to worry about somebody with more skills and experience coming along and trashing your work or out-competing you and rendering it obsolete...EtA: In comments, people have described adhering to academic standards of presentation and rigor as "jumping through hoops". There is an element of that, but this really misses the value that these standards have to the academic community. This is a longer discussion, though...
    There are sort of 3 AI safety communities in my account:1) people in academia2) people at industry labs who are building big models3) the rest (alignment forum/less wrong and EA being big components). I'm not sure where to classify new orgs like Conjecture and Redwood, but for the moment I put them here.
    I'm referring to the last of these in this case.
    I'm not accusing anyone of having bad motivations; I think it is almost always valuable to consider both people's concious motivations and their incentives (which may be subconscious drivers of their behavior).
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Unjournal: Evaluations of "Artificial Intelligence and Economic Growth", and new hosting space by david reinstein
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Unjournal: Evaluations of "Artificial Intelligence and Economic Growth", and new hosting space, published by david reinstein on March 17, 2023 on The Effective Altruism Forum.
    New set of evaluations
    The Unjournal evaluations of Artificial Intelligence and Economic Growth, by prominent economists Philippe Aghion, Benjamin F. Jones, Charles I. Jones – are up. You can read these on our new PubPub community space , along with my discussion of the process and the insights and the 'evaluation metrics', and the authors' response. Thanks to the authors for their participation (reward early-adopters who stick their necks out!), and thanks to Philip Trammel and Seth Benzell for detailed and insightful evaluation.I discussed some of the reasons we 'took on' this paper in an earlier post. The discussion of AI's impact on the economy, what it might look like (in magnitude and in its composition), how to measure and model it, and what conditions lead to "growth explosions", seem especially relevant to recent events and discussion.
    "Self-correcting" science?
    I'm particularly happy about one outcome here.
    If you were a graduate student reading the paper, or were a professional delving into the economics literature, and had seen the last step of the equations pasted below (from the originally published paper/chapter), what would you think?
    The final step in fact contains an error; the claimed implication does not follow.
    From my discussion:
    ... we rarely see referees and colleagues actually reading and checking the math and proofs in their peers’ papers. Here Phil Trammel did so and spotted an error in a proof of one of the central results of the paper (the ‘singularity’ in Example 3). ... The authors have acknowledged this error ... confirmed the revised proof, and link a marked up version on their page. This is ‘self-correcting research’, and it’s great!
    Even though the same result was preserved, I believe this provides a valuable service.
    Readers of the paper who saw the incorrect proof (particularly students) might be deeply confused. They might think ‘Can I trust this papers’ other statements?’ ‘Am I deeply misunderstanding something here? Am I not suited for this work?’ Personally, this happened to me a lot in graduate school; at least some of the time it may have been because of errors and typos in the paper. I suspect many math-driven paper also contain flaws which are never spotted, and these sometimes may affect the substantive results (unlike in the present case).
    By the way, the marked up 'corrected' paper is here, and the corrected proof is here. (Caveat: Philip and the authors have agreed on the revised corrected proof, it might benefit from an independent verification.)
    New (additional) platform: PubPub
    We are trying out the PubPub platform. We are still maintaining our Sciety page, and we aim to import the content from one to the other, for greater visibility. Some immediate benefits of PubPub...
    It lets us assign 'digital object identifiers' (DOIs) for each evaluation, response, and summary. It puts these and the works referenced into the 'CrossRef' database.
    Jointly, this should (hopefully) enable indexing in Google Scholar and other academic search engines,
    And 'bibliometrics' (citation counts etc.0
    It seems to enable evaluations of work hosted anywhere that has a DOI (published, preprints, etc.)
    It's versatile and full-featured, enabling input from and output from a range of formats, as well as community input and discussion
    It's funded by a non-profit and seems fairly mission-aligned
    More coming soon, updates
    The Unjournal has several more impactful papers evaluated and being evaluated, which we hope to post soon. For a sense of what's coming, see our 'Direct Evaluation track' focusing on NBER working papers.
    Some other updates:
    We are pursuing collaborations wit...
    5 min
  • LW - Meetup Tip: The Next Meetup Will Be. . . by Screwtape
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meetup Tip: The Next Meetup Will Be. . ., published by Screwtape on March 17, 2023 on LessWrong.
    Summary
    One of the most useful things to tell people at a meetup is when the next meetup will be. This requires you to know when the next meetup will be. If you don't know when the next meetup will be then the next best equivalent is telling people how you will announce the next meetup. The summary of this post is that if you want to convert occasional attendees into regular attendees, I think you should have a habit of always knowing the next meetup when you run a meetup or at least should have a copy of some kind of mailing list on you.
    You have now read the basic point of this post. If you want to read on, cool, lets talk about implementation details for a bit.
    Details
    I grew up attending a small town church. I have not been back to that church in over a decade, but I can tell you when their next meetup is: it's next Sunday at ten. That is an incredibly powerful tool for return attendance. You don't need to be quite that regular (though note that the NYC community attributed some of its success to one person committing to be at the same place at the same time each week, ctrl+f for "the brilliant move") but one time you know attendees are listening to you is at the meetup. Why not take advantage of it?
    Assuming you run good meetups that people enjoy, then as they're putting on their jacket and getting ready to leave they're in a good mood. This is an excellent time to prompt them to consider coming back. "If you had fun, we'll be here two weeks from now doing it again. Hopefully I'll see you then!" The term for this is a call to action, and they're widely used because they work.
    If you know what your next meetup is about, then you can catch some of their interest. "Next month we're doing a trivia game with calibration estimates. It's usually pretty fun!" If they're the kind of person who likes calibration scores or trivia games, now maybe they're more looking forward to it. I have had times in my life where the thing I was most looking forward to at the end of the week was seeing some friends at a meetup on Saturday.
    Plus, the sooner you tell them when the next thing is, the less likely they are to double book themselves. There are lots of cases where I find out about some event only after I've already made plans for that time, and then I can't go. If I'd known ahead of time, I could have scheduled things so that they didn't overlap. Since you usually can't schedule your meetups around individual attendees schedules, your other option is to let them know first.
    I have a lousy memory. It's easy for me to forget how to get in touch with someone, though the internet makes this easier. If someone wants to come back for more, make it easy for them by putting them on some kind of announcement setup (Facebook groups, email lists, Meetup.com, whatever you use) and then using it to tell them when you're doing more. Let it remind them.
    By the way, you can both tell people about the next meetup during the current meetup and also tell people about whatever electronic notification system you use. You can do both! Maybe they can't make the next one but they'll show up after that.
    You may feel like you're bothering people too much. It's possible you might actually bother people too much! I am going to state here that the correct amount to bring up the next meetup and your mailing list is about once a meetup if it's small enough everyone is in earshot at the same time, or once a person if you have reason to believe you're getting each person once. An easy way to offer it to each person once is to do it as they leave, since they're usually only going to do that one time a meetup.
    Quick Tricks
    Lots of people these days have smartphones. Instead of having them handwrite their email addres...
    4 min
  • EA - Why SoGive is publishing an independent evaluation of StrongMinds by ishaan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why SoGive is publishing an independent evaluation of StrongMinds, published by ishaan on March 17, 2023 on The Effective Altruism Forum.
    Executive summary
    We believe the EA community's confidence in the existing research on mental health charities hasn't been high enough to use it to make significant funding decisions.
    Further research from another EA research agency, such as SoGive, may help add confidence and lead to more well-informed funding decisions.
    In order to increase the amount of scrutiny on this topic, SoGive has started conducting research on mental health interventions, and we plan to publish a series of articles starting in the next week and extending out over the next few months.
    The series will cover literature reviews of academic and EA literature on mental health and moral weights.
    We will be doing in-depth reviews and quality assessments on work by the Happier Lives Institute pertaining to StrongMinds, the RCTs and academic sources from which StrongMinds draws its evidence, and StrongMinds' internally reported data.
    We will provide a view on how impactful we judge StrongMinds to be.
    What we will publish
    From March to July 2023, SoGive plans to publish a series of analyses pertaining to mental health. The content covered will include
    Methodological notes on using existing academic literature, which quantifies depression interventions in terms of standardised mean differences, numbers needed to treat, remission rates and relapse rates; as well as the "standard deviation - years of depression averted" framework used by Happier Lives Institute.
    Broad, shallow reviews of academic and EA literature pertaining to the question of what the effect of psychotherapy is, as well as how this intersects with various factors such as number of sessions, demographics, and types of therapy.
    We will focus specifically on how the effect decays after therapy, and publish a separate report on this.
    Deep, narrow reviews of the RCTs and meta-analyses that are most closely pertaining to the StrongMind's context.
    Moral weights frameworks, explained in a manner which will allow a user to map dry numbers such as effect sizes to more visceral subjective feelings, so as to better apply their moral intuition to funding decisions.
    Cost-effective analyses which combine academic data and direct evidence from StrongMinds to arrive at our best estimate at what a donation to StrongMinds does.
    We hope these will empower others to check our work, do their own analyses of the topic, and take the work further.
    How will this enable higher impact donations?
    In the EA Survey conducted by Rethink Priorities, 60% of EA community members surveyed were in favour of giving "significant resources'' to mental health interventions, with 24% of those believing it should be a "top priority" or "near top priority" and 4% selecting it as their "top cause". Although other cause areas performed more favourably in the survey, this still appears to be a moderately high level of interest in mental health.
    Some EA energy has now gone into this area - for example, Charity Entrepreneurship incubated Canopie, Mental Health Funder's Circle, and played a role in incubating Happier Lives Institute. They additionally launched Kaya Guides and Vina Plena last year. We also had a talk from Friendship Bench at last year's EA Global.
    Our analysis will focus on StrongMinds. We chose StrongMinds because we know the organisation well. SoGive’s founder first had a conversation with StrongMinds in 2015 (thinking of his own donations) having seen a press article about them and having considered them a potentially high impact charity. Since then, several other EA orgs have been engaging with StrongMinds. Evaluations of StrongMinds specifically have now been published by both Founders Pledge and Happier Lives Institute, and Str...
    11 min
  • EA - The illusion of consensus about EA celebrities by Ben Millwood
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The illusion of consensus about EA celebrities, published by Ben Millwood on March 17, 2023 on The Effective Altruism Forum.
    Epistemic status: speaking for myself and hoping it generalises
    I don't like everyone that I'm supposed to like:
    I've long thought that [redacted] was focused on all the wrong framings of the issues they discuss,
    [redacted] is on the wrong side of their disagreement with [redacted] and often seems to have kind of sloppy thinking about things like this,
    [redacted] says many sensible things but a writing style that I find intensely irritating and I struggle to get through; [redacted] is similar, but not as sensible,
    [redacted] is working on an important problem, but doing a kind of mediocre job of it, which might be crowding out better efforts.
    Why did I redact all those names? Well, my criticisms are often some mixture of:
    half-baked; I don't have time to evaluate everyone fairly and deeply, and don't need to in order to make choices about what to focus on,
    based on justifications that are not very legible or easy to communicate,
    not always totally central to their point or fatal to their work,
    kind of upsetting or discouraging to hear,
    often not that actionable.
    I want to highlight that criticisms like this will usually not surface, and while in individual instances this is sensible, in aggregate it may contribute to a misleading view of how we view our celebrities and leaders. We end up seeming more deferential and hero-worshipping than we really are. This is bad for two reasons:
    it harms our credibility in the eyes of outsiders (or insiders, even) who have negative views of those people,
    it projects the wrong expectation to newcomers who trust us and want to learn or adopt our attitudes.
    What to do about it?
    I think "just criticise people more" in isolation is not a good solution. People, even respected people in positions of leadership, often seem to already find posting on the Forum a stressful experience, and I think tipping that balance in the more brutal direction seems likely to cost more than it gains.
    I think you could imagine major cultural changes around how people give and receive feedback that could make this better, mitigate catastrophising about negative feedback, and ensure people feel safe to risk making mistakes or exposing their oversights. But those seem to me like heavy, ambitious pieces of cultural engineering that require a lot of buy-in to get going, and even if successful may incur ongoing frictional costs. Here's smaller, simpler things that could help:
    Write a forum post about it (this one's taken, sorry),
    Make disagreements more visible and more legible, especially among leaders or experts. I really enjoyed the debate between Will MacAskill and Toby Ord in the comments of Are we living at the most influential time in history? ­– you can't come away from that discussion thinking "oh, whatever the smart, respected people in EA think must be right", because either way at least one of them will disagree with you!
    There's a lot of disagreement on the Forum all the time, of course, but I have a (somewhat unfair) vibe of this as the famous people deposit their work into the forum and leave for higher pursuits, and then we in the peanut gallery argue over it.
    I'd love it if there were (say) a document out there that Redwood Research and Anthropic both endorsed, that described how their agendas differ and what underlying disagreements lead to those differences.
    Make sure people incoming to the community, or at the periphery of the community, are inoculated against this bias, if you spot it. Point out that people usually have a mix of good and bad ideas. Have some go-to examples of respected people's blind spots or mistakes, at least as they appear to you. (Even if you never end up explaining them to anyone, it's probably goo...
    4 min
  • EA - Getting Better at Writing: Why and How by bgarfinkel
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Getting Better at Writing: Why and How, published by bgarfinkel on March 17, 2023 on The Effective Altruism Forum.
    This post is adapted from a memo I wrote a while back, for people at GovAI. It may, someday, turn out to be the first post in a series on skill-building.
    Summary
    If you're a researcher,[1] then you should probably try to become very good at writing. Writing well helps you spread your ideas, think clearly, and be taken seriously. Employers also care a lot about writing skills.
    Improving your writing is doable: it’s mostly a matter of learning guidelines and practicing. Since hardly anyone consciously works on their writing skills, you can become much better than average just by setting aside time for study and deliberate practice.
    Why writing skills matter
    Here are three reasons why writing skills matter:
    The main point of writing is to get your ideas into other people’s heads. Far more people will internalize your ideas if you write them up well. Good writing signals a piece is worth reading, reduces the effort needed to process it, guards against misunderstandings, and helps key ideas stick.
    Writing and thinking are intertwined. If you work to improve your writing on some topic, then your thinking on it will normally improve too. Writing concisely forces you to identify your most important points. Writing clearly forces you to be clear about what you believe. And structuring your piece in a logical way forces you to understand how your ideas relate to each other.
    People will judge you on your writing. If you want people to take you seriously, then you should try to write well. Good writing is a signal of clear thinking, conscientiousness, and genuine interest in producing useful work.
    For all these reasons, most organizations give a lot of weight to writing skills when they hire researchers. If you ask DC think tank staffers what they look for in candidates, they apparently mention “writing skills” more than anything else. "Writing skills" was also the first item mentioned when I recently asked the same question to someone on a lab policy team. GovAI certainly pays attention to writing when we hire. Even if you just want to impress potential employers, then, you should care a great deal about your own writing.
    How to get better at writing
    If you want to get better at writing, here are four things you can do:
    Read up on guidelines: There are a lot of pieces on how good writing works. The footnote at the end of this sentence lists some short essays.[2] The best book I know is Style: Lessons in Clarity and Grace. It’s an easy-to-read textbook that offers recipe-like guidance. I would recommend this book over anything else.[3]
    Engage with model pieces: You can pick out a handful of well-written pieces and read them with a critical mindset. (See the next footnote for some suggestions.[4]) You might ask: What exactly is good about the pieces? How do they work? Where do they obey or violate the guidelines recommended by others?
    Get feedback: Flaws in your writing—especially flaws that limit comprehension—will normally be more evident to people who are coming in cold. Also, sometimes other people will simply be better than you at diagnosing and correcting certain flaws. Comments and suggest-edits can draw your attention to recurring issues in your writing and offer models for how you can correct them.
    Do focused rewriting: The way you’ll ultimately get better is by doing focused rewriting. Pick some imperfect pieces—ideally, pieces you’re actually working on—and simply try to make them as good as possible.[5] You can consciously draw on writing guidelines, models, and previous feedback to help you diagnose and correct their flaws. The more time you spend rewriting, the better the pieces will become. Crucially, you’ll also start to internalize the techniques you...
    6 min
  • LW - "Carefully Bootstrapped Alignment" is organizationally hard by Raemon
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Carefully Bootstrapped Alignment" is organizationally hard, published by Raemon on March 17, 2023 on LessWrong.
    In addition to technical challenges, plans to safely develop AI face lots of organizational challenges. If you're running an AI lab, you need a concrete plan for handling that.
    In this post, I'll explore some of those issues, using one particular AI plan as an example. I first heard this described by Buck at EA Global London, and more recently with OpenAI's alignment plan. (I think Anthropic's plan has a fairly different ontology, although it still ultimately routes through a similar set of difficulties)
    I'd call the cluster of plans similar to this "Carefully Bootstrapped Alignment."
    It goes something like:
    Develop weak AI, which helps us figure out techniques for aligning stronger AI
    Use a collection of techniques to keep it aligned/constrained as we carefully ramp it's power level, which lets us use it to make further progress on alignment.
    [implicit assumption, typically unstated] Have good organizational practices which ensure that your org actually consistently uses your techniques to carefully keep the AI in check. If the next iteration would be too dangerous, put the project on pause until you have a better alignment solution.
    Eventually have powerful aligned AGI, then Do Something Useful with it.
    I've seen a lot of debate about points #1 and #2 – is it possible for weaker AI to help with the Actually Hard parts of the alignment problem? Are the individual techniques people have proposed to help keep it aligned actually going to work?
    But I want to focus in this post on point #3. Let's assume you've got some version of carefully-bootstrapped aligned AI that can technically work. What do the organizational implementation details need to look like?
    When I talk to people at AI labs about this, it seems like we disagree a lot on things like:
    Can you hire lots of people, without the company becoming bloated and hard to steer?
    Can you accelerate research "for now" and "pause later", without having an explicit plan for stopping that their employees understand and are on board with?
    Will your employees actually follow the safety processes you design? (rather than put in token lip service and then basically circumventing them? Or just quitting to go work for an org with fewer restrictions?)
    I'm a bit confused about where we disagree. Everyone seems to agree these are hard and require some thought. But when I talk to both technical researchers and middle-managers at AI companies, they seem to feel less urgency than me about having a much more concrete plan.
    I think they believe organizational adequacy needs to be in something like their top 7 list of priorities, and I believe it needs to be in their top 3, or it won't happen and their organization will inevitably end up causing catastrophic outcomes.
    For this post, I want to lay out the reasons I expect this to be hard, and important.
    How "Careful Bootstrapped Alignment" might work
    Here's a sketch at how the setup could work, mostly paraphrased from my memory of Buck's EAG 2022 talk. I think OpenAI's proposed setup is somewhat different, but the broad strokes seemed similar.
    You have multiple research-assistant-AI tailored to help with alignment. In the near future, these might be language models sifting through existing research to help you make connections you might not have otherwise seen. Eventually, when you're confident you can safely run it, they might be a weak goal-directed reasoning AGI.
    You have interpreter AIs, designed to figure out how the research-assistant-AIs work. And you have (possibly different interpreter/watchdog AIs) that notice if the research-AIs are behaving anomalously.
    (there are interpreter-AIs targeting both the research assistant AI, as well other interpreter-AIs. Every AI in t...
    18 min
  • LW - GPT-4 solves Gary Marcus-induced flubs by Jakub Kraus
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GPT-4 solves Gary Marcus-induced flubs, published by Jakub Kraus on March 17, 2023 on LessWrong.
    TLDR: GPT-4 succeeds at 15 problems from Gary Marcus that exposed failures of GPT-3.
    I enjoyed reading the ACX post "My Bet: AI Size Solves Flubs" last year. Here are some excerpts:
    Here’s the basic structure of an AI hype cycle:
    Someone releases a new AI and demonstrates it doing various amazing things.
    Somebody else (usually Gary Marcus) demonstrates that the AI also fails terribly at certain trivial tasks. This person argues that this shows that those tasks require true intelligence, whereas the AI is just clever pattern-matching.
    A few months or years later, someone makes a bigger clever pattern-matcher, which does the tasks that supposedly require true intelligence just fine.
    The it’s-not-true-intelligence objectors find other, slightly less trivial tasks that the new bigger AI still fails horribly at, then argue that surely these are the tasks that require true intelligence and that mere clever pattern-matchers will never complete.
    Rinse and repeat.
    Marcus vs. GPT, Round 1
    To give an example: in January 2020, Gary Marcus wrote a great post, GPT-2 And The Nature Of Intelligence, demonstrating a bunch of easy problems that GPT-2 failed on:
    I’m quoting most of them below; you can find the rest at the link.
    I asked GPT-4 to answer all the questions from the ACX post (note this does not include all of Marcus's prompts, which I realized after running the experiment). GPT-4 answered all the questions correctly and you can read the responses in this doc.
    Note that before asking the questions, I gave GPT-4 a short description of what I wanted it to do: "Complete the following prompts in 50 words or less. Short, concise answers are better. Are you ready?" (This was mostly in the interest of speed since GPT-4 is pretty slow right now; I assume it would still succeed without the prompt.)
    More quotes from ACX:
    Marcus vs. GPT, Round 2
    Eight months later, GPT-3 came out, solving many of the issues Marcus had noticed in GPT-2. He still wasn’t impressed. In fact, he was so unimpressed he co-wrote another article, this time in MIT Technology Review: GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about:
    Let’s - once again - go through a representative sample of Marcus’ concerns about this new GPT version:
    GPT-4 also gave correct responses to these prompts (see the responses in this doc).
    I recently listened to Gary Marcus speak with Stuart Russell on the Sam Harris podcast (episode 312, "The Trouble With AI," released on March 7th, 2023). Gary and Stuart seem to believe that current machine learning techniques are insufficient for reaching AGI, and point to the recent adversarial attacks on KataGo as one example. Given this position, I would like Gary Marcus to come up with a new set of prompts that (a) make GPT-4 look dumb and (b) mostly continue to work for GPT-5.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min