The Nonlinear Library

The Nonlinear Library

By The Nonlinear FundEducation
Download on the App Store

All The Nonlinear Library episodes

  • LW - How seriously should we take the hypothesis that LW is just wrong on how AI will impact the 21st century? by Noosphere89
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How seriously should we take the hypothesis that LW is just wrong on how AI will impact the 21st century?, published by Noosphere89 on February 16, 2023 on LessWrong.
    This question is inspired by 1a3orn's comment on how there are troubling signs of epistemic issues in LW's Alignment field.
    I'll quote the comment here to tell you what I mean:
    I think that the above is also a good explanation for why many ML engineers working on AI or AGI don't see any particular reason to engage with or address arguments about high p(doom).
    When from a distance one views a field that:
    Has longstanding disagreements about basic matters
    Has theories -- but many of the theories have not resulted in really any concrete predictions that differentiate from standard expectations, despite efforts to do so.
    Will continue to exist regardless of how well you criticize any one part of it.
    There's basically little reason to engage with it. These are all also evidence that there's something epistemically off with what is going on in the field.
    Maybe this evidence is wrong! But I do think that it is evidence, and not-weak evidence, and it's very reasonable for a ML engineer to not deeply engage with arguments because of it.
    So I want to ask a question: How seriously should we take the hypothesis that LW is totally wrong on AI?
    Specifically, this splits into several subquestions:
    What's the chance that AI doesn't have that much of an impact on the world by 2100?
    What's the chance that we do have massive impacts, but alignment is so easy that standard ML techniques work?
    How well does the epistemic process on LW work? Are there any changes you would make to LW's epistemic processes?
    I welcome all answers, and I especially welcome any critics of LW/negative answers to at least answer one of the questions I have.
    Edit: For people that don't have a specific scenario in mind, I'll ask a specific question. It doesn't have to be answered, but any answers on this question are appreciated, especially from critics of the "AI is significant" idea.
    1a. What probability will the Explosion or Collapse scenario from Cold Takes happen by 2100?
    Link to the scenarios below:
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic) by Lawrence Chan
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic), published by Lawrence Chan on February 16, 2023 on The AI Alignment Forum.
    This is a followup to what I cheekily call Anthropic's "just try to get the large model to do what you want" research agenda. (Previously: A General Language Assistant as a Laboratory for Alignment, Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Language Models (Mostly) Know What They Know)
    The most interesting takeaway for me is that this is the first paper where Anthropic benchmarks their 175B parameter language model (probably a Claude variant). Previous papers only benchmarked up to 52B parameters. However, we don't have the performance of this model on standard benchmarks (the only benchmarked model from Anthropic is a 52B parameter one called standford-online-all-v4-s3). They also don't give details about its architecture or pretraining procedure.
    In this paper (Ganguli and Askell et al.), the authors study what happens when you just ... ask the language model to be less biased (that is, change their answers based on protected classes such as age or gender). They consider several setups: asking questions directly (Q), adding in the instruction to not be biased (Q+IF), giving it the instruction + chain of thought (Q+IF+CoT), and in some cases, asking it to match particular statistics.
    They find that as you scale the parameter count of their RLHF'ed language models, the models become more biased, but they also become increasingly capable of correcting for their biases:
    They also report how their model changes as you take more RLHF steps:
    First, this suggests that RLHF is having some effect on instruction following: the gap between the Q and Q+IF setups increases as you scale the number of RLHF steps, for both BBQ and admissions discrimination. (I'm not sure what's happening for the gender bias one?) However, simply giving the language model instructions and prompting it to do CoT, even after 50 RLHF steps, seems to have a significantly larger effect than RLHF.
    I was also surprised at how few RLHF steps are needed to get instruction following -- the authors only consider 50-1000 steps of RLHF, and see instruction following even after 50 RLHF steps. I wonder if this is a property of their pretraining process, a general fact about pretrained models (PaLM shows significant 0-shot instruction following capabilities, for example), or if RLHF is just that efficient?
    The authors caution that they've done some amount of prompt engineering, and "have not systematically tested for this in any of our experiments."
    They use the same RLHF procedure as in Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • AF - EIS V: Blind Spots In AI Safety Interpretability Research by Stephen Casper
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS V: Blind Spots In AI Safety Interpretability Research, published by Stephen Casper on February 16, 2023 on The AI Alignment Forum.
    Part 5 of 12 in the Engineer’s Interpretability Sequence.
    Thanks to Anson Ho, Chris Olah, Neel Nanda, and Tony Wang for some discussions and comments.
    TAISIC = “the AI safety interpretability community”
    MI = “mechanistic interpretability”
    Most AI safety interpretability work is conducted by researchers in a relatively small number of places, and TAISIC is closely-connected by personal relationships and the AI alignment forum. Much of the community is focused on a few specific approaches like circuits-style MI, mechanistic anomaly detection, causal scrubbing, and probing. But this is a limited set of topics, and TAISIC might benefit from broader engagement. In the Toward Transparent AI survey (Räuker et al., 2022), we wrote 21 subsections of survey content. Only 1 was on circuits, and only 4 consisted in significant part of works from TAISIC.
    I have often heard people in TAISIC explicitly advising more junior researchers to not focus much on reading from the literature and instead to dive into projects. Obviously, experience working on projects is irreplaceable. But understanding the broader literature and community is a recipe for developing insularity and blind spots. I am quick to push back against advice that doesn’t emphasize the importance of engaging with outside work.
    Within TAISIC, I have heard interpretability research described as dividing into two sets: mechanistic interpretability and, somewhat pejoratively, “traditional interpretability.” I will be the first to say that some paradigms in interpretability research are unproductive (see EIS III-IV). But I give equal emphasis to the importance of TAISIC not being too parochial. Reasons include maintaining relevance and relationships in the broader community, drawing useful inspiration from past works, making less-correlated bets with what we focus on, and most importantly – not reinventing, renaming, and repeating work that has already been done outside of TAISIC.
    TAISIC has reinvented, reframed, or renamed several paradigms
    Mechanistic interpretability requires program synthesis, program induction, and/or programming language translation
    “Circuits”-style MI is arguably the most popular and influential approach to interpretability in TAISIC. Doing this work requires iteratively (1) generating hypotheses for what a network is doing and then (2) testing how valid these hypotheses explain its internal mechanisms. Step 2 may not be that difficult, and causal scrubbing (discussed below) seems like a type of solution that will be useful for it. But step 1 is hard. Mechanistic hypothesis generation is a lot like doing program synthesis, program induction, and/or programming language translation.
    Generating mechanistic hypotheses requires synthesizing programs to explain a network using its behavior and/or structure. If a method for this involves synthesizing programs based on the task or I/O from the network, it is a form of program synthesis or induction. And if a method is based on using a network’s structure to write down a program to explain it, it is very similar to programming language translation.
    In general, program synthesis and program induction are very difficult and currently fail to scale to large problems. This is well-understood, and these fields are mature enough so that we have textbooks on them and how difficult they are (e.g. Gulwani et al., 2017). Meanwhile, programming language translation is very challenging too. In practice, translating between common languages (e.g. Python and Java) is only partially automatable and relies on many hand-coded rules (Qiu, 1999), and using large language models has had very limited successes (Roziere et al.). And in cases like ...
    19 min
  • EA - Transitioning to an advisory role by MaxDalton
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transitioning to an advisory role, published by MaxDalton on February 16, 2023 on The Effective Altruism Forum.
    I’m writing to announce that I’ve resigned from my role as CEA’s Executive Director, and will be transitioning to an advisory role.
    Basically, my mental health has been bad for the last 3 months. Starting in November, my role changed from one that I love - building a team and a product, building close working relationships with people, executing - to one that I find really stressful: dealing with media attention, stakeholders, and lawyers at long unpredictable hours and wrestling with strategic uncertainty. I think I’m also not so good at the latter sort of work, relative to the former.
    I've been getting lots of advice, therapy, and support, but recently I've been close to a crisis – struggling to get out of bed, feeling terror at the idea of sitting at my desk. I really wish that I were strong enough to keep doing this job, especially right now – I care so much about CEA’s work to help more people tackle the important problems we face, and I care deeply about the team we’ve built.
    But I’m just not able to keep going in my current role, and I don't think that pretending to be stronger or struggling on will be good for CEA or for me, because I’m not able to perform as well as I would like and there’s a risk that I’ll burn out with no handover. So I think it’s best to move into an advisory role and allow someone else to direct CEA.
    The boards of Effective Ventures UK and Effective Ventures US, which govern CEA, will appoint an interim Executive Director soon. Once they’re appointed I plan to continue advising and working with them and the CEA team to ensure a smooth transition and help find a new permanent ED. I hope that moving from an executive to advisory role will help alleviate some of the pressure and allow me to contribute more productively to our shared work going forward.
    For a while now I've been trying to build up the leadership team as the body running CEA, with me as one member. I think that the leadership team is very strong: people disagree with each other directly but with care, have complementary strengths, and show strong leadership for their own programs. I think that they will be able to do a great job leading CEA together with the interim ED and the new permanent ED.
    Of course, FTX and subsequent events have highlighted some important issues in EA. I’ve been working with the team to reflect on how this might impact our work and necessitate changes, and I hope that they’ll be able to share more on these conversations and plans in the future. Although I’m very sad not to be able to see through that work in my current role with CEA, I think that the work we’ve done so far will set the new leadership team up well. I also plan to continue to reflect, will discuss my thinking with new leadership, and may publish some of my personal reflections.
    Despite the setbacks of these last few months, I'm very proud of what we've achieved together over the last four years. Compared to 2019, the number of new connections we’re making at events is 5x higher, and people are spending 10x time engaging with the Forum (which also has a lot more interesting content). Overall, I think that we’ve helped hundreds of people to reflect on how they can best contribute to making the world a better place, and begin to work on these critical problems.
    I’m also incredibly grateful to have been a part of this team: CEA staff are incredibly talented, caring, and dedicated. I’ve loved to be a part of a culture where staff are valued and empowered to do things.
    I look forward to seeing the impact which they continue to have over the coming months and years under new leadership.
    This has been true for many people, especially EV board members and some staff who have jumped in t...
    4 min
  • EA - Qualities that alignment mentors value in junior researchers by Akash
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Qualities that alignment mentors value in junior researchers, published by Akash on February 14, 2023 on The Effective Altruism Forum.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    1 min
  • EA - Please don't throw your mind away by TsviBT
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Please don't throw your mind away, published by TsviBT on February 15, 2023 on The Effective Altruism Forum.
    Dialogue
    [Warning: the following dialogue contains an incidental spoiler for "Music in Human Evolution" by Kevin Simler. That post is short, good, and worth reading without spoilers, and this post will still be here if you come back later. It's also possible to get the point of this post by skipping the dialogue and reading the other sections.]
    Pretty often, talking to someone who's arriving to the existential risk / AGI risk / longtermism cluster, I'll have a conversation like the following.
    Tsvi: "So, what's been catching your eye about this stuff?"
    Arrival: "I think I want to work on machine learning, and see if I can contribute to alignment that way."
    T: "What's something that got your interest in ML?"
    A: "It seems like people think that deep learning might be on the final ramp up to AGI, so I should probably know how that stuff works, and I think I have a good chance of learning ML at least well enough to maybe contribute to a research project."
    T: "That makes sense. I guess I'm fairly skeptical of AGI coming very soon, compared to people around here, or at least I'm skeptical that most people have good reasons for believing that. Also I think it's pretty valuable to not cut yourself off from thinking about the whole alignment problem, whether or not you expect to work on an already-existing project. But what you're saying makes sense too. I'm curious though if there's something you were thinking about recently that just strikes you as fun, or like it's in the back of your mind a bit, even if you're not trying to think about it for some purpose."
    A: "Hm... Oh, I saw this video of an octopus doing a really weird swirly thing. Here, let me pull it up on my phone."
    T: "Weird! Maybe it's cleaning itself, like a cat licking its fur? But it doesn't look like it's actually contacting itself that much."
    A: "I thought it might be a signaling display, like a mating dance, or for scaring off predators by looking like a big coordinated army. Like how humans might have scared off predators and scavenging competitors in the ancestral environment by singing and dancing in unison."
    T: "A plausible hypothesis. Though it wouldn't be getting the benefit of being big, like a spread out group of humans."
    A: "Yeah. Anyway yeah I'm really into animal behavior. Haven't been thinking about that stuff recently though because I've been trying to start learning ML."
    T: "Ah, hm, uh... I'm probably maybe imagining things, but something about that is a bit worrying to me. It could make sense, consequentialist backchaining can be good, and diving in deep can be good, and while a lot of that research doesn't seem to me like a very hopeworthy approach, some well-informed people do. And I'm not saying not to do that stuff. But there's something that worries me about having your little curiosities squashed by the backchained goals. Like, I think there's something really good about just doing what's actually interesting to you, and I think it would be bad if you were to avoid putting a lot of energy into stuff that's caught your attention in a deep way, because that would tend to sacrifice a lot of important stuff that happens when you're exploring something out of a natural urge to investigate."
    A: "That took a bit of a turn. I'm not sure I know what you mean. You're saying I should just follow my passion, and not try to work towards some specific goal?"
    T: "No, that's not it. More like, when I see someone coming to this social cluster concerned with existential risk and so on, I worry that they're going to get their mind eaten. Or, I worry that they'll think they're being told to throw their mind away. I'm trying to say, don't throw your mind away."
    A: "I... don't think I'm being told to t...
    28 min
  • EA - Anyone who likes the idea of EA & meeting EAs but doesn't want to discuss EA concepts IRL? by antisocial-throwaway
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Anyone who likes the idea of EA & meeting EAs but doesn't want to discuss EA concepts IRL?, published by antisocial-throwaway on February 15, 2023 on The Effective Altruism Forum.
    Hi all, using a throwaway as this feels a bit anti-social to post!
    Basically, I'm aligned with & live via the basic concepts of EA (use your career for good + donate some of your earnings to effective causes), but that's where my interest really kind of ends.
    I'm not interested in using my free time to read into EA further, I don't feel motivated to learn more about all the concepts that people use/ discuss, etc (utilitarianism, expected value, etc etc). I really like having non-EA friends and don't get any enjoyment from having philosophical discussions with people.
    I really like the core ethos of EA, but when I've gone to in-person meetups (including Prague Fall Season) I've felt like a fraud because I'm not at all versed in the language, and actually have no interest in discussion all the forum talking points. I just want to meet cool people who care about doing good!
    Of course a clear rebuttal here would be "ok then dude just talk to people about other stuff", but I've often felt at these events like people are there to discuss this kind of stuff, and to talk about more normal/ "mundane" stuff would make people think I'm wasting their time.
    So I guess my question is like - is there anyone else out there who feels this way? Any tips? I'd really like to make friends through the EA community but in the same breath I only want to be involved in a straightforward way (career + key ideas), rather than scouring the forums & LessWrong and hitting all the squares on the EA bingo card. Also, is it kind of in my head that you need to be a hardcore EA who has strong opinions on ethics & philosophy & etc, or is that the general mood?
    (I also appreciate that the people who end up reading this will be more "hardcore EA" types who check the forum regularly...)
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    3 min
  • EA - Nobody Wants to Read Your Sht: my favorite book of writing advice, condensed for your convenience by jvb
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nobody Wants to Read Your Sht: my favorite book of writing advice, condensed for your convenience, published by jvb on February 15, 2023 on The Effective Altruism Forum.
    Nobody Wants to Read Your Sht by Steven Pressfield is my favorite book of writing advice. Its core insight is expressed in the title. The best thing you can do for your writing is to internalize this deep truth.
    Pressfield did it by writing ad copy. You can’t avoid internalizing that nobody wants to read your shit when you’re writing ads, which everybody hates and nobody ever wants to read. Maybe you don’t have to go write ad copy to understand this; maybe you can just read the book, or just this post.
    When you understand that nobody wants to read your shit, your mind becomes powerfully concentrated. You begin to understand that writing/reading is, above all, a transaction. The reader donates his time and attention, which are supremely valuable commodities. In return, you the writer must give him something worthy of his gift to you.
    When you understand that nobody wants to read your shit, you develop empathy. [...] You learn to ask yourself with every sentence and every phrase: Is this interesting? Is it fun or challenging or inventive? Am I giving the reader enough? Is she bored? Is she following where I want to lead her?
    What should you do about the fact that nobody wants to read your shit?
    Streamline your message. Be as clear, simple, and easy to understand as you possibly can.
    Make it fun. Or sexy or interesting or scary or informative. Fun writing saves lives.
    Apply this insight to all forms of communication.
    Pressfield wrote this book primarily for fiction writers, who are at the most serious risk of forgetting that nobody wants to read their shit (source: am fiction writer). But the art of empathy applies to all communication, and so do many other elements of fiction:
    Nonfiction is fiction. If you want your factual history or memoir, your grant proposal or dissertation or TED talk to be powerful and engaging and to hold the reader and audience's attention, you must organize your material as if it were a story and as if it were fiction. [...]
    What are the universal structural elements of all stories? Hook. Build. Payoff. This is the shape any story must take. A beginning that grabs the listener. A middle that escalates in tension, suspense, and excitement. And an ending that brings it all home with a bang. That's a novel, that's a play, that's a movie. That's a joke, that's a seduction, that's a military campaign. It's also your TED talk, your sales pitch, your Master's thesis, and the 890-page true saga of your great-great-grandmother's life.
    And your whitepaper, and your grant proposal, and your EA forum post. For this reason, I do recommend going out and grabbing this book, even though much of it concerns fiction. It only takes about an hour to read, because Pressfield knows we don’t want to read his shit. Finally:
    All clients have one thing in common. They're in love with their product/company/service. In the ad biz, this is called Client's Disease. [...]
    What the ad person understands that the client does not is that nobody gives a damn about the client or his product. [...]
    The pros understand that nobody wants to read their shit. They will start from that premise and employ all their arts and all their skills to come up with some brilliant stroke that will cut through that indifference.
    The relevance of this quote to EA writing is left as an exercise to the reader.
    Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
    4 min
  • LW - Hashing out long-standing disagreements seems low-value to me by So8res
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hashing out long-standing disagreements seems low-value to me, published by So8res on February 16, 2023 on LessWrong.
    (Status: a short write-up of some things that I find myself regularly saying in person. In this case, I'm writing up my response to the question of why I don't spend a bunch more time trying to resolve disagreements with people in the community who disagree with me about the hopefulness of whatever research direction. I’m not particularly happy with it, but it’s been languishing in my draft folder for many months now and published is better than perfect.)
    When I first joined the AI alignment community almost ten years ago, there were lots of disagreements—between groups like MIRI and Open Phil, between folks like Eliezer Yudkowsky and Paul Christiano, etc. At that time, I was optimistic about resolving a bunch of those disagreements. I invested quite a few hours in this project, over the years.
    I didn't keep track exactly, but extremely roughly, I think the people with very non-MIRI-ish perspectives I spent the most time trying to converge with (including via conversation, reading and writing blog posts, etc.) were:
    Paul Christiano (previously at OpenAI, now at ARC): 150 hours? (Maybe as low as 100 or as high as 300.)
    Daniel Dewey (then at Open Phil): 40 hours? (Possibly 100+.)
    Nick Beckstead (then at Open Phil): 30 hours?
    Holden Karnofsky (Open Phil): 20 hours?
    Tom Davidson (Open Phil): 15 hours?
    Another non-MIRI person I’ve spent at least a few hours trying to sync with about AI is Rohin Shah at DeepMind.
    (Note that these are all low-confidence ass numbers. I have trouble estimating time expenditures when they’re spread across days in chunks that are spread across years, and when those chunks blur together in hindsight. Corrections are welcome.)
    I continue to have some conversations like this, but my current model is that attempting to resolve older and more entrenched disagreements is not worth the time-cost.
    It's not that progress is impossible. It's that we have a decent amount of evidence of what sorts of time-investment yield what amounts of progress, and it just isn't worth the time.
    On my view, Paul is one of the field’s most impressive researchers. Also, he has spent lots of time talking and working with MIRI researchers, and trying to understand our views.
    If even Paul and I can’t converge that much over hundreds of hours, then I feel pretty pessimistic about the effects of a marginal hour spent trying to converge with other field leaders who have far less context on what MIRI-ish researchers think and why we think it. People do regularly tell me that I’ve convinced them of some central AI claim or other, but it’s rarely someone whose views are as distant from mine as Paul’s are, and I don’t recall any instance of it happening on purpose (as opposed to somebody cool who I didn’t have in mind randomly approaching me later to say “I found your blog post compelling”).
    And I imagine the situation is pretty symmetric, at this level of abstraction. Since I think I’m right and I think Paul’s wrong, and we’ve both thought hard about these questions, I assume Paul is making some sort of mistake somewhere. But such things can be hard to spot. From his perspective, he should probably view me as weirdly entrenched in my views, and therefore not that productive to talk with. I suspect that he should at least strongly consider this hypothesis, and proportionally downgrade his sense of how useful it is to spend an hour trying to talk some sense into me!
    As long as your research direction isn't burning the commons, I recommend just pursuing whatever line of research you think is fruitful, without trying to resolve disagreements with others in the field.
    Note that I endorse writing up what you believe! Articulating your beliefs is an important tool for refi...
    7 min
  • LW - Speedrunning 4 mistakes you make when your alignment strategy is based on formal proof by Quinn
    Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Speedrunning 4 mistakes you make when your alignment strategy is based on formal proof, published by Quinn on February 16, 2023 on LessWrong.
    Lots of people think formal (i.e. computer verified) proofs are a part of viable alignment strategies. This trick never works. If you think it might work, I can save your time in four different ways.
    I previously wrote on this, but people are still making these mistakes and this version is clearer and shorter. Recall the deep wisdom that nobody wants to read your shit.
    Epistemic status: four years deep into proof assistants. Once a customer paid me a lot of money to let formal verification rip. I tried and failed to solve alignment 4 or 5 times since 2021, each attempt lasting at least a couple months.
    Formal verification is a software strategy based on knowing a specification (spec) a priori, then showing that your implementation is perfect (correct). The evergreen comment is: why not just capture important alignment properties in these “specs” and endeavor to implement the system “correctly”. It made me optimistic, and if you’re not careful, it’ll make you optimistic.
    Mistake 1: The gap between the world and the spec hurts you more than the gap between the spec and the implementation
    Formal verification lures you into a false sense of security. Your spec is the map, not the territory. It is not valuable to implement software to the map correctly. Reminiscent of bog standard hardware security wisdom.
    Mistake 2: Proofs are Bayesian evidence
    A proof you don’t understand does not obligate you to believe anything; it is Bayesian evidence like anything else. If an alien sends a 1GB Coq file Riemann.v, running it on your computer does not obligate you to believe that the Riemann hypothesis is true. If you're ever in that situation, do not let anyone tell you that Coq is so awesome that you don't roll to disbelieve. 1GB of plaintext is too much, you'll get exhausted before you understand anything. Do not ask the LLM to summarize the proof.
    Mistake 3: The proof is not the product
    Proofs, users; pick one. A shop that cares about proofs attracts staff that cares about proofs and turns off staff that cares about users. The only extinction-level threat models I entertain are those that are deployed by shops that make something people want. If you want an intuition pump go and look at the price of Ethereum (move fast break things) vs the price of Cardano (formal verification)[1]. Go ahead and write a proof that you're right about alignment, it literally won't affect the real world.
    Corollary: Proof obligations are slow and expensive
    You solved the social problem of hiring both folks excited about proofs and folks excited about users, and they get along. Your proof team is too slow for your product team, everyone starts resenting each other, you face investor pressure to fire the proof engineers. Maybe the FDA-But-For-Software makes it illegal to fire the proof engineers. That buys you real assurance in some threat models.
    But in others, humanity has to be lucky every time. Paperclips only have to be lucky once. It doesn’t matter if the FDA-But-For-Software would have punished you if literally everyone is dead.
    Mistake 4: What you care about and what the spec allows you to express are not friends
    You can’t actually specify what you care about (in a competitive specification language that has a viable tooling ecosystem attached to it).
    I spent almost a week thinking about circuits and polytopes. I thought that if we could verify across trustless channels that semantic units had been learned, we'd be setting a precedent for verifying things that matter. For example, things like non-deception (in the Hubinger sense). It's bad enough to pin down alignment properties to the standards of mathematicians, doing it to the standard of computers is wo...
    5 min