Pigeon Hour

Table of contents

Note: links take you to the corresponding section below; links to the original episode can be found there.

* Laura Duffy solves housing, ethics, and more [00:01:16]

* Arjun Panickssery solves books, hobbies, and blogging, but fails to solve the Sleeping Beauty problem because he's wrong on that one [00:10:47]

* Nathan Barnard on how financial regulation can inform AI regulation [00:17:16]

* Winston Oswald-Drummond on the tractability of reducing s-risk, ethics, and more [00:27:48]

* Nathan Barnard (again!) on why general intelligence is basically fake [00:34:10]

* Daniel Filan on why I'm wrong about ethics (+ Oppenheimer and what names mean in like a hardcore phil of language sense) [00:56:54]

* Holly Elmore on AI pause, wild animal welfare, and some cool biology things I couldn't fully follow but maybe you can [01:04:00]

* Max Alexander and I solve ethics, philosophy of mind, and cancel culture once and for all [01:24:43]

* Sarah Woodhouse on discovering AI x-risk, Twitter, and more [01:30:56]

* Pigeon Hour x Consistently Candid pod-crossover: I debate moral realism with Max Alexander and Sarah Hastings-Woodhouse [01:41:08]

Intro [00:00:00]

To wrap up the year of Pigeon Hour, the podcast, I put together some clips from each episode to create a best-of compilation. This was inspired by 80,000 Hours, a podcast that did the same with their episodes, and I thought it was pretty cool and tractable enough.

It's important to note that the clips I chose range in length significantly. This does not represent the quality or amount of interesting content in the episode. Sometimes there was a natural place to break the episode into a five-minute chunk, and other times it wouldn't have made sense to take a five-minute chunk out of what really needed to be a 20-minute segment. I promise I'm not just saying that.

So without further ado, please enjoy.

#1: Laura Duffy solves housing, ethics, and more [00:01:16]

In this first segment, Laura, Duffy, and I discuss the significance and interpretation of Aristotle's philosophical works in relation to modern ethics and virtue theory.

AARON: Econ is like more interesting. I don't know. I don't even remember of all the things. I don't know, it seems like kind of cool. Philosophy. Probably would have majored in philosophy if signaling wasn't an issue. Actually, maybe I'm not sure if that's true. Okay. I didn't want to do the old stuff though, so I'm actually not sure. But if I could aristotle it's all wrong. Didn't you say you got a lot out of Nicomachi or however you pronounce that?

LAURA: Nicomachian ethics guide to how you should live your life. About ethics as applied to your life because you can't be perfect. Utilitarians. There's no way to be that.

AARON: But he wasn't even responding to utilitarianism. I'm sure it was a good work given the time, but like, there's like no other discipline in which we care. So people care so much about like, what people thought 2000 years ago because like the presumption, I think the justified presumption is that things have iterated and improved since then. And I think that's true. It's like not just a presumption.

LAURA: Humans are still rather the same and what our needs are for living amongst each other in political society are kind of the same. I think America's founding is very influenced by what people thought 2000 years ago.

AARON: Yeah, descriptively that's probably true. But I don't know, it seems like all the whole body of philosophers have they've already done the work of, like, compressing the good stuff. Like the entire academy since like, 1400 or whatever has like, compressed the good stuff and like, gotten rid of the bad stuff. Not in like a high fidelity way, but like a better than chance way. And so the stuff that remains if you just take the state of I don't know if you read the Oxford Handbook of whatever it is, like ethics or something, the takeaways you're going to get from that are just better than the takeaways you're going to get from a summary of the state of the knowledge in any prior year. At least. Unless something weird happened. And I don't know. I don't know if that makes sense.

LAURA: I think we're talking about two different things, though. Okay. In terms of knowledge about logic or something or, I don't know, argumentation about trying to derive the correct moral theory or something, versus how should we think about our own lives. I don't see any reason as to why the framework of virtue theory is incorrect and just because it's old. There's many virtue theorists now who are like, oh yeah, they were really on to something and we need to adapt it for the times in which we live and the kind of societies we live in now. But it's still like there was a huge kernel of truth in at least the way of thinking that Aristotle put forth in terms of balancing the different virtues that you care about and trying to find. I think this is true. Right? Like take one virtue of his humor. You don't want to be on one extreme where you're just basically a meme your entire life. Everybody thinks you're funny, but that's just not very serious. But you don't want to be a boar and so you want to find somewhere in the middle where it's like you have a good sense of humor, but you can still function and be respected by other people.

AARON: Yeah. Once again, I agree. Well, I don't agree with everything. I agree with a lot of what you just said. I think there was like two main points of either confusion or disagreement. And like, the first one is that I definitely think, no, Aristotle shouldn't be discounted or like his ideas or virtue ethics or anything like that shouldn't be discounted because they were canonical texts or something were written a long time ago. I guess it's just like a presumption that I have a pretty strong presumption that conditional on them being good, they would also be written about today. And so you don't actually need to go back to the founding texts and then in fact, you probably shouldn't because the good stuff will be explained better and not in weird it looks like weird terms. The terms are used differently and they're like translations from Aramaic or whatever. Probably not Aramaic, probably something else. And yeah, I'm not sure if you.

LAURA: Agree with this because we have certain assumptions about what words like purpose mean now that we're probably a bit richer in the old conception of them like telos or happiness. Right. Udaimnia is much better concept and to read the original text and see how those different concepts work together is actually quite enriching compared to how do people use these words now. And it would take like I don't know, I think there just is a lot of value of looking at how these were originally conceived because popularizers of the works now or people who are seriously doing philosophy using these concepts. You just don't have the background knowledge that's necessary to understand them fully if you don't read the canonical text.

AARON: Yeah, I think that would be true. If you are a native speaker. Do you know Greek? If you know Greek, this is like dumb because then you're just right.

LAURA: I did take a quarter of it.

AARON: Oh God. Oh my God. I don't know if that counts, but that's like more than anybody should ever take. No, I'm just kidding. That's very cool. No, because I was going to say if you're a native speaker of Greek and you have the connotations of the word eudaimonia and you were like living in the temper shuttle, I would say. Yeah, that's true actually. That's a lot of nuanced, connotation and context that definitely gets lost with translation. But once you take the jump of reading English translations of the texts, not you may as well but there's nothing super special. You're not getting any privileged knowledge from saying the word eudaimonia as opposed to just saying some other term as a reference to that concept or something. You're absorbing the connotation in the context via English, I guess, via the mind of literally the translators who have like.

LAURA: Yeah, well see, I tried to learn virtue theory by any other route than reading Aristotle.

AARON: Oh God.

LAURA: I took a course specifically on Plato and Aristotle.

AARON: Sorry, I'm not laughing at you. I'm just like the opposite type of philosophy person.

LAURA: But keep going. Fair. But she had us read his physics before we read Nicomachi.

AARON: Think he was wrong about all that.

LAURA: Stuff, but it made you understand what he meant by his teleology theory so much better in a way that I could not get if I was reading some modern thing.

AARON: I don't know, I feel like you probably could. No, sorry, that's not true. I don't think you could get what Aristotle the man truly believed as well via a modern text. But is that what you? Depends. If you're trying to be a scholar of Aristotle, maybe that's important. If you're trying to find the best or truest ethics and learn the lessons of how to live, that's like a different type of task. I don't think Aristotle the man should be all that privileged in that.

LAURA: If all of the modern people who are talking about virtue theory are basically Aristotle, then I don't see the difference.

AARON: Oh, yeah, I guess. Fair enough. And then I would say, like, oh, well, they should probably start. Is that in fact the state of the things in virtue theory? I don't even know.

LAURA: I don't know either.

#2 Arjun Panickssery solves books, hobbies, and blogging, but fails to solve the Sleeping Beauty problem because he's wrong on that one [00:10:47]

All right, next, Arjun Panixery and I explore the effectiveness of reading books in retaining and incorporating knowledge, discussing the value of long form content and the impact of great literary works on understanding and shaping personal worldviews.

ARJUN: Oh, you were in the book chat, though. The book rant group chat, right?

AARON: Yeah, I think I might have just not read any of it. So do you want to fill me in on what I should have read?

ARJUN: Yeah, it's group chat of a bunch of people where we were arguing about a bunch of claims related to books. One of them is that most people don't remember pretty much anything from books that they read, right? They read a book and then, like, a few months later, if you ask them about it, they'll just say one page's worth of information or maybe like, a few paragraphs. The other is that what is it exactly? It's that if you read a lot of books, it could be that you just incorporate the information that's important into your existing models and then just forget the information. So it's actually fine. Isn't this what you wrote in your blog post or whatever? I think that's why I added you to that.

AARON: Oh, thank you. I'm sorry I'm such a bad group chat participant. Yeah, honestly, I wrote that a while ago. I don't fully remember exactly what it says, but at least one of the things that it said was and that I still basically stand by, is that it's basically just like it's increasing the salience of a set of ideas more so than just filling your brain with more facts. And I think this is probably true insofar as the facts support a set of common themes or ideas that are kind of like the intellectual core of it. It would be really hard. Okay, so this is not a book, but okay. I've talked about how much I love an 80,000 hours podcast, and I've listened to, I don't think every episode, but at least 100 of the episodes. And no, you're just, like, not going to definitely I've forgotten most of the actual almost all of the actual propositional pieces of information said, but you're just not going to convince me that it's completely not affecting either model of the world or stuff that I know or whatever. I mean, there are facts that I could list. I think maybe I should try.

ARJUN: Sure.

AARON: Yeah. So what's your take on book other long form?

ARJUN: Oh, I don't know. I'm still quite confused or I think the impetus for the group chat's creation was actually Hanania's post where he wrote the case against most books or most was in parentheses or something. I mean, there's a lot of things going on in that post. He just goes off against a bunch of different categories of books that are sort of not closely related. Like, he goes off against great. I mean, this is not the exact take he gives, but it's something like the books that are considered great are considered great literature for some sort of contingent reason, not because they're the best at getting you information that you want.

AARON: This is, like, another topic. But I'm, like, anti great books. In fact, I'm anti great usually just means old and famous. So insofar as that's what we mean by I'm like, I think this is a bad thing, or, like, I don't know, aristotle is basically wrong about everything and stuff like that.

ARJUN: Right, yeah. Wait, we could return to this. I guess this could also be divided into its component categories. He spends more time, though, I think, attacking a certain kind of nonfiction book that he describes as the kind of book that somebody pitches to a publisher and basically expands a single essay's worth of content into with a bunch of anecdotes and stuff. He's like, most of these books are just not very useful to read, I guess. I agree with that.

AARON: Yeah. Is there one that comes to mind as, like, an? Mean, I think of Malcolm Gladwell as, like, the kind of I haven't actually read any of his stuff in a while, but I did, I think, when I started reading nonfiction or with any sort of intent, I read. A bunch of his stuff or whatever and vaguely remember that this is basically what he like for better or.

ARJUN: Um yeah, I guess so. But he's almost, like, trying to do it on purpose. This is the experience that you're getting by reading a Malcolm Gladwell book. It's like talib. Right? It's just him just ranting. I'm thinking, I guess, of books that are about something. So, like, if you have a book that's know negotiation or something, it'll be filled with a bunch of anecdotes that are of dubious usefulness. Or if you get a book that's just about some sort of topic, there'll be historical trivia that's irrelevant. Maybe I can think of an example.

AARON: Yeah. So the last thing I tried to read, maybe I am but haven't in a couple of weeks or whatever, is like, the Derek Parfit biography. And part of this is motivated because I don't even like biographies in general for some reason, I don't know. But I don't know. He's, like, an important guy. Some of the anecdotes that I heard were shockingly close to home for me, or not close to home, but close to my brain or something. So I was like, okay, maybe I'll see if this guy's like the smarter version of Aaron Bergman. And it's not totally true.

ARJUN: Sure, I haven't read the book, but I saw tweet threads about it, as one does, and I saw things that are obviously false. Right. It's the claims that he read, like, a certain number of pages while brushing his teeth. That's, like, anatomically impossible or whatever. Did you get to that part? Or I assumed no, I also saw.

AARON: That tweet and this is not something that I do, but I don't know if it's anatomically impossible. Yeah, it takes a little bit of effort to figure out how to do that, I guess. I don't think that's necessarily false or whatever, but this is probably not the most important.

ARJUN: Maybe it takes long time to brush his teeth.

#3: Nathan Barnard on how financial regulation can inform AI regulation [00:17:16]

In this next segment, Nathan Barnard and I dive into the complexities of AI regulation, including potential challenges and outcomes of governing AI in relation to economic growth and existential security. And we compare it to banking regulation as well.

AARON: Yeah, I don't know. I just get gloomy for, I think justified reasons when people talk about, oh yeah, here's the nine step process that has to take place and then maybe there's like a 20% chance that we'll be able to regulate AI effectively. I'm being facetious or exaggerating, something like that, but not by a gigantic amount.

NATHAN: I think this is pretty radically different to my mainline expectation.

AARON: What's your mainline expectation?

NATHAN: I suppose I expect like AI to come with an increasing importance past economy and to come up to really like a very large fraction of the economy before really crazy stuff starts happening and this world is going very anonymous. Anonymous, anonymous, anonymous. I know the word is it'd be very unusual if this extremely large sector economy which was impacted like a very large number of people's lives remains like broadly unregulated.

AARON: It'll be regulated, but just maybe in a stupid way.

NATHAN: Sure, yes, maybe in a stupid way. I suppose critically, do you expect the stupid way to be like too conservative or too like the specific question of AI accenture it's basically too conservative or too lenient or I just won't be able to interact with this.

AARON: I guess generally too lenient, but also mostly on a different axis where just like I don't actually know enough. I don't feel like I've read learned about various governance proposals to have a good object level take on this. But my broad prior is that there are just a lot of ways to for anything. There's a lot of ways to regulate something poorly. And the reason insofar as anything isn't regulated poorly it's because of a lot of trial and error.

NATHAN: Maybe.

AARON: I mean, there's probably exceptions, right? I don't know. Tax Americana is like maybe we didn't just kept winning wars starting with World War II. I guess just like maybe like a counterexample or something like that.

NATHAN: Yeah, I think I still mostly disagree with this. Oh, cool. Yeah. I suppose I see a much like broader spectrum between bad regulation and good regulation. I agree it's like very small amount. The space of optimal regulation is very small. But I think we have to hit that space for regulation to be helpful. Especially in this especially if you consider that if you sort of buy the AI extension safety risk then the downsides of it's not this quite fine balancing act between too much whether consumer protection and siphoning competition and cycling innovation too much. It's like trying to end this quite specific, very bad outcome which is maybe much worse than going somewhat slowering economic growth, at least somewhat particularly if we think we're going to get something. This is very explosive rates for economic growth really quite soon. And the cost of slowing down economic growth by weather even by quite a large percentage, very small compared to the cost of sort of an accidental catastrophe. I sort of think of Sony iconic growth as the main cost of main way regulation goes wrong currently.

AARON: I think in an actual sense that is correct. There's the question of like okay, Congress in the states like it's better than nothing. I'm glad it's not anarchy in terms of like I'm glad we have a legislature.

NATHAN: I'm also glad the United States.

AARON: How reasons responsive is Congress? I don't think reasons responsive enough to make it so that the first big law that gets passed insofar as there is one or if there is one is on the pareto frontier trading off between economic growth and existential security. It's going to be way inside of that production frontier or whatever. It's going to suck on every action, maybe not every act but at least like some relevant actions.

NATHAN: Yeah that doesn't seem like obviously true to me. I think Dodge Frank was quite a good law.

AARON: That came after 2008, right?

NATHAN: Yeah correct. Yeah there you go. No, I agree. I'm not especially confident about doing regulation before there's some quite bad before there's a quite bad warning shot and yes, if we're in world where we have no warning shots and we're just like blindsided by everyone getting turned into everyone getting stripped their Athens within 3 seconds, this is not good. Both in law we do have one of those shots and I think Glass Seagull is good law. Not good law is a technical term. I think Glass Steagall was a good piece of legislation. I think DoD Frank was a good piece of legislation. I think the 2008 Seamless Bill was good piece of legislation. I think the Troubled Assets Relief Program is a good piece of piece of legislation.

AARON: I recognize these terms and I know some of them and others I do not know the contents of.

NATHAN: Yeah so Glass Eagle was the financial regulation passed in 1933 after Great Depression. The Tropical Asset Relief Program was passed in I think 2008, moved 2009 to help recapitalize banks. Dodge Frank was the sort of landmark post financial cris piece of legislation passed in 2011. I think these are all good pieces of legislation now. I think like financial regulation is probably unusually good amongst US legislation. This is like a quite weak take, I guess. It's unusually.

AARON: So. I don't actually know the pre depression financial history at all but I feel like the more relevant comparison to the 21st century era is what was the regulatory regime in 1925 or something? I just don't know.

NATHAN: Yeah, I know a bit. I haven't read this stuff especially deeply and so I don't want to don't want to be so overcompensant here but sort of the core pieces which were sort of important for the sort of the Great Depression going very badly was yeah, no distinction between commercial banks and investment banks. Yes, such a bank could take much riskier. Much riskier. Things with like custom deposits than they could from 1933 until the Peel Glass Eagle. And combine that with no deposit insurance and if you sort of have the combination of banks being able to do quite risky things with depositors money and no deposit insurance, this is quite dangerously known. And glassy repeal.

AARON: I'm an expert in the sense that I have the Wikipedia page up. Well, yeah, there was a bunch of things. Basically. There's the first bank of the United States. There's the second bank of the United States. There's the free banking era. There was the era of national banks. Yada, yada, yada. It looks like 19. Seven was there was some panic. I vaguely remember this from like, AP US history, like seven years ago or.

NATHAN: Yes, I suppose in short, I sort of agree that the record of sort of non post Cris legislation is like, not very good, but I think record of post Cris legislation really, at least in the financial sector, really is quite good. I'm sure lots of people disagree with this, but this is my take.

#4 Winston Oswald-Drummond on the tractability of reducing s-risk, ethics, and more [00:27:48]

Up next, Winston Oswald Drummond and I talk about the effectiveness and impact of donating to various research organizations, such as suffering-focused S-risk organizations. We discuss tractability, expected value, and essentially where we should give our money.

AARON: Okay, nice. Yeah. Where to go from here? I feel like largely we're on the same page, I feel like.

WINSTON: Yeah. Is your disagreement mostly tractability? Then? Maybe we should get into the disagreement.

AARON: Yeah. I don't even know if I've specified, but insofar as I have one, yes, it's trapped ability. This is the reason why I haven't donated very much to anywhere for money reasons. But insofar as I have, I have not donated to Clrcrs because I don't see a theory of change that connects the research currently being done to actually reducing s risks. And I feel like there must be something because there's a lot of extremely smart people at both of these orgs or whatever, and clearly they thought about this and maybe the answer is it's very general and the outcome is just so big in magnitude that anything kind.

WINSTON: Of that is part of it, I think. Yeah, part of it is like an expected value thing and also it's just very neglected. So it's like you want some people working on this, I think, at least. Even if it's unlikely to work. Yeah, even that might be underselling it, though. I mean, I do think there's people at CRS and Clr, like talking to people at AI labs and some people in politics and these types of things. And hopefully the research is a way to know what to try to get done at these places. You want to have some concrete recommendations and I think obviously people have to also be willing to listen to you, but I think there is some work being done on that and research is partially just like a community building thing as well. It's a credible signal that you were smart and have thought about this, and so it gives people reason to listen to you and maybe that mostly pays off later on in the future.

AARON: Yeah, that all sounds like reasonable. And I guess one thing is that I just don't there's definitely things I mean, first of all, I haven't really stayed up to date on what's going on, so I haven't even done I've done zero research for this podcast episode, for example. Very responsible and insofar as I've know things about these. Orgs. It's just based on what's on their website at some given time. So insofar as there's outreach going on, not like behind the scenes, but just not in a super public way, or I guess you could call that behind the scenes. I just don't have reason to, I guess, know about that. And I guess, yeah, I'm pretty comfortable. I don't even know if this is considered biting a bullet for the crowd that will be listening to this, if that's anybody but with just like yeah, saying a very small change for a very large magnitude, just, like, checks out. You can just do expected value reasoning and that's basically correct, like a correct way of thinking about ethics. But even I don't know how much you know specifically or, like, how much you're allowed want to reveal, but if there was a particular alignment agenda that I guess you in a broad sense, like the suffering focused research community thought was particularly promising and relative to other tractable, I guess, generic alignment recommendations. And you were doing research on that and trying to push that into the alignment mainstream, which is not very mainstream. And then with the hope that that jumps into the AI mainstream. Even if that's kind of a long chain of events. I think I would be a lot more enthusiastic about I don't know that type of agenda, because it feels like there's like a particular story you're telling where it cashes out in the end. You know what I mean?

WINSTON: Yeah, I'm not the expert on this stuff, but I do think you just mean I think there's some things about influencing alignment and powerful AI for sure. Maybe not like a full on, like, this is our alignment proposal and it also handles Sris. But some things we could ask AI labs that are already building, like AGI, we could say, can you also implement these sort of, like, safeguards so if you failed alignment, you fail sort of gracefully and don't cause lots of suffering.

AARON: Right?

WINSTON: Yeah. Or maybe there are other things too, which also seem potentially more tractable. Even if you solve alignment in some sense, like aligning with whatever the human operator tells the AI to do, then you can also get the issue that malevolent actors can take control of the AI and then what they want also causes lots of suffering that type of alignment wouldn't. Yeah, and I guess I tend to be somewhat skeptical of coherent extrapolated volition and things like this, where the idea is sort of like it'll just figure out our values and do the right thing. So, yeah, there's some ways to push on this without having a full alignment plan, but I'm not sure if that counts as what you were saying.

AARON: No, I guess it does. Yeah, it sounds like it does. And it could be that I'm just kind of mistaken about the degree to which that type of research and outreach is going on. That sounds like it's at least partially true.

#5: Nathan Barnard (again!) on why general intelligence is basically fake [00:34:10]

Up next, Nathan Barnard is back for his second episode. And we talked about the nature of general intelligence, its relationship with language and the implications of specialized brain functions on the understanding of human cognitive abilities.

NATHAN: Yes. This like symbolic like symbolic, symbolic reasoning stuff. Yeah. So I think if I was, like, making the if I was, like, making the case for general intelligence being real, I wouldn't have symbolic reasoning, but I would have language stuff. I'd have this hierarchical structure thing, which.

AARON: I would probably so I think of at least most uses of language and central examples as a type of symbolic reasoning because words mean things. They're like yeah. Pointers to objects or something like that.

NATHAN: Yeah, I think it's like, pretty confidence isn't where this isn't a good enough description of general intelligence. So, for instance so if you bit in your brain called, I'm using a checklist, I don't fuck this up vernacular, I'm not making this cool. Lots of connects to use words like pointers as these arbitrary signs happens mostly in this area of the brain called Berkeley's area. But very famously, you can have Berkeley's epaxics who lose the ability to do language comprehension and use the ability to consistently use words as pointers, as signs to point to things, but still have perfect good spatial reasoning abilities. And so, conversely, people with brokers of fascia who fuck up, who have the broker's reason their brain fucks up will not be able to form fluent sentences and have some problems like unsigned syntax, and they'll still be able to have very good spatial reasoning. It could still, for instance, be like, good engineers. Would you like many problems which, like, cost engineering?

AARON: Yeah, I totally buy that. I don't think language is the central thing. I think it's like an outgrowth of, like I don't know, there's like a simplified model I could make, which is like it's like an outgrowth of whatever general intelligence really is. But whatever the best spatial or graphical model is, I don't think language is cognition.

NATHAN: Yes, this is a really big debate in psycholinguistics as to whether language is like an outgrowth of other abilities like the brain has, whether language whether there's very specialized language modules. Yeah, this is just like a very live debate in psycholinguistics moments. I actually do lean towards the reason I've been talking about this actually just going to explain this hierarchical structure thing? Yeah, I keep talking about it. So one theory for how you can comprehend new sentences, like, the dominant theory in linguistics, how you can comprehend new sentences, um, is you break them up into, like you break them up into, like, chunks, and you form these chunks together in this, like, tree structure. So something like, if you hear, like, a totally novel sentence like the pit bull mastiff flopped around deliciously or something, you can comprehend what the sentence means despite the fact you've never heard it. Theory behind this is you saw yes, this can be broken up into this tree structure, where the different, like, ah, like like bits of the sentence. So, like like the mastiff would be like, one bit, and then you have, like, another bit, which is like, the mastiff I can't remember I said rolled around, so that'd be like, another bit, and then you'd have connectors to our heart.

AARON: Okay.

NATHAN: So the massive rolling around one theory of one of the sort of distinctive things that humans have disabilities is like, this quite general ability to break things up into these these tree structures. This is controversial within psycholinguistics, but it's broadly an area which I broadly buy it because we do see harms to other areas of intelligence. You get much worse at, like, Ravens Progressive Matrices, for instance, when you have, like, an injury to brokers area, but, like, not worse at, like, tests like tests of space, of, like, spatial reasoning, for instance.

AARON: So what is like, is there, like, a main alternative to, like, how humans.

NATHAN: Understand language as far as this specificity of how we pass completely novel sentences, as far as where this is just like this is just like the the academic consensus. Okay.

AARON: I mean, it sounds totally like right? I don't know.

NATHAN: Yeah. But yeah, I suppose going back to saying, how far is language like an outgrowth of general intelligence? An outgrowth like general intelligence versus having much more specialized language modules? Yeah, I lean towards the latter, despite yeah, I still don't want to give too strong of a personal opinion here because I'm not a linguistic this is a podcast.

AARON: You're allowed to give takes. No one's going to say this is like the academic we want takes.

NATHAN: We want takes. Well, gone to my head is.

AARON: I.

NATHAN: Think language is not growth of other abilities. I think the main justification for this, I think, is that the loss of other abilities we see when you have damage to broker's area and verca's area.

AARON: Okay, cool. So I think we basically agree on that. And also, I guess one thing to highlight is I think outgrowth can mean a couple of different things. I definitely think it's plausible. I haven't read about this. I think I did at some point, but not in a while. But outgrowth could mean temporarily or whatever. I think I'm kind of inclined to think it's not that straightforward. You could have coevolution where language per se encourages both its own development and the development of some general underlying trait or something.

NATHAN: Yeah. Which seems likely.

AARON: Okay, cool. So why don't humans have general intelligence?

NATHAN: Right. Yeah. As I was sort of talking about previously.

AARON: Okay.

NATHAN: I think I think I'd like to use go back to like a high level like a high level argument is there appears to be very surprised, like, much higher levels of functional specialization in brains than you expect. You can lose much more specific abilities than you expect to be able to lose. You can lose specifically the ability a famous example is like facebindness, actually. You probably lose the ability to specifically recognize things which you're, like, an expert in.

AARON: Who does it or who loses this ability.

NATHAN: If you've damaged your fuse inform area, you'll lose the ability to recognize faces, but nothing else.

AARON: Okay.

NATHAN: And there's this general pattern that your brain is much more you can lose much more specific abilities than you expect. So, for instance, if you sort of have damage to your ventral, medial, prefrontal cortex, you can say the reasoning for why you shouldn't compulsively gamble but still compulsively gamble.

AARON: For instance okay, I understand this not gambling per se, but like executive function stuff at a visceral level. Okay, keep going.

NATHAN: Yeah. Some other nice examples of this. I think memory is quite intuitive. So there's like, a very famous patient called patient HM who had his hippocampus removed and so as a result, lost all declarative memory. So all memory of specific facts and things which happened in his life. He just couldn't remember any of these things, but still perfectly functioning otherwise. I think at a really high level, I think this functional specialization is probably the strongest piece of evidence against the general intelligence hypothesis. I think fundamentally, general intelligence hypothesis implies that, like, if you, like yeah, if you was, like, harm a piece of your brain, if you have some brain injury, you might like generically get worse at tasks you like, generically get worse at, like at like all task groups use general intelligence. But I think suggesting people, including general intelligence, like the ability to write, the ability to speak, maybe not speak, the ability to do math, you do have.

AARON: This it's just not as easy to analyze in a Cogsy paper which IQ or whatever. So there is something where if somebody has a particular cubic centimeter of their brain taken out, that's really excellent evidence about what that cubic centimeter does or whatever, but that non spatial modification is just harder to study and analyze. I guess we'll give people drugs, right? Suppose that set aside the psychometric stuff. But suppose that general intelligence is mostly a thing or whatever and you actually can ratchet it up and down. This is probably just true, right? You can probably give somebody different doses of, like, various drugs. I don't know, like laughing gas, like like, yeah, like probably, probably weed. Like I don't know.

NATHAN: So I think this just probably isn't true. Your working memory corrects quite strongly with G and having better working memory generic can make you much better at lots of tasks if you have like.

AARON: Yeah.

NATHAN: Sorry, but this is just like a specific ability. It's like just specifically your working memory, which is improved if you go memory to a drugs. Improved working memory. I think it's like a few things like memory attention, maybe something like decision making, which are all like extremely useful abilities and improve how well other cognitive abilities work. But they're all separate things. If you improved your attention abilities, your working memory, but you sort of had some brain injury, which sort of meant you sort of had lost ability to pass syntax, you would not get better at passing syntax. And you can also use things separately. You can also improve attention and improve working memory separately, which just it's not just this one dial which you can turn up.

AARON: There's good reason to expect that we can't turn it up because evolution is already sort of like maximizing, given the relevant constraints. Right. So you would need to be looking just like injuries. Maybe there are studies where they try to increase people's, they try to add a cubic centimeter to someone's brain, but normally it's like the opposite. You start from some high baseline and then see what faculties you lose. Just to clarify, I guess.

NATHAN: Yeah, sorry, I think I've lost the you still think there probably is some general intelligence ability to turn up?

AARON: Honestly, I think I haven't thought about this nearly as much as you. I kind of don't know what I think at some level. If I could just write down all of the different components and there are like 74 of them and what I think of a general intelligence consists of does that make it I guess in some sense, yeah, that does make it less of an ontologically legit thing or something. I think I think the thing I want to get the motivating thing here is that with humans yet you can like we know humans range in IQ, and there's, like, setting aside a very tiny subset of people with severe brain injuries or development disorders or whatever. Almost everybody has some sort of symbolic reasoning that they can do to some degree. Whereas the smartest maybe I'm wrong about this, but as far as I know, the smartest squirrel is not going to be able to have something semantically represent something else. And that's what I intuitively want to appeal to, you know what I mean?

NATHAN: Yeah, I know what you're guessing at. So I think there's like two interesting things here. So I think one is, could a squirrel do this? I'm guessing a squirrel couldn't do this, but a dog can, or like a dog probably can. A chimpanzee definitely can.

AARON: Do what?

NATHAN: Chimpanzees can definitely learn to associate arbitrary signs, things in the world with arbitrary signs.

AARON: Yes, but maybe I'm just adding on epicentercles here, but I feel like correct me if I'm wrong, but I think that maybe I'm just wrong about this, but I would assume that Chicken Tees cannot use that sign in a domain that is qualitatively different from the ones they've been in. Right. So, like, a dog will know that a certain sign means sit or whatever, but maybe that's not a good I.

NATHAN: Don'T know think this is basically not true.

AARON: Okay.

NATHAN: And we sort of know this from teaching.

AARON: Teaching.

NATHAN: There's like a famously cocoa de guerrilla. Also a bonobo whose name I can't remember were taught sign language. And the thing they were consistently bad at was, like, putting together sentences they could learn quite large vocabularies learning to associate by large, I mean in the hundreds of words, in the low hundreds of words which they could consistently use consistently use correctly.

AARON: What do you mean by, like, in what sense? What is bonobo using?

NATHAN: A very famous and quite controversial example is like, coco gorilla was like, saw a swan outside and signed water bird. That's like, a controversial example. But other things, I think, which are controversial here is like, the syntax part of putting water and bird together is the controversial part, but it's not the controversial part that she could see a swan and call that a bird.

AARON: Yeah, I mean, this is kind of just making me think, okay, maybe the threshold for D is just like at the chimp level or something. We are like or whatever the most like that. Sure. If a species really can generate from a prefix and a suffix or whatever, a concept that they hadn't learned before.

NATHAN: Yeah, this is a controversial this is like a controversial example of that the addition to is the controversial part. Yeah, I suppose maybe brings back to why I think this matters is will there be this threshold which AIS cross such that their reasoning after this is qualitatively different to their reasoning previously? And this is like two things. One, like a much faster increase in AI capabilities and two, alignment techniques which worked on systems which didn't have g will no longer work. Systems which do have g. Brings back to why I think this actually matters. But I think if we're sort of accepting it, I think elephants probably also if you think that if we're saying, like, g is like a level of chimpanzees, chimpanzees just, like, don't don't look like quantitatively different to, like, don't look like that qualitatively different to, like, other animals. Now, lots of other animals live in similar complex social groups. Lots of other animals use tools.

AARON: Yeah, sure. For one thing, I don't think there's not going to be a discontinuity in the same way that there wasn't a discontinuity at any point between humans evolution from the first prokaryotic cells or whatever are eukaryotic one of those two or both, I guess. My train of thought. Yes, I know it's controversial, but let's just suppose that the sign language thing was legit with the waterbird and that's not like a random one off fluke or something. Then maybe this is just some sort of weird vestigial evolutionary accident that actually isn't very beneficial for chimpanzees and they just stumbled their way into and then it just enabled them to it enables evolution to bootstrap Shimp genomes into human genomes. Because at some the smartest or whatever actually, I don't know. Honestly, I don't have a great grasp of evolutionary biology or evolution at all. But, yeah, it could just be not that helpful for chimps and helpful for an extremely smart chimp that looks kind of different or something like that.

NATHAN: Yeah. So I suppose just like the other thing she's going on here, I don't want to keep banging on about this, but you can lose the language. You can lose linguistic ability. And it's just, like, happens this happens in stroke victims, for instance. It's not that rare. Just, like, lose linguistic ability, but still have all the other abilities which we sort of think of as like, general intelligence, which I think would be including the general intelligence, like, hypothesis.

AARON: I agree that's, like, evidence against it. I just don't think it's very strong evidence, partially because I think there is a real school of thought that says that language is fundamental. Like, language drives thought. Language is, like, primary to thought or something. And I don't buy that. If you did buy that, I think this would be, like, more damning evidence.

#6 Daniel Filan on why I'm wrong about ethics (+ Oppenheimer and what names mean in like a hardcore phil of language sense) [00:56:54]

[Note: I forgot to record an intro segment here. Sorry!]

AARON: Yeah. Yes. I'm also anti scam. Right, thank you. Okay, so I think that thing that we were talking about last time we talked, which is like the thing I think we actually both know stuff about instead of just like, repeating New York Times articles is my nuanced ethics takes and why you think about talk about that and then we can just also branch off from there.

DANIEL: Yeah, we can talk about that.

AARON: Maybe see where that did. I luckily I have a split screen up, so I can pull up things. Maybe this is kind of like egotistical or something to center my particular view, but you've definitely given me some of the better pushback or whatever that I haven't gotten that much feedback of any kind, I guess, but it's still interesting to hear your take. So basically my ethical position or the thing that I think is true is that which I think is not the default view. I think most people think this is wrong is that total utilitarianism does not imply that for some amount of suffering that could be created there exists some other extremely large arbitrarily, large amount of happiness that could also be created which would morally justify the former. Basically.

DANIEL: So you think that even under total utilitarianism there can be big amounts of suffering such that there's no way to morally tip the calculus. However much pleasure you can create, it's just not going to outweigh the fact that you inflicted that much suffering on some people.

AARON: Yeah, and I'd highlight the word inflicted if something's already there and you can't do anything about it, that's kind of neither here nor there as it pertains to your actions or something. So it's really about you increasing, you creating suffering that wouldn't have otherwise been created. Yeah. It's also been a couple of months since I've thought about this in extreme detail, although I thought about it quite a bit. Yeah.

DANIEL: Maybe I should say my contrary view, I guess, when you say that, I don't know, does total utilitarianism imply something or not? I'm like, well, presumably it depends on what we mean by total utilitarianism. Right. So setting that aside, I think that thesis is probably false. I think that yeah. You can offset great amounts of suffering with great amounts of pleasure, even for arbitrary amounts of suffering.

AARON: Okay. I do think that position is like the much more common and even, I'd say default view. Do you agree with that? It's sort of like the implicit position of people who are of self described total utilitarians who haven't thought a ton about this particular question.

DANIEL: Yeah, I think it's probably the implicit default. I think it's the implicit default in ethical theory or something. I think that in practice, when you're being a utilitarian, I don't know, normally, if you're trying to be a utilitarian and you see yourself inflicting a large amount of suffering, I don't know. I do think there's some instinct to be like, is there any way we can get around this?

AARON: Yeah, for sure. And to be clear, I don't think this would look like a thought experiment. I think what it looks like in practice and also I will throw in caveats as I see necessary, but I think what it looks like in practice is like, spreading either wild animals or humans or even sentient digital life through the universe. That's in a non as risky way, but that's still just maybe like, say, making the earth, making multiple copies of humanity or something like that. That would be an example that's probably not like an example of what an example of creating suffering would be. For example, just creating another duplicate of earth. Okay.

DANIEL: Anything that would be like so much suffering that we shouldn't even the pleasures of earth outweighs.

AARON: Not necessarily, which is kind of a cop out. But my inclination is that if you include wild animals, the answer is yes, that creating another earth especially. Yeah, but I'm much more committed to some amount. It's like some amount than this particular time and place in human industry is like that or whatever.

DANIEL: Okay, can I get a feel of some other concrete cases to see?

AARON: Yeah.

DANIEL: So one example that's on my mind is, like, the atomic bombing of Hiroshima and Nagasaki, right? So the standard case for this is, like, yeah, what? A hundred OD thousand people died? Like, quite terrible, quite awful. And a lot of them died, I guess a lot of them were sort of some people were sort of instantly vaporized, but a lot of people died in extremely painful ways. But the countercase is like, well, the alternative to that would have been like, an incredibly grueling land invasion of Japan, where many more people would have died or know regardless of what the actual alternatives were. If you think about the atomic bombings, do you think that's like the kind of infliction of suffering where there's just not an offsetting amount of pleasure that could make that okay?

AARON: My intuition is no, that it is offsettable, but I would also emphasize that given the actual historical contingencies, the alternative, the implicit case for the bombing includes reducing suffering elsewhere rather than merely creating happiness. There can definitely be two bad choices that you have to make or something. And my claim doesn't really pertain to that, at least not directly.

#7: Holly Elmore on AI pause, wild animal welfare, and some cool biology things I couldn't fully follow but maybe you can [01:04:00]

Up next, Holly Elmore and I discuss the complexities and implications of AI development and open sourcing. We talk about protests and ethical considerations around her, um, uh, campaign to pause the development of frontier AI systems until, until we can tell that they're safe.

AARON: So what's the plan? Do you have a plan? You don't have to have a plan. I don't have plans very much.

HOLLY: Well, right now I'm hopeful about the UK AI summit. Pause AI and I have planned a multi city protest on the 21 October to encourage the UK AI Safety Summit to focus on safety first and to have as a topic arranging a pause or that of negotiation. There's a lot of a little bit upsetting advertising for that thing that's like, we need to keep up capabilities too. And I just think that's really a secondary objective. And that's how I wanted to be focused on safety. So I'm hopeful about the level of global coordination that we're already seeing. It's going so much faster than we thought. Already the UN Secretary General has been talking about this and there have been meetings about this. It's happened so much faster at the beginning of this year. Nobody thought we could talk about nobody was thinking we'd be talking about this as a mainstream topic. And then actually governments have been very receptive anyway. So right now I'm focused on other than just influencing opinion, the targets I'm focused on, or things like encouraging these international like, I have a protest on Friday, my first protest that I'm leading and kind of nervous that's against Meta. It's at the Meta building in San Francisco about their sharing of model weights. They call it open source. It's like not exactly open source, but I'm probably not going to repeat that message because it's pretty complicated to explain. I really love the pause message because it's just so hard to misinterpret and it conveys pretty clearly what we want very quickly. And you don't have a lot of bandwidth and advocacy. You write a lot of materials for a protest, but mostly what people see is the title.

AARON: That's interesting because I sort of have the opposite sense. I agree that in terms of how many informational bits you're conveying in a particular phrase, pause AI is simpler, but in some sense it's not nearly as obvious. At least maybe I'm more of a tech brain person or whatever. But why that is good, as opposed to don't give extremely powerful thing to the worst people in the world. That's like a longer everyone.

HOLLY: Maybe I'm just weird. I've gotten the feedback from open source ML people is the number one thing is like, it's too late, there's already super powerful models. There's nothing you can do to stop us, which sounds so villainous, I don't know if that's what they mean. Well, actually the number one message is you're stupid, you're not an ML engineer. Which like, okay, number two is like, it's too late, there's nothing you can do. There's all of these other and Meta is not even the most powerful generator of models that it share of open source models. I was like, okay, fine. And I don't know, I don't think that protesting too much is really the best in these situations. I just mostly kind of let that lie. I could give my theory of change on this and why I'm focusing on Meta. Meta is a large company I'm hoping to have influence on. There is a Meta building in San Francisco near where yeah, Meta is the biggest company that is doing this and I think there should be a norm against model weight sharing. I was hoping it would be something that other employees of other labs would be comfortable attending and that is a policy that is not shared across the labs. Obviously the biggest labs don't do it. So OpenAI is called OpenAI but very quickly decided not to do that. Yeah, I kind of wanted to start in a way that made it more clear than pause AI. Does that anybody's welcome something? I thought a one off issue like this that a lot of people could agree and form a coalition around would be good. A lot of people think that this is like a lot of the open source ML people think know this is like a secret. What I'm saying is secretly an argument for tyranny. I just want centralization of power. I just think that there are elites that are better qualified to run everything. It was even suggested I didn't mention China. It even suggested that I was racist because I didn't think that foreign people could make better AIS than Meta.

AARON: I'm grimacing here. The intellectual disagreeableness, if that's an appropriate term or something like that. Good on you for standing up to some pretty bad arguments.

HOLLY: Yeah, it's not like that worth it. I'm lucky that I truly am curious about what people think about stuff like that. I just find it really interesting. I spent way too much time understanding the alt. Right. For instance, I'm kind of like sure I'm on list somewhere because of the forums I was on just because I was interested and it is something that serves me well with my adversaries. I've enjoyed some conversations with people where I kind of like because my position on all this is that look, I need to be convinced and the public needs to be convinced that this is safe before we go ahead. So I kind of like not having to be the smart person making the arguments. I kind of like being like, can you explain like I'm five. I still don't get it. How does this work?

AARON: Yeah, no, I was thinking actually not long ago about open source. Like the phrase has such a positive connotation and in a lot of contexts it really is good. I don't know. I'm glad that random tech I don't know, things from 2004 or whatever, like the reddit source code is like all right, seems cool that it's open source. I don't actually know if that was how that right. But yeah, I feel like maybe even just breaking down what the positive connotation comes from and why it's in people's self. This is really what I was thinking about, is like, why is it in people's self interest to open source things that they made and that might break apart the allure or sort of ethical halo that it has around it? And I was thinking it probably has something to do with, oh, this is like how if you're a tech person who makes some cool product, you could try to put a gate around it by keeping it closed source and maybe trying to get intellectual property or something. But probably you're extremely talented already, or pretty wealthy. Definitely can be hired in the future. And if you're not wealthy yet I don't mean to put things in just materialist terms, but basically it could easily be just like in a yeah, I think I'll probably take that bit out because I didn't mean to put it in strictly like monetary terms, but basically it just seems like pretty plausibly in an arbitrary tech person's self interest, broadly construed to, in fact, open source their thing, which is totally fine and normal.

HOLLY: I think that's like 99 it's like a way of showing magnanimity showing, but.

AARON: I don't make this sound so like, I think 99.9% of human behavior is like this. I'm not saying it's like, oh, it's some secret, terrible self interested thing, but just making it more mechanistic. Okay, it's like it's like a status thing. It's like an advertising thing. It's like, okay, you're not really in need of direct economic rewards, or sort of makes sense to play the long game in some sense, and this is totally normal and fine, but at the end of the day, there's reasons why it makes sense, why it's in people's self interest to open source.

HOLLY: Literally, the culture of open source has been able to bully people into, like, oh, it's immoral to keep it for yourself. You have to release those. So it's just, like, set the norms in a lot of ways, I'm not the bully. Sounds bad, but I mean, it's just like there is a lot of pressure. It looks bad if something is closed source.

AARON: Yeah, it's kind of weird that Meta I don't know, does Meta really think it's in their I don't know. Most economic take on this would be like, oh, they somehow think it's in their shareholders interest to open source.

HOLLY: There are a lot of speculations on why they're doing this. One is that? Yeah, their models aren't as good as the top labs, but if it's open source, then open source quote, unquote then people will integrate it llama Two into their apps. Or People Will Use It And Become I don't know, it's a little weird because I don't know why using llama Two commits you to using llama Three or something, but it just ways for their models to get in in places where if you just had to pay for their models too, people would go for better ones. That's one thing. Another is, yeah, I guess these are too speculative. I don't want to be seen repeating them since I'm about to do this purchase. But there's speculation that it's in best interests in various ways to do this. I think it's possible also that just like so what happened with the release of Llama One is they were going to allow approved people to download the weights, but then within four days somebody had leaked Llama One on four chan and then they just were like, well, whatever, we'll just release the weights. And then they released Llama Two with the weights from the beginning. And it's not like 100% clear that they intended to do full open source or what they call Open source. And I keep saying it's not open source because this is like a little bit of a tricky point to make. So I'm not emphasizing it too much. So they say that they're open source, but they're not. The algorithms are not open source. There are open source ML models that have everything open sourced and I don't think that that's good. I think that's worse. So I don't want to criticize them for that. But they're saying it's open source because there's all this goodwill associated with open source. But actually what they're doing is releasing the product for free or like trade secrets even you could say like things that should be trade secrets. And yeah, they're telling people how to make it themselves. So it's like a little bit of a they're intentionally using this label that has a lot of positive connotations but probably according to Open Source Initiative, which makes the open Source license, it should be called something else or there should just be like a new category for LLMs being but I don't want things to be more open. It could easily sound like a rebuke that it should be more open to make that point. But I also don't want to call it Open source because I think Open source software should probably does deserve a lot of its positive connotation, but they're not releasing the part, that the software part because that would cut into their business. I think it would be much worse. I think they shouldn't do it. But I also am not clear on this because the Open Source ML critics say that everyone does have access to the same data set as Llama Two. But I don't know. Llama Two had 7 billion tokens and that's more than GPT Four. And I don't understand all of the details here. It's possible that the tokenization process was different or something and that's why there were more. But Meta didn't say what was in the longitude data set and usually there's some description given of what's in the data set that led some people to speculate that maybe they're using private data. They do have access to a lot of private data that shouldn't be. It's not just like the common crawl backup of the Internet. Everybody's basing their training on that and then maybe some works of literature they're not supposed to. There's like a data set there that is in question, but metas is bigger than bigger than I think well, sorry, I don't have a list in front of me. I'm not going to get stuff wrong, but it's bigger than kind of similar models and I thought that they have access to extra stuff that's not public. And it seems like people are asking if maybe that's part of the training set. But yeah, the ML people would have or the open source ML people that I've been talking to would have believed that anybody who's decent can just access all of the training sets that they've all used.

AARON: Aside, I tried to download in case I'm guessing, I don't know, it depends how many people listen to this. But in one sense, for a competent ML engineer, I'm sure open source really does mean that. But then there's people like me. I don't know. I knew a little bit of R, I think. I feel like I caught on the very last boat where I could know just barely enough programming to try to learn more, I guess. Coming out of college, I don't know, a couple of months ago, I tried to do the thing where you download Llama too, but I tried it all and now I just have like it didn't work. I have like a bunch of empty folders and I forget got some error message or whatever. Then I tried to train my own tried to train my own model on my MacBook. It just printed. That's like the only thing that a language model would do because that was like the most common token in the training set. So anyway, I'm just like, sorry, this is not important whatsoever.

HOLLY: Yeah, I feel like torn about this because I used to be a genomicist and I used to do computational biology and it was not machine learning, but I used a highly parallel GPU cluster. And so I know some stuff about it and part of me wants to mess around with it, but part of me feels like I shouldn't get seduced by this. I am kind of worried that this has happened in the AI safety community. It's always been people who are interested in from the beginning, it was people who are interested in singularity and then realized there was this problem. And so it's always been like people really interested in tech and wanting to be close to it. And I think we've been really influenced by our direction, has been really influenced by wanting to be where the action is with AI development. And I don't know that that was right.

AARON: Not personal, but I guess individual level I'm not super worried about people like you and me losing the plot by learning more about ML on their personal.

HOLLY: You know what I mean? But it does just feel sort of like I guess, yeah, this is maybe more of like a confession than, like a point. But it does feel a little bit like it's hard for me to enjoy in good conscience, like, the cool stuff.

AARON: Okay. Yeah.

HOLLY: I just see people be so attached to this as their identity. They really don't want to go in a direction of not pursuing tech because this is kind of their whole thing. And what would they do if we weren't working toward AI? This is a big fear that people express to me with they don't say it in so many words usually, but they say things like, well, I don't want AI to never get built about a pause. Which, by the way, just to clear up, my assumption is that a pause would be unless society ends for some other reason, that a pause would eventually be lifted. It couldn't be forever. But some people are worried that if you stop the momentum now, people are just so luddite in their insides that we would just never pick it up again. Or something like that. And, yeah, there's some identity stuff that's been expressed. Again, not in so many words to me about who will we be if we're just sort of like activists instead of working on.

AARON: Maybe one thing that we might actually disagree on. It's kind of important is whether so I think we both agree that Aipause is better than the status quo, at least broadly, whatever. I know that can mean different things, but yeah, maybe I'm not super convinced, actually, that if I could just, like what am I trying to say? Maybe at least right now, if I could just imagine the world where open eye and Anthropic had a couple more years to do stuff and nobody else did, that would be better. I kind of think that they are reasonably responsible actors. And so I don't know. I don't think that actually that's not an actual possibility. But, like, maybe, like, we have a different idea about, like, the degree to which, like, a problem is just, like, a million different not even a million, but, say, like, a thousand different actors, like, having increasingly powerful models versus, like, the actual, like like the actual, like, state of the art right now, being plausibly near a dangerous threshold or something. Does this make any sense to you?

HOLLY: Both those things are yeah, and this is one thing I really like about the pause position is that unlike a lot of proposals that try to allow for alignment, it's not really close to a bad choice. It's just more safe. I mean, it might be foregoing some value if there is a way to get an aligned AI faster. But, yeah, I like the pause position because it's kind of robust to this. I can't claim to know more about alignment than OpenAI or anthropic staff. I think they know much more about it. But I have fundamental doubts about the concept of alignment that make me think I'm concerned about even if things go right, like, what perverse consequences go nominally right, like, what perverse consequences could follow from that. I have, I don't know, like a theory of psychology that's, like, not super compatible with alignment. Like, I think, like yeah, like humans in living in society together are aligned with each other, but the society is a big part of that.

#8: Max Alexander and I solve ethics, philosophy of mind, and cancel culture once and for all [01:24:43]

Now, up next, Max Alexander, and I talk about Twitter, how it's changed as a platform, and many fun things besides.

AARON: Wait, maybe. Like, what else? I don't know. What do you think about Twitter, like, in general? I don't know. Because this is how we met.

MAX: Yeah.

AARON: We have not met in real life.

MAX: Worse as a platform than it was two years ago or something.

AARON: Okay.

MAX: Stability wise, and there are small changes that make it worse or something, but largely my experience is unchanged, I think.

AARON: Do you think it's good, bad? I don't know. Do you think people should join Twitter on the market?

MAX: I think EA should join EA. Twitter. I'm not sure if you join Twitter rather than other social medias or something. I think sort of the area of social media we're on is uniquely quite good or something.

AARON: I agree.

MAX: And some of this is like, you get interactions with people, which is good, and people are very nice or something, and very civil where we are. And it's less clear the sorts of personal ability or something and niceness that you get where we are in, like, are elsewhere in Twitter because I don't go elsewhere. But basically you should join Twitter, I guess, if you're going to enter a small community or something, if you're just going to use it to browse memes or something, it's not clear this is better than literally any other social media that has no.

AARON: Yeah, I agree. Well, I guess our audience is, of all, maybe four people, is largely from Twitter. But you never know. There's like a non zero chance that somebody from the wider world will be listening. I think it's at least worth an experiment. Right. Maybe you could tell me something that I should experiment with. Is there anything else like Twitter that we don't have in common that you think that maybe I don't do? It's like, oh, he's an idiot for not doing.

MAX: Oh, probably not. I mean, I'm sure you do better things than I do. Probably.

AARON: Well, I mean, probably this is a large. Right? Like, I don't know.

MAX: I think a benefit of using Twitter is like, it kind of opens you up or something. Probably is the case. It probably does literally build your social skills or something. I mean, maybe not in an obviously useful way, because it's like you're probably not necessarily that much better at doing in person stuff or something as a result of these Twitter. Maybe it improves you very slightly or something, but it's a different skill, texting versus talking.

AARON: Actually, here's something I want your thoughts on recently. Maybe this is outing me as a true Twitter addict, but no, I, by and large, have had a really good experience and I stand by that. I think it's net on net. Not just on net, but just in general, added value to my life and stuff. And it's great, especially given the community that I'm in. The communities that I'm in. But yeah, this is going to kind of embarrassing. I've started thinking in tweets. I'm not 100% of the time, not like my brain is only stuck on Twitter mode, but I think on the margin there's been a chef toward a thought verbalizes an Aaron's brain as something that could be a tweet. And I'm not sure this is a positive.

MAX: Like it is the case. I've had my friends open Twitter in front of me, like my Twitter and go through and read my tweets. Actually, many people in my life do this. I don't know why. I don't really want them to do that. And it does change the way you talk. Certainly part of that is probably character element, and part of it is probably like culture or something. So that's the case. I don't know if I experienced that or I do sometimes if I thought of a really stupid pun. Normally you don't do anything with that, but now I can or something. Right. It's worth holding on for the 6 seconds it takes to open my phone. But I think I actually kind of maybe think in tweets already or something. Like, if you read my writing, I've gotten feedback that it's both very poetic or something. And poems are short or something. It's like very stanza or something, which is kind of how Twitter works also. Right. I think if you looked at the formatting of some of my writing, you would see that it's very twitter like or something. In some sense, there's no character limit, and so maybe this is just the sort of thing you're experiencing or something. Or maybe it's more intense.

AARON: Yeah, probably not exactly. Honestly, I don't think this is that big of a deal. One thing is, I think this is a causal effect. I've blogged less and. And I think it's like, not a direct replacement. Like, I think Scooter has been like an outlet for my ideas that actually feels less effortful and takes less. So it's not like a one for one thing. So other more worky things have filled in the gap for blogging. But I think it has been a causal reason that I haven't blogged as much as I would like to. Really would like have to or something. Yeah, I can see that being thing that is like ideas, there's no strong signal that a particular tweet is an important idea that's worth considering. Whereas if you've written a whole blog post on it and you have 200 subscribers or whatever, you put in a lot of effort. People are at least going to say like, oh, this is me. At least plausibly like an important idea. Like when they're coming into it or something like that.

MAX: Yeah. And if you think something is valuable or something, maybe this is different for you or something. But I get like three likes on all my tweets. It's very rare I get ten likes or something. The number of followers. It's just stuck there forever.

#9: Sarah Woodhouse on discovering AI x-risk, Twitter, and more [01:30:56]

Here, Sarah Woodhouse and I talked about how she got into the field of AI safety, starting from concerned about job automation, and how this led her to be a Twitter influencer.

SARAH: Well, I realized that a chatbot could very easily do my job and that my employers either hadn’t noticed this or they had noticed, but they were just being polite about it and they didn’t want to fire me because they’re too nice. And I was like, I should find out what AI development is going to be like over the next few years so that I know if I should go and get good at some other stuff.

SARAH: I just had a little innocent Google. And then within a few clicks, I’d completely doom pilled myself. I was like, we’re all going to die. I think I found Geoffrey Hinton because he was on the news at the time, because he just quit his job at Google. And he was there saying things that sounded very uncertain, very alarming. And I was like, well, he’s probably the pessimist, but I’m sure that there are loads of optimists to counteract that because that’s how it usually goes. You find a doomer and then you find a bunch of more moderate people, and then there’s some consensus in the middle that everything’s basically fine.

SARAH: I was like, if I just keep looking, I’ll find the consensus because it’s there. I’m sure it’s there. So I just kept looking and looking for it. I looked for it for weeks. I just didn’t find it. And then I was like, nobody knows what’s going on. This seems really concerning. So then I started lurking on Twitter, and then I got familiar with all the different accounts, whatever. And then at some point, I was like, I’m going to start contributing to this conversation, but I didn’t think that anybody would talk back to me. And then at some point, they started talking back to me and I was like, this is kind of weird.

SARAH: And then at some point, I was having an existential crisis and I had a couple of glasses of wine or something, and I just decided to type this big, long thread. And then I went to bed. I woke up the next morning slightly grouchy and hungover. I checked my phone and there were all these people messaging me and all these people replying to my thread being like, this is so relatable. This really resonated with me. And I was like, what is going on?

AARON: You were there on Twitter before that thread right? I’m pretty sure I was following you.

SARAH: I think, yeah, I was there before, but no one ever really gave me any attention prior to that. I think I had a couple of tweets that blew up before that, but not to the same extent. And then after that, I think I was like, okay, so now I have an audience. When I say an audience, like, obviously a small one, but more of an audience than I’ve ever had before in my life. And I was like, how far can I take this?

SARAH: I was a bit like, people obviously started following me because I’m freFreaking out about AI, but if I post an outfit, what’s going to happen? How far can I push this posting, these fit checks? I started posting random stuff about things that were completely unrelated. I was like, oh, people are kind of here for this, too. Okay, this is weird. So now I’m just milking it for all its worth, and I really don’t know why anybody’s listening to me. I’m basically very confused about the whole thing.

AARON: I mean, I think it’s kind of weird from your perspective, or it’s weird in general because there aren’t that many people who just do that extremely logical thing at the beginning. I don’t know, maybe it’s not obvious to people in every industry or whatever that AI is potentially a big deal, but there’s lots of truckers or whatever. Maybe they’re not the best demographic or the most conducive demographic, like, getting on Twitter or whatever, but there’s other jobs that it would make sense to look into that. It’s kind of weird to me that only you followed the rabbit hole all the way down.

SARAH: I know! This is what I…Because it’s not that hard to complete the circle. It probably took me like a day, it took me like an afternoon to get from, I’m worried about job automation to I should stop saving for retirement. It didn’t take me that long. Do you know what I mean? No one ever looks. I literally don’t get it. I was talking to some people. I was talking to one of my coworkers about this the other day, and I think I came up in conversation. She was like, yeah, I’m a bit worried about AI because I heard on the radio that taxi drivers might be out of a job. That’s bad. And I was like, yeah, that is bad. But do you know what else? She was like, what are the AI companies up to that we don’t know about? And I was like, I mean, you can go on their website. You can just go on their website and read about how they think that their technology is an extinction risk. It’s not like they’re hiding. It’s literally just on there and no one ever looks. It’s just crazy.

AARON: Yeah. Honestly, I don’t even know if I was in your situation, if I would have done that. It’s like, in some sense, I am surprised. It’s very few people maybe like one, but at another level, it’s more rationality than most humans have or something. Yeah. You regret going down that rabbit hole?

SARAH: Yeah, kind of. Although I’m enjoying the Twitter thing and it’s kind of fun, and it turns out there’s endless comedic material that you can get out of impending doom. The whole thing is quite funny. It’s not funny, but you can make it funny if you try hard enough. But, yeah, what was I going to say? I think maybe I was more primed for doom pilling than your average person because I already knew what EA was and I already knew, you know what I mean. That stuff was on my radar.

AARON: That’s interesting.

SARAH: I think had it not been on my radar, I don’t think I would have followed the pipeline all the way.

AARON: Yeah. I don’t know what browser you use, but it would be. And you should definitely not only do this if you actually think it would be cool or whatever, but this could be in your browser history from that day and that would be hilarious. You could remove anything you didn’t want to show, but if it’s like Google Chrome, they package everything into sessions. It’s one browsing session and it’ll have like 10,000 links.

SARAH: Yeah, I think for non-sketchy reasons, I delete my Google history more regularly than that. I don’t think I’d be able to find that. But I can remember the day and I can remember my anxiety levels just going up and up somewhere between 01:00 p.m. and 07:00 p.m. And by the evening I’m like, oh, my God.

AARON: Oh, damn, that’s wild.

SARAH: It was really stressful.

AARON: Yeah, I guess props for, I don’t know if props…Is the right word, I guess, impressed? I’m actually somewhat surprised to hear that you said you regret it. I mean, that sucks though, I guess. I’m sorry.

SARAH: If you could unknow this, would you?

AARON: No, because I think it’s worth maybe selfishly, but not overall because. Okay, yeah, I think that would plausibly be the selfish thing to do. Actually. No, actually, hold on. No, I actually don’t think that’s true. I actually think there’s enough an individual can do selfishly such that it makes sense. Even the emotional turmoil.

SARAH: It would depend how much you thought that you were going to personally move the needle by knowing about it. I personally don’t think that I’m going to be able to do very much. I was going to tip the scales. I wouldn’t selfishly unknow it and sacrifice the world. But me being not particularly informed or intelligent and not having any power, I feel like if I forgot that AI was going to end the world, it would not make much difference.

AARON: You know what I mean? I agree that it’s like, yes, it is unlikely for either of us to tip the scales, but.

SARAH: Maybe you can’t.

AARON: No, actually, in terms of, yeah, I’m probably somewhat more technically knowledgeable just based on what I know about you. Maybe I’m wrong.

SARAH: No, you’re definitely right.

AARON: It’s sort of just like a probabilities thing. I do think that ‘doom’ - that word - is too simplified, often too simple to capture what people really care about. But if you just want to say doom versus no doom or whatever, AI doom versus no AI doom. Maybe there’s like a one in 100,000 chance that one of us tips the scales. And that’s important. Maybe even, like, one in 10,000. Probably not. Probably not.

SARAH: One in 10,000. Wow.

#10: Pigeon Hour x Consistently Candid pod-crossover: I debate moral realism* with Max Alexander and Sarah Hastings-Woodhouse [01:41:08]

And for our last segment, this is a segment of Sarah, from the previous segments, new podcast called Consistently Candid, excellent name. And basically, Max Alexander and I are debating moral realism or something like that, as you'll see, with Sarah being the moderator and judge.

Aaron: Yeah. Max, do you want to go first or second?Max: I'll go second.Sarah: Okay.

Aaron: Well, the first thing is that. The thing I always say is that I simply deferred to Sharon Hewitt Rawlette, who was on the 80,000 Hours Podcast. They had a whole podcast episode about this, and she's an actual philosopher who made the actual case for this.

Aaron: And so everything I say is just basically, like…the actual case is that you just listen to her. Well, I guess one thing is that, okay, what's the base rate of people being correct given that they're on the 80,000 hours podcast? Pretty high. Probably, like, 99%.

Max: I don't know if that's right.Aaron: Not that.Sarah: Has no one ever been on the 80,000 Podcast and argued the opposite?Max: Sam Bankman Fried was on the 80,000 hours podcast.Sarah: Oh, yeah, that's true. That was embarrassing.Aaron: Well, that's why I said 99%, not 100%.Max: Yeah, that was their one episode.Aaron: Yeah, everything else - wait, I'm sorry. I was mostly joking about that, but no, I am serious: maybe I'll find a way to reference the URL in the show description or something, or just like Google “80,000 Hours podcast moral realism.”

Aaron: First of all, my actual point of view is a weak version of moral realism. I believe that truly normative statements, such as "a person should do X or Y," are not objectively true or false. However, I do think that sometimes, at least occasionally, statements that objectively order worlds or actions can be true or false. For example, saying "world A is objectively better than world B."

Aaron: The most intuitive argument, or perhaps intuition pump, that I can gesture to in favor of my point of view is this idea of comparing and objectively ordering worlds or actions. It's just like, okay, so you have one world and then another world, which is exactly the same, except it also creates a person who's feeling a lot of pain, and that's the only difference. And I want to say that this world is objectively worse, and the reason why it's objectively worse is just because it is built into both, sort of, semantically, the meaning of what we say, pain or suffering, but also not mean. That's true.

Aaron: But another perspective on this. It's sort of like a brute fact of the universe, in the same way that facts about physics are that suffering and pain are bad. And so if you just add some of this bad stuff, or on the other side, add some objectively good stuff, you get an objectively better shade of the world. And so I will leave it there for Max to tell me why I'm wrong.Sarah: All right, well, okay, can I ask a question first? Max, you want to go. How do I phrase this? Sorry, I'm just clarifying. So you're basically saying that you can't make truth claims about what people ought to do, but you can about which states of affairs are better or worse than others.

Sarah: But if you can definitely say this circumstance is better than this one, objectively speaking, then if you could find some way of empirically determining which actions brought about more pleasure or pain, even if, I mean, maybe we can never actually determine which actions would do that. But say, if you could, then would those things not be like, would you not be able to make a claim about what you should do?Aaron: I think you can make a claim. In fact, I think they actually would make the claim. But then what I wouldn't be able to say, at least what I currently think, is that those wouldn't be objectively true or false in the same way. I'm less sure about this, for what it's worth. I'm like, less.Sarah: How can it be objectively the case that one situation could be better than the other, but it's not objectively true that you should do the thing that is most likely to bring about the better one?Aaron: No, this is a good question. I actually just had this debate on Twitter, sort of.Sarah: Okay.Aaron: Although I think the person ended up agreeing with me. One thing is, I think some people just have the sense those two statements are basically just saying are just like, rewording of the same thing. And that's just not my sense. But maybe I'm the weird one, and everybody else has the sense that when they say, oh, x is better than y, and the statement like, oh, you should act so as to bring about x, that these are just exactly the same thing. It's just reworded. Is that your sense?Sarah: I think they're exactly the same thing, but. Well, actually, no, I kind of do. I don't really understand how it can be true that X is better than y, and at the same time, it's not true that you should try and make X happen.

Outro [1:47:04]

All right, that's it; thank you so much to all my guests!

And if you're listening to this, like there's like a 99% chance that you are invited as a guest. It's just a matter of me working my way through some people who I'm being unnecessarily slow with.

Thanks so much for listening and you know where to find me - if you're listening to this, you almost certainly know where to find me.

Take care.

Get full access to Aaron's Blog at www.aaronbergman.net/subscribe

Intro

At the gracious invitation of AI Safety Twitter-fluencer Sarah Hastings-Woodhouse, I appeared on the very first episode of her new podcast “Consistently Candid” to debate moral realism (or something kinda like that, I guess; see below) with fellow philosophy nerd and EA Twitter aficionado Max Alexander, alongside Sarah as moderator and judge of sorts.

What I believe

In spite of the name of the episode and the best of my knowledge/understanding a few days ago, it turns out my stance may not be ~genuine~ moral realism.

Here’s my basic meta-ethical take:

* Descriptive statements that concern objective relative goodness or badness (e.g., "it is objectively for Sam to donate $20 than to buy an expensive meal that costs $20 more than a similar, less fancy meal”) can be and sometimes are true; but

* Genuinely normative claims like “Sam should (!) donate $20 and should not buy that fancy meal” are never objectively true.

Of course the label per se doesn’t really matter. But for a bunch of reasons it still seems wise to figure out which label really does work best.

Some definitions

Stanford Encyclopedia of Philosophy:

Moral realists are those who think that, in these respects, things should be taken at face value—moral claims do purport to report facts and are true if they get the facts right. Moreover, they hold, at least some moral claims actually are true. That much is the common and more or less defining ground of moral realism (although some accounts of moral realism see it as involving additional commitments, say to the independence of the moral facts from human thought and practice, or to those facts being objective in some specified way).

Wikipedia:

The robust model of moral realism commits moral realists to three theses:

* The semantic thesis: The primary semantic role of moral predicates (such as "right" and "wrong") is to refer to moral properties (such as rightness and wrongness), so that moral statements (such as "honesty is good" and "slavery is unjust") purport to represent moral facts, and express propositions that are true or false (or approximately true, largely false, and so on).

* The alethic thesis: Some moral propositions are in fact true.

* The metaphysical thesis: Moral propositions are true when actions and other objects of moral assessment have the relevant moral properties (so that the relevant moral facts obtain), where these facts and properties are robust: their metaphysical status, whatever it is, is not relevantly different from that of (certain types of) ordinary non-moral facts and properties.

And to settle the matter, here’s what some LLM have to say.

My prompt:

Suppose a person believes the following:

1: Descriptive statements that concern objective relative goodness or badness (e.g., "it is objectively for Sam to donate $20 than to buy an expensive meal that costs $20 more than a similar, less fancy meal”) can be and sometimes are true; but

2: Genuinely normative claims like “Sam should donate $20 and should not buy that fancy meal” are never objectively true.

Is this person best categorized as a moral realist?

ChatGPT

Given these two beliefs, the person's stance doesn't align completely with traditional moral realism, which typically asserts that there are objective truths about both moral facts (like the goodness or badness of actions) and moral duties (what one should or should not do). However, it also doesn't fully align with moral anti-realism, since the person believes in some form of objective truth regarding moral value.

One possible categorization for this stance could be a form of "value realism" combined with "normative anti-realism" or "normative subjectivism." This would mean that the person believes in objective truths about what is better or worse (value realism) but thinks that what one ought to do (normative claims) is not a matter of objective truth but perhaps of subjective judgment or societal conventions.

This position is quite sophisticated and might be best understood as a hybrid or intermediate position in the landscape of moral philosophy, rather than fitting neatly into the category of moral realism or its standard alternatives.

Great, so much for trying to fit things into nice little buckets.

Claude, fortunately, is more definitive:

No, the view described is not best categorized as moral realism. Here's why:

Moral realism is the view that there are objective moral facts or truths. A moral realist believes that some moral claims, including normative claims about what one should or shouldn't do, can be objectively and mind-independently true.

The view in the example accepts that there can be objective truths about relative degrees of goodness or badness (claim 1). However, it denies that there are objective normative truths about what one should or shouldn't do (claim 2).

By rejecting that genuine normative 'should/shouldn't' claims can be objectively true, the view denies a key commitment of moral realism - namely that moral claims can state objective normative facts. At best it allows for objective evaluative facts about goodness/badness, not prescriptive facts about obligations.

So this view would be more accurately categorized as a form of moral anti-realism like non-cognitivism or error theory. It does not qualify as moral realism because of its rejection of the existence of objective normative truths that could make strong normative claims justified or true. The commitment to objective evaluative facts alone does not sufficiently qualify the view as realist in the metaethical sense.

In summary, the view accepts objective "good/bad" facts but denies objective "should/shouldn't" facts. This combination puts it at odds with core commitments of moral realism, so it is better seen as an anti-realist view.

I’m gonna disagree with Claude here, at least in that the bolded claim above (objective normative facts are a key part of moral realism) just seems…wrong? Like all the definitions could replace the term “moral fact” or “moral claim” with “normative fact” and “normative claim” but they do not!

So I really don’t have a great answer to which label works best.

One of two ~human level chatbots giving me a definitive-sounding “anti-realism” answer is too much evidence for me, whose only formal philosophy training is an undergrad minor, to rule that one out. There are also good arguments, I think, for the “realist label,” as well as for “neither” (i.e., ‘secret third thing’). In fact all of these seem pretty similar in terms of argument convincingness/correctness.

So, in sum, 🤷‍♂️.

Some tweets on the matter, both of which are images that link to the original:

Ok now you can vote:

Anyway, long story short, I believe something kinda sorta like moral realism, maybe, and that is what we discuss. Without further ado, here’s the (very imperfect) transcript of the episode.

Please note that some of the long blocks of text have been cleaned up a bit via minor rewording. I am too lazy to do that for everything.

Transcript

Sarah: Hello, and welcome to my first experiment in low effort podcasting. In this episode of the podcast, which I'm now calling Consistently Candid because some people thought that was funny, I talked to Aaron Bergman and Max Alexander about moral realism.

Sarah: They kind of debate it. And I, having read the Wikipedia page about five minutes previously, a, occasionally chime in with some opinions that I hadn't thought out very well. So enjoy!

Sarah: Anyway, I guess this is my podcast now, but I don't have a name for it yet.Max: That's a good podcast nameSarah: Introduce it.Aaron: Can I broadcast this on Pigeon Hour as well?Sarah: Yeah, sure.Max: Okay, cool.Aaron: Sweet.Sarah: But I also want to make my own thing because people.Aaron: No, totally. But yeah, you can say no, you can copyright it and then sue me.Sarah: No. Well, that's fine. This is totally, like, anyone can broadcast it anywhere they want.Max: You can text or whatever, get on the Trump website.Sarah: Yeah. So you guys have a disagreement, apparently, about moral realism. I have briefly skimmed the Wikipedia page, and I don't have an opinion, but I thought we have it out.Aaron: No, I feel like the format should be that we try to convince you…Sarah: So, yeah, you try and convince me that you each try and convince me you're right, and I will come to a conclusion and let you know who I'm persuaded by. And if at any point I have, like, a thought that's worth articulating, I'll weigh in with that. But I think that's kind of unlikely because I don't really know anything; I'm playing a moderating role here.Max: Well, confusion is worth pointing out or something like that, right?Sarah: Yeah, I can do that at regular intervals. I can tell you how confused I am. That's definitely doable.Aaron: Maybe you should start with, like, do you have an initial take at all, or are you really 50/50?Sarah: I mean, from very briefly reading the Wikipedia, it liked doesn't sound true to me.Max: Oh, hell yeah!Aaron: No. Okay, podcast over.Max: Way over the Wikipedia just to see what says. Did you read actual Wikipedia?Sarah: Wikipedia? Yeah, it says “moral realism (also, ethical realism) is the position that ethical Sentences express propositions that refer to objective features of the world. That is, features independent of subjective opinion.”Aaron: Yeah, facts.Max: Good summary from Wikipedia.Sarah: Fake.Max: My job is going to be easy.Sarah: Then, but I'm totally open to be persuaded.Aaron: Okay. The first thing is that I recognize that it sounds fake, it sounds very sus, but then it actually surprisingly checks out. So I just want to get that on the table.Sarah: Okay, what about if each of you do, like, a little opening spiel about why you think you're right, and then you can yell at each other about it afterwards.Aaron: Yeah. Max, do you want to go first or second?Max: I'll go second.Sarah: Okay.

Aaron: Well, the first thing is that. The thing I always say is that I simply deferred to Sharon Hewitt Rawlette, who was on the 80,000 Hours Podcast. They had a whole podcast episode about this, and she's an actual philosopher who made the actual case for this.

Aaron: And so everything I say is just basically, like…the actual case is that you just listen to her. Well, I guess one thing is that, okay, what's the base rate of people being correct given that they're on the 80,000 hours podcast? Pretty high. Probably, like, 99%.

Max: I don't know if that's right.Aaron: Not that.Sarah: Has no one ever been on the 80,000 Podcast and argued the opposite?Max: Sam Bankman Fried was on the 80,000 hours podcast.Sarah: Oh, yeah, that's true. That was embarrassing.Aaron: Well, that's why I said 99%, not 100%.Max: Yeah, that was their one episode.Aaron: Yeah, everything else - wait, I'm sorry. I was mostly joking about that, but no, I am serious: maybe I'll find a way to reference the URL in the show description or something, or just like Google “80,000 Hours podcast moral realism.”

Aaron: First of all, my actual point of view is a weak version of moral realism. I believe that truly normative statements, such as "a person should do X or Y," are not objectively true or false. However, I do think that sometimes, at least occasionally, statements that objectively order worlds or actions can be true or false. For example, saying "world A is objectively better than world B."

Aaron: The most intuitive argument, or perhaps intuition pump, that I can gesture to in favor of my point of view is this idea of comparing and objectively ordering worlds or actions. It's just like, okay, so you have one world and then another world, which is exactly the same, except it also creates a person who's feeling a lot of pain, and that's the only difference. And I want to say that this world is objectively worse, and the reason why it's objectively worse is just because it is built into both, sort of, semantically, the meaning of what we say, pain or suffering, but also not mean. That's true.

Aaron: But another perspective on this. It's sort of like a brute fact of the universe, in the same way that facts about physics are that suffering and pain are bad. And so if you just add some of this bad stuff, or on the other side, add some objectively good stuff, you get an objectively better shade of the world. And so I will leave it there for Max to tell me why I'm wrong.Sarah: All right, well, okay, can I ask a question first? Max, you want to go. How do I phrase this? Sorry, I'm just clarifying. So you're basically saying that you can't make truth claims about what people ought to do, but you can about which states of affairs are better or worse than others.

Sarah: But if you can definitely say this circumstance is better than this one, objectively speaking, then if you could find some way of empirically determining which actions brought about more pleasure or pain, even if, I mean, maybe we can never actually determine which actions would do that. But say, if you could, then would those things not be like, would you not be able to make a claim about what you should do?Aaron: I think you can make a claim. In fact, I think they actually would make the claim. But then what I wouldn't be able to say, at least what I currently think, is that those wouldn't be objectively true or false in the same way. I'm less sure about this, for what it's worth. I'm like, less.Sarah: How can it be objectively the case that one situation could be better than the other, but it's not objectively true that you should do the thing that is most likely to bring about the better one?Aaron: No, this is a good question. I actually just had this debate on Twitter, sort of.Sarah: Okay.Aaron: Although I think the person ended up agreeing with me. One thing is, I think some people just have the sense those two statements are basically just saying are just like, rewording of the same thing. And that's just not my sense. But maybe I'm the weird one, and everybody else has the sense that when they say, oh, x is better than y, and the statement like, oh, you should act so as to bring about x, that these are just exactly the same thing. It's just reworded. Is that your sense?Sarah: I think they're exactly the same thing, but. Well, actually, no, I kind of do. I don't really understand how it can be true that X is better than y, and at the same time, it's not true that you should try and make X happen.Aaron: Yeah, if they're not semantic and don't mean the exact same thing, then there's the question of what else you need to get from one to the other. If you've established one, what else do you need?

Aaron: For instance, if you're unsure, my somewhat unconfident perspective is that statements that are normative, like "you should do X," are their own thing. We might just not live in a universe where those have meaning outside of the social realm or above it. We can use them as useful fictions in the social world, but they're not fundamental in the same way that physics is or something like that.

Max: You're saying moral claims are. Sorry, moral claims are like this.Aaron: True normative claims. So you ought to do x. Yeah.Max: Well, I mean, depending on what you mean by them being different from air quotes, physics or something like this, kind of sounds like you might be an anti realist, maybe.Sarah: I was going to say it does kind of sound like you.Aaron: No, I know in some ways I'm like, maybe I am, but then if so, I just want to defend my actual position, which is that, okay, fine, you can call me whatever you want. But then I still think that we have objective ordering of states of the world, and that's kind of what really.Max: As Sarah introduced, moral realism is about the truth of valuable propositions. This is the core of it. But then there's the actual core, which is something like "murder is wrong" evaluates to true sometimes. So, if I said murder is wrong and I'm talking about Sarah shooting me with a gun, that's true; it's bad for her to shoot me with a gun, all things considered.

Max: This is what people really focus on. It doesn't matter if your position is that objectively, conscious states imply an ordering of preferability or something to that effect. Stubbing your toe is preferable to being shot with a gun. Objectively, this is just what it is to experience suffering. And the same thing applies to positive experiences. It's better to have a birthday party than it is to eat one cookie. This is just what it is to experience things, and it applies to ordering states of affairs.

Max: I can accept this under some definitions. It's not objective per se, but it is true based on my preference function. Objectively, I prefer these things this way. And maybe it's true more broadly than this. But if you don't have that bit about being shot by a gun being wrong for Sarah to do, then you're not practically a realist, in my opinion.Aaron: There's this sort of meta debate, which may be important. I kind of think it actually ends up with a couple different ways. There's the meta debate, and there's the meta-meta debate of, like, okay, is this actually an important question or not? Do you think it's not important to establish? Do you think it's sort of irrelevant unless you have true normativity?

Aaron: Sorry, honestly, I got distracted and kind of lost my train of thought in the last few seconds, so I'll let you take it wherever.

Max: Yeah, I mean, I guess Sarah where. Having heard these very basic things for it, though, I guess Aaron hasn't done the best job defending moral realism. If you'd like. Or like defending standard moral realism or something like that. I should say defending the best version.Sarah: He hasn't defended.Aaron: Making the best argument, the most convincing argument for moral realism. I'm just trying to claim or defend what I think is true.Sarah: Yeah, no, I guess I still don't really understand how believing that there are states which are objectively more preferable than others is compatible with believing that there aren't actions that are objectively right and objectively wrong. I just don't really understand how those two things.Aaron: Okay, I feel like maybe we should just table that and set aside the question. I feel like the question of whether normativity is legit or not. We can say, okay, maybe objective ordering implies normativity. Maybe it doesn't. But then we can sell this debate like objective ordering, which I think Max does not. I think our most core disagreement. Max does not think objective ordering is a thing.Max: I actually. Sorry. You go, Sarah.Sarah: No, that's a good clarification. So carry on.Max: I'm not sure that's our core disagreement, but I am happy to debate that. Just in that, I would say that if I bought Aaron's thing about objective ordering, this does not make me a moral realist or something, or at least not in a really strong sense of the term. But also if I can convince Aaron that the objective ordering thing isn't the case, I guess this also works.Aaron: Actually, I want to jump in and say I feel like actual court. The reason why moral realism is the question about it is sort of important, is that people want to be able to say, "Oh, these are just like my values or whatever. I can just assert them as the fundamental truth of the matter." And that's sort of like the core. I don't want to say that's the core, but that is certainly a core, an important part of the debate, which is like, I want to say no.

Aaron: If we're discussing, once we decided that we want to try to do what's right or whatever, then it's like an investigative, not empirical, but like quasi empirical, sort of similar ish to an empirical question. We have to uncover the truth, just like assert it or whatever. And that's like a core reason why realism is important.

Aaron: But if you have someone who's arguing like, "No, I just don't care what I ought to do or whether I'm just going to reject any claims about whether I ought to act morally or not," I feel like it doesn't actually matter. Maybe in some sense it does, but in a practical sense, it doesn't matter whether they're objectively acting wrong or not. There's no good way to respond regardless of whether realism is true. You know what I mean?Max: Well, I suppose that's a slightly different thing. But in defending that or something, you kind of referenced ought statements, right?Aaron: Yeah.Max: Which I think is a bit to my point of, like, the relevant thing is here is normativity, the thing. People disagree. Assuming somebody says, I will do what I ought to morally or something like that. Right. It's not the case that an ordering of preferable states matters, unless that then implies that they need to take certain actions. Okay. Yeah, there's this ordering of pain versus suffering. Like different states of that. But why shouldn't I just blow up a city or something like that? You need to be able to tell me that or the reason.Aaron: Do I.Max: If you don't want me to blow up the city.Aaron: No, but you haven't. Right. And so neither has anybody else.Max: This debate has an action relevant thing.Aaron: No. Okay. So I disagree there because I think evidently people, it is like a lucky, I mean, not exactly lucky, but I'll just say lucky. Fact of the matter that people are, in fact, a lot of people are inclined to try to act morally. The relevant hard question is figuring out what that means, not trying to convince people whether they ought to or not.Max: Sure. But what I'm saying is that.Aaron: You'Re.Max: Saying here's this objective ordering thing or whatever. Basically, kind of what you're saying is here's this thing that actually what you're not saying is, here's this thing that leads to actions you should take. But what you're kind of trying to say is, this is the thing for you. This is the thing that leads to what actions should be taken. Like this sort of objective ordering of pain.Aaron: What I'm saying is, let's just take is given. Or if you happen to want to act morally, whatever that means, then I can lay out at least like a framework, or not like a framework, but criteria.Max: There's a question or something which is like, why shouldn't I be a. I guess it seems like you're a hedonist to some degree like that. Right? Why shouldn't I be a preference-ist? Like a preference utilitarian.Aaron: Oh, yeah. So this is another different.Max: But I can objectively.Sarah: What's a preference utilitarian?Aaron: Please somebody who's wrong.Sarah: Okay.Aaron: Go ahead.Max: Aaron thinks something like, for a world state. So like, just a way the world could be to be more preferable or to be better, I guess I should say to be better is to have more happiness in it, which roughly is like the sort of doing drugs happiness. I don't know if that's unfair to Aaron, but that is a type of.Aaron: No, that totally, yeah.Max: Okay. Do more drugs and being stabbed would make it less preferable. That could kind of sharp pain and a preference theory of welfare. You don't have to be utilitarian. Just what it says for welfare to be what welfare is for. A preference theory is that when your preferences are satisfied, that is better. So if I have a preference for being covered in chocolate, jumping into a bathtub full of chocolate is like a really good state of affairs. Whereas for Aaron it would be if I did a bunch of weed or something is like the better thing.Aaron: I mean, that's like the worst. That's like a very inefficient pharmacological route.Max: Yeah, but it is legal where am.Aaron: Okay.Sarah: So, Aaron, you think that people.Aaron: Sorry, go ahead. Okay.Sarah: Surely it's just the case that different people can achieve different states of welfare by doing different things. Everyone wants to be high all the time.Aaron: No, totally.Sarah: What's your point?Aaron: My thing is that preferences don't intrinsically matter. They certainly matter in the real world as how we actually do things or whatever, but there's no intrinsic value. Or insofar as preferences matter, they only matter because at the end of the day, they achieve, they increase happiness and decrease.Max: I mean, Aaron might not be committed to this for various ways, but like a very naive, hedonic theory of welfare. That is kind of in the drug style of it. Like, dopamine is pleasure or something like that. Might say that you shouldn't go around force feeding people drugs, but if it was the case that everybody magically became on drugs, this would actually be quite good, assuming they don't die as a result. If you just could kind of bring that state of affairs about by pushing a button, you should do that. Even though it's the case, many people have a preference not for.Sarah: This is like the AGI comes and plugs us all into weird endorphin machines, forever.Aaron: Adding in, if you could just stipulate that also, maybe this inhibits a population growth, that would be good. And maybe this is also, like an x risk in some ways, because, okay, then we can't defend against asteroids or whatever, but if you just set those aside, then, yeah.Max: And then, like, a preference theory would say, since people don't have a preference for this, this actually means it's not as good.Sarah: Okay, I get that. So if you're not a preference utilitarian, then you think that people might not actually understand what states they would experience the most happiness in, and therefore their preferences might not be the best metric for, like, people might think, oh, I would be happier if I did this thing. But they don't know what would actually make them happier. So it isn't necessarily right to try and bring about their preference.Aaron: I'm getting really tempted to decide to walk, maybe. Should I walk back?Sarah: I don't know. What's your preference?Max: I think, as, like, a vague quibble, probably a preference theorist would say, it's not so much about happiness, it's about welfare, because those are different. Whereas high welfare is like, high preference satisfaction for a preference theory. Whereas happiness might not be like, you might have a preference for making lots of paintings, but it's not the case that this makes you feel the emotion happy, or something like that.Aaron: I think that scenario max prevented was like, I bite the bullet on that and said, yeah, if everybody could be magically happy, even if they didn't want to be, that would be good. But the other side of the coin. The other side of the coin is like, okay, you can also imagine just, like, a being that really wants to suffer and be tortured, in fact. And then there's nothing incoherent about that.

Aaron: And so should we, in fact, cause this immense pain if we invent this creature? And I think, no, we shouldn't. And I think that's a pretty compelling objection to preference theories of welfare.Max: Wait, sorry, what's your objection again?Aaron: Wait, sorry. What was that?Max: What's the compelling objection?Aaron: If there's, like, a pig that wants to be tortured, should be tortured. The pig.Max: I see what you're saying. Yeah. And a preference theory might say you should.Aaron: Yeah. And I think, pretty obviously, I think it becomes more clear if you imagine that, okay, we have genetic control over the pig. We can control. You're like, choose whether to create the pig, kid. Like, I want to create the pig. And then I do. It's like, seems bad.Max: Yeah. I mean, for what it's worth, if I was a moral realist or whatever, I would not be a preference theorist, probably, but I might be a preference theorist because I'm not a realist.Aaron: This is interesting. Also, in a minute, I'm going to have to maybe not actually go, but stop being a good conversationalist or a bad conversationalist. I'm going to become an even worse conversationalist.Max: Nice.Aaron: For, like, I don't know, ten minutes at least.Max: Yeah.Aaron: So you guys can talk about. You can gossip about me.Max: Just in the middle of the podcast is gossip.Aaron: I mean, we can rearrange it.Max: Right. That's.Sarah: I can use those ten minutes to speed run developing some actual opinions. I didn't have any coming in. I still kind of don't.Max: If you want, I can give you a vague overview of the other prominent welfare theories.Sarah: Okay. That would be useful. And then I'll sort of pick one with it.Max: The third one is often something like objective list theory, which probably the easiest way to explain it is something like Aristotle's concept of eudiomia, I think, is.Sarah: How you say it.Max: So, something that is good for humans to live can be thought of as living a very fulfilling life, which is the height of welfare. Some other intuitions about this are related to the ideas of higher and lower level pleasures, as proposed by Bentham or another utilitarian philosopher. The concept suggests that it's better to be a dissatisfied human than a very happy pig.

Max: The reason for this is that although rolling around in the mud is pleasurable for a pig, there's something about being human and experiencing a wide range of emotions and experiences that might be more intense than what a pig can feel. Of course, someone might argue that pigs have their own unique experiences, but humans can do things like math and create TV shows, which are fulfilling on a higher order level. Engaging in these higher order pleasures contributes to a good human life.

Max: Living a fulfilling life involves engaging in objectively welfare-enhancing activities, such as loving well and embodying virtues like compassion and honor. Instead of welfare being limited by how many drugs one can consume or how many preferences one can fulfill, it's about living an "objectively good" human life. Many people have intuitions about this concept, and it resonates with their understanding of what it means to lead a meaningful existence.Sarah: So is the thing that makes us able to live more fulfilling lives than pigs? Does that hinge on intelligence?Max: Kind of depends who you like. The thing about the pigs is not from Aristotle, as I was taught, like Aristotle. He'd say something like, there is this thing that it is to live a very good life as a pig. And this is different from what it's.Sarah: Like to live a very good life as a human.Max: I don't know where he goes on. Like, is it more preferable to be a pig or a human? But most people think it's more preferable to be a human than a pig. And by most people, I mean most humans, not most people.Sarah: Yeah. That seems like a ridiculously difficult thing to ascertain. Fair enough. Okay.Max: And I guess the reason you might doubt this a bit is that it can be a bit elitist. Well, this isn't necessarily a reason to doubt it, but consider this: you could make an objective list theory that says it's really important to listen to opera. To be a good human and live a good human life is to kind of go to the opera and really get into it.

Max: It's like, I don't know, why would that be the case? That's kind of weird, right? You might be like, oh, to live a really good human life is to have a monogamous, straight relationship with four children or something. It's like, I don't know, why would that be the case?

Max: I mean, maybe I'm kind of just making up obviously wrong objective lists, but why is it the case that what it is to be good is to have these very specific sorts of things? Like, how do you actually go about making a list? Because a hedonist or a preference theorist can say, oh, it's by having lots of utility, like joy, or it's by having lots of fulfilling preferences. And I'm not quite sure how an objective list theorist goes about justifying their list outside of appeals to intuition or something like that.Sarah: Okay, got you. Okay, so we've got pump yourself full of drugs, follow your preferences, and make a random, arbitrary list of things that you think a good person or a fulfilled person might do, and then do those and also project that onto everyone else in a sort of, like, controlling, elitist type way.Max: I like how you've described all of them unfairly. That's not really the right word. Caricature. You've, like, caricatureized them all. That's how I describe them in private.Sarah: Cool. Now I've got to pick one. Which one of these resonates with me? I mean, I guess the preference one seems like the way that most people actually go about life. Maybe it's that one.Max: To be honest, I think it's probably all of them or something like what people think. That's why they all have obvious issues if you take them to the extreme. At the baseline, they might seem appealing. For example, what does it mean to live a good life? I don't know. Fulfilling my preferences seems pretty good. But wait, I have some bad preferences sometimes.

Max: Maybe it's to be really happy all the time. However, I don't want to be on drugs constantly. Well, perhaps it's to live a fulfilling human life. But now I don't know how to define this, and it's kind of just about defining it by my preferences. And now I'm back there, and you can go in a circle like this, around and around.Sarah: Yeah. Difficult being a person isn't.Max: Yeah.Sarah: What a.Max: You know, Aaron would say tougher bolts. The alternative?Sarah: Being a pig.Max: Well, being a person.Sarah: Being a person is preferable to being a pig.Max: No, I meant to like not being.Sarah: To not being a person.Max: To just being. Not existing.Sarah: Dead.Max: Yeah.Sarah: Right. Yeah.Max: I mean, I think it's preferable to be me than it is to be dead, but not objectively, I guess.Sarah: Yeah, this reminds me of a conversation I had with someone on Twitter when I made a poll about whether people thought that the world was net negative or not. A surprising percentage of people did, by the way, which is pressing. Someone made a point about people having a survival instinct, and if people want to survive, then surviving is good.

Sarah: Someone else responded by saying that people have preferences that don't accurately reflect the best state of the world. So, people have a survival instinct, but actually, they ought not to have a survival instinct because they think it's better to exist. But actually, they're wrong about that. That's kind of what I thought they were saying. I feel like if you've backed yourself into that corner, something deep in my gut tells me that you've gone wrong somewhere.Max: What I would say is something like what they're running into is the fact that moral realism might be fake. Right. Well, I guess this isn't strictly the case. Maybe the issue you're pointing out is what I'm describing here, though.

Max: If moral realism is correct, then there's a fact of the matter about some of this stuff. Like there's a right theory of welfare, if there's realism about welfare anyway. But if there's not a right theory of welfare, then you might be like, "Oh, the world is kind of worth it because people have this survival instinct," and someone's like, "Well, no, because of this other thing," and you're like, "Oh, that's kind of convincing."

Max: But then someone can just say, "Oh, but this other convincing thing," and you just go around in a circle forever because there's nothing to really ground yourself on since there's no actual fact of the matter.

Sarah: Well, that sounds like all of philosophy.Max: I mean, that's what I'd say.Sarah: I'm starting to get that unmoored feeling right now. It's kind of disorientating, though.Max: It is the case that you can get better at articulating and maybe even thinking what you think or something like that. Maybe you just have to be kind of careful about it or something. Maybe if you thought about a lot about what welfare is or something, you kind of get a sense about the life you want to lead or something like that. And then you just have to be a bit careful about people coming in and gaslighting isn't the right word, but maybe gaslighting you about what you want to do with your life or something like that.Sarah: Open to being gaslit as long as someone's telling me what to do.Max: I would have done that online if I were you.Sarah: Yeah, probably not. Please, I didn't mean that. Nobody gaslight me. I don't want to be gaslit. Do you want to give me any more philosophy crash courses?Max: I mean, I can. I do think it's interesting how people think about realism, and the number of people who are "right" is higher than I'd expect. I say "right" because I'm an antirealist. So, you might think that people would have a default towards realism.

Max: However, this gets technical. I think religion is a subjectivist realist theory. By subjectivist, it means that God is the one telling you what to do, and it's realist because he's real. Some philosophers may argue that religion is also antirealist, but that's debatable. Many people are religious and believe it's objectively bad to kill your mom, for example.

Max: It's interesting that I encounter people my age who think antirealism seems right, or that realism seems false after some thought. This is especially strange considering many philosophers are realists, so it might be a bit weird.Sarah: So realism is like a well subscribed position amongst professional philosophers.Aaron: Let me google that again back.Max: Hello. There is some data on this. Is this the right thing? Okay, so meta ethics, moral realism or moral antirealism? 62% accept or accept or lean towards moral realism. So 24% lean towards it, 37% accept it, and then 11.5% accept antirealism. And 14.5%. You're welcome. Sorry. Lean towards antirealism, and then, like, 12% do some other stuff, whatever that means. So, like, agnostic.Sarah: That's pretty surprising.Max: Yeah. I mean, one reason would be, like, why are you doing moral philosophy as your job if you are an antirealist? Right. Is like a relatively compelling question, especially when you could be paid more to do something else with your life. So there might be, like a selection effect to some degree.Sarah: Yeah. I feel like I wish I had done philosophy at uni instead of doing English. That probably would have been more fun, and then I would know more things and I could hold my own in a conversation. Well, okay.Max: So part of it is like, this is my hot philosophy take, I'm going to say, comes a bit from antirealism. So if you're an anti realist right. You think there's kind of no fact of the matter about or whatever you're an antirealist about. So you could be like an antirealist about art or something. So there's no objectively good art. Right.Sarah: Yeah.Max: So, whatever domain you're an anti-realist about, there's no fact of the matter. It comes down to who can squabble best or something like that. This really advantages people who have advantages, like me. I can squabble best, I think, because I've thought about this stuff a lot, know the terms, and can be confusing if I need to. Maybe I can talk somewhat well or something, so I can outmaneuver others.

Max: If you've ever seen Ben Shapiro debate, like in "Ben Shapiro owns liberals," what he's really doing isn't winning an argument. He's stumping them and making it difficult for them to respond. If you can do this in a conversation, you can just kind of win. A lot of philosophy, or conversations about philosophy, is actually about that.

Sarah: Yeah. Although the Ben Shapiro thing just depends what your algorithm is giving you, because if you're on the other side of TikTok, then you get the liberals owning Ben Shapiro. It's always just like the same interviews with different clips cut out of them or whatever.Max: I'm glad to know there are clips of that happening.Sarah: I've seen at least a few. I don't know. There was one where he went to the Cambridge union or something, and some girl yelled at him about how he's wrong about being pro life or something, and then everyone cheered. I don't know. To be honest, I obviously agreed with her. I don't think her arguments were that good. I think she was mostly just shouting at him. But given that she was right, it was still kind of satisfying to watch anyway. Yeah. I don't know. Sometimes I'm good at arguing with people, other times not. It just depends whether I'm in the zone or not. I'm kind of not right now, really.Max: What I'm saying is you should be more confident or something, and not because you're really good or something like that. I mean, that's probably true as well, but more because if there's no fact of the matter, it's better to stand your ground, usually, and rather than kind of get somebody to override what you think by just talking louder at you or something. If you kind of find it to be the case where it's like, oh, I disagree with this, but I don't know how to say it. You should just be like. You should kind of make a mental note to be like, I'm going to stick to my position, or something like that.Sarah: So you just always double down on everything, is what you're saying? Yeah, because nobody's right anyway.Max: Well, so this isn't true about some empirical things. Kind of like, if you were like, I think it's the case empirically that dogs weigh twelve pounds on average. I don't know, maybe you don't have to double down on that. If somebody's like, actually, the National Dog Institute studied this and then hints, you study, but otherwise you can double down, I give you permission.Sarah: Okay, excellent. I will be doing that.Max: It makes convincing people harder, actually.Sarah: As long as you're fine with me referring people back to you. When I'm a stubborn, belligerent pain in the ass, I can be like, well, Max told me that I was allowed to double down on every stupid take that I just came up with five minutes ago. So you can take it up with.Max: Him for business inquiries, please see my twitter.Aaron: I'm sort of back. I'm only sort of back.Max: Hello?Sarah: Okay.Aaron: Honestly, I should be banned from any podcast forever.Sarah: No, this is really funny.Max: They should make you, like, the 80k host, but you never go to the studio. You just do your errands.Aaron: No, I applied to be the CEO.Max: Did they reject you already?Aaron: Not yet. I don't think they should choose me, but I wasn't joking. You never know.Max: Nice.Sarah: It's a shame that this wasn't live when you submitted your application because it would have been such excellent evidence.Aaron: Well, I did interview Rob Wiblin and their producer Kieran Harris, like, a year ago.Sarah: You did?Aaron: Yeah. It's like the coolest thing I've ever done. And so I'm always, like, smug when I tell people.Sarah: That's so then follow you on Twitter.Max: Brag about it more, or like a resume.Aaron: It's a line on my resume.Sarah: That should be more prominent in your online presence. That's so cool.Aaron: Maybe I should put it on Pigeon Hour. Yeah, I'll send you guys. No, it's weird because it's not that big of a deal. But the other shit, it's cool. I'm not going to deny that.Sarah: That is a massive deal. What are you talking about? How did you swing that?Max: You don't get to know.Sarah: Well, I don't know. I don't know who won this argument. I don't know if you guys finished having your argument, but I would say we really did.Max: But I think you started in favor of what I thought and then Darren didn't convince you. So that's kind.Sarah: I don't think he has convinced me. I'm confused about how the objective ordering thing can exist independently of there being things that it is right or wrong to do. I still don't get that.Max: I actually can. Well, I don't know if I can give Aaron.Aaron: Hello? Yeah, I don't know. I'll just be a fly on the wall, I guess. Keep going.Max: Yeah, so think about it like this: you can rank every person based on height. That's something you can actually do. I can give you the ordering of heights of all people alive right now. However, there are two things that just giving you this ordered set of elements doesn't provide.

Max: One, it doesn't tell you what tallness is. It tells you who is taller than other people, like five foot nine is taller than five foot eight. You get that information, but you don't know who's tall because is tall 6ft? Is it 4ft? You don't know. That's an additional thing.

Max: It also doesn't tell you how tall you should want to be. It just tells you how tall you are. So, it's entirely possible to have this ordering of states of the world based on suffering or something like that, right? And it just doesn't have the next thing, which is what states of affairs are good. You can rank them, but you just don't get that information from that.Sarah: If you were like having to choose between two actions and each of them would bring about a different state of the world, and it was objectively true that one was better than the other and you knew that, even though the tallness analogy doesn't map onto that, because then it's not like you wouldn't have to determine the cutoff point of where do things stop being bad and when do they start being good. If you had two states of affairs and you knew one was better than the other, then surely you would be able to say, oh, it is right for me to do the one that brings about state b as opposed to.Aaron: State a. I think that makes a lot of sense. But if somebody was just committed to not arguing with you or just saying, you know what, I really don't care, then I don't know. That's where you get into the question of whether you have any response or whether there is a legitimate response to, well, maybe Max disagrees.Max: So I guess, first, to Sarah's point, you might say this is a bit of like, so really what Aaron might be claiming. I guess it depends on what he thinks he's claiming. He could be claiming, you can objectively rank states based on a betterness relationship, and that betterness relationship is, like, the objective one you ought to use. Or what he's saying is you can rank states based on a betterness relationship of suffering. You could also rank them based on a betterness relationship of preference function as well. Right. And so there's kind of this question about which is the right ordering.Aaron: I think there's, like, one true betterness function.Max: Okay, but why is it the one true one? I guess say that.Aaron: Oh, it's just, like, built into the meaning of, like, wait, is there a one true? So that's actually something I don't have. I think it's possible there's, like, some indeterminate ones, but there's, like, approximately one or something like that. Okay, so why is that? Yeah, it's just, like, the fact of the matter that suffering is good, happiness is bad, and nothing else is either intrinsically good or intrinsically bad. That is the short answer.Max: Um, yeah. I think this makes Sarah's question really important, though. Like, why this just.Aaron: I feel like this is sort of a bad thing for me to be arguing about, because, hey, I think it's great. Nobody wants to take this as a normative thing. I think it makes a lot of sense. It's like, okay, two thumbs up. You know what I mean? It's like, in some level, maybe. I don't think I can defend it at some intense metaphysical, fundamental level. Kind of, who cares?Max: Well, you should care, probably because it's okay. Probably important for your worldview or something would be like, okay, darn.Sarah: Okay. I feel like I'm still confused, but that's what I expected.Max: Just know that your starting intuitions were right.Sarah: I think they were right.Aaron: No, I think your starting intuitions are, like, sensible and wrong.Sarah: Sensible and wrong.Max: I remembered your thing, Aaron. The question was something like, if someone's like, oh, I don't care about what's objectively moral, good, what do you say to them? And so I guess what a philosopher would say is, you can make mistakes, right? Like, I could say, if you want to earn $100, invest in bitcoin or whatever. That's a horrible example. Sorry. If you want to earn $100. Invest in us treasury bonds.

Max: And I've done all the math or whatever, and this is just objectively the case. This is like the way you could earn $100 and you could just be like, I don't give a shit, I'm going to go invest in bitcoin instead, and then lose all your money. It's like, sure, whatever, but you've made a mistake, right? It's the case you did the wrong action.Aaron: If your goal was to bring whatever.Max: And you only have these two options available to you, this is the set of things. And I kind of, like, supposed that also investing in treasury bonds actually does net you $100.Sarah: But then isn't that, like, in that situation, the person has the goal of making more money, and in Aaron's scenario, they don't care about whether they're bringing about the best state of affairs.Max: Yeah, that's fair. I think kind of what you say there is like, what would we say about a murderer or something, I guess. Especially if you're like a realist. They don't care about the fact they shouldn't murder. They're murdering, but they're still making a mistake or something.Sarah: You might say different to, like, if someone murdered someone under the impression that by doing that they were, I don't know, say, saving five other people, that would be different to the person that murdered someone just because they didn't care whether or not that person, I think, to live or not.Max: The way to say this would be that a realist theory is kind of necessarily paternalistic or something. It's saying you need to be a certain way or something like that. And that makes it sound bad or something like that. But there are times when parents are right. Like parents are paternalistic in saying you should eat your vegetables. And in terms of health outcomes, this is true or something.

Max: And so moral theories are paternalistic about what actions you should take to better the world. And you don't have to want to better the world, I guess, but you ought to want to better the world or something. Just like we often kind of think.Aaron: Like, well, I mean, then you get into just recursion. I do feel like that's why realism.Max: Doesn't work or whatever.Aaron: I do think we're going off or not going off. But this particular question about whether the relation.Max: Sorry. One final thing is something like, I think what philosophers will often say is something like what it means to be rational or to be a rational agent, or to be a certain, whatever synonym or qualifying word, type of person, like a moral agent. Or whatever.

Max: To be like a person is to do this sort of stuff, like a sort of kantian view of ethics or whatever says by being a properly rational agent, you kind of wrong about everything. I mean, yeah, but that's the answer, right? You could say, if you're Aristotle, it'd be like to be a proper human or whatever, you kind of have to care about these sorts of things. But, yeah, I mean, you're right. You do just kind of get into a recursion thing, or you can.Aaron: I know this isn't a direct response. I just keep thinking that this sub question about what objective ordering imply, like, normativity is interesting. I just don't think it's like that. It's not like the main feel like it's like a little bit of a sideshow. And also one that's like, I feel like I'm trying to argue no, but also I'm sort of like thumbs up, no, but I'm trying to make my maximally defensible or something. Whereas if the answer was yes, that would sort of be convenient or something that would make me a full throated moral realist. You know what I mean?Sarah: But I feel like the answer just is yes. I don't get how it isn't. It just seems like. Can't speak incoherent for the answer to not be yes. So maybe you should just take the strong version, because it kind of seems like. Well, that's the logical.Aaron: I just want to ask where you think - okay, yeah - where you think. I think remembering Becca, you said you don't equate those two statements: "X would be better" and "you ought to do X." So, it seems like there's a gap between the former and the latter. The question is, how do you bridge that gap? Where do you get the extra bit of information from?

Aaron: Honestly, I look around and I don't see it. It's sort of like my juice. I'm just curious. They vibe associate. They don't merely vibe associate, but associate very strongly or something. However, there's still a little gap or something. I'm not doing a good job of explicating.Sarah: Yeah. So how do you get from X would be better than y to. You should bring about X. Maybe you're right and I am, in fact, wrong.Aaron: That was directed of Max.Max: Nice.Sarah: No, but Max, can you say something smart, please?Max: Yeah, sometimes I can.Sarah: Do you do it now to respond to that? I don't have a response. I'm getting.Max: I mean, like, it's kind of the case that some orderings imply normativity and some don't, I guess, is the case.Aaron: Like what.Max: Utility functions in the preference theory sense of that implies normativity.Aaron: That's like what we're talking about.Max: Yeah, but not all orderings imply normativity. You can order a set of numbers, and that doesn't have any normativity. And you could order colors in the world based on wavelength or something, and that doesn't tell you which colors you ought to bring about or something. Right. But an ordering of preferences based on your preference function or, sorry, states of the world based on your preference function does tell you what you should bring about or something.Aaron: No, maybe. But then there's the question of how do you get the really substantive should bit? You're an anti realist. I feel like the natural thing in an anti realist world, I think, is to say, it's like, where does this normativity come from? It's like, is there a God? Like, maybe. But setting that aside, what I would.Max: Say is there's normativity insofar as you want there to be some, like what ought you to do or want to do, like whatever you want to ought to do. Was that sufficiently smart, Sarah?Sarah: That sounded very sus, because, hang on, I'm confused again. Maybe my question was more like, I don't know. I don't have the technical language to express this, but it seems like, Aaron, what you're saying is there's like this unbridgeable gap between making a claim about states that are better than others and making a claim about a normative claim about what people ought to do. And there's like two fundamentally different categories, and it's impossible to kind of bridge the gap between them.

Sarah: But then I don't understand how if you believe that, then that seems like that would make you definitely not a moral realist, because you're basically saying that you can't use the. What's the opposite of normative? There's a word for that, right?Max: Somebody give me the word.Aaron: Sometimes it's positive.Sarah: But yeah, you can't use the descriptive claims to make the normative claims. So if you think that, then doesn't that just make you not a realist?Aaron: Well, I'm claiming that there are descriptive moral claims, and, for example, x is morally better than y.Sarah: It's like a descriptive x is better than y. Doesn't really seem like a moral claim. But you should do x seems like.Aaron: A moral claim, right?Sarah: If you're saying one thing is better than the other. But that doesn't imply anything about what you should do. To me, that's not a moral.Aaron: This is like a semantic question. Yeah, I guess I honestly don't really know how most people use the words. If I want to say, all things considered, you have these two worlds, and one is better and one is better. One is morally better. I think it would be great if people brought about the better one, but I don't think in some ultimate fundamental sense, there's this normativity thing.

Aaron: I think it's a very useful social thing to invoke. I think it would be great or whatever. Personally, I prefer that. But it's not like some fundamental truth of the universe that seems like, substantively seems like I'm talking about something that's at least related to morality.Sarah: I feel like I'm getting more and more confused the further into this conversation.Aaron: Welcome to philosophy.Max: Yeah.Sarah: This is horrible. Why do you guys do this?Aaron: Acts of getting a PhD?Max: Not actually. That's not a literally at the mean, not currently enrolled at a PhD program. But if you're on a hiring committee, feel free to hit me up.Aaron: Yes. If Wilma cast will listen to this, and I feel like there's at least a .01% chance he is, that's then plausible, I guess.Sarah: Come on. It's at least 0.2.Aaron: Yeah. If we have a campaign to get them to listen.Max: On average.Aaron: No, there's definitely a couple of philosophy people who will be listening.Sarah: Wait, will, if you're listening, I have a copy of what we owe the future under my tv, right on top of my high school musical two dvd.Max: Is that your. Thanks for the content of them by quality?Sarah: No, I would have put the dvd on top, obviously, if I was objectively ordering them.Max: Okay, yeah, inversely ordered.Aaron: I want to debate Will McCaskill. If you can get him on the podcast, that would be so sweet.Sarah: Maybe I should finish his book first. I only read.Aaron: I actually have critiques about what we are the future. And I feel like for non EA people that critique Will MacAskill, I am, like, the ultimate will MacAskill defender. But I also think he's wrong about stuff in what we are the future. So if you can get him on the pond, that would be so sweet.Max: Did you ever put that on the.Aaron: No, I only have 16 million drafts that will never see the light of day.Max: Well, if you put one of them up the chance.Aaron: Okay. I have, like, miscellaneous. It's in my list of. I have a twitter thread that's like, EA takes, and it's, like, in there.Sarah: That's it?Aaron: Yeah.Max: At some point, maybe you could just dm him the Twitter thread and say, want to go on a podcast to debate this? The next time he's on 80K, send it to Rob Woodland and just have him read out the takes.Aaron: Yeah, no, I definitely will.Max: Okay, good. I'll hold you to that. It's in writing now, but audibly written.Sarah: Cool. Well, I feel like we should wrap up soon. I have to decide who I agree with. Actually, I thought the whole point was for me to.Aaron: Oh, wait. Yes, you do. Sorry.Sarah: I thought we're going to solve this whole question.Aaron: Yes.Sarah: Just get on this.Aaron: I was wrong. You're objectively right, obviously.Sarah: I'm always. Never been wrong. Not even once.Max: Okay.Sarah: Who do I think is right?Max: I don't know.Sarah: I'm really confused. I don't know who's right.Aaron: Do you think Max has made, like, a more compelling case? Although I do feel like I've been mildly handicapped by my traveling circumstances.Max: Yeah, you probably did that because you knew I'd win no matter what.Sarah: Yeah, it seems like you guys don't really disagree about that much. So it seems like, Aaron, you think there's, like, this objective ordering thing that is true, but doesn't have any practical implications because it doesn't actually tell anyone anything about what they should.Aaron: No, no. Can I defend my. Okay, you got to add the normativity thing. I'm sorry to be annoying, but just to jump in, I think it has practical implications. Whenever somebody says, oh, I just assert that I'm a moral subjectivist, and I think that I just intrinsically value, say, human lives over other animal lives for reasons that go above and beyond, like sentience or something like that. And I want to be able to pull this out and say you're objectively. It's like the kind of thing that I'm interested.Sarah: To. I didn't mean to say that it didn't have it. It clearly does have practical implications, like the one you just named. So I'm sorry for misrepresenting you. And then, Max, you think that there is no objective ordering of worlds, and also that you can't make any normative claims about what people should or shouldn't do objectively. Objectively speaking, who do I agree with? I don't know. I don't feel qualified to make the call because my brain feels a bit scrambled.Aaron: As a reminder. Unfortunately, the world doesn't have to obey what you declare to be correct, so the stakes might be lower than you think.Sarah: Oh, I thought I was solving this issue once. And for what you're telling me it doesn't even matter what I say.Aaron: You were like, I'm not qualified. And I was like, well, maybe, but luckily the world, unfortunately, Oxford philosophy isn't going to stamp it solved for better and for worse.Sarah: Unbelievable.Aaron: I know it's bad because you have less power, but it does lower the stakes.Sarah: Okay, well, no, that's a relief. I feel a little bit less pressure now. I guess we never even really discussed how you do the world ordering or how you justify which worlds are better than others. Maybe that would have been important to get into, but let's not do that right now. I think I agree with Max.Max: No.Sarah: Sorry.Aaron: No. Can we edit in? I don't know, some bad sound effects of me, like falling or something.Max: Oh, yeah.Sarah: I don't know how to do that. If you can teach me how to.Aaron: Do that, then sure, yeah. That is one thing. Well, I can do it on my end. I don't know what you're going to do on your chromebook.Sarah: Yeah, I didn't think that through. Really? There's got to be something I can. Oh, I'll figure it out. I'll figure it out.Max: It all works.Sarah: Yeah. Okay, cool. Thanks for arguing about moral realism.Aaron: It's been really, anytime, literally just wake me up in the middle of the night. I'm, like, barely even joking.Sarah: I also think what I might have to do is sort of timestamp all the moments where I said a coherent sentence because there weren't that many of them.Max: At least four or five.Sarah: I actually don't know if I said a single coherent thing this whole time.Aaron: No, you definitely did.Sarah: Yeah.Aaron: Awesome. Not more than I was expecting, but more than I would have expected. An arbitrary person. If we just pick a random person. This is, like, much better than that.Sarah: You did better than I thought you would.Aaron: No, you did better than the fake median human.Sarah: Thank you.Aaron: In fact, probably. Definitely better than the 75th percentile. Probably, like, better than that, in fact.Sarah: Wow. High price.Aaron: Once we're getting into, like, 99.99. We can debate that.Sarah: Yeah. We can quibble over the. Quibble over whether it's 99.8 or 99.9, but I'll take that. Okay, cool. Thanks, guys.Aaron: Lovely. All right. Pip pip cheerio.Sarah: Have a good rest of your days. I'm going to go to bed. Yeah.Aaron: Okay, cool. Adios.Sarah: Cool. Bye.

Get full access to Aaron's Blog at www.aaronbergman.net/subscribe

Note: I can’t seem to edit or remove the “transcript” tab. I recommend you ignore that and just look at the much higher quality, slightly cleaned up one below.

Most importantly, follow Sarah on Twitter!

Summary

(Written by chatGPT, as you can probably tell)

In this episode of Pigeon Hour host Aaron delves deep into the world of AI safety with his guest, Sarah Woodhouse. Sarah shares her unexpected journey from fearing job automation to becoming a recognized voice on AI safety Twitter. Her story starts with a simple Google search that led her down a rabbit hole of existential dread and unexpected fame on social media. As she narrates her path from lurker to influencer, Sarah reflects on the quirky dynamics of the AI safety community, her own existential crisis, and the serendipitous tweet that resonated with thousands.

Aaron and Sarah’s conversation takes unexpected turns, discussing everything from the peculiarities of EA rationalists to the surprisingly serious topic of shrimp welfare. They also explore the nuances of AI doom probabilities, the social dynamics of tech Twitter, and Sarah’s unexpected viral fame as a tween. This episode is a rollercoaster of insights and anecdotes, perfect for anyone interested in the intersection of technology, society, and the unpredictable journey of internet fame.

Topics discussed

Discussion on AI Safety and Personal Journeys:

* Aaron and Sarah discuss her path to AI safety, triggered by concerns about job automation and the realization that AI could potentially replace her work.

* Sarah's deep dive into AI safety started with a simple Google search, leading her to Geoffrey Hinton's alarming statements, and eventually to a broader exploration without finding reassuring consensus.

* Sarah's Twitter engagement began with lurking, later evolving into active participation and gaining an audience, especially after a relatable tweet thread about an existential crisis.

* Aaron remarks on the rarity of people like Sarah, who follow the AI safety rabbit hole to its depths, considering its obvious implications for various industries.

AI Safety and Public Perception:

* Sarah discusses her surprise at discovering the AI safety conversation happening mostly in niche circles, often with a tongue-in-cheek attitude that could seem dismissive of the serious implications of AI risks.

* The discussion touches on the paradox of AI safety: it’s a critically important topic, yet it often remains confined within certain intellectual circles, leading to a lack of broader public engagement and awareness.

Cultural Differences and Personal Interests:

* The conversation shifts to cultural differences between the UK and the US, particularly in terms of sincerity and communication styles.

* Personal interests, such as theater and musicals (like "Glee"), are also discussed, revealing Sarah's background and hobbies.

Effective Altruism (EA) and Rationalist Communities:

* Sarah points out certain quirks of the EA and rationalist communities, such as their penchant for detailed analysis, hedging statements, and the use of probabilities in discussions.

* The debate around the use of "P(Doom)" (probability of doom) in AI safety discussions is critiqued, highlighting how it can be both a serious analytical tool and a potentially alienating jargon for outsiders.

Shrimp Welfare and Ethical Considerations:

* A detailed discussion on shrimp welfare as an ethical consideration in effective altruism unfolds, examining the moral implications and effectiveness of focusing on animal welfare at a large scale.

* Aaron defends his position on prioritizing shrimp welfare in charitable giving, based on the principles of importance, tractability, and neglectedness.

Personal Decision-Making in Charitable Giving:

* Strategies for personal charitable giving are explored, including setting a donation cutoff point to balance moral obligations with personal needs and aspirations.

Transcript

AARON: Whatever you want. Okay. Yeah, I feel like you said this on Twitter. The obvious thing is, how did you learn about AI safety? But maybe you’ve already covered that. That’s boring. First of all, do you want to talk about that? Because we don’t have to.

SARAH: I don’t mind talking about that.

AARON: But it’s sort of your call, so whatever. I don’t know. Maybe briefly, and then we can branch out?

SARAH: I have a preference for people asking me things and me answering them rather than me setting the agenda. So don’t ever feel bad about just asking me stuff because I prefer that.

AARON: Okay, cool. But also, it feels like the kind of thing where, of course, we have AI. Everyone already knows that this is just like the voice version of these four tweets or whatever. But regardless. Yes. So, Sarah, as Pigeon Hour guest, what was your path through life to AI safety Twitter?

SARAH: Well, I realized that a chatbot could very easily do my job and that my employers either hadn’t noticed this or they had noticed, but they were just being polite about it and they didn’t want to fire me because they’re too nice. And I was like, I should find out what AI development is going to be like over the next few years so that I know if I should go and get good at some other stuff.

SARAH: I just had a little innocent Google. And then within a few clicks, I’d completely doom pilled myself. I was like, we’re all going to die. I think I found Geoffrey Hinton because he was on the news at the time, because he just quit his job at Google. And he was there saying things that sounded very uncertain, very alarming. And I was like, well, he’s probably the pessimist, but I’m sure that there are loads of optimists to counteract that because that’s how it usually goes. You find a doomer and then you find a bunch of more moderate people, and then there’s some consensus in the middle that everything’s basically fine.

SARAH: I was like, if I just keep looking, I’ll find the consensus because it’s there. I’m sure it’s there. So I just kept looking and looking for it. I looked for it for weeks. I just didn’t find it. And then I was like, nobody knows what’s going on. This seems really concerning. So then I started lurking on Twitter, and then I got familiar with all the different accounts, whatever. And then at some point, I was like, I’m going to start contributing to this conversation, but I didn’t think that anybody would talk back to me. And then at some point, they started talking back to me and I was like, this is kind of weird.

SARAH: And then at some point, I was having an existential crisis and I had a couple of glasses of wine or something, and I just decided to type this big, long thread. And then I went to bed. I woke up the next morning slightly grouchy and hungover. I checked my phone and there were all these people messaging me and all these people replying to my thread being like, this is so relatable. This really resonated with me. And I was like, what is going on?

AARON: You were there on Twitter before that thread right? I’m pretty sure I was following you.

SARAH: I think, yeah, I was there before, but no one ever really gave me any attention prior to that. I think I had a couple of tweets that blew up before that, but not to the same extent. And then after that, I think I was like, okay, so now I have an audience. When I say an audience, like, obviously a small one, but more of an audience than I’ve ever had before in my life. And I was like, how far can I take this?

SARAH: I was a bit like, people obviously started following me because I’m freFreaking out about AI, but if I post an outfit, what’s going to happen? How far can I push this posting, these fit checks? I started posting random stuff about things that were completely unrelated. I was like, oh, people are kind of here for this, too. Okay, this is weird. So now I’m just milking it for all its worth, and I really don’t know why anybody’s listening to me. I’m basically very confused about the whole thing.

AARON: I mean, I think it’s kind of weird from your perspective, or it’s weird in general because there aren’t that many people who just do that extremely logical thing at the beginning. I don’t know, maybe it’s not obvious to people in every industry or whatever that AI is potentially a big deal, but there’s lots of truckers or whatever. Maybe they’re not the best demographic or the most conducive demographic, like, getting on Twitter or whatever, but there’s other jobs that it would make sense to look into that. It’s kind of weird to me that only you followed the rabbit hole all the way down.

SARAH: I know! This is what I…Because it’s not that hard to complete the circle. It probably took me like a day, it took me like an afternoon to get from, I’m worried about job automation to I should stop saving for retirement. It didn’t take me that long. Do you know what I mean? No one ever looks. I literally don’t get it. I was talking to some people. I was talking to one of my coworkers about this the other day, and I think I came up in conversation. She was like, yeah, I’m a bit worried about AI because I heard on the radio that taxi drivers might be out of a job. That’s bad. And I was like, yeah, that is bad. But do you know what else? She was like, what are the AI companies up to that we don’t know about? And I was like, I mean, you can go on their website. You can just go on their website and read about how they think that their technology is an extinction risk. It’s not like they’re hiding. It’s literally just on there and no one ever looks. It’s just crazy.

AARON: Yeah. Honestly, I don’t even know if I was in your situation, if I would have done that. It’s like, in some sense, I am surprised. It’s very few people maybe like one, but at another level, it’s more rationality than most humans have or something. Yeah. You regret going down that rabbit hole?

SARAH: Yeah, kind of. Although I’m enjoying the Twitter thing and it’s kind of fun, and it turns out there’s endless comedic material that you can get out of impending doom. The whole thing is quite funny. It’s not funny, but you can make it funny if you try hard enough. But, yeah, what was I going to say? I think maybe I was more primed for doom pilling than your average person because I already knew what EA was and I already knew, you know what I mean. That stuff was on my radar.

AARON: That’s interesting.

SARAH: I think had it not been on my radar, I don’t think I would have followed the pipeline all the way.

AARON: Yeah. I don’t know what browser you use, but it would be. And you should definitely not only do this if you actually think it would be cool or whatever, but this could be in your browser history from that day and that would be hilarious. You could remove anything you didn’t want to show, but if it’s like Google Chrome, they package everything into sessions. It’s one browsing session and it’ll have like 10,000 links.

SARAH: Yeah, I think for non-sketchy reasons, I delete my Google history more regularly than that. I don’t think I’d be able to find that. But I can remember the day and I can remember my anxiety levels just going up and up somewhere between 01:00 p.m. and 07:00 p.m. And by the evening I’m like, oh, my God.

AARON: Oh, damn, that’s wild.

SARAH: It was really stressful.

AARON: Yeah, I guess props for, I don’t know if props…Is the right word, I guess, impressed? I’m actually somewhat surprised to hear that you said you regret it. I mean, that sucks though, I guess. I’m sorry.

SARAH: If you could unknow this, would you?

AARON: No, because I think it’s worth maybe selfishly, but not overall because. Okay, yeah, I think that would plausibly be the selfish thing to do. Actually. No, actually, hold on. No, I actually don’t think that’s true. I actually think there’s enough an individual can do selfishly such that it makes sense. Even the emotional turmoil.

SARAH: It would depend how much you thought that you were going to personally move the needle by knowing about it. I personally don’t think that I’m going to be able to do very much. I was going to tip the scales. I wouldn’t selfishly unknow it and sacrifice the world. But me being not particularly informed or intelligent and not having any power, I feel like if I forgot that AI was going to end the world, it would not make much difference.

AARON: You know what I mean? I agree that it’s like, yes, it is unlikely for either of us to tip the scales, but.

SARAH: Maybe you can’t.

AARON: No, actually, in terms of, yeah, I’m probably somewhat more technically knowledgeable just based on what I know about you. Maybe I’m wrong.

SARAH: No, you’re definitely right.

AARON: It’s sort of just like a probabilities thing. I do think that ‘doom’ - that word - is too simplified, often too simple to capture what people really care about. But if you just want to say doom versus no doom or whatever, AI doom versus no AI doom. Maybe there’s like a one in 100,000 chance that one of us tips the scales. And that’s important. Maybe even, like, one in 10,000. Probably not. Probably not.

SARAH: One in 10,000. Wow.

AARON: But that’s what people do. People vote, even though this is old 80k material I’m regurgitating because they basically want to make the case for why even if you’re not. Or in some article they had from a while ago, they made a case for why doing things that are unlikely to counterfactually matter can still be amazingly good. And the classic example, just voting if you’re in a tight race, say, in a swing state in the United States, and it could go either way. Yeah. It might be pretty unlikely that you are the single swing vote, but it could be one in 100,000. And that’s not crazy.

SARAH: It doesn’t take very much effort to vote, though.

AARON: Yeah, sure. But I think the core justification, also, the stakes are proportionally higher here, so maybe that accounts for some. But, yes, you’re absolutely right. Definitely different amounts of effort.

SARAH: Putting in any effort to saving the world from AI. I wouldn’t say that. I wouldn’t say that I’m sacrificing.

AARON: I don’t even know if I like. No. Maybe it doesn’t feel like a sacrifice. Maybe it isn’t. But I do think there’s, like, a lot. There’s at least something to be. I don’t know if this really checks out, but I would, like, bet that it does, which is that more reasonably, at least calibrated. I wanted to say reasonably well informed. But really what it is is, like, some level of being informed and, like, some level of knowing what you don’t know or whatever, and more just like, normal. Sorry. I hope normal is not like a bat. I’m saying not like tech Bros, I guess so more like non tech bros. People who are not coded as tech bros. Talking about this on a public platform just seems actually, in fact, pretty good.

SARAH: As long as we like, literally just people that aren’t men as well. No offense.

AARON: Oh, no, totally. Yeah.

SARAH: Where are all the women? There’s a few.

AARON: There’s a few that are super. I don’t know, like, leaders in some sense, like Ajeya Cotra and Katja Grace. But I think the last EA survey was a third. Or I could be butchering this or whatever. And maybe even within that category, there’s some variation. I don’t think it’s 2%.

SARAH: Okay. All right. Yeah.

AARON: Like 15 or 20% which is still pretty low.

SARAH: No, but that’s actually better than I would have thought, I think.

AARON: Also, Twitter is, of all the social media platforms, especially mail. I don’t really know.

SARAH: Um.

AARON: I don’t like Instagram, I think.

SARAH: I wonder, it would be interesting to see whether or not that’s much, if it’s become more male dominated since Elon Musk took.

AARON: It’s not a huge difference, but who knows?

SARAH: I don’t know. I have no idea. I have no idea. We’ll just be interesting to know.

AARON: Okay. Wait. Also, there’s no scheduled time. I’m very happy to keep talking or whatever, but as soon as you want to take a break or hop off, just like. Yeah.

SARAH: Oh, yeah. I’m in no rush.

AARON: Okay, well, I don’t know. We’ve talked about the two obvious candidates. Do you have a take or something? Want to get out to the world? It’s not about AI or obesity or just a story you want to share.

SARAH: These are my two pet subjects. I don’t know anything else.

AARON: I don’t believe you. I know you know about house plants.

SARAH: I do. A secret, which you can’t tell anyone, is that I actually only know about house plants that are hard to kill, and I’m actually not very good at taking care of them.

AARON: Well, I’m glad it’s house plants in that case, rather than pets. Whatever.

SARAH: Yeah. I mean, I have killed some sea monkeys, too, but that was a long time ago.

AARON: Yes. So did I, actually.

SARAH: Did you? I feel like everyone has. Everyone’s got a little sea monkey graveyard in their past.

AARON: New cause area.

SARAH: Are there more shrimp or more sea monkeys? That’s the question.

AARON: I don’t even know what even. I mean, are they just plankton?

SARAH: No, they’re not plankton.

AARON: I know what sea monkeys are.

SARAH: There’s definitely a lot of them because they’re small and insignificant.

AARON: Yeah, but I also think we don’t. It depends if you’re talking about in the world, which I guess probably like sea monkeys or farmed for food, which is basically like. I doubt these are farmed either for food or for anything.

SARAH: Yeah, no, you’re probably right.

AARON: Or they probably are farmed a tiny bit for this niche little.

SARAH: Or they’re farmed to sell in aquariums for kids.

AARON: Apparently. They are a kind of shrimp, but they were bred specifically to, I don’t know, be tiny or something. I’m just skimming that, Wikipedia. Here.

SARAH: Sea monkeys are tiny shrimp. That is crazy.

AARON: Until we get answers, tell me your life story in whatever way you want. It doesn’t have to be like. I mean, hopefully not. Don’t straight up lie, but wherever you want to take that.

SARAH: I’m not going to lie. I’m just trying to think of ways to make it spicier because it’s so average. I don’t know what to say about it.

AARON: Well, it’s probably not that average, right? I mean, it might be average among people you happen to know.

SARAH: Do you have any more specific questions?

AARON: Okay, no. Yeah, hold on. I have a meta point, which is like, I think the people who are they have a thing on the top of their mind, and if I give any sort of open ended question whatsoever, they’ll take it there and immediately just start giving slinging hot takes. But thenOther people, I think, this category is very EA. People who aren’t, especially my sister, they’re like, “No, I have nothing to talk about. I don’t believe that.” But they’re not, I guess, as comfortable.

SARAH: No, I mean, I have. Something needs to trigger them in me. Do you know what I mean? Yeah, I need an in.

AARON: Well, okay, here’s one. Is there anything you’re like, “Maybe I’ll cut this. This is kind of, like narcissistic. I don’t know. But is there anything you want or curious to ask?” This does sound kind of weird. I don’t know. But we can cut it if need be.

SARAH: What does the looking glass in your Twitter name mean? Because I’ve seen a bunch of people have this, and I actually don’t know what it means, but I was like, no.

AARON: People ask this. I respond to a tweet that’s like, “What does that like?” At least, I don’t know, once every month or two. Or know basically, like Spencer Greenberg. I don’t know if you’re familiar with him. He’s like a sort of.

SARAH: I know the know.

AARON: He literally just tweeted, like a couple years ago. Put this in your bio to show that you really care about finding the truth or whatever and are interested in good faith conversations. Are you familiar with the scout mindset?

SARAH: Yeah.

AARON: Julia Galef. Yeah. That’s basically, like the short version.

SARAH: Okay.

AARON: I’m like, yeah, all right. And there’s at least three of us who have both a magnifying glass. Yeah. And a pause thing, which is like, my tightest knit online community I guess.

SARAH: I think I’ve followed all the pause people now. I just searched the emoji on Twitter, and I just followed everyone. Now I can’t find. And I also noticed when I was doing this, that some people, if they’ve suspended their account or they’re taking time off, then they put a pause in their thing. So I was, like, looking, and I was like, oh, these are, like, AI people. But then they were just, like, in their bio, they were, like, not tweeting until X date. This is a suspended account. And I was like, I see we have a messaging problem here. Nice. I don’t know how common that actually.

AARON: Was. I’m glad. That was, like, a very straightforward question. Educated the masses. Max Alexander said Glee. Is that, like, the show? You can also keep asking me questions, but again, this is like.

SARAH: Wait, what did he say? Is that it? Did he just say glee? No.

AARON: Not even a question mark. Just the word glee.

SARAH: Oh, right. He just wants me to go off about Glee.

AARON: Okay. Go off about. Wait, what kind of Glee are we? Vaguely. This is like a show or a movie or something.

SARAH: Oh, my God. Have you not seen it?

AARON: No. I mean, I vaguely remember, I think, watching some TV, but maybe, like, twelve years ago or something. I don’t know.

SARAH: I think it stopped airing in, like, maybe 2015?

AARON: 16. So go off about it. I don’t know what I. Yeah, I.

SARAH: Don’t know what to say about this.

AARON: Well, why does Max think you might have a take about Glee?

SARAH: I mean, I don’t have a take about. Just see the thing. See? No, not even, like, I am just transparently extremely lame. And I really like cheesy. I’m like. I’m like a musical theater kid. Not even ironically. I just like show tunes. And Glee is just a show about a glee club at a high school where they sing show tunes and there’s, like, petty drama, and people burst into song in the hallways, and I just think it’s just the most glorious thing on Earth. That’s it. There are no hot takes.

AARON: Okay, well, that’s cool. I don’t have a lot to say, unfortunately, but.

SARAH: No, that’s totally fine. I feel like this is not a spicy topic for us to discuss. It’s just a good time.

AARON: Yeah.

SARAH: Wait.

AARON: Okay. Yeah. So I do listen to Hamilton on Spotify.

SARAH: Okay.

AARON: Yeah, that’s about it.

SARAH: I like Hamilton. I’ve seen it three times. Oh.

AARON: Live or ever. Wow. Cool. Yeah, no, that’s okay. Well, what do people get right or wrong about theater kids?

SARAH: Oh, I don’t know. I think all the stereotypes are true.

AARON: I mean, that’s generally true, but usually, it’s either over moralized, there’s like a descriptive thing that’s true, but it’s over moralized, or it’s just exaggerated.

SARAH: I mean, to put this in more context, I used to be in choir. I went every Sunday for twelve years. And then every summer we do a little summer school and we go away and put on a production. So we do a musical or something. So I have been. What have I been? I was in Guys and Dolls. I think I was just in the chorus for that. I was the reverend in Anything Goes. But he does unfortunately get kidnapped in like the first five minutes. So he’s not a big presence. Oh, I’ve been Tweedle dumb in Alice in Wonderland. I could go on, but right now as I’m saying this, I’m looking at my notice board and I have two playbills from when I went to Broadway in April where I saw Funny Girl and Hadestown.

SARAH: I went to New York.

AARON: Oh, cool. Oh yeah. We can talk about when you’re moving to the United States. However.

SARAH: I’m not going to do that. Okay.

AARON: I know. I’m joking. I mean, I don’t know.

SARAH: I don’t think I’m going to do that. I don’t know. It just seems like you guys have got a lot going on over there. It seems like things aren’t quite right with you guys. Things aren’t quite right with us either.

AARON: No, I totally get this. I think it would be cool. But also I completely relate to not wanting to. I’ve lived within 10 miles of one. Not even 10 miles, 8 miles in one location. Obviously gone outside of that. But my entire life.

SARAH: You’ve just always lived in DC.

AARON: Yeah, either in DC or. Sorry. But right now in Maryland, it’s like right next to DC on the Metro or at Georgia University, which is in the trying to think would I move to the UK. Like I could imagine situations that would make me move to the UK. But it would still be annoying. Kind of.

SARAH: Yeah, I mean, I guess it’s like they’re two very similar places, but there are all these little cultural things which I feel like kind of trip you up.

AARON: I don’t to. Do you want to say what?

SARAH: Like I think people, I just like, I don’t know. I don’t have that much experience because I’ve only been to America twice. But people seem a lot more sincere in a way that you don’t really get that. Like people are just never really being upfront. And in America, I just got the impression that people just have less of a veneer up, which is probably a good thing. But it’s really hard to navigate if you’re not used to it or something. I don’t know how to describe that.

AARON: Yeah, I’ve definitely heard this at least. And yeah, I think it’s for better and for worse.

SARAH: Yeah, I think it’s generally a good thing.

AARON: Yeah.

SARAH: But it’s like there’s this layer of cynicism or irony or something that is removed and then when it’s not there, it’s just everything feels weak. I can’t describe it.

AARON: This is definitely, I think, also like an EA rationalist thing. I feel like I’m pretty far on the spectrum. Towards the end of surgical niceties are fine, but I don’t know, don’t obscure what you really think unless it’s a really good reason to or something. But it can definitely come across as being rude.

SARAH: Yeah. No, but I think it’s actually a good rule of thumb to obscure what you. It’s good to try not to obscure what you think most of the time, probably.Ably, I don’t know, but I would love to go over temporarily for like six months or something and just hang out for a bit. I think that’d be fun. I don’t know if I would go back to New York again. Maybe. I like the bagels there.

AARON: I should have a place. Oh yeah. Remember, I think we talked at some point. We can cut this out if you like. Don’t if either of us doesn’t want it in. But we discussed, oh yeah, I should be having a place. You can. I emailed the landlord like an hour before this. Hopefully, probably more than 50%. That is still an offer. Yeah, probably not for all six months, but I don’t know.

SARAH: I would not come and sleep on your sofa for six months. That would be definitely impolite and very weird.

AARON: Yeah. I mean, my roommates would probably grumble.

SARAH: Yeah. They would be like.

AARON: Although I don’t know. Who knows? I wouldn’t be shocked if people were actually like, whatever somebody asked for as a question. This is what he said. I might also be interested in hearing how different backgrounds. Wait, sorry. This is not good grammar. Let me try to parse this. Not having a super hardcore EA AI rationalist background shape how you think or how you view AI as rationality?

SARAH: Oh, that’s a good question. I think it’s more happening the other way around, the more I hang around in these circles. You guys are impacting how I think.

AARON: It’s definitely true for me as well.

SARAH: Seeping into my brain and my language as well. I’ve started talking differently. I don’t know. That’s a good question, though. Yeah. One thing that I will say is that there are certain things that I find irritating about the EA way of style of doing things. I think one specific, I don’t know, the kind of like hand ring about everything. And I know that this is kind of the point, right? But it’s kind of like, you know, when someone’s like, I want to take a stance on something, but then whenever they want to take a stance on something, they feel the need to write like a 10,000 word blog post where they’re thinking about the second and order and third and fifth order effects of this thing. And maybe this thing that seems good is actually bad for this really convoluted reason. That’s just so annoying.

AARON: Yeah.

SARAH: Also understand that maybe that is a good thing to do sometimes, but it just seems like, I don’t know how anyone ever gets anywhere. It seems like everyone must be paralyzed by indecision all the time because they just can’t commit to ever actually just saying anything.

AARON: I think this kind of thing is really good if you’re trying to give away a billion dollars. Oh yes, I do want the billion dollar grantor to be thinking through second and third order effects of how they give away their billion dollars. But also, no, I am super. The words on the tip of my tongue, not overwhelmed but intimidated when I go on the EA forum because the posts, none of them are like normal, like five paragraph essays. Some of them are like, I think one of them I looked up for fun because I was going to make a meme about it and still will. Probably was like 30,000 words or something. And even the short form posts, which really gets me kind of not even annoyed. I don’t know, maybe kind of annoyed is that the short form posts, which is sort of the EA forum version of Twitter, are way too high quality, way too intimidating. And so maybe I should just suck it up and post stuff anyway more often. It just feels weird. I totally agree.

SARAH: I was also talking to someone recently about how I lurked on the EA forum and less wrong for months and months and I couldn’t figure out the upvoting system and I was like, am I being stupid or why are there four buttons? And I was like, well, eventually I had to ask someone because I couldn’t figure it out. And then he explained it to me and I was like, that is just so unnecessary. Like, just do it.

AARON: No, I do know what you mean.

SARAH: I just tI think it’s annoying. It pisses me off. I just feel like sometimes you don’t need to add more things. Sometimes less is good. Yeah, that’s my hot take. Nice things.

AARON: Yeah, that’s interesting.

SARAH: But actually, a thing that I like that EA’s do is the constant hedging and caveatting. I do find it kind of adorable. I love that because it’s like you’re having to constantly acknowledge that you probably didn’t quite articulate what you really meant and that you’re not quite making contact with reality when you’re talking. So you have to clarify that you probably were imprecise when you said this thing. It’s unnecessary, but it’s kind of amazing.

AARON: No, it’s definitely. I am super guilty of this because I’ll give an example in a second. I think I’ve been basically trained to try pretty hard, even in normal conversation with anybody, to just never say anything that’s literally wrong. Or at least if I do caveat it.

AARON: I was driving home, me and my parents and I, unless visited, our grandparents were driving back, and we were driving back past a cruise ship that was in a harbor. And my mom, who was driving at the time, said, “Oh, Aaron, can you see if there’s anyone on there?” And I immediately responded like, “Well, there’s probably at least one person.” Obviously, that’s not what she meant. But that was my technical best guess. It’s like, yes, there probably are people on there, even though I couldn’t see anybody on the decks or in the rooms. Yeah, there’s probably a maintenance guy. Felt kind of bad.

SARAH: You can’t technically exclude that there are, in fact, no people.

AARON: Then I corrected myself. But I guess I’ve been trained into giving that as my first reaction.

SARAH: Yeah, I love that. I think it’s a waste of words, but I find it delightful.

AARON: It does go too far. People should be more confident. I wish that, at least sometimes, people would say, “Epistemic status: Want to bet?” or “I am definitely right about this.” Too rarely do we hear, "I’m actually pretty confident here.

SARAH: Another thing is, people are too liberal with using probabilities. The meaning of saying there is an X percent chance of something happening is getting watered down by people constantly saying things like, “I would put 30% on this claim.” Obviously, there’s no rigorous method that’s gone into determining why it’s 30 and not 35. That’s a problem and people shouldn’t do that. But I kind of love it.

AARON: I can defend that. People are saying upfront, “This is my best guess. But there’s no rigorous methodology.” People should take their word for that. In some parts of society, it’s seen as implying that a numeric probability came from a rigorous model. But if you say, “This is my best guess, but it’s not formed from anything,” people should take their word for that and not refuse to accept them at face value.

SARAH: But why do you have to put a number on it?

AARON: It depends on what you’re talking about. Sometimes probabilities are relevant and if you don’t use numbers, it’s easy to misinterpret. People would say, “It seems quite likely,” but what does that mean? One person might think “quite reasonably likely” means 70%, the other person thinks it means 30%. Even though it’s weird to use a single number, it’s less confusing.

SARAH: To be fair, I get that. I’ve disagreed with people about what the word “unlikely” means. Someone’s pulled out a scale that the government uses, or intelligence services use to determine what “unlikely” means. But everyone interprets those words differently. I see what you’re saying. But then again, I think people in AI safety talking about P Doom was making people take us less seriously, especially because people’s probabilities are so vibey.

AARON: Some people are, but I take Paul Cristiano’s word seriously.

SARAH: He’s a 50/50 kind of guy.

AARON: Yeah, I take that pretty seriously.Obviously, it’s not as simple as him having a perfect understanding of the world, even after another 10,000 hours of investigation. But it’s definitely not just vibes, either.

SARAH: No, I came off wrong there. I don’t mean that everyone’s understanding is just vibes.

AARON: Yeah.

SARAH: If you were looking at it from the outside, it would be really difficult to distinguish between the ones that are vibes and the ones that are rigorous, unless you carefully parsed all of it and evaluated everyone’s background, or looked at the model yourself. If you’re one step removed, it looks like people just spitting out random, arbitrary numbers everywhere.

AARON: Yeah. There’s also the question of whether P doom is too weird or silly, or if it could be easily dismissed as such.

SARAH: Exactly, the moment anyone unfamiliar with this discussion sees it, they’re almost definitely going to dismiss it. They won’t see it as something they need to engage with.

AARON: That’s a very fair point. Aside from the social aspect, it’s also a large oversimplification. There’s a spectrum of outcomes that we lump into doom and not doom. While this binary approach can be useful at times, it’s probably overdone.

SARAH: Yeah, because when some people say doom, they mean everyone dies, while others mean everyone dies plus everything is terrible. And no one specifies what they mean. It is silly. But, I also find it kind of funny and I kind of love it.

AARON: I’m glad there’s something like that. So it’s not perfect. The more straightforward thing would be to say P existential risk from AI comes to pass. That’s the long version, whatever.

SARAH: If I was in charge, I would probably make people stop using PDOOm. I think it’s better to say it the long way around. But obviously I’m not in charge. And I think it’s funny and kind of cute, so I’ll keep using it.

AARON: Maybe I’m willing to go along and try to start a new norm. Not spend my whole life on it, but say, I think this is bad for X, Y, and Z reasons. I’ll use this other phrase instead and clarify when people ask.

SARAH: You’re going to need Twitter premium because you’re going to need a lot more characters.

AARON: I think there’s a shorthand which is like PX risk or P AiX risk.

SARAH: Maybe it’s just the word doom that’s a bit stupid.

AARON: Yeah, that’s a term out of the Bay Area rationalists.

SARAH: But then I also think it kind of makes the whole thing seem less serious. People should be indignant to hear that this meme is being used to trade probabilities about the likelihood that they’re going to die and their families are going to die. This has been an in-joke in this weird niche circle for years and they didn’t know about it. I’m not saying that in a way to morally condemn people, but if you explain this to people…People just go to dinner parties in Silicon Valley and talk about this weird meme thing, and what they really mean is the ODs know everyone’s going to prematurely die. People should be outraged by that, I think.

AARON: I disagree that it’s a joke. It is a funny phrase, but the actual thing is people really do stand by their belief.

SARAH: No, I totally agree with that part. I’m not saying that people are not being serious when they give their numbers, but I feel like there’s something. I don’t know how to put this in words. There’s something outrageous about the fact that for outsiders, this conversation has been happening for years and people have been using this tongue-in-cheek phrase to describe it, and 99.9% of people don’t know that’s happening. I’m not articulating this very well.

AARON: I see what you’re saying. I don’t actually think it’s like. I don’t know a lot of jargon.

SARAH: But when I first found out about this, I was outraged.

AARON: I honestly just don’t share that intuition. But that’s really good.

SARAH: No, I don’t know how to describe this.

AARON: I think I was just a little bit indignant, perhaps.

SARAH: Yeah, I was indignant about it. I was like, you guys have been at social events making small talk by discussing the probability of human extinction all this time, and I didn’t even know. I was like, oh, that’s really messed up, guys.

AARON: I feel like I’m standing by the rational tier because, it was always on. No one was stopping you from going on less wrong or whatever. It wasn’t behind closed.

SARAH: Yeah, but no one ever told me about it.

AARON: Yeah, that’s like a failure of outreach, I suppose.

SARAH: Yeah. I think maybe I’m talking more about. Maybe the people that I’m mad at is the people who are actually working on capabilities and using this kind of jargon. Maybe I’m mad at those people. They’re fine.

AARON: Do we have more questions? I think we might have more questions. We have one more. Okay, sorry, but keep going.

SARAH: No, I’m going to stop making that point now because I don’t really know what I’m trying to say and I don’t want to be controversial.

AARON: Controversy is good for views. Not necessarily for you. No, thank you for that. Yes, that was a good point. I think it was. Maybe it was wrong. I think it seems right.

SARAH: It was probably wrong.

Shrimp Welfare: A Serious Discussion

AARON: I don’t know what she thinks about shrimp welfare. Oh, yeah. I think it’s a general question, but let’s start with that. What do you think about shrimp? Well, today.

SARAH: Okay. Is this an actual cause area or is this a joke about how if you extrapolate utilitarianism to its natural conclusion, you would really care about shrimp?

AARON: No, there’s a charity called the Shrimp Welfare Initiative or project. I think it’s Shrimp Welfare Initiative. I can actually have a rant here about how it’s a meme that people find amusing. It is a serious thing, but I think people like the meme more than they’re willing to transfer their donations in light of it. This is kind of wrong and at least distasteful.

No, but there’s an actual, if you Google, Shrimp Welfare Project. Yeah, it’s definitely a thing, but it’s only a couple of years old. And it’s also kind of a meme because it does work in both ways. It sort of shows how we’re weird, but in the sense that we are willing to care about things that are very different from us. Not like we’re threatening other people. That’s not a good description.

SARAH: Is the extreme version of this position that we should put more resources into improving the lives of shrimp than into improving the lives of people just because there are so many more shrimp? Are there people that actually believe that?

AARON: Well, I believe some version of that, but it really depends on who the ‘we’ is there.

SARAH: Should humanity be putting more resources?

AARON: No one believes that as far as I know.

SARAH: Okay. Right. So what is the most extreme manifestation of the shrimp welfare position?

AARON: Well, I feel like my position is kind of extreme, and I’m happy to discuss it. It’s easier than speculating about what the more extreme ones are. I don’t think any of them are that extreme, I guess, from my perspective, because I think I’m right.

SARAH: Okay, so what do you believe?

AARON: I think that most people who have already decided to donate, say $20, if they are considering where to donate it and they are better morally, it would be better if they gave it to the shrimp welfare project than if they gave it to any of the commonly cited EA organizations.

SARAH: Malaria nets or whatever.

AARON: Yes. I think $20 of malaria nets versus $20 of shrimp. I can easily imagine a world where it would go the other way. But given the actual situation, the $20 of shrimp is much better.

SARAH: Okay. Is it just purely because there’s just more shrimp? How do we know how much shrimp suffering there is in the world?

AARON: No, this is an excellent question. The numbers are a key factor, but no, it’s not as simple. I definitely don’t think one shrimp is worth one human.

SARAH: I’m assuming that it’s based on the fact that there are so many more shrimp than there are people that I don’t know how many shrimp there are.

AARON: Yeah, that’s important, but at some level, it’s just the margin. What I think is that when you’re donating money, you should give to wherever it does the most good, whatever that means, whatever you think that means. But let’s just leave it at that. The most good is morally best at the margin, which means you’re not donating where you think the world should or how you think the world should expend its trillion dollar wealth. All you’re doing is adding $20 at this current level, given the actual world. And so part of it is what you just said, and also including some new research from Rethink Priorities.Measuring suffering in reasonable ranges is extremely hard to do. But I believe it’s difficult to do a better job than raising priorities on that, given what I’ve seen. I can provide some links. There are a few things to consider here: numbers, times, and the enormity of suffering. I think there are a couple of key elements, including tractability.

Are you familiar with the three-pronged concept people sometimes discuss, which encompasses tractability, and neglectedness?

SARAH: Okay.

AARON: Importance is essentially what we just mentioned. Huge numbers and plausible amounts of suffering. When you try to do the comparison, it seems like they’re a significant concern. Tractability is another factor. I think the best estimates suggest that a one-dollar donation could save around 10,000 shrimp from a very painful death.

SARAH: In that sense…

AARON: You could imagine that even if there were a hundred times more shrimp than there actually are, we have direct control over how they live and die because we’re farming them. The industry is not dominated by wealthy players in the United States. Many individual farmers in developing nations, if educated and provided with a more humane way of killing the shrimp, would use it. There’s a lot of potential for improvement here. This is partly due to the last prong, neglectedness, which is really my focus.

SARAH: You’re saying no one cares about the shrimp.

AARON: I’m frustrated that it’s not taken seriously enough. One of the reasons why the marginal cost-effectiveness is so high is because large amounts of money are donated to well-approved organizations. But individual donors often overlook this. They ignore their marginal impact. If you want to see even a 1% shift towards shrimp welfare, the thing to do is to donate to shrimp welfare. Not donate $19 to human welfare and one dollar to shrimp welfare, which is perhaps what they think the overall portfolio should be.

SARAH: Interesting. I don’t have a good reason why you’re wrong. It seems like you’re probably right.

AARON: Let me put the website in the chat. This isn’t a fair comparison since it’s something I know more about.

SARAH: Okay.

AARON: On the topic of obesity, neither of us were more informed than the other. But I could have just made stuff up or said something logically fallacious.

SARAH: You could have told me that there were like 50 times the number of shrimp in the world than there really are. And I would have been like, sure, seems right.

AARON: Yeah. And I don’t know, if I…If I were in your position, I would say, “Oh, yeah, that sounds right.” But maybe there are other people who have looked into this way more than me that disagree, and I can get into why I think it’s less true than you’d expect in some sense.

SARAH: I just wonder if there’s like… This is like a deeply non-EA thing to say. So I don’t know, maybe I shouldn’t say it, but are there not any moral reasons? Is there not any good moral philosophy behind just caring more about your own species than other species? If you’re sorry, but that’s probably not right, is it? There’s probably no way to actually morally justify that, but it seems like it feels intuitively wrong. If you’ve got $20 to be donating 19 of them to shrimp and one to children with malaria, that feels like there should be something wrong with that, but I can’t tell you what it is.

AARON: Yeah, no, there is something wrong, which is that you should donate all 20 because they’re acting on the margin, for one thing. I do think that doesn’t check out morally, but I think basically me and everybody I know in terms of real life or whatever, I do just care way more about humans. I don’t know, for at least the people that it’s hard to formalize or specify what you mean by caring about or something. But, yeah, I think you can definitely basically just be a normal human who basically cares a lot about other humans. And still that’s not like, negated by changing your $20 donation or whatever. Especially because there’s nothing else that I do for shrimp. I think you should be like a kind person or something. I’m like an honest person, I think. Yeah, people should be nice to other humans. I mean, you should be nice in the sense of not beating them. But if you see a pigeon on the street, you don’t need to say hi or whatever, give it a pet, because. I don’t know. But yeah, you should be basically like, nice.

SARAH: You don’t stop to say hi to every pigeon that you see on the way to anywhere.

AARON: I do, but I know most normal people don’t.

SARAH: This is why I’m so late to everything, because I have to do it. I have to stop for every single one. No exceptions.

AARON: Yeah. Or how I think about it is sort of like a little bit of compartmentalization, which I think is like… Which is just sort of like a way to function normally and also sort of do what you think really checks out at the end of the day, just like, okay, 99% of the time I’m going to just be like a normal person who doesn’t care about shrimp. Maybe I’ll refrain from eating them. But actually, even that is like, I could totally see a person just still eating them and then doing this. But then during the 1% of the time where you’re deciding how to give money away and none of those, the beneficiaries are going to be totally out of sight either way. This is like a neutral point, I guess, but it’s still worth saying, yeah, then you can be like a hardcore effective altruist or whatever and then give your money to the shrimp people.

SARAH: Do you have this set up as like a recurring donation?

AARON: Oh, no. Everybody should call me out as a hypocrite because I haven’t donated much money, but I’m trying to figure out actually, given that I haven’t had a stable income ever. And maybe, hopefully I will soon, actually. But even then, it’s still a part-time thing. I haven’t been able to do sort of standard 10% or more thing, and I’m trying to figure out what the best thing to do or how to balance, I guess, not luxury, not like consumption on things that I… Well, to some extent, yeah. Maybe I’m just selfish by sometimes getting an Uber. That’s totally true. I think I’m just a hypocrite in that respect. But mostly I think the trade-off is between saving, investing, and giving. Beast of the money that I have saved up and past things. So this is all sort of a defense of why I don’t have a recurring donation going on.

SARAH: I’m not asking you to defend yourself because I do not do that either.

AARON: I think if I was making enough money that I could give away $10,000 a year and plan on doing that indefinitely, I would be unlikely to set up a recurring donation. What I would really want to do is once or twice a year, really try to prioritize deciding on how to give it away rather than making it the default. This has a real cost for charities. If you set up a recurring donation, they have more certainty in some sense of their future cash flow. But that’s only good to do if you’re really confident that you’re going to want to keep giving there in the future. I could learn new information that says something else is better. So I don’t think I would do that.

SARAH: Now I’m just thinking about how many shrimp did you say it was per dollar?

AARON: Don’t quote me. I didn’t say an actual thing.

SARAH: It was like some big number. Right. Because I just feel like that’s such a brainworm. Imagine if you let that actually get in your head and then every time you spend some unnecessary amount of money on something you don’t really need, you think about how many shrimp you just killed by getting an Uber or buying lunch out. That is so stressful. I think I’m going to try not to think about that.

AARON: I don’t mean to belittle this. This is like a core, I think you’re new to EA type of thinking. It’s super natural and also troubling when you first come upon it. Do you want me to talk about how I, or other people deal with that or take action?

SARAH: Yeah, tell me how to get the shrimp off my conscience.

AARON: Well, for one thing, you don’t want to totally do that. But I think the main thing is that the salience of things like this just decreases over time. I would be very surprised if, even if you’re still very engaged in the EA adjacent communities or EA itself in five years, that it would be as emotionally potent. Brains make things less important over time. But I think the thing to do is basically to compartmentalize in a sort of weird sense. Decide how much you’re willing to donate. And it might be hard to do that, but that is sort of a process. Then you have that chunk of money and you try to give it away the best you can under whatever you think the best ethics are. But then on the daily, you have this other set pot of money. You just are a normal person. You spend it as you wish. You don’t think about it unless you try not to. And maybe if you notice that you might even have leftover money, then you can donate the rest of it. But I really do think picking how much to give should sort of be its own project. And then you have a pile of money you can be a hardcore EA about.

SARAH: So you pick a cut off point and then you don’t agonize over anything over and above that.

AARON: Yeah. And then people, I mean, the hard part is that if somebody says their cut off point is like 1% of their income and they’re making like $200,000, I don’t know. Maybe their cut off point should be higher. So there is a debate. It depends on that person’s specific situation. Maybe if they have a kid or some super expensive disease, it’s a different story. If you’re just a random guy making $200,000, I think you should give more.

SARAH: Maybe you should be giving away enough to feel the pinch. Well, not even that. I don’t think I’m going to do that. This is something that I do actually want to do at some point, but I need to think about it more and maybe get a better job.

AARON: Another thing is, if you’re wanting to earn to give as a path to impact, you could think and strive pretty hard. Maybe talk to people and choose your education or professional development opportunities carefully to see if you can get a better paying job. That’s just much more important than changing how much you give from 10% to 11% or something. You should have this macro level optimization. How can I have more money to spend? Let me spend, like, I don’t know, depends what life stage you are, but if you had just graduated college or maybe say you’re a junior in college or something. It could make sense to spend a good amount of time figuring out what that path might look like.

AARON: I’m a huge hypocrite because I definitely haven’t done all this nearly as much as I should, but I still endorse it.

SARAH: Yeah, I think it’s fine to say what you endorse doing in an ideal world, even if you’re not doing that, that’s fine.

AARON: For anybody listening, I tweeted a while ago, asking if anyone has resources on how to think about giving away wealth. I’m not very wealthy but have some amount of savings. It’s more than I really need. At the same time, maybe I should be investing it because EA orgs don’t feel like, or they think they can’t invest it because there’s potentially a lot of blowback if they make poor investments, even though it would be higher expected value.

There’s also the question of, okay, having some amount of savings allows me to take higher, potentially somewhat higher risk, but higher value opportunities because I have a cushion. But I’m very confused about how to give away what I should do here. People should DM me on Twitter or anywhere they have ideas.

SARAH: I think you should calculate how much you need to cover your very basic needs. Maybe you should work out, say, if you were working 40 hours a week in a minimum wage job, like how much would you make then? And then you should keep that for yourself. And then the rest should definitely all go to the shrimp. Every single penny. All of it.

AARON: This is pretty plausible. Just to make it more complicated, there’s also the thing that I feel like my estimates or my best guesses of the best charities to give to over time has changed. And so there’s like two competing forces. One is that I might get wiser and more knowledgeable as time goes on. The other one is that in general, giving now is better than giving later. All else equal, because I think for a couple of reasons, the main one just being that the charities don’t know that you’re going to give later.

AARON: So it’s like they can plan for the future much better if they get money now. And also there’s just higher leverage opportunities or higher value per dollar opportunities now in general than there will be later for a couple of reasons I don’t really need to. This is what makes it really complicated. So I’ve donated in the past to places that I don’t think, or I don’t think even at the time were the best to. So then there’s a question of like, okay, how long do I save this money? Do I sit on it for months until I’m pretty confident, like a year.

AARON: I do think that probably over the course of zero to five years or something, becoming more confident or changing your mind is like the stronger effect than how much good you give to the, or how much better it is for the charities to give now instead of later. But also that’s weird because you’re never committing at all.Sometimes you might decide to give it away, and maybe you won’t. Maybe at that time you’re like, “Oh, that’s what I want. A car, I have a house, whatever.” It’s less salient or something. Maybe something bad happened with EA and you no longer identify that way. Yeah, there’s a lot of really thorny considerations. Sorry, I’m talking way too much.

SARAH: Long, are you factoring AI timelines into this?

AARON: That makes it even more sketchy. But that could also go both ways. On one hand, you have the fact that if you don’t give away your money now and you die with it, it’s never going to do any good. The other thing is that it might be that especially high leverage opportunities come in the future or something potentially you need, I don’t know, whatever I can imagine I could make something up about. OpenPhil needs as much money as it can get to do X, Y and Z. It’s really important right now, but I won’t know that until a few years down the line. So just like everything else, it doesn’t neatly wash out.

SARAH: What do you think the AGI is going to do to the shrimp? I reckon it’s probably pretty neat, like one shrimp per paperclip. Maybe you could get more. I wonder what the sort of shrimp to paperclip conversion rate is.

AARON: Has anyone looked into that morally? I think like one to zero. I don’t think in terms of money. You could definitely price that. I have no idea.

SARAH: I don’t know. Maybe I’m not taking this as seriously as I should be because I’m.

AARON: No, I mean, humor is good. When people are giving away money or deciding what to do, they should be serious. But joking and humor is good. Sorry, go ahead.

SARAH: No, you go ahead.

AARON: I had a half-baked idea. At EA Global, they should have a comedy show where people roast everybody, but it’s a fundraiser. You have to pay to get 100 people to attend. They have a bidding contest to get into the comedy show. That was my original idea. Or they could just have a normal comedy show. I think that’d be cool.

SARAH: Actually, I think that’s a good idea because you guys are funny. There is a lot of wit on this side of Twitter. I’m impressed.

AARON: I agree.

SARAH: So I think that’s a very good idea.

AARON: Okay. Dear Events team: hire Aaron Bergman, professional comedian.

SARAH: You can just give them your Twitter as a source for how funny you are, and that clearly qualifies you to set this up. I love it.

AARON: This is not important or related to anything, but I used to be a good juggler for entertainment purposes. I have this video. Maybe I should make sure the world can see it. It’s like a talent show. So maybe I can do that instead.

SARAH: Juggling. You definitely should make sure the world has access to this footage.

AARON: It had more views than I expected. It wasn’t five views. It was 90 or something, which is still nothing.

SARAH: I can tell you a secret right now if you want. That relates to Max asking in the chat about glee.

AARON: Yes.

SARAH: This bit will also have to edit out, but me having a public meltdown over AI was the second time that I’ve ever blown up on the Internet. The first time being. I can’t believe I’m telling you this. I think I’m delirious right now. Were you ever in any fandoms, as a teenager?

AARON: No.

SARAH: Okay. Were you ever on Tumblr?

AARON: No. I sort of know what the cultural vibes were. I sort of know what you’re referring to. There are people who like Harry Potter stuff and bands, like Kpop stuff like that.

SARAH: So people would make these fan videos where they’d take clips from TV shows and then they edit them together to music. Sometimes people would edit the clips to make it look like something had happened in the plot of the show that hadn’t actually happened. For example, say, what if X character had died? And then you edit the clips together to try and make it look like they’ve died. And you put a sad song, how to save a life by the fray or something, over the top. And then you put it on YouTube.

AARON: Sorry, tell me what…"Hat I should search or just send the link here. I’m sending my link.

SARAH: Oh, no, this doesn’t exist anymore. It does not exist anymore. Right? So, say if you’re, like, eleven or twelve years old and you do this, and you don’t even have a mechanism to download videos because you don’t know how to do technology. Instead, you take your little iPod touch and you just play a YouTube video on your screen, and you literally just film the screen with your iPod touch, and that’s how you’re getting the clips. It’s kind of shaky because you’re holding the camera anyway.

SARAH: Then you edit together on the iMovie app of your iPod touch, and then you put it on the Internet, and then you just forget about it. You forget about it. Two years later, you’re like, oh, I wonder what happened to that YouTube account? And you log in and this little video that you’ve made with edited clips that you’ve filmed off the screen of your laptop to ‘How To Save Life’ by The Fray with clips from Glee in it, has nearly half a million views.

AARON: Nice. Love it.

SARAH: Embarrassing because this is like, two years later. And then all the comments were like, oh, my God, this was so moving. This made me cry. And then obviously, some of them were hating and being like, do you not even know how to download video clips? Like, what? And then you’re so embarrassed.

AARON: I could totally seem it. Creative, but only a reasonable solution. Yeah.

SARAH: So that’s my story of how I went viral when I was like, twelve.

AARON: It must have been kind of overwhelming.

SARAH: Yeah, it was a bit. And you can tell that my time, it’s like 20 to eleven at night, and now I’m starting to really go off on one and talk about weird things.

AARON: Like an hour. So, yeah, we can wrap up. And I always say this, but it’s actually true. Which is that low standard, like, low stakes or low threshold. Low bar for doing that in recording some of the time.

SARAH: Yeah, probably. We’ll have to get rid of the part about how I went viral on YouTube when I was twelve. I’ll sleep on that.

AARON: Don’t worry. I’ll send the transcription at some point soon.

SARAH: Yeah, cool.

AARON: Okay, lovely. Thank you for staying up late into the night for this.

SARAH: It’s not that late into the night. I’m just like, lame and go to bed early.

AARON: Okay, cool. Yeah, I know. Yeah, for sure. All right, bye.

Get full access to Aaron's Blog at www.aaronbergman.net/subscribe

* Follow ⁠Max on Twitter⁠

* And read his ⁠blog⁠

* Listen here or on Spotify or Apple Podcasts

* RIP Google Podcasts 🪦🪦🪦

Summary

In this philosophical and reflective episode, hosts Aaron and Max engage in a profound debate over the nature of consciousness, moral realism, and subjective experience. Max, a skeptic of moral realism, challenges Aaron on the objective moral distinction between worlds with varying levels of suffering. They ponder the hard problem of consciousness, discussing the possibility of philosophical zombies and whether computations could account for consciousness. As they delve into the implications of AI on moral frameworks, their conversation extends to the origins of normativity and the nonexistence of free will.

The tone shifts as they discuss practical advice for running an Effective Altruism group, emphasizing the importance of co-organizers and the balance between being hospitable and maintaining normalcy. They exchange views on the potential risks and benefits of being open in community building and the value of transparency and honest feedback.

Transitioning to lighter topics, Max and Aaron share their experiences with social media, the impact of Twitter on communication, and the humorous side of office gossip. They also touch on the role of anonymity in online discussions, pondering its significance against the backdrop of the Effective Altruism community.

As the episode draws to a close, they explore the consequences of public online behavior for employment and personal life, sharing anecdotes and contemplating the broader implications of engaging in sensitive discourses. Despite their digressions into various topics, the duo manages to weave a coherent narrative of their musings, leaving listeners with much to reflect upon.

Transcript

AARON: Without any ado whatsoever. Max Alexander and I discuss a bunch of philosophy things and more.

MAX: I don't think moral realism is true or something.

AARON: Okay, yeah, we can debate this.

MAX: That's actually an issue then, because if it's just the case that utilitarianism and this an axiology, which is true or something, whether or not I'm bothered by or would make certain traits personally doesn't actually matter. But if you had the godlike AI or like, I need to give it my axiological system or something, and there's not an objective one, then this becomes more of a problem that you keep running into these issues or something.

AARON: Okay, yeah, let's debate. Because you think I'm really wrong about this, and I think you're wrong, but I think your position is more plausible than you think. My position is probably. I'm at like 70%. Some version of moral realism is true. And I think you're at, like, what? Tell me. Like, I don't know, 90 or something.

MAX: I was going to probably 99% or something. I've yet to hear a thing that's plausible or something here.

AARON: Okay, well, here, let's figure it out once and for all. So you can press a button that doesn't do Nick. The only thing that happens is that it creates somebody in the world who's experiencing bad pain. There's no other effect in the world. And then you have to order these two worlds. There's no normativity involved. You only have to order them according to how good they are. This is my intuition pump. This isn't like a formal argument. This is my intuition pump that says, okay, the one without that suffering person and no other changes. Subjectively, not subjectively. There's a fact of the matter as to which one is better is, like, not. I mean, I feel like, morally better and better here just are synonyms. All things considered. Better, morally better, whatever. Do you have a response, or do you just want to say, like, no, you're a formal argument.

MAX: What makes this fact of the matter the case or something like that?

AARON: Okay, I need to get into my headspace where I've done this or had this debate before. I do know. I'll defer to Sharon Roulette not too long ago, like ADK podcast guest who basically made the case for hedonic moral realism and hedonic value being the one thing that intrinsically matters and a moral realist view based on that. And I basically disagree with her. Okay. It's like settling in now. Yeah. So it is just the fact of the matter that pleasure is moral is good. And if you say that's not true, then you're wrong and pain is bad. And if you say that that's not true, you're just wrong. That's kind of the argument. That's it. And then I can build on top of it. Where do you get ordering of the world from? But that's the core of the argument here.

MAX: Yeah. I think you need an explanation for why this is the fact of the matter or something.

AARON: Okay. I mean, do I need an explanation for why one equals one or something like that? Do you need an explanation?

MAX: Yes, I think yes. Really? Because we take this to be the case or something, but the symbols one plus one equals two or something is like by itself not true or something. It's like just a bunch of lines, really, or something. Like there's all these axioms and things we build on with the mathematical system and you could do other ones. There are like a bunch of other systems.

AARON: I guess if you're a true epistemological nihilist and you think there are no statements that are true, then I'm probably not going to convince you. Is that the case for you?

MAX: I don't think it's the case that there aren't things that are true or something.

AARON: Do you think there's anything that is true? Can you give me an example? Is there a bed behind you?

MAX: I'll say yes, but that's probably couched. You could probably, I think, represent the universe or something as a bunch of a matrix of atom positions or subatomic particle positions and these things, and maybe the rules that govern the equations that govern how they interact or something.

AARON: Yeah, I agree.

MAX: Make claims that are truth valuable based on that matrix or something. And then you could be like, we can then draw fuzzy concepts around certain things in this matrix and then say more true and true things or whatever.

AARON: So I think our disagreement here is that maybe, I don't know if it's a disagreement, but the hard problem of consciousness introduces the fact that that description of the world is just not complete. You have subjective experience also. Are you a phenomenal realist?

MAX: What's the definition of that again?

AARON: So you think qualia is like a real legit thing?

MAX: Is qualia just experience or something?

AARON: I'm sorry, I feel like I'm just depending, assuming that, you know, every single term that has ever been used in philosophy. I feel like the thing that I want to say is qualia is like. Yeah, I'll say it's like subjective experience, basically. But then people will say, like, oh, qualia exists, but not in the sense that people normally think. And. But I want to use the strong version of qualia that people argue about. It's real. It's, like, genuine. There's no illusions going on. There's no such thing as functional qualia. If there's functional pain, it is like a different thing than what people mean when they say pain. Most people mean when they say pace and pain. Most of the time, there's, like, a real, genuine, legit subjective experience going on. Do you think this thing is real?

MAX: I would say yes, but it does seem like what I would say subjective experiences or something is like a type of computation or something.

AARON: I actually lean towards functionalism. Are you familiar with functionalism and theory of philosophy of mind or whatever?

MAX: Yeah. Wouldn't that just be the. I think that's what I said. Right. It's just computations or whatever.

AARON: So I'm actually not super sure about this. So I apologize to the philosophical community if I'm getting this wrong, but my sense is that when people say functionalism, sometimes they mean that, as in not exactly empirical, but sort of empirical fact of the world is that if you have some computations, you get Qualia, and other people mean that they are just identical. They are just the same thing. There's no, like, oh, you get one and you get the other. They're just the same thing. And I think to say that computations are identical just mean the same thing as qualia is just not true, because it's at least conceivable that. Tell me if you disagree with this. I claim that it's at least conceivable that you have computers that do some sort of computation that you hypothesize might be conscious, but they are not, in fact, conscious. And I think this is conceivable.

MAX: Yeah, I agree. Though I would say when it is the case, something, it doesn't seem that means there's something besides just computations going on, or, like, the specific type of computations, like, really, what you said there is. It's conceivable you could have computations that.

AARON: Look like we want. Okay, yeah, sorry. You're right. Actually, what I mean is that. Sorry, not what I mean, but what I should have said is that it is conceivable that functionalism is false. Meaning that you can get. Meaning that you can have two sets of systems doing computations, and one of them has qualia and the other one does not. One of them is conscious, the other one is not. Do you think this is conceivable?

MAX: Well, do you mean like identical computations or something? I think that'd be the necessary thing or something, because my computer is doing computations right now, but I don't think it's conscious. And I'm doing computations right now, and I do think I'm conscious.

AARON: So if you think that you could run your brain's program on an arbitrarily powerful large computer, do you think it's conceivable that that hypothetical computer would not have conscious experience?

MAX: I do, but I think this is like the real question. Or the reason I would say it's conceivable that's not having conscious experience is because I would think your simulation just isn't doing the right sort of thing. You think it is, but it's not. For whatever reason, carbon and atoms have interactions that we didn't realize, but they do.

AARON: Yeah, actually, this is a good point. I'm actually very sympathetic to this. When people say, I feel like functionalism and I forget what the term is, but substance based, at some point you're just going to get down to like, oh, no, it needs to be like, yeah, if you really want the quarks to be doing the exact same things, then you're just getting two identical physical systems. Physical. I don't like that word so much, but whatever, I'll use it. I think we just reinvented the zombies thing. Like, are zombies conceivable? And I claim yes.

MAX: I would think no, or something. Okay, I guess I don't know the P zombie example super well, but I would guess at a certain level of your ability to, like, if you knew all the physical things about the world, like all the physical things that were knowable, p zombies would not be possible for something like you'd be able to tell.

AARON: But are they conceivable? I think that's the crux. I think.

MAX: I think P zombies are like an epistemic problem or something. Right? Is probably what I would say. It's a lack of knowledge or something. Like if you knew all the relevant things and you would be able to.

AARON: Tell, maybe, yeah, I think the fact that it's an epistemic problem is like a demonstration of its conceivability. I don't think it's conceivable that there's like, wait, what's a good example of something that's inconceivable? I feel like they're all very abstract, like a circular square or something. But I don't have a better example at hand. But no, one thing is, you don't know that I'm. I guess you know that you're a conscious, right. But, like, it's really, like, if you just stipulate that there's another version of you that is, like, yeah, in another everettian branch, maybe that's not a good example. I don't know. Because. I don't know. I feel like it's just weird with physics, but as close as you could possibly make it to you in just, like, whatever way, just, like, throw in all the stipulations you want. Do you think there's any chance it's not conscious, but you are?

MAX: I guess I'm pretty confident it would.

AARON: Be conscious or something.

MAX: Like, Spock Teleporter thing. Yeah, I think we're probably both conscious. Or I was wrong about me being conscious in the first place, I guess. Or there is a very small chance that's incredibly. Probably not worth mentioning. But I'm already mentioning it, that I've lost my mind and the whole universe is just my brain making shit up or whatever. And so it's all just like, I'm.

AARON: The only real thing connecting this back to moral realism. Yes. My original claim was that the matrix of quarks and positions and physical laws is just not a complete description of the universe. And I think that you want to say that it is and that valence. So, like, valence is just encoded in that description somewhere.

MAX: Yeah. Or, like, the thing you're interested in is, like, because you have to zoom in or something. I don't know if you've heard of Spinoza's God or something, but he got excommunicated from Judaism for this.

AARON: I didn't even know you could get excommunicated.

MAX: I didn't know either, but he did, and they still aren't over it. Somebody wrote a letter to someone high up in whatever jewish thing, and they were like, documentary, and they're like, we'll never talk about Spinoza. So his claim is, like, God is the universe. So if the totality of all physical interactions and matter is, like, the universe, he probably won't put it that way, but that's his idea. And so if that's true, the rock over there is not conscious, but I am. And so it's like a merger. Property of a smaller part of a system is consciousness or something. It's not just like the universe being conscious. Yeah, I would say that it is. The smaller. It's just a bunch of interactions is what Wally is or whatever.

AARON: Yeah. I don't think, unfortunately, that we're going to solve this year. I actually do know we disagree on. Now, more fundamentally, I don't know what the appropriate term is, but I think that you're wrong about Valence just being, I guess, logically implied by the whole mathematical thing, like all the quirks and physical laws. You think it's like a logical necessity that falls out of that, that valence is real and happens, in fact.

MAX: I don't know if logical. I don't know enough logic, I guess, to say this is the case, but it is like given the physical rules governing our universe and all the.

AARON: Oh, no. You know, but they're not given. That was a part of the description.

MAX: What do you mean?

AARON: The description of how quirks interact, or like physical laws or whatever is like part of the model that I'm referring to. And I think you want to say that given this whole picture of the physical world, and I'll just use physical to mean what most people think as physical. And yet the thing that science is concerned with, just things that are subject to observation and causality and measurable causality, I guess you think that it's like, okay, we have a whole picture of the universe under this world, under this view, and then it just is definitely the case, given all this, that I know that you're sentient. I'm sentient. Both sentience and dalence is like, wherever it is happening is just like directly implied by this whole description of the universe or whatever.

MAX: Yeah. Or like, insofar as it does exist or something.

AARON: Yeah. Okay. Unfortunately, I don't think we're going to resolve this debate, but I do think this is like the crux of disagreement.

MAX: I think probably that sort of approach that I take you to be taken is probably the most convincing way one could go about doing it.

AARON: Yeah, I'm not convinced of it, but.

MAX: I do think this is the case.

AARON: Can I get you to 2%?

MAX: Because I could give you this example. Right. And what you're saying maybe gets you out of it or something, or can get you more out of it than someone else would be able to. And so, like a typical way you might go about saying moral realism is true or something, there's like this book, or it's like a series of lectures, they turn into a book by Christine Korsgaard, I think is her name. She's like a know.

AARON: I don't believe she's a.

MAX: She is called kantian. I don't know if she's literally a.

AARON: Okay.

MAX: But it uses whatever and like the idea is sort of the way you get normativity, which is the thing that tells you. You can imagine the space of possible ethical systems or something like, there's a bunch. One tells you to stab everyone. We probably shouldn't do that. And the way you get to pick which one is normativity that tells you what you ought to do. And the way you get normativity is like, it follows from human reason or something like this, right? We kind of have this natural reason, whatever Kant called it. He used a different term, I think. And you can make these arguments, and it follows from this, right? And then the question is like, well, what if there are aliens? Right? And then their reason leads them to pick a different ethical theory out of the available ones they could have picked. And these two ethical theories clash with each other. They say they do different things. Who gets to win that clash or something? Like, who ought to. You need now a meta normative theory or something. And I think what your response would be is like, oh, the answer is in qualia or whatever, or something.

AARON: No, I don't think normativity is. I'm actually not super sure about this, but I think, yeah, when I say moral realism, I mean, as far as I know, I'm, like, the only person who espouses this. I'm sure that's not true. There's definitely, like, 10,000 phds, but I don't know who they are. But my claim is that objective ordering of worlds exists, normativity per se. Not necessarily.

MAX: I would say that. Okay, it's not clear what the difference is here or something, because earlier I kind of claimed that ethical theories are just isomorphic to orderings of possible worlds or something.

AARON: More formally, the claim, it would be good if x happened, or even it would be good if you did x sometimes has an objective truth value, true or false. But the claim, like, you should do x. I don't know. I'm actually pretty divided. Or not, I don't know. But it seems like there's, like, an extra thing being added there when you go from, it would be good if you did x objectively to, you should objectively do x. And I kind of not willing to make that jump.

MAX: I guess the problem is you have to or something at a certain point.

AARON: Or it's like.

MAX: That'S fair, what you get out of being like, there's this objective ordering or something, but it's not the case you have to do anything with. It's just like, I have this thing here.

AARON: Oh, no, actually, yeah, this actually makes a lot of sense, I guess. I haven't thought about that much. Yeah, you might think, okay, if you're a moral realist, that includes normativity, but you don't think there's any form of divine punishment, then maybe just we're describing the same thing and we're just using different words or something like that. Because. Yeah. It's like, okay, I don't know. At the end of the day, no one's going to make you do it. There's no punishment. You really should or something. And I want to claim you really should. Well, it would be good if you did x. You should. Maybe you should, but I don't know if you objectively should or something. Yeah, maybe this is just the same thing.

MAX: Yeah. It's, like less clear where we disagree then. Because I might be willing to say you have a preference function. Right. Maybe you could call it objective. It emerges objectively from the fact that there's an errand, that there's this betterness ordering of possible worlds based on the.

AARON: Things for me or in my judgment of the world. Okay. Yeah, sure.

MAX: And then I could say that. Right. And if that's what we're calling moral realism, sure. But probably what people really mean by moral realism is like, you have one of those two orderings, whichever one.

AARON: Oh, no, that's not what I mean by moral realism. That's a psychological fact that is very contingent. What I mean is that one of those is true. My claim is that, in fact, my ordering of worlds. Well, there's two claims. One of them is, like, the normative ethical position, which is like, oh, I think it's true. But the other one is that, okay, conditional on it being true, it is objectively true. I feel like. Like kind of lost in word salad now.

MAX: Yeah, I guess what is objectively true here? Mean or something at this man.

AARON: Yeah. I don't know.

MAX: Just like, is I'm trying to remember my McDowell or whatever and my Mackie. These are like people who wrote a bunch of articles against each other about this sort of objective, subjective thing.

AARON: Maybe we should find another thing to discuss, argue about. I don't know. Do you have any ideas? It can be something like, really not deep.

MAX: I mean, the other thing I'm almost certain is true is like, there's no free will or something.

AARON: Yeah. Nobody thinks that that's not true.

MAX: People really like free will. Even people who don't, who think free will is false. Like free will.

AARON: Oh, yeah. I mean, I'm kind of sympathetic to what's the word for it. Yeah. I think that compatibilism has like some. There's like, no, hold on, hold on. Before you block me on Twitter. I don't think it's like saying anything fundamental about the universe or like the nature of reality in the way that moral realism is. I do think it's like a useful semantic distinction to make between. I don't know if useful is like maybe something like slightly stronger than useful, but the sense in which, okay, I can choose to pick up my phone here, even though there's no libertarian free will that is meaningfully different than somebody like my dad running over here and making me do that physically or something like that.

MAX: I think you would just call that like coerced or uncoursed or something. I mean, the reason I say that is.

AARON: Yeah, sure.

MAX: When you say free will, most people think libertarian free will. And that carries things with it that compatibilism doesn't or something. And people basically just.

AARON: Bailey. Yes, people with this. I agree with this. I think nobody. Yeah, basically nobody. Compatibilism is like a fake philosopher thing. Not entirely fake, but like basically fake philosopher thing.

MAX: Or it's used to justify things that shouldn't be justifiable or something like that.

AARON: Yeah. Although honestly, I kind of like it as a psychological crutch. So I'm just going to keep on doing that. Well, you can't stop me. Okay. Sorry.

MAX: That's true. I could if I put like a button in your brain or a tumor or whatever.

AARON: Yeah. But you can't do that, at least for now. I hope you'll choose not to if you do get the ability to do that. Yeah. Okay. Are there any other deep philosophy takes before, I don't know, say something else?

MAX: No, I think you can move it.

AARON: No. Okay. There's actually nothing particular on my mind. As usual, I did not prepare for this conversation. So sorry about that. Yeah. So do you want to tell me your life story or like something interesting about your life story that you want to discuss on a podcast? Not necessarily the whole thing, an arbitrary part, but is there. And one of these could be your experience in college, like running an EA group, for example.

MAX: I guess I could give advice to EAA group organizers or something.

AARON: Yes, do that.

MAX: One is get co organizers because it's really difficult to do alone, especially if you're a certain type of person, which I am, which is like, I don't like sending emails. I can do it kind of, but I'll put them off sometimes too much and sometimes you're just like I have other things to do. I can't make this presentation. And if you have to do literally all of it yourself and all the tabling and these things, it can get demoralizing and just difficult to do. So if you're like, oh, I should do this high impact thing. Start a K group or something. Make sure you have other people with you and that they're like, the value aligned is probably the wrong way to say it, but committed or something. Or committed enough to put in at least the amount of time you're putting in or something.

AARON: No, I totally agree with this. Even though I'm like a much less, I feel like committed community builder. I did have my stint in college and luckily I had a very good type, a co organizer who's also extroverted. I'm like me. So this was very, extremely good.

MAX: Yeah, I think probably so there's kind of this community sentiment now. Maybe, or like maybe where we are, I should say in the community, people should be more normal. Why can't you just be normal or something like this?

AARON: I'll represent the other position here.

MAX: Well, I'm going to say something like, I think that's kind of bad to some degree. I think what you should do is be hospitable or something, which maybe means isn't that much different. But I think, I think lots of weird things. I may even say lots of weird things a lot of the time and these sorts of things, but one can be nice and understanding and welcoming while doing this. And that probably does mean there are certain types of weird you can't do. Like if you're a group organizer, probably don't start a group house with your members or something. I guess though if you're in college, maybe it's slightly different because dorms are kind of like a weird space. And is it bad if a bunch of EA's live in the same dorm or friends live in the same room? I don't know. That's fine. But if you're a group organizer, probably be careful about flirting with your members or something. Don't do power imbalances, I guess is the thing, but I think it's okay.

AARON: To be like, don't do power. Kids. Do not do power imbalances. Okay. No, I agree with all this. Sorry, keep going.

MAX: And maybe sometimes this does mean you shouldn't always say what you think is the case. Probably at the first EA meeting you don't go like, I think utilitarianism is true and we should donate the US budget to shrimp or something.

AARON: I don't think that's true, actually. Only 90%.

MAX: Okay. Yeah, see, that's reasonable. But most people will need room. And I think it is the case that part of letting people explore things and think for themselves is not weighing them down with what you think or something. And so you can be pushed back when they think things to help. Basically, you might think there are certain objectively true philosophy things. I don't necessarily think there's philosophy stuff, but you might take this to be the case. Good philosophy professor will not indoctrinate or something their students or very strongly convince them of this thing. They'll give them the space to sort of think through things rather than just tell them the right answer or something. And good group organizing looks like this too, I think. Especially when you're onboarding new eas or something.

AARON: Yeah, I largely agree with this. I do think this can actually, and I'm not sure if we. Tell me what your thoughts are on this. I feel like people are going to say, oh, Aaron's pro indoctrination. I am not pro indoctrination. I love me a good critic. Everybody should write to me and tell me what I'm really wrong about. But I actually think that just being kind of upfront about what you think is actually sort of. Sorry, let me back up. Sometimes what people will go really hard on the don't indoctrinate and what that looks like is kind of refusing to let on what they think is the case on a forgiven subject. And I think this is actually not necessarily, I don't want to say literally in all cases that I can possibly imagine, but basically, yeah, as a rule of thumb, no, you should be okay telling people what you think and then if it's in fact the case that people disagree, especially in the EA context, in the EA community say that too. But yeah, I actually think community builders, this is a generalization that I'm not super confident in, but we think that community builders are a little too hesitant to just say like, oh yeah, no, I think this is true. Here's why. These are some of the other plausible views. Tell me why I'm wrong.

MAX: Yeah, I think my guess would be that's maybe true in some sense, but there's like a sort of line you want to walk or something, and being on one side of the line could be very bad or lead to certain issues that I don't read the EA form that much. I'm a fake EA, but those are doing community building better or something post where people I think kind of the takeaway. I might have the title wrong, but it seems like the takeaway of the post I'm thinking of is like, epistemics are sometimes bad in community things or something.

AARON: Was this kind of recent ish?

MAX: Yeah, I think it's like people are way too into AI or something.

AARON: Okay. I think this is like a good faith criticism that's absolutely wrong. And I actually have a twitter thread that I'll like. Maybe I'll actually, I'll see if I can find it right now, but if I can, like 2 seconds. But yeah, wait, keep going.

MAX: Yeah, I mean, like, what I would say is there are failure modes that look like that or something, and you might want to avoid it for various reasons or. Yeah.

AARON: Yes. Also, empirically, I did get like one round of feedback that was like, oh, no, thanks, Aaron, for not being so coy. So maybe I'm going hard. I'm like, the fact that I got ten people or whatever, I think probably.

MAX: It'S like you'd expect it to be a dynamic system or something. Like the amount of coinness you should have goes down as number of sessions you've interacted with this person go up or something like that. The way to do there's not a catch all for how to run a discussion group or something. It's kind of based on the people you have in your group or something. And so I think you do actually want more tibbinness towards the start. You want to be more normal, more. Not just saying what you think is the case, or something like the first session to get people warmed up, kind of understand where their boundaries are or something like this. Boundaries probably isn't the right word, but how they are, because it can be intimidating. I think if the person, the group organizer is like, this is what I think is the case, tell me what is wrong. You just might not want to say why it's wrong. Because you're scared or something. You don't want them to dislike you for normal human reasons.

AARON: Yeah. No, to be clear, I very much buy into the thing. Do not say what. Just because you think an arbitrary proposition to be true doesn't mean that it is a good idea to say that. There's trivial examples here. If you go to something I can make up, I don't know. You don't like your aunt's dress at a family gathering. You think that's like a true statement, that it's ugly? No one's going to say, oh yeah, well, maybe you shouldn't lie. If she really asks you, we can argue about that. And I think I'll probably defend that. You probably shouldn't lie, but you should be as nice as possible or whatever, but you don't like it if she in fact asks you point blank. But no, you shouldn't just say, oh, let me raise my hand like, hi, Aunt Mary, your dress is ugly, or whatever. On the other hand, then we could get into, okay, if somebody asks you, okay, what do you think about wild animals or whatever? I don't know. Yeah, you should be as polite and normal sounding as possible. But even if you're not directly lying, you shouldn't just say things that are like people would naturally take to be like a conclusion that you don't believe in or whatever.

MAX: Yeah, I agree with this, though, probably there is another thing here. This is not really what you're saying or something, but if somebody maybe advice, I would say or something, which I think people often get, is also don't derail conversations or something. I guess even the pursuit of true things, like if you mention, if you're doing a discussion on animal welfare or something, and you're like, yes, in passing, you frame it for. You give like a 92nd spiel. Like framing. You're like, yes. And people also kind of care about animal welfare here. You could read Brian Tamask or something. If anyone's interested, I can send you links. And someone asks later on, what are your thoughts here? Maybe don't talk about animal welfare, wild animal welfare in the context. Because maybe you just think of derail it because you have to be like, here's all this stuff or something. It's just more productive to be like, we'll talk about it later or something.

AARON: Yeah, okay. No, I agree. We're sort of debating vibes, right?

MAX: We're like debating.

AARON: I think we're like, no, sorry, I was using that sort of like maybe facetiously is the right word, like sarcastically or whatever. Unlike an analytic philosophy where we get to have propositions and then you say p and I say, not p, and then we yell at each other. Unfortunately, this isn't as conducive to that. Although I do think also for the case of EA fellowships per se, and maybe like the general case of group discussions, you also shouldn't let. There's like a thing where at least I've noticed, and maybe this is just like n equals three or whatever, but there will be people who are well meaning and everything. They just don't know what they're talking about and they'll derail the conversation. And then you're afraid to rerail the conversation because. I don't know, because that would be like indoctrination or something. And I think sometimes rerailing is just, like the right thing to do.

MAX: Yeah. I mean, I haven't experienced it that much, I think. But see, no issues. That or something.

AARON: I think rerailing is also. Maybe this is like a general point. I feel like I'm talking from the position as the leader or whatever, but no, just even in the other side of things, I feel like in general, in college, I feel like that's the biggest thing I could think of, like that category, taking college classes or whatever. I would want to know whether the professor thinks that the thing I said is a good point or a bad point. Right. I feel like a lot of times they would just say good point no matter what, and then it's no evidence as to whether it's actually a good point or not. Well, I'm being slightly overstating this a little bit.

MAX: I think you're conflating good and true or something. Maybe.

AARON: Yeah, sure. But, yeah, I don't remember any specific examples that I can give, but all I'm trying to say is that it's not just from the perspective of somebody who's trying to get people to lead them to the right answer or whatever. No. I don't know. I feel like it's also, well, in one sense, if you really value your epistemics, it might be like, help. You might want people to give you more people who, in fact, more knowledgeable and have thought a lot about some subject to be more upfront about what they think, but also as sort of a matter of just like, respect is not exactly the right word. I don't think the professors were being disrespectful to me, but it's like if we're just friends talking, or it puts you more on an equal playing field in some way. If the one person who is ostensibly and in fact more knowledgeable and ostensibly in a position of de facto power or whatever, leading this discussion or whatever, it puts them on a pedestal. If they try to be super coy about what they think and then not give you legible feedback as to whether a point you said is plausible or not. You can imagine, I don't know, just making an argument that is just kind of terrible. And then, I don't know, I would kind of want the pressure to say no. That's being polite about it, probably, but say, like, no, bad or something.

MAX: Yeah. I mean, I guess there's something where maybe I do think professors should not just say everything's good or something. They should only say it when it's good or something. Where good doesn't mean true, in my mind. Yeah. But I think I probably lean towards this, which is like risk aversion in sort of community building and classroom building is valuable or something, because you might take the approach that if you're like, I think it is a more risky or something, risky probably isn't the right thing to say there. I mean like risking, like a risk averse versus risk, sort of like, yeah, it's more risky to sort of do the things you're describing and maybe it's like risk neutral or something. The EV is positive because you'll get lots of really engaged people as a result, and you'll also push people out and discounts out or something. I think I tend to favor being risk neutral here. So you maybe aren't making as many, really. You're losing out some value at one end, but you are including more people. And I sort of anticipate this on net being better or something like producing better results, at least like in community health sorts of senses.

AARON: Yeah, I actually think. I agree with that. I do think that people in social situations are just like, in fact, I don't know if it's exactly a bias formally, but people are just like risk averse. Because the whole evos thing, it's like, oh, we used to if you angered somebody in your tribe, they might kill you or whatever, but now it's like, okay, somebody leaves your club, it's fine. No, maybe it's not fine. Right. That is in fact a loss potentially. Maybe it's not. But I think we sort of have to adjust, make ourselves adjust a little bit more in the pro risk direction.

MAX: Yeah. Though I think probably if you take the project of EA seriously or something, there are probably good reasons to want various types of diversity and that sort of approach of, I guess being more risk averse is more likely to get you diversity or something.

AARON: Yes, good point. Excellent. Good job. Everything you said is true, in fact and good. Any other hot or cold takes, in fact, or medium or lukewarm takes?

MAX: Um, I mean, certainly. Right is the question.

AARON: I don't know. It doesn't have to be related to the space is like wider than you probably initially think the space here is like. As long as it's not an info hazard. Not illegal to say. Yeah, you can kind of bring up whatever and not like singling out like random people and not mean, not mean.

MAX: Like global parties gossip. I, like, know what Toby or did last.

AARON: Done. Okay. Unfortunately, I'm not hip to that.

MAX: And I'm sure there is gossip, like somebody didn't change coffee pot or something. But I don't work there, so I don't know.

AARON: Maybe. Hopefully you will soon. CEa, if you're listening to this.

MAX: Well, they're different organizations.

AARON: I don't know. Oxford, every. Ea.org, if you're listening to this.

MAX: Yeah. They're all.

AARON: Wait, maybe. Like, what else? I don't know. What do you think about Twitter, like, in general? I don't know. Because this is how we met.

MAX: Yeah.

AARON: We have not met in real life.

MAX: Worse as a platform than it was two years ago or something.

AARON: Okay.

MAX: Stability wise, and there are small changes that make it worse or something, but largely my experience is unchanged, I think.

AARON: Do you think it's good, bad? I don't know. Do you think people should join Twitter on the market?

MAX: I think EA should join EA. Twitter. I'm not sure if you join Twitter rather than other social medias or something. I think sort of the area of social media we're on is uniquely quite good or something.

AARON: I agree.

MAX: And some of this is like, you get interactions with people, which is good, and people are very nice or something, and very civil where we are. And it's less clear the sorts of personal ability or something and niceness that you get where we are in, like, are elsewhere in Twitter because I don't go elsewhere. But basically you should join Twitter, I guess, if you're going to enter a small community or something, if you're just going to use it to browse memes or something, it's not clear this is better than literally any other social media that has no.

AARON: Yeah, I agree. Well, I guess our audience is, of all, maybe four people, is largely from Twitter. But you never know. There's like a non zero chance that somebody from the wider world will be listening. I think it's at least worth an experiment. Right. Maybe you could tell me something that I should experiment with. Is there anything else like Twitter that we don't have in common that you think that maybe I don't do? It's like, oh, he's an idiot for not doing.

MAX: Oh, probably not. I mean, I'm sure you do better things than I do. Probably.

AARON: Well, I mean, probably this is a large. Right? Like, I don't know.

MAX: I think a benefit of using Twitter is like, it kind of opens you up or something. Probably is the case. It probably does literally build your social skills or something. I mean, maybe not in an obviously useful way, because it's like you're probably not necessarily that much better at doing in person stuff or something as a result of these Twitter. Maybe it improves you very slightly or something, but it's a different skill, texting versus talking.

AARON: Actually, here's something I want your thoughts on recently. Maybe this is outing me as a true Twitter addict, but no, I, by and large, have had a really good experience and I stand by that. I think it's net on net. Not just on net, but just in general, added value to my life and stuff. And it's great, especially given the community that I'm in. The communities that I'm in. But yeah, this is going to kind of embarrassing. I've started thinking in tweets. I'm not 100% of the time, not like my brain is only stuck on Twitter mode, but I think on the margin there's been a chef toward a thought verbalizes an Aaron's brain as something that could be a tweet. And I'm not sure this is a positive.

MAX: Like it is the case. I've had my friends open Twitter in front of me, like my Twitter and go through and read my tweets. Actually, many people in my life do this. I don't know why. I don't really want them to do that. And it does change the way you talk. Certainly part of that is probably character element, and part of it is probably like culture or something. So that's the case. I don't know if I experienced that or I do sometimes if I thought of a really stupid pun. Normally you don't do anything with that, but now I can or something. Right. It's worth holding on for the 6 seconds it takes to open my phone. But I think I actually kind of maybe think in tweets already or something. Like, if you read my writing, I've gotten feedback that it's both very poetic or something. And poems are short or something. It's like very stanza or something, which is kind of how Twitter works also. Right. I think if you looked at the formatting of some of my writing, you would see that it's very twitter like or something. In some sense, there's no character limit, and so maybe this is just the sort of thing you're experiencing or something. Or maybe it's more intense.

AARON: Yeah, probably not exactly. Honestly, I don't think this is that big of a deal. One thing is, I think this is a causal effect. I've blogged less and. And I think it's like, not a direct replacement. Like, I think Scooter has been like an outlet for my ideas that actually feels less effortful and takes less. So it's not like a one for one thing. So other more worky things have filled in the gap for blogging. But I think it has been a causal reason that I haven't blogged as much as I would like to. Really would like have to or something. Yeah, I can see that being thing that is like ideas, there's no strong signal that a particular tweet is an important idea that's worth considering. Whereas if you've written a whole blog post on it and you have 200 subscribers or whatever, you put in a lot of effort. People are at least going to say like, oh, this is me. At least plausibly like an important idea. Like when they're coming into it or something like that.

MAX: Yeah. And if you think something is valuable or something, maybe this is different for you or something. But I get like three likes on all my tweets. It's very rare I get ten likes or something. The number of followers. Gross. It's just stuck there forever.

AARON: Feel like it's not true. Should I read all your bangers? I have a split screen going on. Should I search from Max Alexander?

MAX: Search for my baggers. That's 20,000 tweets or posts.

AARON: How many?

MAX: The Ted like ones you'll find over Ted likes are know very small percentage of the total do.

AARON: So why is your handle absurdly, Max? Is there a story you don't have to answer mean?

MAX: It's very simple. So I think existentialism is right or something. And absurdism specifically. And my name is.

AARON: Wait, really? Wait, what even is absurdism?

MAX: Humans have this inherent search for meaning, and there's no inherent meaning in the universe.

AARON: This is just moral realism in continental flavor.

MAX: Well, and then you have to. So there is no moral truth. Right. And you have to make your own meaning or you could kill yourself.

AARON: I guess this is not true. Okay, whatever, but whatever. This is just continental bullshit.

MAX: If you're a moral antirealist or something, you probably end up being an existentialist or don't care about philosophy, I suppose.

AARON: Oh, this is a good one. If anyone at OpenAI follows me, I just want to say that I'd probably pay $20 a month, maybe even more, for a safely aligned super intelligence. I will actually second that. So, open AI. We can promise you $40 if you do that.

MAX: A month, in fact.

AARON: Yes. Yeah. There's lots of bangers here. You guys should all follow Max and look up his bangers.

MAX: I would be surprised if somebody's listening to this and isn't already.

AARON: You never know. I'm going to have to check my listening data after this and we'll see how big our audience is. Kind of forget. Yeah. So once again, the space of. Also, I'm happy to take a break. There's no formalized thing here. Structure.

MAX: I mean, I'll go for hours.

AARON: Oh, really? Okay, cool. No. Are there any topics that are, like. I feel like. Yeah, the space is very large. Here's something. Wait, no, I was going to say, is there anything I do that you disagree with?

MAX: That's like a classic.

AARON: Yeah, I find you very unobjectionable. That's a boring compliment. I'm just kidding. Thank you. It's actually not.

MAX: I have, like, I suppose writing takes or something, but I don't know if I can find the book.

AARON: Oh, wait, no. We have those two similar blog posts, but you know what I'm talking about. Okay, can I just give the introduction to this?

MAX: Okay.

AARON: No, I think we just have similar blog posts that I will hopefully remember to link that I think are substantively very similar, except they have totally different vibes. And yours is very positive and mine is very negative. In fact, mine is called on suffering. I don't remember what yours is called.

MAX: It's a wonderful life.

AARON: There you go. Those are the vibes, but they're both like, oh, no. Hedonic value is actually super meaningful and important, but then we take it in opposite directions. That's it.

MAX: Yeah.

AARON: I don't know if I feel like your title actually appears to something. I feel like your title is bad, but besides that, the piece is good.

MAX: Well, there's a movie called it's a wonderful life and the post is, like Christmas themed.

AARON: Oh, I feel like I keep not getting things like that.

MAX: It's okay.

AARON: I feel like I vaguely knew that was like a phrase people said, but wasn't sure where it came from. I remember there's like an EA thing called non trivial that I helped on a little bit and I didn't realize it was a reference to non trivial pursuits, which is like a board game or something. No, I think it was actually originally called non trivial pursuits, I think. I'm sorry, Peter McIntyre, if you're listening to this, I apologize for any false, like, I don't know, like a reasonable amount of time, and I had no idea. And then the guy who I was working under brought this up and I was like, wait, that's a board game. Or something. Anyway, this is not an important aside, I'm kind of a bad podcaster because Dwarkesh Patel, who's about 100,000 times better at podcasting than me, goes for like six or 8 hours. That stresses me out so much. Even from an out. Like, not even doing it, doing it, I would just die. I would simply fizzle out of existence. But even thinking about it is very stressful.

MAX: It depends who guessed it, but I could probably talk to somebody for six continuous hours or something.

AARON: Tell me we don't have to discuss this all but one thing could be the virtues of being anonymous. Not anonymous.

MAX: Sure. Okay. I think the virtue is probably like comfortableness or something is the primary one, and maybe some risk aversion or something.

AARON: Yeah.

MAX: Probably. It's not that common or something, but it's probably more common than people think. But being socially ostracized or being very publicly canceled or something, and maybe for bad reasons, one might say as well does occur. And you might be the sort of person who wants to avoid this. I mean, in some sense you've kind of been piled on by various points and it seems like you just were fine with this.

AARON: TBD. So I have not been formally hired by anyone since I've discussed this. No, there was like the tweet where I was like, we should bury poor people underground. Just kidding. That's not what I said. That is what a subset of people who piled on me said. I said, which is not in fact what I said. I said, I asked a question, which was like, why are we not building underground more? No, but yeah, I feel like this is definitely just like a personal taste thing. I don't know. I'm sure there are, but on very broadly, don't even want to say like EA Twitter, but like extended, I don't know, like broadly, like somewhat intellectual ish, English speaking Twitter or something like that. Are there examples of people with under, say, 5000 followers who have said something that is not legitimately indicative? I mean, this is doing a lot of work here, I want to say not legitimately indicative of them being a terrible person. Right. That's doing a lot of work. Right. But I think that neither of us are. So then this is the question.

MAX: My guess would be, like, teachers or something, maybe that sometimes happens to, or like, I think the more, I think probably the sorts of jobs we end up wanting or something like this are more okay with online presence. Because I don't know, everyone at your, ethink priorities is terminally online, right? And so they're not going to mind if you are too.

AARON: I'm not counting on getting an EA or a job.

MAX: Like corporate and public sector jobs I think are generally are like you should make your social media private or something.

AARON: Yeah.

MAX: They tell you that not just the.

AARON: Corporate world is like a big category, right? If you're a software engineer, first of all, I think if you actually have reprehensible opinions, I think it is in your self interest to be an alt. If you want to say them out loud. Not being a bad person is doing a lot of work here. But for what it's worth, I really do think I'm extending there are communists that I would say are genuinely not bad people. I really think they're wrong about the whole communism thing, but that does not fall into my category of automatically makes it such that you should be an alt to avoid being exposed.

MAX: I think probably there are good reasons in EA specifically to maybe make an alt that are not the case elsewhere. And this is like the community is homogeneous and lots of people with lots of power are also on Twitter, basically. Maybe if somebody rethink it's a bad vibe of you or something, that's indicative they shouldn't hire you. I guess maybe. But maybe it doesn't matter how I use Twitter. I don't think this is true, but maybe I use Twitter in a way that's abrasive or maybe they just don't like my puns or something. This is information that will make them change their opinion about me in an interview. And maybe it's not that impactful and maybe it is a little impactful and maybe sometimes it's positive it will make them like me more or something. But the community is so insular, such that this is more of a problem or something.

AARON: Yeah, I feel like my sense is just that it's more symmetrical than this is giving it credit for. Yeah, I guess if you have a lot of heterodox views that are on hot especially, I don't know. Yeah. If you disagree with a lot of people in EA who are powerful, not just not in a weird way, just have hiring power or whatever, on topics that are really emotionally salient or whatever. Yeah. I would probably say if you want to very frankly discuss your opinions and get hired at the orgs where these people have hiring power, it's like probably make an alt. I just don't think that's true for that many people.

MAX: My guess is this is more important the smaller the is or something. My guess would be open philanthropy, like where you think I don't know, global parties institute. It doesn't matter if you make an alternate, really, you're okay. But there are some EA orgs that are like three people or something. Right. And probably your online presence matters, but.

AARON: I mean that's also. They also hire a fewer open spots. Yeah. I just feel like the upside is or maybe the downside. I think we probably disagree on the downside somewhat. I mean, just like stepping away from the abstract level. I just personally think that you, in fact, I can take this out, but can we just say that your last name is not in fact Alexander? Yeah, that's okay. I feel like you could just use your real name and your life would.

MAX: Well, it is my real name. It just my middle name.

AARON: Okay. Wow. Okay. Very, very slate sarcodex esque.

MAX: That's why I did it, because.

AARON: Oh, nice.

MAX: But it was like really nice.

AARON: I actually don't know why I didn't connect the dots there. Yeah, I feel like just like you personally in expectation, it would probably not be a dramatic change to your life if I were you and I was suddenly cast into your shoes. But with my beliefs, I would just use my real name or something like that.

MAX: Yeah, I think it's like at the point now it doesn't matter or something. There's no real upside to changing it and there's no real downside.

AARON: Yeah, it's not a huge deal.

MAX: Yeah, I think probably, well, some people like to be horny on Twitter or something and probably if you want to do that, you should be anonymous or.

AARON: I mean, but once again, even I feel like it depends what you mean by be horny. Right. If you're going to post nude photos on Twitter. Yeah. Actually had a surprisingly good dm conversation with a porn account. I did not follow them, for what it's worth.

MAX: They're presumably run by real.

AARON: No, no, it was actually. No, she was very polite. Yes, I think I'm correctly assuming the name clearly. I won't say the name, but like accounting. But it was clearly indicative. It was a woman. She basically objected to me suggesting that minimum wage jobs in the US are uncommon. And I think this is actually, in fact, I deleted my tweet because it was like, I think giving a false impression that, yeah, I live in a wealthy area. It's true that server sector jobs generally pay better than minimum wage, but elsewhere in the US it's not true. It's a very polite interaction. Anyway, sorry, a total side story. Oh yeah, horny on Twitter. Yes, probably. Yeah, I agree. I don't know, but that can mean multiple things. I guess I have participated somewhat in gender discourse. I guess. I don't think I've been extremely horny. Yeah, fair enough. I don't think I've been very horny on me.

MAX: Yeah. I think probably there's maybe something to be the case that, and you see this as prestige increases or something. This isn't totally true because I think Elie Iser Zukowski posted about whatever or something, but Will McCaskill and Peter Wildefer are not engaging in gender discourse or whatever. Right. Publicly anyway.

AARON: Wait, do I want to put my money or not my money, my social reputation doesn't use Twitter very much, which I think, I don't know. I would in fact take this question much more seriously if I was him. Like the question of whether to just be more frank and open and just, I don't know, maybe he doesn't want to do. I'm like totally hypothesizing here. Wait, Peter totally does sometimes, at least in one case. I know. No, I have disagreed with Peter on gender discourse and that's just like somewhat. I don't know. Right.

MAX: I just don't see. He's not starting gender discourse, I guess I should say.

AARON: Well, I mean, Adam and Eve started gender. No, but I don't know. I don't want to single anybody out here. I feel like, yeah, if you're like the one face of a social movement, yeah, you should probably take it pretty seriously. At least the question of like, yeah, you should be more risk averse. I will buy into that. I don't think my position here is like absolute whatsoever. I think, yeah. For people with fewer than 5000 followers, it's like a gradient, right? I just think on the margin people are general or like in general, people maybe are more risk averse than they have to be or something like that.

MAX: Yeah. Though I'm not sure there are that many major examples. Because one reason you might be anonymous is not because you are scared about people who follow you finding you elsewhere. It's because you don't want the inverse to be the case or something.

AARON: Wait, what's the, I'm sorry, you don't.

MAX: Want your parents to google you Twitter account and read your tweets? Yeah, everyone I know knows my Twitter. I don't even know how they all found it, but.

AARON: That'S interesting. Okay, well, I mean, yeah.

MAX: Read stupid tweets of me in front of me.

AARON: Well, there you go. That's like a good real life example of like, okay. That's like a real downside, I guess. Or maybe it's not because it's kind of a funny thing, but.

MAX: I think you do get this. People will joke about this sometimes, I think on Twitter or something, or I've seen it ever where it's like, I hope my boss is not using Twitter today because they'll be like, why weren't you working or something?

AARON: Oh, yeah.

MAX: Literally just tweeting or something.

AARON: Yeah. I know. If you've called in sick to work and you're, like, lying about that, you're.

MAX: Just tweeting a bunch, they might be kind of suspicious or something. If your boss could see you all the time, you open Twitter for like a minute to tweet or something, they'd probably be like a little judgy or something, right? Maybe they wouldn't.

AARON: Yeah, yeah, sure. I mean, it's just like a matter of degree.

MAX: These are probably the most real world scenarios, is like somebody, you know, gets information that isn't really that damaging but slightly inconveniences you or something.

AARON: Yeah. Oh, actually, I talked about this more with Nathan Young, but then I didn't record his audio. I'm sorry. I've apologized for this before, but I'm so bad at me. This is before. I really hope that I've solved this for this particular, I think I have for this episode. But, yeah, on my one viral tweet or whatever, one thing is just like, oh, yeah. A lot of people basically, I think they were earnest. Not earnest, exactly. That's like, maybe too generous, but they genuinely thought that I was saying something bad, like something immoral. I think they were wrong about that, but I don't think they were lying. But then, in fact, okay, several of my real life friends either, it's like, come up somehow and none of them actually thought I was saying anything bad. And in fact, I met somebody basically at a party who was talking about this. And then it was like, oh, that's me. I was the one who posted that. But basically the point here is that nothing bad in my. As far as I can tell, maybe I'm wrong, but as far as I can tell, nothing bad in my life has come with a viral tweet that mainly was viral from people quotating it, saying that. I was saying something like immoral. N equals one.

MAX: Yeah. I don't know how come it would be or something. I think, like Contra points or something. I believe it's her. Has videos about canceling and stuff like this. I think she was canceled at various. Lindsay Elsie, you may have heard of, I think, kind of got run off the Internet.

AARON: Actually don't know this person.

MAX: She does, like, video, or she did video essays about media and stuff. She's a fiction author now. There are orders of magnitude more well known than we are for something.

AARON: There's, like, canceling that really genuinely ruins people's lives. And then there's, like, Barry Weiss. I don't know, she has like a substack or like, I don't even know, like, IDW person or I think. I don't even know if she got. There's people like this. I'm sure they have other academics and they're professors at universities who don't like them, but they have successful, profitable substacs. And it seems to me like their lives aren't made a lot worse by being, quote unquote, canceled. No, but then there really are. Right. I'm sure that there's definitely cases of normalish people. Yeah.

MAX: I don't know, but maybe it's just more psychologically damaging, regardless of consequences, to be piled on when it's your real face or something.

AARON: Yeah. Also, I think 90. I don't know about at least 90%. Probably not like 99 and then like ten, nine. But something in between these two numbers of pylons just don't have any real world consequences whatsoever. Canceling is like a much less common phenomenon.

MAX: Yeah, that seems right to me.

AARON: Yeah. Let it be known that we are two white. We can also take this part out if you don't want this identifying information. 220 something white males discussing being canceled on a podcast, which is possibly the most basic thing that has ever happened in the universe.

MAX: I mean, I want to be known for this. When I, my children come up to me and say, we found the one podcast interview you did.

AARON: Yeah. I feel like I want to reiterate that there's a core thing here, which is, like, if you really hate jewish people and you think they should die, there's like, a fundamental thing there where if you're open and honest about your opinions, people are going to correctly think that you're a bad person. Whereas I think neither of us hold any views that are like that.

MAX: Agree. I think.

AARON: Yeah, no, maybe I wouldn't know, which is fine. But I'm sort of using an extreme example that I think very few people, at least, who I would interact with, hold. But if you're in fact, like, a person who has views that I think really indicate immoral behavior or something. Sorry. Maybe I'll even take that part out because it could be like, clip eclipsed or whatever, but if you just say like, oh yeah, I steal things from street vendors or whatever. I don't know, that's a bad thing. And you're like being honest, don't do that. And then people are like, yeah, you have to make, I guess, have some confidence that you're like a person who's like, I don't know, doesn't do very, doesn't do pretty things or have views that true legitimately indicate that you either do or would do very immoral things that are regarded to be immoral by a large number of people or whatever. This is doing a lot of the.

MAX: Mean, like I think this goes back to earlier things or something. I just thought of it or something. I mean, it's not clear the sign of this or something, but I wrote a whole blog post about joining EA or something which I think has some genuinely very good parts in it that are really good explanations of why someone should be motivated to do EA or something. There's lots of really embarrassing stuff about life in high school or something in it.

AARON: Yeah, no, I'm pretty sure everyone in.

MAX: 80 who works at 80k. Not everyone, but I have good reason to think many people at read this, which is like, I don't know what to make of that. It's just kind of weird. Right, people?

AARON: Yeah, no, I agree. It's kind of weird. I think like object level. I don't know. I don't even know if embarrassed. Embarrassing is like the right word. Yeah, it's like genuinely, I guess, more vulnerable than a lot of people are in their public blogs or whatever. But I don't really think it reflects poorly on you when take it in aggregate. Right. I mean maybe you disagree with that, but especially I don't think so.

MAX: Or if you think it does, you probably are judgy to a degree I think is unreasonable or something from when I was like 13 or something.

AARON: Yeah, sure.

MAX: That bad? Really?

AARON: What bad things? Should I. No, there's something that I actually don't even think is immoral, but it's like somewhat embarrassing. Maybe I'll even say it on the next podcast episode after I've thought about it for ten minutes instead of 1 minute or something like that. But I don't know, I think if I said it, it would be okay. Nothing that bad would happen. I guess. You're going to continue being Max Alexander instead of Max.

MAX: I mean, like it's the brand. Like I can't.

AARON: Wait. I feel like this is not a good reason. I feel like pretty quickly. Yeah. I feel like path dependence is like a real thing, but in this particular case, it's just like, not as big as maybe scenes or something like that. I don't know. Yeah.

MAX: I just don't care what the upside is or something.

AARON: Yeah, true. Yeah. I'm thinking maybe we wrap up.

Get full access to Aaron's Blog at www.aaronbergman.net/subscribe

* Listen on Spotify or Apple Podcasts

* Be sure to check out and follow Holly’s Substack and org Pause AI.

Blurb and summary from Clong

Blurb

Holly and Aaron had a wide-ranging discussion touching on effective altruism, AI alignment, genetic conflict, wild animal welfare, and the importance of public advocacy in the AI safety space. Holly spoke about her background in evolutionary biology and how she became involved in effective altruism. She discussed her reservations around wild animal welfare and her perspective on the challenges of AI alignment. They talked about the value of public opinion polls, the psychology of AI researchers, and whether certain AI labs like OpenAI might be net positive actors. Holly argued for the strategic importance of public advocacy and pushing the Overton window within EA on AI safety issues.

Detailed summary

* Holly's background - PhD in evolutionary biology, got into EA through New Atheism and looking for community with positive values, did EA organizing at Harvard

* Worked at Rethink Priorities on wild animal welfare but had reservations about imposing values on animals and whether we're at the right margin yet

* Got inspired by FLI letter to focus more on AI safety advocacy and importance of public opinion

* Discussed genetic conflict and challenges of alignment even with "closest" agents

* Talked about the value of public opinion polls and influencing politicians

* Discussed the psychology and motives of AI researchers

* Disagreed a bit on whether certain labs like OpenAI might be net positive actors

* Holly argued for importance of public advocacy in AI safety, thinks we have power to shift Overton window

* Talked about the dynamics between different AI researchers and competition for status

* Discussed how rationalists often dismiss advocacy and politics

* Holly thinks advocacy is neglected and can push the Overton window even within EA

* Also discussed Holly's evolutionary biology takes, memetic drive, gradient descent vs. natural selection

Full transcript (very imperfect)

AARON

You're an AI pause, Advocate. Can you remind me of your shtick before that? Did you have an EA career or something?

HOLLY

Yeah, before that I was an academic. I got into EA when I was doing my PhD in evolutionary biology, and I had been into New Atheism before that. I had done a lot of organizing for that in college. And while the enlightenment stuff and what I think is the truth about there not being a God was very important to me, but I didn't like the lack of positive values. Half the people there were sort of people like me who are looking for community after leaving their religion that they grew up in. And sometimes as many as half of the people there were just looking for a way for it to be okay for them to upset people and take away stuff that was important to them. And I didn't love that. I didn't love organizing a space for that. And when I got to my first year at Harvard, harvard Effective Altruism was advertising for its fellowship, which became the Elite Fellowship eventually. And I was like, wow, this is like, everything I want. And it has this positive organizing value around doing good. And so I was totally made for it. And pretty much immediately I did that fellowship, even though it was for undergrad. I did that fellowship, and I was immediately doing a lot of grad school organizing, and I did that for, like, six more years. And yeah, by the time I got to the end of grad school, I realized I was very sick in my fifth year, and I realized the stuff I kept doing was EA organizing, and I did not want to keep doing work. And that was pretty clear. I thought, oh, because I'm really into my academic area, I'll do that, but I'll also have a component of doing good. I took giving what we can in the middle of grad school, and I thought, I actually just enjoy doing this more, so why would I do anything else? Then after grad school, I started applying for EA jobs, and pretty soon I got a job at Rethink Priorities, and they suggested that I work on wild animal welfare. And I have to say, from the beginning, it was a little bit like I don't know, I'd always had very mixed feelings about wild animal welfare as a cause area. How much do they assume the audience knows about EA?

AARON

A lot, I guess. I think as of right now, it's a pretty hardcore dozen people. Also. Wait, what year is any of this approximately?

HOLLY

So I graduated in 2020.

AARON

Okay.

HOLLY

Yeah. And then I was like, really?

AARON

Okay, this is not extremely distant history. Sometimes people are like, oh, yeah, like the OG days, like four or something. I'm like, oh, my God.

HOLLY

Oh, yeah, no, I wish I had been in these circles then, but no, it wasn't until like, 2014 that I really got inducted. Yeah, which now feels old because everybody's so young. But yeah, in 2020, I finished my PhD, and I got this awesome remote job at Rethink Priorities during the Pandemic, which was great, but I was working on wild animal welfare, which I'd always had some. So wild animal welfare, just for anyone who's not familiar, is like looking at the state of the natural world and seeing if there's a way that usually the hedonic so, like, feeling pleasure, not pain sort of welfare of animals can be maximized. So that's in contrast to a lot of other ways of looking at the natural world, like conservation, which are more about preserving a state of the world the way preserving, maybe ecosystem balance, something like that. Preserving species diversity. The priority with wild animal welfare is the effect of welfare, like how it feels to be the animals. So it is very understudied, but I had a lot of reservations about it because I'm nervous about maximizing our values too hard onto animals or imposing them on other species.

AARON

Okay, that's interesting, just because we're so far away from the margin of I'm like a very pro wild animal animal welfare pilled person.

HOLLY

I'm definitely pro in theory.

AARON

How many other people it's like you and formerly you and six other people or whatever seems like we're quite far away from the margin at which we're over optimizing in terms of giving heroin to all the sheep or I don't know, the bugs and stuff.

HOLLY

But it's true the field is moving in more my direction and I think it's just because they're hiring more biologists and we tend to think this way or have more of this perspective. But I'm a big fan of Brian domestics work. But stuff like finding out which species have the most capacity for welfare I think is already sort of the wrong scale. I think a lot will just depend on how much. What are the conditions for that species?

AARON

Yeah, no, there's like seven from the.

HOLLY

Coarseness and the abstraction, but also there's a lot of you don't want anybody to actually do stuff like that and it would be more possible to do the more simple sounding stuff. My work there just was consisted of being a huge downer. I respect that. I did do some work that I'm proud of. I have a whole sequence on EA forum about how we could reduce the use of rodenticide, which I think was the single most promising intervention that we came up with in the time that I was there. I mean, I didn't come up with it, but that we narrowed down. And even that just doesn't affect that many animals directly. It's really more about the impact is from what you think you'll get with moral circle expansion or setting precedents for the treatment of non human animals or wild animals, or semi wild animals, maybe like being able to be expanded into wild animals. And so it all felt not quite up to EA standards of impact. And I felt kind of uncomfortable trying to make this thing happen in EA when I wasn't sure that my tentative conclusion on wild animal welfare, after working on it and thinking about it a lot for three years, was that we're sort of waiting for transformative technology that's not here yet in order to be able to do the kinds of interventions that we want. And there are going to be other issues with the transformative technology that we have to deal with first.

AARON

Yeah, no, I've been thinking not that seriously or in any formal way, just like once in a while I just have a thought like oh, I wonder how the field of, like, I guess wild animal sorry, not wild animal. Just like animal welfare in general and including wild animal welfare might make use of AI above and beyond. I feel like there's like a simple take which is probably mostly true, which is like, oh, I mean the phrase that everybody loves to say is make AI go well or whatever that but that's basically true. Probably you make aligned AI. I know that's like a very oversimplification and then you can have a bunch of wealth or whatever to do whatever you want. I feel like that's kind of like the standard line, but do you have any takes on, I don't know, maybe in the next couple of years or anything more specifically beyond just general purpose AI alignment, for lack of a better term, how animal welfare might put to use transformative AI.

HOLLY

My last work at Rethink Priorities was like looking a sort of zoomed out look at the field and where it should go. And so we're apparently going to do a public version, but I don't know if that's going to happen. It's been a while now since I was expecting to get a call about it. But yeah, I'm trying to think of what can I scrape from that?

AARON

As much as you can, don't reveal any classified information. But what was the general thing that this was about?

HOLLY

There are things that I think so I sort of broke it down into a couple of categories. There's like things that we could do in a world where we don't get AGI for a long time, but we get just transformative AI. Short of that, it's just able to do a lot of parallel tasks. And I think we could do a lot we could get a lot of what we want for wild animals by doing a ton of surveillance and having the ability to make incredibly precise changes to the ecosystem. Having surveillance so we know when something is like, and the capacity to do really intense simulation of the ecosystem and know what's going to happen as a result of little things. We could do that all without AGI. You could just do that with just a lot of computational power. I think our ability to simulate the environment right now is not the best, but it's not because it's impossible. It's just like we just need a lot more observations and a lot more ability to simulate a comparison is meteorology. Meteorology used to be much more of an art, but it became more of a science once they started just literally taking for every block of air and they're getting smaller and smaller, the blocks. They just do Bernoulli's Law on it and figure out what's going to happen in that block. And then you just sort of add it all together and you get actually pretty good.

AARON

Do you know how big the blocks are?

HOLLY

They get smaller all the time. That's the resolution increase, but I don't know how big the blocks are okay right now. And shockingly, that just works. That gives you a lot of the picture of what's going to happen with weather. And I think that modeling ecosystem dynamics is very similar to weather. You could say more players than ecosystems, and I think we could, with enough surveillance, get a lot better at monitoring the ecosystem and then actually have more of a chance of implementing the kinds of sweeping interventions we want. But the price would be just like never ending surveillance and having to be the stewards of the environment if we weren't automating. Depending on how much you want to automate and depending on how much you can automate without AGI or without handing it over to another intelligence.

AARON

Yeah, I've heard this. Maybe I haven't thought enough. And for some reason, I'm just, like, intuitively. I feel like I'm more skeptical of this kind of thing relative to the actual. There's a lot of things that I feel like a person might be skeptical about superhuman AI. And I'm less skeptical of that or less skeptical of things that sound as weird as this. Maybe because it's not. One thing I'm just concerned about is I feel like there's a larger scale I can imagine, just like the choice of how much, like, ecosystem is like yeah, how much ecosystem is available for wild animals is like a pretty macro level choice that might be not at all deterministic. So you could imagine spreading or terraforming other planets and things like that, or basically continuing to remove the amount of available ecosystem and also at a much more practical level, clean meat development. I have no idea what the technical bottlenecks on that are right now, but seems kind of possible that I don't know, AI can help it in some capacity.

HOLLY

Oh, I thought you're going to say that it would increase the amount of space available for wild animals. Is this like a big controversy within, I don't know, this part of the EA animal movement? If you advocate diet change and if you get people to be vegetarians, does that just free up more land for wild animals to suffer on? I thought this was like, guys, we just will never do anything if we don't choose sort of like a zone of influence and accomplish something there. It seemed like this could go on forever. It was like, literally, I rethink actually. A lot of discussions would end in like, okay, so this seems like really good for all of our target populations, but what about wild animals? I could just reverse everything. I don't know. The thoughts I came to on that were that it is worthwhile to try to figure out what are all of the actual direct effects, but I don't think we should let that guide our decision making. Only you have to have some kind of theory of change, of what is the direct effect going to lead to? And I just think that it's so illegible what you're trying to do. If you're, like, you should eat this kind of fish to save animals. It doesn't lead society to adopt, to understand and adopt your values. It's so predicated on a moment in time that might be convenient. Maybe I'm not looking hard enough at that problem, but the conclusion I ended up coming to was just like, look, I just think we have to have some idea of not just the direct impacts, but something about the indirect impacts and what's likely to facilitate other direct impacts that we want in the future.

AARON

Yeah. I also share your I don't know. I'm not sure if we share the same or I also feel conflicted about this kind of thing. Yeah. And I don't know, at the very least, I have a very high bar for saying, actually the worst of factory farming is like, we should just like, yeah, we should be okay with that, because some particular model says that at this moment in time, it has some net positive effect on animal welfare.

HOLLY

What morality is that really compatible with? I mean, I understand our morality, but maybe but pretty much anyone else who hears that conclusion is going to think that that means that the suffering doesn't matter or something.

AARON

Yeah, I don't know. I think maybe more than you, I'm willing to bite the bullet if somebody really could convince me that, yeah, chicken farming is actually just, in fact, good, even though it's counterintuitive, I'll be like, all right, fine.

HOLLY

Surely there are other ways of occupying.

AARON

Yeah.

HOLLY

Same with sometimes I would get from very classical wild animal suffering people, like, comments on my rodenticide work saying, like, well, what if it's good to have more rats? I don't know. There are surely other vehicles for utility other than ones that humans are bent on destroying.

AARON

Yeah, it's kind of neither here nor there, but I don't actually know if this is causally important, but at least psychologically. I remember seeing a mouse in a glue trap was very had an impact on me from maybe turning me, like, animal welfare pills or something. That's like, neither here nor there. It's like a random anecdote, but yeah, seems bad. All right, what came after rethink for you?

HOLLY

Yeah. Well, after the publication of the FLI Letter and Eliezer's article in Time, I was super inspired by pause. A number of emotional changes happened to me about AI safety. Nothing intellectual changed, but just I'd always been confused at and kind of taken it as a sign that people weren't really serious about AI risk when they would say things like, I don't know, the only option is alignment. The only option is for us to do cool, nerd stuff that we love doing nothing else would. I bought the arguments, but I just wasn't there emotionally. And seeing Eliezer advocate political change because he wants to save everyone's lives and he thinks that's something that we can do. Just kind of I'm sure I didn't want to face it before because it was upsetting. Not that I haven't faced a lot of upsetting and depressing things like I worked in wild animal welfare, for God's sake, but there was something that didn't quite add up for me, or I hadn't quite grocked about AI safety until seeing Eliezer really show that his concern is about everyone dying. And he's consistent with that. He's not caught on only one way of doing it, and it just kind of got in my head and I kept wanting to talk about it at work and it sort of became clear like they weren't going to pursue that sort of intervention. But I kept thinking of all these parallels between animal advocacy stuff that I knew and what could be done in AI safety. And these polls kept coming out showing that there was really high support for Paws and I just thought, this is such a huge opportunity, I really would love to help out. Originally I was looking around for who was going to be leading campaigns that I could volunteer in, and then eventually I thought, it just doesn't seem like somebody else is going to do this in the Bay Area. So I just ended up quitting rethink and being an independent organizer. And that has been really I mean, honestly, it's like a tough subject. It's like a lot to deal with, but honestly, compared to wild animal welfare, it's not that bad. And I think I'm pretty used to dealing with tough and depressing low tractability causes, but I actually think this is really tractable. I've been shocked how quickly things have moved and I sort of had this sense that, okay, people are reluctant in EA and AI safety in particular, they're not used to advocacy. They kind of vaguely think that that's bad politics is a mind killer and it's a little bit of a threat to the stuff they really love doing. Maybe that's not going to be so ascendant anymore and it's just stuff they're not familiar with. But I have the feeling that if somebody just keeps making this case that people will take to it, that I could push the Oberson window with NEA and that's gone really well.

AARON

Yeah.

HOLLY

And then of course, the public is just like pretty down. It's great.

AARON

Yeah. I feel like it's kind of weird because being in DC and I've always been, I feel like I actually used to be more into politics, to be clear. I understand or correct me if I'm wrong, but advocacy doesn't just mean in the political system or two politicians or whatever, but I assume that's like a part of what you're thinking about or not really.

HOLLY

Yeah. Early on was considering working on more political process type advocacy and I think that's really important. I totally would have done it. I just thought that it was more neglected in our community to do advocacy to the public and a lot of people had entanglements that prevented them from doing so. They work sort of with AI labs or it's important to their work that they not declare against AI labs or something like that or be perceived that way. And so they didn't want to do public advocacy that could threaten what else they're doing. But I didn't have anything like that. I've been around for a long time in EA and I've been keeping up on AI safety, but I've never really worked. That's not true. I did a PiBBs fellowship, but.

AARON

I've.

HOLLY

Never worked for anybody in like I was just more free than a lot of other people to do the public messaging and so I kind of felt that I should. Yeah, I'm also more willing to get into conflict than other EA's and so that seems valuable, no?

AARON

Yeah, I respect that. Respect that a lot. Yeah. So like one thing I feel like I've seen a lot of people on Twitter, for example. Well, not for example. That's really just it, I guess, talking about polls that come out saying like, oh yeah, the public is super enthusiastic about X, Y or Z, I feel like these are almost meaningless and maybe you can convince me otherwise. It's not exactly to be clear, I'm not saying that. I guess it could always be worse, right? All things considered, like a poll showing X thing is being supported is better than the opposite result, but you can really get people to say anything. Maybe I'm just wondering about the degree to which the public how do you imagine the public and I'm doing air quotes to playing into policies either of, I guess, industry actors or government actors?

HOLLY

Well, this is something actually that I also felt that a lot of EA's were unfamiliar with. But it does matter to our representatives, like what the constituents think it matters a mean if you talk to somebody who's ever interned in a congressperson's office, one person calling and writing letters for something can have actually depending on how contested a policy is, can have a largeish impact. My ex husband was an intern for Jim Cooper and they had this whole system for scoring when calls came in versus letters. Was it a handwritten letter, a typed letter? All of those things went into how many points it got and that was something they really cared about. Politicians do pay attention to opinion polls and they pay attention to what their vocal constituents want and they pay attention to not going against what is the norm opinion. Even if nobody in particular is pushing them on it or seems to feel strongly about it. They really are trying to calibrate themselves to what is the norm. So those are always also sometimes politicians just get directly convinced by arguments of what a policy should be. So yeah, public opinion is, I think, underappreciated by ya's because it doesn't feel like mechanistic. They're looking more for what's this weird policy hack that's going to solve what's? This super clever policy that's going to solve things rather than just like what's acceptable discourse, like how far out of his comfort zone does this politician have to go to advocate for this thing? How unpopular is it going to be to say stuff that's against this thing that now has a lot of public support?

AARON

Yeah, I guess mainly I'm like I guess I'm also I definitely could be wrong with this, but I would expect that a lot of the yeah, like for like when politicians like, get or congresspeople like, get letters and emails or whatever on a particular especially when it's relevant to a particular bill. And it's like, okay, this bill has already been filtered for the fact that it's going to get some yes votes and some no votes and it's close to or something like that. Hearing from an interested constituency is really, I don't know, I guess interesting evidence. On the other hand, I don't know, you can kind of just get Americans to say a lot of different things that I think are basically not extremely unlikely to be enacted into laws. You know what I mean? I don't know. You can just look at opinion. Sorry. No great example comes to mind right now. But I don't know, if you ask the public, should we do more safety research into, I don't know, anything. If it sounds good, then people will say yes, or am I mistaken about this?

HOLLY

I mean, on these polls, usually they ask the other way around as well. Do you think AI is really promising for its benefits and should be accelerated? They answer consistently. It's not just like, well now that sounds positive. Okay. I mean, a well done poll will correct for these things. Yeah. I've encountered a lot of skepticism about the polls. Most of the polls on this have been done by YouGov, which is pretty reputable. And then the ones that were replicated by rethink priorities, they found very consistent results and I very much trust Rethink priorities on polls. Yeah. I've had people say, well, these framings are I don't know, they object and wonder if it's like getting at the person's true beliefs. And I kind of think like, I don't know, basically this is like the kind of advocacy message that I would give and people are really receptive to it. So to me that's really promising. Whether or not if you educated them a lot more about the topic, they would think the same is I don't think the question but that's sometimes an objection that I get. Yeah, I think they're indicative. And then I also think politicians just care directly about these things. If they're able to cite that most of the public agrees with this policy, that sort of gives them a lot of what they want, regardless of whether there's some qualification to does the public really think this or are they thinking hard enough about it? And then polls are always newsworthy. Weirdly. Just any poll can be a news story and journalists love them and so it's a great chance to get exposure for the whatever thing. And politicians do care what's in the news. Actually, I think we just have more influence over the political process than EA's and less wrongers tend to believe it's true. I think a lot of people got burned in AI safety, like in the previous 20 years because it would be dismissed. It just wasn't in the overton window. But I think we have a lot of power now. Weirdly. People care what effective altruists think. People see us as having real expertise. The AI safety community does know the most about this. It's pretty wild now that's being recognized publicly and journalists and the people who influence politicians, not directly the people, but the Fourth Estate type, people pay attention to this and they influence policy. And there's many levels of I wrote if people want a more detailed explanation of this, but still high level and accessible, I hope I wrote a thing on EA forum called The Case for AI Safety Advocacy. And that kind of goes over this concept of outside versus inside game. So inside game is like working within a system to change it. Outside game is like working outside the system to put pressure on that system to change it. And I think there's many small versions of this. I think that it's helpful within EA and AI safety to be pushing the overton window of what I think that people have a wrong understanding of how hard it is to communicate this topic and how hard it is to influence governments. I want it to be more acceptable. I want it to feel more possible in EA and AI safety to go this route. And then there's the public public level of trying to make them more familiar with the issue, frame it in the way that I want, which is know, with Sam Altman's tour, the issue kind of got framed as like, well, AI is going to get built, but how are we going to do it safely? And then I would like to take that a step back and be like, should AI be built or should AGI be just if we tried, we could just not do that, or we could at least reduce the speed. And so, yeah, I want people to be exposed to that frame. I want people to not be taken in by other frames that don't include the full gamut of options. I think that's very possible. And then there's a lot of this is more of the classic thing that's been going on in AI safety for the last ten years is trying to influence AI development to be more safety conscious. And that's like another kind of dynamic. There, like trying to change sort of the general flavor, like, what's acceptable? Do we have to care about safety? What is safety? That's also kind of a window pushing exercise.

AARON

Yeah. Cool. Luckily, okay, this is not actually directly responding to anything you just said, which is luck. So I pulled up this post. So I should have read that. Luckily, I did read the case for slowing down. It was like some other popular post as part of the, like, governance fundamentals series. I think this is by somebody, Zach wait, what was it called? Wait.

HOLLY

Is it by Zach or.

AARON

Katya, I think yeah, let's think about slowing down AI. That one. So that is fresh in my mind, but yours is not yet. So what's the plan? Do you have a plan? You don't have to have a plan. I don't have plans very much.

HOLLY

Well, right now I'm hopeful about the UK AI summit. Pause AI and I have planned a multi city protest on the 21 October to encourage the UK AI Safety Summit to focus on safety first and to have as a topic arranging a pause or that of negotiation. There's a lot of a little bit upsetting advertising for that thing that's like, we need to keep up capabilities too. And I just think that's really a secondary objective. And that's how I wanted to be focused on safety. So I'm hopeful about the level of global coordination that we're already seeing. It's going so much faster than we thought. Already the UN Secretary General has been talking about this and there have been meetings about this. It's happened so much faster at the beginning of this year. Nobody thought we could talk about nobody was thinking we'd be talking about this as a mainstream topic. And then actually governments have been very receptive anyway. So right now I'm focused on other than just influencing opinion, the targets I'm focused on, or things like encouraging these international like, I have a protest on Friday, my first protest that I'm leading and kind of nervous that's against Meta. It's at the Meta building in San Francisco about their sharing of model weights. They call it open source. It's like not exactly open source, but I'm probably not going to repeat that message because it's pretty complicated to explain. I really love the pause message because it's just so hard to misinterpret and it conveys pretty clearly what we want very quickly. And you don't have a lot of bandwidth and advocacy. You write a lot of materials for a protest, but mostly what people see is the title.

AARON

That's interesting because I sort of have the opposite sense. I agree that in terms of how many informational bits you're conveying in a particular phrase, pause AI is simpler, but in some sense it's not nearly as obvious. At least maybe I'm more of a tech brain person or whatever. But why that is good, as opposed to don't give extremely powerful thing to the worst people in the world. That's like a longer everyone.

HOLLY

Maybe I'm just weird. I've gotten the feedback from open source ML people is the number one thing is like, it's too late, there's already super powerful models. There's nothing you can do to stop us, which sounds so villainous, I don't know if that's what they mean. Well, actually the number one message is you're stupid, you're not an ML engineer. Which like, okay, number two is like, it's too late, there's nothing you can do. There's all of these other and Meta is not even the most powerful generator of models that it share of open source models. I was like, okay, fine. And I don't know, I don't think that protesting too much is really the best in these situations. I just mostly kind of let that lie. I could give my theory of change on this and why I'm focusing on Meta. Meta is a large company I'm hoping to have influence on. There is a Meta building in San Francisco near where yeah, Meta is the biggest company that is doing this and I think there should be a norm against model weight sharing. I was hoping it would be something that other employees of other labs would be comfortable attending and that is a policy that is not shared across the labs. Obviously the biggest labs don't do it. So OpenAI is called OpenAI but very quickly decided not to do that. Yeah, I kind of wanted to start in a way that made it more clear than pause AI. Does that anybody's welcome something? I thought a one off issue like this that a lot of people could agree and form a coalition around would be good. A lot of people think that this is like a lot of the open source ML people think know this is like a secret. What I'm saying is secretly an argument for tyranny. I just want centralization of power. I just think that there are elites that are better qualified to run everything. It was even suggested I didn't mention China. It even suggested that I was racist because I didn't think that foreign people could make better AIS than Meta.

AARON

I'm grimacing here. The intellectual disagreeableness, if that's an appropriate term or something like that. Good on you for standing up to some pretty bad arguments.

HOLLY

Yeah, it's not like that worth it. I'm lucky that I truly am curious about what people think about stuff like that. I just find it really interesting. I spent way too much time understanding the alt. Right. For instance, I'm kind of like sure I'm on list somewhere because of the forums I was on just because I was interested and it is something that serves me well with my adversaries. I've enjoyed some conversations with people where I kind of like because my position on all this is that look, I need to be convinced and the public needs to be convinced that this is safe before we go ahead. So I kind of like not having to be the smart person making the arguments. I kind of like being like, can you explain like I'm five. I still don't get it. How does this work?

AARON

Yeah, no, I was thinking actually not long ago about open source. Like the phrase has such a positive connotation and in a lot of contexts it really is good. I don't know. I'm glad that random tech I don't know, things from 2004 or whatever, like the reddit source code is like all right, seems cool that it's open source. I don't actually know if that was how that right. But yeah, I feel like maybe even just breaking down what the positive connotation comes from and why it's in people's self. This is really what I was thinking about, is like, why is it in people's self interest to open source things that they made and that might break apart the allure or sort of ethical halo that it has around it? And I was thinking it probably has something to do with, oh, this is like how if you're a tech person who makes some cool product, you could try to put a gate around it by keeping it closed source and maybe trying to get intellectual property or something. But probably you're extremely talented already, or pretty wealthy. Definitely can be hired in the future. And if you're not wealthy yet I don't mean to put things in just materialist terms, but basically it could easily be just like in a yeah, I think I'll probably take that bit out because I didn't mean to put it in strictly like monetary terms, but basically it just seems like pretty plausibly in an arbitrary tech person's self interest, broadly construed to, in fact, open source their thing, which is totally fine and normal.

HOLLY

I think that's like 99 it's like a way of showing magnanimity showing, but.

AARON

I don't make this sound so like, I think 99.9% of human behavior is like this. I'm not saying it's like, oh, it's some secret, terrible self interested thing, but just making it more mechanistic. Okay, it's like it's like a status thing. It's like an advertising thing. It's like, okay, you're not really in need of direct economic rewards, or sort of makes sense to play the long game in some sense, and this is totally normal and fine, but at the end of the day, there's reasons why it makes sense, why it's in people's self interest to open source.

HOLLY

Literally, the culture of open source has been able to bully people into, like, oh, it's immoral to keep it for yourself. You have to release those. So it's just, like, set the norms in a lot of ways, I'm not the bully. Sounds bad, but I mean, it's just like there is a lot of pressure. It looks bad if something is closed source.

AARON

Yeah, it's kind of weird that Meta I don't know, does Meta really think it's in their I don't know. Most economic take on this would be like, oh, they somehow think it's in their shareholders interest to open source.

HOLLY

There are a lot of speculations on why they're doing this. One is that? Yeah, their models aren't as good as the top labs, but if it's open source, then open source quote, unquote then people will integrate it llama Two into their apps. Or People Will Use It And Become I don't know, it's a little weird because I don't know why using llama Two commits you to using llama Three or something, but it just ways for their models to get in in places where if you just had to pay for their models too, people would go for better ones. That's one thing. Another is, yeah, I guess these are too speculative. I don't want to be seen repeating them since I'm about to do this purchase. But there's speculation that it's in best interests in various ways to do this. I think it's possible also that just like so what happened with the release of Llama One is they were going to allow approved people to download the weights, but then within four days somebody had leaked Llama One on four chan and then they just were like, well, whatever, we'll just release the weights. And then they released Llama Two with the weights from the beginning. And it's not like 100% clear that they intended to do full open source or what they call Open source. And I keep saying it's not open source because this is like a little bit of a tricky point to make. So I'm not emphasizing it too much. So they say that they're open source, but they're not. The algorithms are not open source. There are open source ML models that have everything open sourced and I don't think that that's good. I think that's worse. So I don't want to criticize them for that. But they're saying it's open source because there's all this goodwill associated with open source. But actually what they're doing is releasing the product for free or like trade secrets even you could say like things that should be trade secrets. And yeah, they're telling people how to make it themselves. So it's like a little bit of a they're intentionally using this label that has a lot of positive connotations but probably according to Open Source Initiative, which makes the open Source license, it should be called something else or there should just be like a new category for LLMs being but I don't want things to be more open. It could easily sound like a rebuke that it should be more open to make that point. But I also don't want to call it Open source because I think Open source software should probably does deserve a lot of its positive connotation, but they're not releasing the part, that the software part because that would cut into their business. I think it would be much worse. I think they shouldn't do it. But I also am not clear on this because the Open Source ML critics say that everyone does have access to the same data set as Llama Two. But I don't know. Llama Two had 7 billion tokens and that's more than GPT Four. And I don't understand all of the details here. It's possible that the tokenization process was different or something and that's why there were more. But Meta didn't say what was in the longitude data set and usually there's some description given of what's in the data set that led some people to speculate that maybe they're using private data. They do have access to a lot of private data that shouldn't be. It's not just like the common crawl backup of the Internet. Everybody's basing their training on that and then maybe some works of literature they're not supposed to. There's like a data set there that is in question, but metas is bigger than bigger than I think well, sorry, I don't have a list in front of me. I'm not going to get stuff wrong, but it's bigger than kind of similar models and I thought that they have access to extra stuff that's not public. And it seems like people are asking if maybe that's part of the training set. But yeah, the ML people would have or the open source ML people that I've been talking to would have believed that anybody who's decent can just access all of the training sets that they've all used.

AARON

Aside, I tried to download in case I'm guessing, I don't know, it depends how many people listen to this. But in one sense, for a competent ML engineer, I'm sure open source really does mean that. But then there's people like me. I don't know. I knew a little bit of R, I think. I feel like I caught on the very last boat where I could know just barely enough programming to try to learn more, I guess. Coming out of college, I don't know, a couple of months ago, I tried to do the thing where you download Llama too, but I tried it all and now I just have like it didn't work. I have like a bunch of empty folders and I forget got some error message or whatever. Then I tried to train my own tried to train my own model on my MacBook. It just printed. That's like the only thing that a language model would do because that was like the most common token in the training set. So anyway, I'm just like, sorry, this is not important whatsoever.

HOLLY

Yeah, I feel like torn about this because I used to be a genomicist and I used to do computational biology and it was not machine learning, but I used a highly parallel GPU cluster. And so I know some stuff about it and part of me wants to mess around with it, but part of me feels like I shouldn't get seduced by this. I am kind of worried that this has happened in the AI safety community. It's always been people who are interested in from the beginning, it was people who are interested in singularity and then realized there was this problem. And so it's always been like people really interested in tech and wanting to be close to it. And I think we've been really influenced by our direction, has been really influenced by wanting to be where the action is with AI development. And I don't know that that was right.

AARON

Not personal, but I guess individual level I'm not super worried about people like you and me losing the plot by learning more about ML on their personal.

HOLLY

You know what I mean? But it does just feel sort of like I guess, yeah, this is maybe more of like a confession than, like a point. But it does feel a little bit like it's hard for me to enjoy in good conscience, like, the cool stuff.

AARON

Okay. Yeah.

HOLLY

I just see people be so attached to this as their identity. They really don't want to go in a direction of not pursuing tech because this is kind of their whole thing. And what would they do if we weren't working toward AI? This is a big fear that people express to me with they don't say it in so many words usually, but they say things like, well, I don't want AI to never get built about a pause. Which, by the way, just to clear up, my assumption is that a pause would be unless society ends for some other reason, that a pause would eventually be lifted. It couldn't be forever. But some people are worried that if you stop the momentum now, people are just so luddite in their insides that we would just never pick it up again. Or something like that. And, yeah, there's some identity stuff that's been expressed. Again, not in so many words to me about who will we be if we're just sort of like activists instead of working on.

AARON

Maybe one thing that we might actually disagree on. It's kind of important is whether so I think we both agree that Aipause is better than the status quo, at least broadly, whatever. I know that can mean different things, but yeah, maybe I'm not super convinced, actually, that if I could just, like what am I trying to say? Maybe at least right now, if I could just imagine the world where open eye and Anthropic had a couple more years to do stuff and nobody else did, that would be better. I kind of think that they are reasonably responsible actors. And so I don't know. I don't think that actually that's not an actual possibility. But, like, maybe, like, we have a different idea about, like, the degree to which, like, a problem is just, like, a million different not even a million, but, say, like, a thousand different actors, like, having increasingly powerful models versus, like, the actual, like like the actual, like, state of the art right now, being plausibly near a dangerous threshold or something. Does this make any sense to you?

HOLLY

Both those things are yeah, and this is one thing I really like about the pause position is that unlike a lot of proposals that try to allow for alignment, it's not really close to a bad choice. It's just more safe. I mean, it might be foregoing some value if there is a way to get an aligned AI faster. But, yeah, I like the pause position because it's kind of robust to this. I can't claim to know more about alignment than OpenAI or anthropic staff. I think they know much more about it. But I have fundamental doubts about the concept of alignment that make me think I'm concerned about even if things go right, like, what perverse consequences go nominally right, like, what perverse consequences could follow from that. I have, I don't know, like a theory of psychology that's, like, not super compatible with alignment. Like, I think, like yeah, like humans in living in society together are aligned with each other, but the society is a big part of that. The people you're closest to are also my background in evolutionary biology has a lot to do with genetic conflict.

AARON

What is that?

HOLLY

Genetic conflict is so interesting. Okay, this is like the most fascinating topic in biology, but it's like, essentially that in a sexual species, you're related to your close family, you're related to your ken, but you're not the same as them. You have different interests. And mothers and fathers of the same children have largely overlapping interests, but they have slightly different interests in what happens with those children. The payoff to mom is different than the payoff to dad per child. One of the classic genetic conflict arenas and one that my advisor worked on was my advisor was David Haig, was pregnancy. So mom and dad both want an offspring that's healthy. But mom is thinking about all of her offspring into the future. When she thinks about how much.

AARON

When.

HOLLY

Mom is giving resources to one baby, that is in some sense depleting her ability to have future children. But for dad, unless the species is.

AARON

Perfect, might be another father in the future.

HOLLY

Yeah, it's in his interest to take a little more. And it's really interesting. Like the tissues that the placenta is an androgenetic tissue. This is all kind of complicated. I'm trying to gloss over some details, but it's like guided more by genes that are active in when they come from the father, which there's this thing called genomic imprinting that first, and then there's this back and forth. There's like this evolution between it's going to serve alleles that came from dad imprinted, from dad to ask for more nutrients, even if that's not good for the mother and not what the mother wants. So the mother's going to respond. And you can see sometimes alleles are pretty mismatched and you get like, mom's alleles want a pretty big baby and a small placenta. So sometimes you'll see that and then dad's alleles want a big placenta and like, a smaller baby. These are so cool, but they're so hellishly complicated to talk about because it involves a bunch of genetic concepts that nobody talks about for any other reason.

AARON

I'm happy to talk about that. Maybe part of that dips below or into the weeds threshold, which I've kind of lost it, but I'm super interested in this stuff.

HOLLY

Yeah, anyway, so the basic idea is just that even the people that you're closest with and cooperate with the most, they tend to be clearly this is predicated on our genetic system. There's other and even though ML sort of evolves similarly to natural selection through gradient descent, it doesn't have the same there's no recombination, there's not genes, so there's a lot of dis analogies there. But the idea that being aligned to our psychology would just be like one thing. Our psychology is pretty conditional. I would agree that it could be one thing if we had a VNM utility function and you could give it to AGI, I would think, yes, that captures it. But even then, that utility function, it covers when you're in conflict with someone, it covers different scenarios. And so I just am like not when people say alignment. I think what they're imagining is like an omniscient. God, who knows what would be best? And that is different than what I think could be meant by just aligning values.

AARON

No, I broadly very much agree, although I do think at least this is my perception, is that based on the right 95 to 2010 Miri corpus or whatever, alignment was like alignment meant something that was kind of not actually possible in the way that you're saying. But now that we have it seems like actually humans have been able to get ML models to understand basically human language pretty shockingly. Well, and so actually, just the concern about maybe I'm sort of losing my train of thought a little bit, but I guess maybe alignment and misalignment aren't as binary as they were initially foreseen to be or something. You can still get a language model, for example, that tries to well, I guess there's different types of misleading but be deceptive or tamper with its reward function or whatever. Or you can get one that's sort of like earnestly trying to do the thing that its user wants. And that's not an incoherent concept anymore.

HOLLY

No, it's not. Yeah, so yes, there is like, I guess the point of bringing up the VNM utility function was that there was sort of in the past a way that you could mathematically I don't know, of course utility functions are still real, but that's not what we're thinking anymore. We're thinking more like training and getting the gist of what and then getting corrections when you're not doing the right thing according to our values. But yeah, sorry. So the last piece I should have said originally was that I think with humans we're already substantially unaligned, but a lot of how we work together is that we have roughly similar capabilities. And if the idea of making AGI is to have much greater capabilities than we have, that's the whole point. I just think when you scale up like that, the divisions in your psyche or are just going to be magnified as well. And this is like an informal view that I've been developing for a long time, but just that it's actually the low capabilities that allows alignment or similar capabilities that makes alignment possible. And then there are, of course, mathematical structures that could be aligned at different capabilities. So I guess I have more hope if you could find the utility function that would describe this. But if it's just a matter of acting in distribution, when you increase your capabilities, you're going to go out of distribution or you're going to go in different contexts, and then the magnitude of mismatch is going to be huge. I wish I had a more formal way of describing this, but that's like my fundamental skepticism right now that makes me just not want anyone to build it. I think that you could have very sophisticated ideas about alignment, but then still just with not when you increase capabilities enough, any little chink is going to be magnified and it could be yeah.

AARON

Seems largely right, I guess. You clearly have a better mechanistic understanding of ML.

HOLLY

I don't know. My PiBBs project was to compare natural selection and gradient descent and then compare gradient hacking to miotic drive, which is the most analogous biological this is a very cool thing, too. Meatic drive. So Meiosis, I'll start with that for everyone.

AARON

That's one of the cell things.

HOLLY

Yes. Right. So Mitosis is the one where cells just divide in your body to make more skin. But Meiosis is the special one where you go through two divisions to make gametes. So you go from like we normally have two sets of chromosomes in each cell, but the gametes, they recombine between the chromosomes. You get different combinations with new chromosomes and then they divide again to bring them down to one copy each. And then like that, those are your gametes. And the gametes eggs come together with sperm to make a zygote and the cycle goes on. But during Meiosis, the point of it is to I mean, I'm going to just assert some things that are not universally accepted, but I think this is by far the best explanation. But the point of it is to take this like, you have this huge collection of genes that might have individually different interests, and you recombine them so that they don't know which genes they're going to be with in the next generation. They know which genes they're going to be with, but which allele of those genes. So I'm going to maybe simplify some terminology because otherwise, what's to stop a bunch of genes from getting together and saying, like, hey, if we just hack the Meiosis system or like the division system to get into the gametes, we can get into the gametes at a higher rate than 50%. And it doesn't matter. We don't have to contribute to making this body. We can just work on that.

AARON

What is to stop that?

HOLLY

Yeah, well, Meiosis is to stop that. Meiosis is like a government system for the genes. It makes it so that they can't plan to be with a little cabal in the next generation because they have some chance of getting separated. And so their best chance is to just focus on making a good organism. But you do see lots of examples in nature of where that cooperation is breaking down. So some group of genes has found an exploit and it is fucking up the species. Species do go extinct because of this. It's hard to witness this happening. But there are several species. There's this species of cedar that has a form of this which is, I think, maternal genome. It's maternal genome elimination. So when the zygote comes together, the maternal chromosomes are just thrown away and it's like terrible because that affects the way that the thing works and grows, that it's put them in a death spiral and they're probably going to be extinct. And they're trees, so they live a long time, but they're probably going to be extinct in the next century. There's lots of ways to hack meiosis to get temporary benefit for genes. This, by the way, I just think is like nail in the coffin. Obviously, gene centered view is the best evolutionarily. What is the best the gene centered view of evolution.

AARON

As opposed to sort of standard, I guess, high school college thing would just be like organisms.

HOLLY

Yeah, would be individuals. Not that there's not an accurate way to talk in terms of individuals or even in terms of groups, but to me, conceptually.

AARON

They'Re all legit in some sense. Yeah, you could talk about any of them. Did anybody take like a quirk level? Probably not. That whatever comes below the level of a gene, like an individual.

HOLLY

Well, there is argument about what is a gene because there's multiple concepts of genes. You could look at what's the part that makes a protein or you can look at what is the unit that tends to stay together in recombination or something like over time.

AARON

I'm sorry, I feel like I cut you off. It's something interesting. There was meiosis.

HOLLY

Meiotic drive is like the process of hacking meiosis so that a handful of genes can be more represented in the next generation. So otherwise the only way to get more represented in the next generation is to just make a better organism, like to be naturally selected. But you can just cheat and be like, well, if I'm in 90% of the sperm, I will be next in the next generation. And essentially meiosis has to work for natural selection to work in large organisms with a large genome and then yeah, ingredient descent. We thought the analogy was going to be with gradient hacking, that there would possibly be some analogy. But I think that the recombination thing is really the key in Meadic Drive. And then there's really nothing like that in.

AARON

There'S. No selection per se. I don't know, maybe that doesn't. Make a whole lot of sense.

HOLLY

Well, I mean, in gradient, there's no.

AARON

G in analog, right?

HOLLY

There's no gene analog. Yeah, but there is, like I mean, it's a hill climbing algorithm, like natural selection. So this is especially, I think, easy to see if you're familiar with adaptive landscapes, which looks very similar to I mean, if you look at a schematic or like a model of an illustration of gradient descent, it looks very similar to adaptive landscapes. They're both, like, in dimensional spaces, and you're looking at vectors at any given point. So the adaptive landscape concept that's usually taught for evolution is, like, on one axis you have fitness, and on the other axis you have well, you can have a lot of things, but you have and you have fitness of a population, and then you have fitness on the other axis. And what it tells you is the shape of the curve there tells you which direction evolution is going to push or natural selection is going to push each generation. And so with gradient descent, there's, like, finding the gradient to get to the lowest value of the cost function, to get to a local minimum at every step. And you follow that. And so that part is very similar to natural selection, but the Miosis hacking just has a different mechanism than gradient hacking would. Gradient hacking probably has to be more about I kind of thought that there was a way for this to work. If fine tuning creates a different compartment that doesn't there's not full backpropagation, so there's like kind of two different compartments in the layers or something. But I don't know if that's right. My collaborator doesn't seem to think that that's very interesting. I don't know if they don't even.

AARON

Know what backup that's like a term I've heard like a billion times.

HOLLY

It's updating all the weights and all the layers based on that iteration.

AARON

All right. I mean, I can hear those words. I'll have to look it up later.

HOLLY

You don't have to full I think there are probably things I'm not understanding about the ML process very well, but I had thought that it was something like yeah, like in yeah, sorry, it's probably too tenuous. But anyway, yeah, I've been working on this a little bit for the last year, but I'm not super sharp on my arguments about that.

AARON

Well, I wouldn't notice. You can kind of say whatever, and I'll nod along.

HOLLY

I got to guard my reputation off the cuff anymore.

AARON

We'll edit it so you're correct no matter what.

HOLLY

Have you ever edited the Oohs and UMS out of a podcast and just been like, wow, I sound so smart? Like, even after you heard yourself the first time, you do the editing yourself, but then you listen to it and you're like, who is this person? Looks so smart.

AARON

I haven't, but actually, the 80,000 Hours After hours podcast, the first episode of theirs, I interviewed Rob and his producer Kieran Harris, and that they have actual professional sound editing. And so, yeah, I went from totally incoherent, not totally incoherent, but sarcastically totally incoherent to sounding like a normal person. Because of that.

HOLLY

I used to use it to take my laughter out of I did a podcast when I was an organizer at Harvard. Like, I did the Harvard Effective Alchruism podcast, and I laughed a lot more than I did now than I do now, which is kind of like and we even got comments about it. We got very few comments, but they were like, girl hosts laughs too much. But when I take my laughter out, I would do it myself. I was like, wow, this does sound suddenly, like, so much more serious.

AARON

Yeah, I don't know. Yeah, I definitely say like and too much. So maybe I will try to actually.

HOLLY

Realistically, that sounds like so much effort, it's not really worth it. And nobody else really notices. But I go through periods where I say like, a lot, and when I hear myself back in interviews, that really bugs me.

AARON

Yeah.

HOLLY

God, it sounds so stupid.

AARON

No. Well, I'm definitely worse. Yeah. I'm sure there'll be a way to automate this. Well, not sure, but probably not too distant.

HOLLY

Future people were sending around, like, transcripts of Trump to underscore how incoherent he is. I'm like, I sound like that sometimes.

AARON

Oh, yeah, same. I didn't actually realize that this is especially bad. When I get this transcribed, I don't know how people this is a good example. Like the last 10 seconds, if I get it transcribed, it'll make no sense whatsoever. But there's like a free service called AssemblyAI Playground where it does free dr

AARONased transcription and that makes sense. But if we just get this transcribed without identifying who's speaking, it'll be even worse than that. Yeah, actually this is like a totally random thought, but I actually spent not zero amount of effort trying to figure out how to combine the highest quality transcription, like whisper, with the slightly less good

AARONased transcriptions. You could get the speaker you could infer who's speaking based on the lower quality one, but then replace incorrect words with correct words. And I never I don't know, I'm.

HOLLY

Sure somebody that'd be nice. I would do transcripts if it were that easy, but I just never have but it is annoying because I do like to give people the chance to veto certain segments and that can get tough because even if I talk you.

AARON

Have podcasts that I don't know about.

HOLLY

Well, I used to have the Harvard one, which is called the turning test. And then yeah, I do have I.

AARON

Probably listened to that and didn't know it was you.

HOLLY

Okay, maybe Alish was the other host.

AARON

I mean, it's been a little while since yeah.

HOLLY

And then on my I like, publish audio stuff sometimes, but it's called low effort. To underscore.

AARON

Oh, yeah, I didn't actually. Okay. Great minds think alike. Low effort podcasts are the future. In fact, this is super intelligent.

HOLLY

I just have them as a way to catch up with friends and stuff and talk about their lives in a way that might recorded conversations are just better. You're more on and you get to talk about stuff that's interesting but feels too like, well, you already know this if you're not recording it.

AARON

Okay, well, I feel like there's a lot of people that I interact with casually that I don't actually they have these rich online profiles and somehow I don't know about it or something. I mean, I could know about it, but I just never clicked their substack link for some reason. So I will be listening to your casual.

HOLLY

Actually, in the 15 minutes you gave us when we pushed back the podcast, I found something like a practice talk I had given and put it on it. So that's audio that I just cool. But that's for paid subscribers. I like to give them a little something.

AARON

No, I saw that. I did two minutes of research or whatever. Cool.

HOLLY

Yeah. It's a little weird. I've always had that blog as very low effort, just whenever I feel like it. And that's why it's lasted so long. But I did start doing paid and I do feel like more responsibility to the paid subscribers now.

AARON

Yeah. Kind of the reason that I started this is because whenever I feel so much I don't know, it's very hard for me to write a low effort blog post. Even the lowest effort one still takes at the end of the day, it's like several hours. Oh, I'm going to bang it out in half an hour and no matter what, my brain doesn't let me do that.

HOLLY

That usually takes 4 hours. Yeah, I have like a four hour and an eight hour.

AARON

Wow. I feel like some people apparently Scott Alexander said that. Oh, yeah. He just writes as fast as he talks and he just clicks send or whatever. It's like, oh, if I could do.

HOLLY

That, I would have written in those paragraphs. It's crazy. Yeah, you see that when you see him in person. I've never met him, I've never talked to him, but I've been to meetups where he was and I'm at this conference or not there right now this week that he's supposed to be at.

AARON

Oh, manifest.

HOLLY

Yeah.

AARON

Nice. Okay.

HOLLY

Cool Lighthaven. They're now calling. It looks amazing. Rose Garden. And no.

AARON

I like, vaguely noticed. Think I've been to Berkeley, I think twice. Right? Definitely. This is weird. Definitely once.

HOLLY

Berkeley is awesome. Yeah.

AARON

I feel like sort of decided consciously not to try to, or maybe not decided forever, but had a period of time where I was like, oh, I should move there, or we'll move there. But then I was like I think being around other EA's in high and rational high concentration activates my status brain or something. It is very less personally bad. And DC is kind of sus that I was born here and also went to college here and maybe is also a good place to live. But I feel like maybe it's actually just true.

HOLLY

I think it's true. I mean, I always like the DCAS. I think they're very sane.

AARON

I think both clusters should be more like the other one a little bit.

HOLLY

I think so. I love Berkeley and I think I'm really enjoying it because I'm older than you. I think if you have your own personality before coming to Berkeley, that's great, but you can easily get swept. It's like Disneyland for all the people I knew on the internet, there's a physical version of them here and you can just walk it's all in walking distance. That's all pretty cool. Especially during the pandemic. I was not around almost any friends and now I see friends every day and I get to do cool stuff. And the culture is sometimes it's like a really annoying near miss for me, but a lot of the times it's just like, oh, wow, how do I know so many people who are so similar to me? This is great.

AARON

Yeah, that's definitely cool. Yeah, I've definitely had that in Eags and stuff. Cool. I feel like you have a party, right?

HOLLY

You don't have to answer that Robin Hansen's talk. I mean, probably know what he's going to say. That's the thing when you know someone's rich online profile so well, it can be weird to see them in person and just hear them say only stuff from that only a subset of those things. I'm not saying Robin's, like like, I don't know, I haven't seen him enough in person. But Stephen Pinker was this way for like I was in the evolutionary biology department, but it was kind of close to the psychology department. And I went to a lab meeting there and I talked to Steve a few times and then he actually was, yeah, like, why don't we have a meeting and talk about your career? And I was such I had read every word he'd ever written at that.

AARON

Um, that's cool.

HOLLY

But I just had nothing to say to him. And then I realized pretty much everything I did say, I knew that he was going to answer because he's not someone who speaks very spontaneously. He pretty much has Cached chunks and loads them. The only spontaneous conversation we ever had was about AI and it was because we.

AARON

Listened to a lot of ADK. But I think I mean, I did talk to like for this other podcast episode and I don't know, I didn't have that. Totally. I feel like it was like I didn't know everything he was going to say, but who else would be like that?

HOLLY

Rob has a lot of off the cuff content. He doesn't say everything he thinks.

AARON

True. Yeah. Oh, we didn't talk about we can cut this part. We didn't talk about whether there's a conspiracy to not fund pause research or pause not research pause stuff. Do you want to have a comment that we can edit out?

HOLLY

I wouldn't call it a conspiracy, but I just think there's like, a reluctance to do it.

AARON

Yeah.

HOLLY

And some of it is like I think people are just being honest about it. They're like, yeah, it would get in the way of what I'm already doing. I'm trying to have a good relationship with AI companies and I feel like this would piss them off. I don't feel like they're giving their reasoning and it could make sense. I just think that they are wrong that their whole organization shouldn't be able to fund other causes.

AARON

If this is OpenPhil, I feel like that's not a good yeah. If you're like a multibillion dollar grant organization, it's very hard to have a single yeah, it's like that's not like a person with views who needs to it's not like a single agent necessarily. I mean, it kind of acts that way.

HOLLY

Yeah. I don't even know not sure how much I can say. Yeah. I'm not sure that AI companies expect that. I'm not sure if it's like that actual that's been communicated to people like OpenPhil and they are acting accordingly, or if they're just afraid of that and acting accordingly. I don't just I feel like there should be some way for OpenPhil or Dustin to fund advocacy interventions. I think part of it is that the people making those decisions aren't convinced of them, aren't convinced that advocacy is good. And I think there are some things like that. I don't know. It's hard for me to ignore that. Holden is married to Daniela Amade and they all used to live with his brother in law, dario Amade of Anthropic. And Daniel is also of like I'm not trying to say that there's something sinister going on, but it's just like, who wants to believe that their wife is doing something really bad if like, who wants to really go there and consider that possibility? I just think that's concerning. Of course, he's probably not thinking as clearly about that as somebody else would. That bothers me. I really was bothered by holden went on that six month sabbatical and came back with his playbook for AI safety. And it was just like, more of the same. He didn't even mention public advocacy. It was like the reason he went on that sabbatical it was because of well, never mind. I'm not sure of the reason he went on that sabbatical, but it was like the news that happened during that sabbatical was all about public is kind of into this now. It just seemed like he should at least engage on that, and he didn't. And he even suggested starting a new AI company. I just thought it just seems so dated. It just wasn't, considering the strategic position we're in now. And I kind of wondered if that was because, I don't know, he's really bought into what Daniela and Dario think.

AARON

About I'm kind of more bought into the perspective of much better than replacement cutting edge AI lab is like, maybe not good or something than you seem to be. I don't have a super strong view on this. I haven't thought about it nearly as much as either you or any of the people you just mentioned, but I don't know, it doesn't seem crazy.

HOLLY

Yeah, I guess I look at it as like that would be. I don't think it's impossible that somebody could just come up with the answer to alignment and if they're able to use that AI to make sure that nobody else makes unaligned AI or something like that, and it doesn't become a totalitarian dictatorship or something, all of those things, I don't think it's impossible. I don't even know how unlikely it is. If you told me in ten years that that's how it turned out, I would be like, oh, wow. But I wouldn't be like no. But as far as the best action to take and to advocate for, I think pause is the best. I think we don't have to help another AI lab get started, but our opportunity now is before we've gone far enough with AGI pursuits, is to implement a pause and have some barrier to if someone breaks the pause they're not like one step away from. I do just think that that's overall the best action to take, but if I'm just dispassionately mapping what could happen, I could see a lot of things happening. I could see alignment by default being true. I could see that we just like I don't know, there's just like something we don't get. Maybe we are just projecting our own instincts onto AI. That would surprise me less than everything going perfect, or like one singleton forming. That was good.

AARON

Yeah, maybe. Also, let me know whatever you want to wrap up much. I don't think I've made this a public take. Not that it's been a secret, but I think maybe even more, at least relative to the other AI safety pilled. Not the other, but relative to the AI safety pilled, like Twitter sphere or something like it. It seems pretty possible that OpenAI is I was going to say net good. I don't have problems with that phrase. epistemically.

HOLLY

It seems like they've done a really good job with the product so far. I'll definitely say that.

AARON

Yeah, I'm just a lot I don't know, I feel like it's easy to and I don't think they've acted perfectly or anthropic, but it's really easy to, I guess, miss it. It seems like in the world where, I don't know, meta and some random I don't know, whatever pick your other the next five labs or whoever would come along in the next five years or whatever, the world where those labs companies are at the cutting edge, it seems like a lot worse for maybe not super explicit reasons or reasons that are meta.

HOLLY

Just seems like less that's, like, all frankly, take that out, because I don't want to be making I want to be very on the up and up with what I'm saying about meta. But, yeah, I mean, just Yan LeCun's way of talking about and there was that article recently that alleged that Zuck just wants to be that he says things about just wanting to win and they think that open source is a way to do it and that Jan Lacoon is not just saying his opinion, it's calculated to undermine all the safety stuff.

AARON

It's so weird. Yeah. Also another just weird thing is that even though all of this is in some sense in some sense, it's like the extreme cutting edge of capitalism. On the other sense, okay, the key movers here have more money. It's like marginal money. Probably doesn't actually per se is probably not actually directly good for them or whatever. Once you have $100 million or whatever, the next million dollars isn't all that great. And it seems like a lot of them are, if not ethically motivated motivated by things beyond pure status, actually. Sorry, not pure status, but maybe at least like pure monetary incentives. Sorry, I sort of lost my train of thought.

HOLLY

I frequently think that people underrate the importance of the motive that just, like, people like doing what they're doing. They like their science, they like their work, and they don't want to think that it's bad. I just think, as simple as that, they really enjoy doing their work. They enjoy the kind of status that it brings, even if it's not financial, even if the wards aren't necessarily financial. The dynamic between Lacoon and Benjio and Hinton is really interesting because I'm just paraphrasing interactions I've remembered, but they seem to be saying, just give it up, Yan. We made a mistake. We need to course correct. And they both express henton and Benjio both expressed a lot of remorse about even though they didn't think that they did it on, but, like, they feel very sad that their life's work might have this legacy. And they seem to think that Yan Mccun is not dealing with that. And this could be a way of insisting that nothing's wrong and everything's good and just pushing harder in the other direction might be, like, a way of getting away from that possibility. I don't know.

AARON

Yeah, it sort of sucks that the psychology of a couple of dudes is quite important. Yeah. I don't know.

HOLLY

This is another area where my history of animal advocacy is interesting because I was a kid, vegetarian, and so I observed over many years how people would react to that and especially how they would react when they didn't think they had to make good arguments. It was one of the ways I first got interested in rationality, actually, because people would just give adults would just give the worst arguments for same so far. Yes, and I'm seeing that a lot with this. People who are unquestionably, the smartest people I knew are now saying the dumbest shit, now that pause is on the table and they're getting better about it. I mean, I think they were just taken aback at first, but they would say just like the dumbest reasons that it wasn't going to work, it just revealed. They obviously didn't want it to be a thing, or they didn't want to think about a new paradigm, or they kind of wanted things to be the way they were, where the focus was on technical stuff. I was having a conversation with somebody about the first instance of the Campaign for AI safety website. That's the Australian AI Safety Advocacy Group. And the first version of that website was a bit amateurish, I will definitely say, but I was in this thread and the people in it were making fun of it and picking on little things about it that didn't even make any sense. There was one line that was like ML engineers could be made to work on AI safety, or instead they could work on AI safety. Retrained was the word they used. And this is very similar. Like in vegan advocacy, you hear this all the time. Like slaughterhouse workers can be retrained in organic farming. It's not a great it's a little sillier in that case, very silly.

AARON

In the first case. I don't think it's that silly.

HOLLY

Yeah, but the point of that kind of thing is we care about the jobs of the people who be affected by this. And there are jobs in our thing.

AARON

Silicon Valley ML experts really struggling to make ends meet.

HOLLY

But that line was picked on and made fun of. And actually one person who was like a very smart person, knows a lot about the topic, was like, this would be like forced labor camps. And they might not have said camp, they might have just said forced labor program or something like that. And I was just like, what the dude? That's the most uncharitable explanation I've ever reaction I've ever heard. The reason that we can't pause or advocate for AI safety in public is that just everybody who wants to do it is too stupid. And so the only thing we can do is what you're doing, I guess, which I guess I won't say what it is because I want to maintain their anonymity. But it really struck me that happened in April and I just thought it was just very recognizable to me as the kind of terrible argument that only makes sense if you just think you have everybody's on your side and you can do a status move to keep people out or to keep something else out. That particular incident influenced me strongly to push for this harder because I don't know, if you're just present, like, making the argument more even if your argument is stupid, people just don't react that dumb.

AARON

No, I'm glad you updated in that. Like, I do think it's very good that AI safety seems NEA. It seems, like, pretty high. I don't know, it depends what status hierarchy you're talking about. But in all relevant domains, it seems pretty high status. And actually, it's kind of crazy how smart everybody is. This is my personal I don't know. Yeah, I feel like technical AI safety people really fucking smart. And so yeah, I've seen some people on Twitter say only once or twice because it's so far from true, but once or twice? Yeah, I guess they're just not smart enough to work in ML. It's like, okay, I don't know. It's like the farthest possible thing from the truth.

HOLLY

Yeah. The ML people, the open source ML people who are trying to hurt my feelings definitely want to go in on, like, I'm not smart enough, or my degree isn't a dumb subject or something. Yeah, it's great to be smart, but there just are more important things, and I just don't think you have to be a genius to see the logic of what I'm saying. Anyway, what I was saying was there's like a status quo or a relative status quo that a lot of people were comfortable even. I think Jan Lacoon was comfortable with being cool ML genius and doesn't want there to be some moral or ethical question with it. At least that's the picture I get from his interaction with the other Turing Prize winners. And then within AI safety, people don't really want to think about switching gears. Or maybe the landscape has shifted and now the next move is something that's not the skills they've spent all their time developing and not the skills that kind of got them into this whole thing, which I don't want anybody working on technical stuff to quit or something.

AARON

Yeah, the soy lent is just adds to the ethos.

HOLLY

Yeah, guys, I've been drinking a soy lent the whole time. It's not that I love them, but I do go through these periods where I feel like kind of nauseous and don't want to eat, and, like, soylent is whatever works.

AARON

Yeah, cool. I think I'm, like, slightly running out.

HOLLY

Of steam, which is like, there by four.

AARON

Okay. Yeah. But you are invited back on pigeon hour anytime. Not literally anytime, but virtually anytime.

HOLLY

We can record one for my thing.

AARON

Oh, yeah, totally. Any closing takes? Thoughts? I don't have any. You don't have to either.

HOLLY

Yeah, it was a fun time. Thank you.

AARON

Oh, cool. Yeah, no, maybe at some other point we can just discuss all your Evo biology takes or whatever, because that was quite interesting.

HOLLY

Oh, yeah. There's going to be maybe this chat cone thing, which is like, the less Wrong did, like, the Miri conversations last year, and they're trying to replicate that for more topics. And there might be one on evolution soon that I might be part of.

AARON

I'll keep an eye on that.

HOLLY

So I don't know if accompanying readings are fun for the podcast. Anyway. Yeah, I should probably go because I also need to pee. I've had three different liquids over here this whole time.

AARON

Okay. That's a great reason. Thank you so much.

HOLLY

Okay, bye. Thank you.

Get full access to Aaron's Blog at www.aaronbergman.net/subscribe

Listen on:

* Spotify

* Apple Podcasts

* Google Podcasts

Note: the core discussion on ethics begins at 7:58 and moves into philosophy of language at ~1:12:19

Daniel’s stuff:

* AI X-risk podcast

* The Filan Cabined podcast

* Personal website and blog

Blurb and bulleted summary from Clong

This wide-ranging conversation between Daniel and Aaron touches on movies, business drama, philosophy of language, ethics and legal theory. The two debate major ethical concepts like utilitarianism and moral realism. Thought experiments around rational beings choosing to undergo suffering feature prominently. meandering tangents explore the semantics of names and references.

* Aaron asserts that total utilitarianism does not imply that any amount of suffering can be morally justified by creating more happiness. His argument is that the affirmative case for this offsetting ability has not been clearly made.

* He proposes a thought experiment - if offered to experience the suffering of all factory farmed animals in exchange for unlimited happiness, even a perfectly rational being would refuse. This indicates there are some levels of suffering not offsettable.

* Aaron links this to experiences like hunger where you realize suffering can be worse than you appreciate normally. This causes his intuition some suffering can't be outweighed.

* Daniel disagrees, believing with the right probabilities and magnitudes of suffering versus happiness, rational beings would take that gamble.

* For example, Daniel thinks the atomic bombing of Japan could be offset by reducing more suffering. Aaron is less sure given the pain inflicted.

* Daniel also proposes offsets for animal farming, but Aaron doesn't think factory farming harm is offsettable by any amount of enjoyment of meat.

* They discuss definitions of rationality and whether evolution pressures against suicide impact the rationality of not killing oneself.

* Aaron ties his argument to siding with what a perfectly rational being would choose to experience, not necessarily what they would prefer.

* They debate whether hypothetical aliens pursuing "schmorality" could point to a concept truly analogous to human morality. Aaron believes not.

Transcript

(Very imperfect)

AARON

O'how's, it going it's going all right.

DANIEL

Yeah, I just so yesterday I saw Barbie and today I saw Oppenheimer, so it's good to oh, cool. That cultural.

AARON

Nice, nice.

DANIEL

Do you have takes? Yeah, I thought it was all right. It was a decent view of Oppenheimer as a person. It was like a how? I don't know. I feel like the public can tend to be taken in by this physicist figures you get this with quotes, right? Like, the guy was just very good at having fun with journalists, and now we get these amazing nuggets of wisdom from Einstein. I don't know. I think that guy was just having good I don't know. The thing that I'm coming away from is I thought I only watched Barbie because it was coming out on the same day as Oppenheimer, right? Like, otherwise it wouldn't have occurred to me to watch it. I was like, yeah, whatever. Barbie is, like, along for the ride, and Oppenheimer is going to be amazing, but in like, maybe Oppenheimer was a bit better than Barbie, but I'm not even sure of that, actually.

AARON

Yeah, I've been seeing people say that on Twitter. I haven't seen either, but I've been seeing several people say that I'm following, say, like, Barbie was exceptional. And also that kind of makes sense because I'm following all these EA people who are probably care more about the subject matter for the latter one. So it's like, I kind of believe that Barbie is, like, aesthetically better or something. That's my take. Right.

DANIEL

Guess. Well, if you haven't seen them, I guess I don't want to spoil them for you. They're trying to do different things aesthetically. Right. Like, I'm not quite sure I'd want to say one is aesthetically better. Probably in some ways, I think Barbie probably has more aesthetic blunders than Oppenheimer does. Okay. But yeah, I don't know if you haven't seen it, I feel like I don't want to spoil it for you.

AARON

Okay. No, that's fine. This isn't supposed to be like probably isn't the most important the most interesting thing we could be talking about is that the bar?

DANIEL

Oh, jeez.

AARON

Oh, no, that's a terrible bar. That was like an overstatement. That would be a very high bar. It would also be, like, kind of paralyzing. I don't know. Actually know what that would be, honestly. Probably some social juicy gossip thing. Not that we necessarily have any.

DANIEL

Yeah, I think your interestingness. Yeah, I think I don't have the know, the closest to gossip thing I saw was like, do you see this bit of Carolyn Elson's diaries and letters to SBF that was leaked to the.

AARON

No, I don't. Was this like today or recently? How recently?

DANIEL

This was like a few days ago.

AARON

I've been seeing her face on Twitter, but I don't actually think I know anything about this. And no, I would not have.

DANIEL

Background of who she is and stuff.

AARON

Yeah, hold on. Let the audience know that I am on a beach family vacation against my will. Just kidding. Not against my will. And I have to text my sister back. Okay, there we go. I mean, I broadly know the FTX story. I know that she was wait, I'm like literally blanking on the Alameda.

DANIEL

That's the name of research.

AARON

Okay. Yeah. So she was CEO, right? Yeah. Or like some sort of like I think I know the basics.

DANIEL

The like, she was one of the OG Stanford EA people and was around.

AARON

Yeah, that's like a generation. Not an actual generation, like an EA generation. Which is what, like six years or.

DANIEL

Like the I don't know, I've noticed like, in the there's like I feel like there's this gap between pre COVID people and post COVID people. No one left their house. Partly people moved away, but also you were inside for a while and never saw anyone in person. So it felt like, oh, there's like this crop of new people or something. Whereas in previous years, there'd be some number of new people per year and they'd get gradually integrated in. Anyway, all that is to say that, I don't know, I think SBF's side of the legal battle leaked some documents to The New York Times, which were honestly just like her saying, like, oh, I feel very stressed and I don't like my job, and I'm sort of glad that the thing is blown up now. I don't know. It honestly wasn't that salacious. But I think that's, like, the way I get in the loop on gossip like some of the New York Times.

AARON

And I eventually I love how it's funny that this particular piece of gossip is, like, running through the most famous and prestigious news organization in the world. Or, like, one of them or something. Yeah. Instead of just being like, oh, yeah, these two people are dating, or whatever. Anyway, okay, I will maybe check that out.

DANIEL

Yeah, I mean, honestly, it's not even that interesting.

AARON

The whole thing is pretty I am pretty. This is maybe bad, but I can't wait to watch the Michael Lewis documentary, pseudo documentary or whatever.

DANIEL

Yeah, it'll be good to read the book. Yeah, it's very surreal. I don't know. I was watching Oppenheimer. Right. And I have to admit, part of what I'm thinking is be if humanity survives, there's going to be this style movie about open AI, presumably, right? And I'm like, oh, man, it'll be amazing to see my friend group depicted on film. But that is going to happen. It's just going to be about FTX and about how they're all criminals. So that's not great.

AARON

Yeah, actually, everybody dunks on crypto now, and it's like low status now or whatever. I still think it's really cool. I never had more than maybe $2,000 or whatever, which is not a trivial I mean, it's not a large amount of my money either, but it's not like, nothing. But I don't know, if it wasn't for all the cultural baggage, I feel like I would be a crypto bro or I would be predisposed to being a crypto bro or something.

DANIEL

Yeah. I should say I was like joking about the greedy crypto people who want their money to not be stolen. I currently have a Monero sticker on the back of my a big I don't know, I'm a fan of the crypto space. It seems cool. Yeah. I guess especially the bit that is less about running weird scams. The bit that's running weird scams I'm less of a fan of.

AARON

Yeah. Yes. I'm also anti scam. Right, thank you. Okay, so I think that thing that we were talking about last time we talked, which is like the thing I think we actually both know stuff about instead of just like, repeating New York Times articles is my nuanced ethics takes and why you think about talk about that and then we can just also branch off from there.

DANIEL

Yeah, we can talk about that.

AARON

Maybe see where that did. I luckily I have a split screen up, so I can pull up things. Maybe this is kind of like egotistical or something to center my particular view, but you've definitely given me some of the better pushback or whatever that I haven't gotten that much feedback of any kind, I guess, but it's still interesting to hear your take. So basically my ethical position or the thing that I think is true is that which I think is not the default view. I think most people think this is wrong is that total utilitarianism does not imply that for some amount of suffering that could be created there exists some other extremely large arbitrarily, large amount of happiness that could also be created which would morally justify the former. Basically.

DANIEL

So you think that even under total utilitarianism there can be big amounts of suffering such that there's no way to morally tip the calculus. However much pleasure you can create, it's just not going to outweigh the fact that you inflicted that much suffering on some people.

AARON

Yeah, and I'd highlight the word inflicted if something's already there and you can't do anything about it, that's kind of neither here nor there as it pertains to your actions or something. So it's really about you increasing, you creating suffering that wouldn't have otherwise been created. Yeah. It's also been a couple of months since I've thought about this in extreme detail, although I thought about it quite a bit. Yeah.

DANIEL

Maybe I should say my contrary view, I guess, when you say that, I don't know, does total utilitarianism imply something or not? I'm like, well, presumably it depends on what we mean by total utilitarianism. Right. So setting that aside, I think that thesis is probably false. I think that yeah. You can offset great amounts of suffering with great amounts of pleasure, even for arbitrary amounts of suffering.

AARON

Okay. I do think that position is like the much more common and even, I'd say default view. Do you agree with that? It's sort of like the implicit position of people who are of self described total utilitarians who haven't thought a ton about this particular question.

DANIEL

Yeah, I think it's probably the implicit default. I think it's the implicit default in ethical theory or something. I think that in practice, when you're being a utilitarian, I don't know, normally, if you're trying to be a utilitarian and you see yourself inflicting a large amount of suffering, I don't know. I do think there's some instinct to be like, is there any way we can get around this?

AARON

Yeah, for sure. And to be clear, I don't think this would look like a thought experiment. I think what it looks like in practice and also I will throw in caveats as I see necessary, but I think what it looks like in practice is like, spreading either wild animals or humans or even sentient digital life through the universe. That's in a non as risky way, but that's still just maybe like, say, making the earth, making multiple copies of humanity or something like that. That would be an example that's probably not like an example of what an example of creating suffering would be. For example, just creating another duplicate of earth. Okay.

DANIEL

Anything that would be like so much suffering that we shouldn't even the pleasures of earth outweighs.

AARON

Not necessarily, which is kind of a cop out. But my inclination is that if you include wild animals, the answer is yes, that creating another earth especially. Yeah, but I'm much more committed to some amount. It's like some amount than this particular time and place in human industry is like that or whatever.

DANIEL

Okay, can I get a feel of some other concrete cases to see?

AARON

Yeah.

DANIEL

So one example that's on my mind is, like, the atomic bombing of Hiroshima and Nagasaki, right? So the standard case for this is, like, yeah, what? A hundred OD thousand people died? Like, quite terrible, quite awful. And a lot of them died, I guess a lot of them were sort of some people were sort of instantly vaporized, but a lot of people died in extremely painful ways. But the countercase is like, well, the alternative to that would have been like, an incredibly grueling land invasion of Japan, where many more people would have died or know regardless of what the actual alternatives were. If you think about the atomic bombings, do you think that's like the kind of infliction of suffering where there's just not an offsetting amount of pleasure that could make that okay?

AARON

My intuition is no, that it is offsettable, but I would also emphasize that given the actual historical contingencies, the alternative, the implicit case for the bombing includes reducing suffering elsewhere rather than merely creating happiness. There can definitely be two bad choices that you have to make or something. And my claim doesn't really pertain to that, at least not directly.

DANIEL

Right. Sorry. But when you said you thought your answer was no, you think you can't offset that with pleasure?

AARON

My intuition is that you can, but I know very little about how painful those deaths were and how long they lasted.

DANIEL

Yeah, so the non offset so it's like, further out than atomic bombing.

AARON

That's my guess, but I'm like.

DANIEL

Okay, sure, that's your guess. You're not super confident. That's fine. I guess another thing would be, like, the animal farming system. So, as you're aware, tons of animals get kept in farms for humans to eat, by many count. Many of them live extremely horrible lives. Is there some amount that humans could enjoy meat such that that would be okay?

AARON

No. So the only reason I'm hesitating is because, like, the question is, like, what the actual alternative is here, but, like, if it's like, if it's, like, people enjoy, like, a meat a normal amount and there's no basically the answer is no. Although, like, what I would actually endorse doing depends on what the alternative is.

DANIEL

Okay, but you think that factory farming is so bad that it's not offsettable by pleasure.

AARON

Yeah, that's right. I'm somewhat maybe more confident than the atomic bombing case, but again, I don't know what it's like to be a factory farm pig. I wouldn't say I'm, like, 99% sure. Probably more than 70% or something. Or 70%, like, conditional on me being right about this thesis, I guess something like that, which I'm like. Yeah, okay. I don't know. Some percent, maybe, not probably not 99% sure, but also more than 60. Probably more than 70% sure or something.

DANIEL

All right. Yeah. So I guess maybe can you tell us a little bit about why you would believe that there's some threshold that you like where you can no longer compensate by permitting pleasure?

AARON

Yes. Let me run through my argument and sort of a motivation, and the motivation actually is sort of more a direct answer to what you just said. So the actual argument that I have and I have a blog post about this that I'll link, it was part of an EA forum post also that you'll also link in the show description is that the affirmative default case doesn't seem to actually be made anywhere. That's not the complete argument, but it's a core piece of it, which is that it seems to be, like, the default received view, which doesn't mean it's wrong, but does mean that we should be skeptical. If you accept that I'm right, that the affirmative case hasn't been made, we can talk about that. Then you should default to some other heuristic. And the heuristic that I assert and sort of argue, but kind of just assert is a good heuristic is. Okay. Is you do the following thought experiment. If I was a maximally or perfectly rational being, would I personally choose to undergo this amount of suffering in compensation or not compensation, exchange for later undergoing or earlier undergoing some arbitrarily large amount of happiness. And I personally have the intuition that there are events or things that certainly conceivable states and almost certainly possible states that I could be in such that even as a rational being, like as a maximum rational being, I would choose to just disappear and not exist rather than undergo both of these things.

DANIEL

Okay.

AARON

Yeah.

DANIEL

Why do you think that?

AARON

Yeah, so good question. I think the answer comes at a couple of different levels. So there's a question of why I'm saying it and why I'm saying it is because I'm pretty sure this is the answer I would actually give if actually given if Credibly offered this option. But that just pushes the question back. Okay, why do I feel that.

DANIEL

Even what option are we talking about here? There exists a thing such that for.

AARON

All pleasures, basically, for example, let's just run with the fact, the assumption that a genie God descends. And I think it's credible, and he offers that I can live the life of every factory, farmed animal in exchange for whatever I want for any amount of time or something like that. Literally, I don't have to give the answer now. It can just be like an arbitrarily good state for an arbitrarily long period of time.

DANIEL

Oh, yeah.

AARON

And not only would I say the words no, I don't want to do that, I think that the words no, I don't want to do that, are selfishly in a non pejorative sense. Correct. And then there's a question of why do I have that intuition? And now I'm introspecting, which is maybe not super reliable. I think part of my intuition that I can kind of maybe sort of access via introspection just comes from basically, I'm very fortunate to not have had a mostly relatively comfortable life, like as a Westerner with access to painkillers, living in the 21st century. Even still, there have definitely been times when I've been suffered, at least not in a relative sense, but just like, in an absolute sense to me, in a pretty bad way. And one example I can give was just like, I was on a backpacking trip, and this is the example I give in another blog post I can link. I was on a backpacking trip, and we didn't have enough food, and I was basically very hungry for like five days. And I actually think that this is a good and I'm rambling on, but I'll finish up. I think it's illustrative. I think there's some level of suffering where you're still able to do at least for me, I'm still able to do something like reasoning and intentionally storing memories. One of the memories I tried to intentionally codify via language or something was like, yeah, this is really bad, this really sucks, or something like, that what.

DANIEL

Sucked about it, you were just like, really hungry yeah.

AARON

For five days.

DANIEL

Okay. And you codified the thought, like, feeling of this hunger I'm feeling, this really sucks.

AARON

Something like that. Right. I could probably explicate it more, but that's basically okay. Actually, hold on. All right. Let me add so not just it really sucks, but it sucks in a way that I can't normally appreciate, so I don't normally have access to how bad it sucks. I don't want to forget about this later or something.

DANIEL

Yeah. The fact that there are pains that are really bad where you don't normally appreciate how bad they are, it's not clear how that implies non offset ability.

AARON

Right, I agree. It doesn't.

DANIEL

Okay.

AARON

I do think that's causally responsible for my intuition that I lend link to a heuristic that I then argue does constitute an argument in the absence of other arguments for offset ability.

DANIEL

Yeah. Okay. So that causes this intuition, and then you give some arguments, and the argument is like, you think that if a genie offered you to live liable factory farmed animals in exchange for whatever you wanted, you wouldn't go for that.

AARON

Yes. And furthermore, I also wouldn't go for it if I was much more rational.

DANIEL

If you were rational, yeah. Okay. Yeah. What do I think about this? One thing I think is that the I think the case of live experience this suffering and then experience this pleasure, to me, I think that this is kind of the wrong way to go about this. Because the thing about experiencing suffering is that it's not just we don't live in this totally dualistic world where suffering just affects only your immaterial mind or something in a way where afterwards you could just be the same. In the real world, suffering actually affects you. Right. Perhaps indelibly. I think instead, maybe the thing I'd want to say is suppose you're offered a gamble, right, where there's like a 1% chance that you're going to have to undergo excruciating suffering and a 99% chance that you get extremely awesome pleasures or something.

AARON

Yeah.

DANIEL

And this is meant to model a situation in which you do some action in which one person is going to undergo really bad suffering and 99 other people are going to undergo really great pleasure. And to me, I guess my intuition is that for any bad thing, you could make the probability small enough and you can make the rest of the probability mass good enough that I want to do that. I feel like that's worth it for me. And now it feels a little bit unsatisfying that we're just going that we're both drilling down to, like, well, this is the choice I would make, and then maybe you can disagree that it's the choice you would make. But yeah, I guess about the gambling case, what do you think about that? Let's say it's literally a one in a million chance that you would have to undergo, let's say, the life of one factory farmed animal.

AARON

Yeah.

DANIEL

Or is that not enough? Do you want it to be like, more?

AARON

Well, I guess it would have to be like one of the worst factory farmed animals. Life, I think would make that like.

DANIEL

Yeah, okay, let's say it's like, maybe literally one in a billion chance.

AARON

First of all, I do agree that these are basically isomorphic or morally equivalent, or if anything, time ordering in my example does mess things up a little bit, I'll be happy to reverse them or say that instead compare one person to 1000 people. So, yeah, you can make the probability small enough that my intuition changes. Yeah. So in fact, 1%, I'm very like, no, definitely not doing that. One in a million. I'm like, I don't know, kind of 50 50. I don't have a strong intuition either way. 100 trillion. I have the intuition. You know what? That's just not going to happen. That's my first order intuition. I do think that considering the case where you live, one being lives both lives, or you have, say, one being undergoing the suffering and then like 100 trillion undergoing the pleasure makes small probabilities more if you agree that they're sort of isomorphic makes them more complete or something like that, or complete more real in some. Not tangible is not the right word, but more right.

DANIEL

You're less tempted to round it to zero.

AARON

Yeah. And so I tend to think that I trust my intuitions more about reasoning. Okay, there's one person undergoing suffering and like 100 trillion undergoing happiness as it pertains to the question of offset ability more than I trust my intuitions about small probabilities.

DANIEL

I guess that's strange because that strikes me as strange because I feel like you're regularly in situations where you make choices that have some probability of causing you quite bad suffering, but a large probability of being fun. Like going to the beach. There could be a shark there. I guess this is maybe against your will, but you can go to a restaurant, maybe get food poisoning, but how often are you like, oh man, if I flip this switch, one person will be poisoned, but 99 people will?

AARON

Well, then you'd have to think that, okay, staying home would actually be safer for some reason, which I don't affirmatively think is true, but this actually does work out for the question of whether you should kill yourself. And there hopefully this doesn't get censored by Apple or whatever, so nobody do that. But there I just think that my lizard brain or there's enough evolutionary pressure to not trust that I would be rational when it comes to the question of whether to avoid a small chance of suffering by unaliving myself, as they say on TikTok.

DANIEL

Hang on, evolution is pressured. So there's some evolutionary pressure to make sure you really don't want to kill yourself, but you think that's like, irrational.

AARON

I haven't actually given this a ton of thought. It gets hard when you loop in altruism and yeah, the question also there's like some chance that of sentient's after death, there's not literally zero or something like that. Yeah, I guess those are kind of cop outs. So I don't know, I feel like it certainly could be. And I agree this is sort of like a strike against my argument or something. I can set up a situation you have no potential to improve the lives of others, and you can be absolutely sure that you're not going to experience any sentience after death. And then I feel like my argument does kind of imply that, yeah, that's like the rational thing to do. I wouldn't do it. Right. So I agree. This is like a strike against me.

DANIEL

Yeah. I guess I just want to make two points. So the first point I want to make is just methodologically. If we're talking about which are you likely to be more rational about gambles of small risks, small probabilities of risk versus large rewards as opposed to situations where you can do a thing that affects a large number of people one way and a small number of people another way? I think the gambles are more like decisions that you make a bunch and you should be rational about and then just the second thing in terms of like, I don't know, I took you to be making some sort of argument along the lines of there's evolutionary pressure to want to not kill yourself. Therefore, that's like a debunking explanation. The fact that there was evolutionary pressure to not kill ourselves means that our instinct that we shouldn't kill ourselves is irrational. Whereas I would tend to look at it and say the fact that there was very strong evolutionary pressure to not kill ourselves is an explanation of why I don't want to kill myself. And I see that as affirming the choice to not kill myself, actually.

AARON

Well, I just want to say I don't think it's an affirmative argument that it is irrational. I think it opens up the question. I think it means it's more plausible that for other I guess not even necessarily for other reasons, but it just makes it more plausible that it is irrational. Well.

DANIEL

Yeah, I take exactly the opposite view. Okay. I think that if I'm thinking about, like, oh, what do I really want? If I consider my true preferences, do I really want to kill myself or something? And then I learn that, oh, evolution has shaped me to not kill myself, I think the inference I should make is like, oh, I guess probably the way evolution did that is that it made it such that my true desires are to not kill myself.

AARON

Yeah. So one thing is I just don't think preferences have any intrinsic value. So I don't know, we might just like I guess I should ask, do you agree with that or disagree with.

DANIEL

That do I think preferences have intrinsic value? No, but so no, but I think like, the whole game here is like, what do I prefer? Or like, what would I prefer if I understood things really clearly?

AARON

Yes. And this is something I didn't really highlight or maybe I didn't say it at all, is that I forget if I really argue it or kind of just assert it, but I at least assert that the answer to hedonic utilitarian. What you should do under hedonic utilitarianism is maybe not identical to, but exactly the same as what a rational agent would do or what a rational agent would prefer if they were to experience everything that this agent would cause. Or something like that. And so these should give you the exact same answers is something I believe sure. Because I do think preferences are like we're built to understand or sort of intuit and reason about our own preferences.

DANIEL

Kind of, yeah. But broadly, I guess the point I'm making at a high level is just like if we're talking about what's ethical or what's good or whatever, I take this to ultimately be a question about what should I understand myself as preferring? Or to the extent that it's not a question of that, then it's like, I don't know, then I'm a bit less interested in the exercise.

AARON

Yeah. It's not ideal that I appeal to this fake and that fake ideally rational being or something. But here's a reason you might think it's more worth thinking about this. Maybe you've heard about I think Tomasic makes an argument about yeah. At least in principle, you can have a pig that's in extreme pain but really doesn't want to be killed still or doesn't want to be taken out of its suffering or whatever, true ultimate preference or whatever. And so at least I think this is pretty convincing evidence that you can have where that's just like, wrong about what would be good for it, you know what I mean?

DANIEL

Yeah, sorry, I'm not talking about preference versus hedonic utilitarianism or anything. I'm talking about what do I want or what do I want for living things or something. That's what I'm talking about.

AARON

Yeah. That language elicits preferences to me and I guess the analogous but the idea.

DANIEL

Is that the answer to what I want for living things could be like hedonic utilitarianism, if you see what I mean.

AARON

Or it could be by that do you mean what hedonic utilitarianism prescribes?

DANIEL

Yeah, it could be that what I want is that just whatever maximizes beings pleasure no matter what they want.

AARON

Yeah. Okay. Yeah, so I agree with that.

DANIEL

Yeah. So anyway, heading back just to the suicide case right. If I learn that evolution has shaped me to not want to kill myself, then that makes me think that I'm being rational in my choice to not kill myself.

AARON

Why?

DANIEL

Because being rational is something like optimally achieving your goals. And I'm a little bit like I sort of roughly know the results of killing myself, right? There might be some question about like, but what are my goals? And if I learned that evolution has shaped my goals such that I would hate killing myself right, then I'm like, oh, I guess killing myself probably ranks really low on the list of states ordered by how much I like them.

AARON

Yeah, I guess then it seems like you have two mutually incompatible goals. Like, one is staying alive and one is hedonic utilitarianism and then you have to choose which of these predominates or whatever.

DANIEL

Yeah, well, I think that to the extent that evolution is shaping me to not want to commit suicide, it looks like the not killing myself one is winning. I think it's evidence. I don't think it's conclusive. Right. Because there could be multiple things going on. But I take evolutionary explanations for why somebody would want X. I think that's evidence that they are rational in pursuing X rather than evidence that they are irrational in pursuing X.

AARON

Sometimes that's true, but not always. Yeah, there's a lot in general it is. Yeah. But I feel like moral anti realistic, we can also get into that. Are going to not think this is like woo or Joe Carl Smith says when he's like making fun of moralists I don't know, in a tongue in cheek way. In one of his posts arguing for explicating his stance on antirealism basically says moral realists want to say that evolution is not sensitive to moral reasons and therefore evolutionary arguments. Actually, I don't want to quote him from memory. I'll just assert that evolution is sensitive to a lot of things, but one of them is not moral reasons and therefore evolutionary arguments are not a good evidence or are not good evidence when it comes to purely, maybe not even purely, but philosophical claims or object level moral claims, I guess, yeah, they can be evidenced by something, but not that.

DANIEL

Yeah, I think that's wrong because I think that evolution why do I think it's wrong? I think it's wrong because what are we talking about when we talk about morality? We're talking about some logical object that's like the completion of a bunch of intuitions we have. Right. And those I haven't thought about intuitions are the product of evolution. The reason we care about morality at all is because of evolution under the standard theory that evolution is the reason our brains are the way they are.

AARON

Yeah, I think this is a very strange coincidence and I am kind of weirded out by this, but yes, I.

DANIEL

Don'T think it's a coincidence or like not a coincidence.

AARON

So it's not a coincidence like conditional honor, evolutionary history. It is like no extremely lucky or something that we like, of course we'd find it earthlings wound up with morality and stuff. Well, of course you would.

DANIEL

Wait. Have you read the metafic sequence by Elizar? Yudkowski.

AARON

I don't think so. And I respect Elias a ton, except I think he's really wrong about ethics and meta ethics in a lot of like I don't even know if I but I have not, so I'm not really giving them full time.

DANIEL

Okay. I don't know. I basically take this from my understanding of the meta ethics sequence, which I recommend people read, but I don't think it's a coincidence. I don't think we got lucky. I think it's a coincidence. There are some species that get evolved, right, and they end up caring about schmorality, right?

AARON

Yeah.

DANIEL

And there are some species that get evolved, right? And they end up caring about the prime numbers or whatever, and we evolved and we ended up caring about morality. And it's not like a total so, okay, partly I'm just like, yeah, each one of them is really glad they didn't turn out to be the other things. The ones that care about two of.

AARON

Them are wrong, but two of them are wrong.

DANIEL

Well, they're morally wrong. Two of them do morally wrong things all the time. Right?

AARON

I want to say that I hate when people say that. Sorry. So what I am saying is that you can call those by different names, but if I'm understanding this argument right, they all think that they're getting at the same core concept, which is like, no, what should we do in some okay, so does schmorality have any sort of normativity?

DANIEL

No, it has schmormativity.

AARON

Okay, well, I don't know what schmormativity is.

DANIEL

You know how normativity I feel like that's good. Schmormativity is about promoting the schmud.

AARON

Okay, so it sounds like that's just normativity, except it's normativity about different propositions. That's what it sounds like.

DANIEL

Well, basically, I don't know, instead of these schmalians wait, no, they're aliens. They're not shmalians. They're aliens. They just do a bunch of schmud things, right? They engage in projects, they try and figure out what the schmud is. They pursue a schmud and then they look at humans, they're like, oh, these humans are doing morally good things. That's horrible. I'm so glad that we pursue the schmood instead.

AARON

Yeah, I don't know if it's incoherent. I don't think they're being incoherent. Your description of a hypothetical let's just take for granted whatever in the thought experiment is in fact happening. I think your description is not correct. And the reason it's not correct is because there is like, what's a good analogy? So when it comes to abstract concepts in general, it is very possible for okay, I feel like it's hard to explain directly, but here an analogy, is you can have two different people who have very different conceptions of justice, but fundamentally are earnestly trying to get at the same thing. Maybe justice isn't well defined or isn't like, actually, I should probably have come up with a good example here. But you know what? I'm happy to change the word for what I use as morality or whatever, but it has the same core meaning, which is like, okay, really, what should you do at the end of the day?

DANIEL

Yeah.

AARON

What should you do?

DANIEL

Whereas they care about morality, which is what they should do, which is a different thing. They have strong desires to do what they should do.

AARON

I don't think it is coherent to say that there are multiple meanings of the word should or multiple kinds. Yeah.

DANIEL

No, there aren't.

AARON

Sorry. There aren't multiple meanings of the word should. Fine.

DANIEL

There's just a different word, which is schmood, which means something different, and that's what their desires are pegged to.

AARON

I don't think it's coherent, given what you've already the entire picture, I think, is incoherent. Given everything else besides the word schmoud, it is incoherent to assert that there is something broadly not analogous, like maybe isomorphic to normativity or, like, the word should. Yeah. There is only what's yeah. I feel like I'm not gonna I'm not gonna be able to verbalize it super well. I do. Yeah. Can you take something can you pick.

DANIEL

A sentence that I said that was wrong or that was incoherent?

AARON

Well, it's all wrong because these aliens don't exist.

DANIEL

The aliens existed.

AARON

Okay, well, then we're debating, like, I actually don't know. It depends. You're asserting something about their culture and psychology, and then the question is, like, are you right or wrong about that? If we just take for granted that you're right, then you're right. All right. I'm saying no, you can't be sure. So conditional on being right, you're right. Then there's a question of, like, okay, what is the probability? So, like, conditional on aliens with something broad, are you willing to accept this phrase, like, something broadly analogous to morality? Is that okay?

DANIEL

Yeah, sure.

AARON

Okay. So if we accept that there's aliens with something broadly analogous to morality, then you want to say that they can have not only a different word, but truly a pointer to a different concept. And I think that's false.

DANIEL

So you think that in conceptual space, there's morality and that there's, like, nothing near it for miles.

AARON

The study, like yeah, basically. At least when we're talking about, like, the like, at the at the pre conclusion stage. So, like, before you get to the point where you're like, oh, yeah, I'm certain that, like, the answer is just that we need, like, we need to make as many tennis balls as possible or whatever the general thing of, like, okay, broadly, what is the right thing to do? What should I do? Would it be good for me to do this cluster of things yeah. Is, like, miles from everything else.

DANIEL

Okay. I think there's something true to that. I think I agree with that in some ways and on others, my other response is I think it's not a total coincidence that humans ended up caring about morality. I think if you look at these evolutionary arguments for why humans would be motivated to pursue morality. They rely on very high level facts. Like, there are a bunch of humans around. There's not one human who's, like, a billion times more powerful than everyone else. We have language. We talk through things. We reason. We need to make decisions. We need to cooperate in certain ways to produce stuff. And it's not about the fact that we're bipedal or something. So in that sense, I think it's not a total coincidence that we ended up caring about morality. And so in some sense, I think because that's true, you could maybe say you couldn't slightly tweak our species that it cared about something other than morality, which is kind of like saying that there's nothing that close to morality in concept space.

AARON

But I think I misspoke earlier what I should have said is that it's very weird that we care about that most people at least partially care about suffering and happiness. I think that's just a true statement. Sorry, that is the weird thing. Why is it weird? The weird thing is that it happens to be correct, even though I only.

DANIEL

Have what do you mean it's correct?

AARON

Now we have to get okay, so this is going into moral realism. I think moral realism is true, at least.

DANIEL

Sorry, what do you mean by moral realism? Wait, different by moral realism?

AARON

Yes. So I actually have sort of a weak version of moral realism, which is, like, not that normative statements are true, but that there is, like, an objective. So if you can rank hypothetical states of the world in an ordinal way such that one is objectively better than another.

DANIEL

Yes. Okay. I agree with that, by the way. I think that's true. Okay.

AARON

It sounds like you're a moral realist.

DANIEL

Yeah, I am.

AARON

Okay. Oh, really? Okay. I don't know. I thought you weren't. Okay, cool.

DANIEL

Lots of people in my reference class aren't. I think most Bay Area rationalists are not moral realists, but I am.

AARON

Okay. Maybe I was confused. Okay, that's weird. Okay. Sorry about that. Wait, so what do I mean by it happens to be true? It's like it happens to coincide with yeah, sorry, go ahead.

DANIEL

You said it happens to be correct that we care about morality or that we care about suffering and pleasure and something and stuff.

AARON

Maybe that wasn't the ideal terminology it happens to so, like, it's not morally correct? The caring about it isn't the morally correct thing. It seems sort of like the caring is instrumentally useful in promoting what happens to be legitimately good or something. Or, like legitimately good or something like that.

DANIEL

But but I think, like so the aliens could say a similar thing, right? They could say, like, oh, hey, we've noticed that we all care about schmurality. We all really care about promoting Schmeasure and avoiding Schmuffering. Right? And they'd say, like, they'd say, like, yeah, what's? What's wrong?

AARON

I feel like it's not maybe I'm just missing something, but at least to me, it's like, only adding to the confusion to talk about two different concepts of morality rather than just like, okay, this alien thinks that you should tile the universe paperclips, or something like that, or even that more reasonably, more plausibly. Justice is like that. Yeah. I guess this gets back to there's only one concept anywhere near that vicinity in concept space or something. Maybe we disagree about that. Yeah.

DANIEL

Okay. If I said paperclips instead of schmorality, would you be happy?

AARON

Yes.

DANIEL

I mean, cool, okay, for doing the.

AARON

Morally correct thing and making me happy.

DANIEL

I strive to. But take the paperclipper species, right? What they do is they notice, like, hey, we really care about making paperclips, right? And, hey, the fact that we care about making paperclips, that's instrumentally useful in making sure that we end up making a bunch of paperclips, right? Isn't that an amazing coincidence that we ended up caring our desires were structured in this correct way that ends up with us making a bunch of paperclips. Is that like, oh, no, total coincidence. That's just what you cared about.

AARON

You left at the part where they assert that they're correct about this. That's the weird thing.

DANIEL

What proposition are they correct about?

AARON

Or sorry, I don't think they're correct implicitly.

DANIEL

What proposition do they claim they're correct about?

AARON

They claim that the world in which there is many paperclips is better than the world in which there is fewer paperclips.

DANIEL

Oh, no, they just think it's more paperclipy. They don't think it's better. They don't care about goodness. They care about paperclips.

AARON

So it sounds like we're not talking about anything remotely like morality, then, because I could say, yeah, morality, morality. It's pretty airy. It's a lot of air in here. I don't know, maybe I'm just confused.

DANIEL

No, what I'm saying is, so you're like, oh, it's like this total coincidence that humans we got so lucky. It's so weird that humans ended up caring about morality, and it's like, well, we had to care about something, right? Like anything we don't care about.

AARON

Oh, wow, sorry, I misspoke earlier. And I think that's generating some confusion. I think it's a weird coincidence that we care about happiness and suffering.

DANIEL

Happiness and suffering, sorry. Yeah, but mutatus mutantus, I think you want to say that's like a weird coincidence. And I'm like, well, we had to care about something.

AARON

Yeah, but it could have been like, I don't know, could it have been otherwise, right? At least conceivably it could have been otherwise.

DANIEL

Yeah, the paperclip guys, they're like, conceivably, we could have ended up caring about pleasure and suffering. I'm so glad we avoided that.

AARON

Yeah, but they're wrong and we're right.

DANIEL

Right about what?

AARON

And then maybe I don't agree. Maybe this isn't the point you're making. I'm sort of saying that in a blunt way to emphasize it. I feel like people should be skeptical when I say, like okay, I have good reason to think that even though we're in a very similar epistemic position, I have reason to believe that we're right and not the aliens. Right. That's like a hard case to make, but I do think it's true.

DANIEL

There's no proposition that the aliens and us disagree on yes.

AARON

The intrinsic value of pleasure and happiness.

DANIEL

Yeah, no, they don't care about value. They care about schmalu, which is just.

AARON

Like, how much paperclips there is. I don't think that's coherent. I don't think they can care about value.

DANIEL

Okay.

AARON

They can, but only insofar as it's a pointer to the exact same not exact, but like, basically the same concept as our value.

DANIEL

So do you reject the orthogonality thesis?

AARON

No.

DANIEL

Okay. I think that is super intelligent.

AARON

Yeah.

DANIEL

So I take the orthogonality thesis to mean that really smart agents can be motivated by approximately any desires. Does that sound right to you?

AARON

Yeah.

DANIEL

So what if the desire is like, produce a ton of paperclips?

AARON

Yeah, it can do that descriptively. It's not morally good.

DANIEL

Oh, no, it's not morally good at all. They're not trying to be morally good. They're just trying to produce a bunch of paperclips.

AARON

Okay, in that case, we don't disagree. Yeah, I agree. This is like a conceivable state of the world.

DANIEL

Yeah. But what I'm trying to say is when you say it's weird that we got lucky the reason you think it's weird is that you're one of the humans who cares about pleasure and suffering. Whereas if you were one of the aliens who cared about paperclips. The analogous shmarin instead of Aaron would be saying, like, oh, it's crazy that we care about paperclips, because that actually causes us to make a ton of paperclips.

AARON

Do they intrinsically care about paperclips or is it a means of cement?

DANIEL

Intrinsically, like, same as in the Orphogonality thesis.

AARON

Do they experience happiness because of the paperclips or is it more of a functional intrinsic value?

DANIEL

I think they probably experience happiness when they create paperclips, but they're not motivated by the happiness. They're motivated by like, they're happy because they succeeded at their goal of making tons of paperclips. If they can make tons of paperclips but not be happy about it, they'd be like, yeah, we should do that. Sorry. No, they wouldn't. They'd say, like, we should do that and then they would do it.

AARON

Would your case still work if we just pretended that they're not sentient?

DANIEL

Yeah, sure.

AARON

Okay. I think this makes it cleaner for both sides. Yeah, in that case, yes. So I think the thing that I reject is that there's an analog term that's anything like morality in their universe. They can use a different word, but it's pointing to the same concept.

DANIEL

When you say anything like morality. So the shared concepts sorry, the shared properties between morality and paperclip promotion is just that you have a species that is dedicated to promoting it.

AARON

I disagree. I think morality is about goodness and badness.

DANIEL

Yes, that's right.

AARON

Okay. And I think it is totally conceivable. Not even conceivable. So humans wait, what's a good example? In some sense I intrinsically seem to value about regular. I don't know if this is a good example. Let's run with it intrinsically value like regulating my heartbeat. It happens to be true that this is conducive to my happiness and at least local non suffering. But even if it weren't, my brain stem would still try really hard to keep my heart beating or something like that. I reject that there's any way in which promoting heart beatingness is an intrinsic moral or schmoral value or even that could be it could be hypothesized as one but it is not in fact one or something like that.

DANIEL

Okay.

AARON

Likewise, these aliens could claim that making paperclips is intrinsically good. They could also just make them and not make that claim. And those are two very different things.

DANIEL

They don't claim it's good. They don't think it's good.

AARON

They think it's claim it schmud.

DANIEL

Which they prefer. Yeah, they prefer.

AARON

Don't. I think that is also incoherent. I think there is like one concept in that space because wait, I feel like also this is just like at some point it has to cash out in the real world. Right? Unless we're talking about really speculative not even physics.

DANIEL

What I mean is they just spend all of their time promoting paperclips and then you send them a copy of Jeremy Bentham's collected writings, they read it and they're like all right, cool. And then they just keep on making paperclips because that's what they want to do.

AARON

Yeah. So descriptively.

DANIEL

Sure.

AARON

But they never claim that. It's like we haven't even introduced objectivity to this example. So did they ever claim that it's objectively the right thing to do?

DANIEL

No, they claim that it's objectively the paperclipy thing to do.

AARON

I agree with that. It is the paperclippy thing to do.

DANIEL

Yeah, they're right about stuff. Yeah.

AARON

So they're right about that. They're just not a right. So I do think this all comes back down to the question of whether there's analogous concepts in near ish morality that an alien species might point at. Because if there's not, then the paperclippiness is just like a totally radically different type of thing.

DANIEL

But why does it like when did I say that they were closely analogous? This is what I don't understand.

AARON

So it seems to be insinuated by the closeness of the word semantic.

DANIEL

Oh yeah, whatever. When I was making it a similar sounding word, all I meant to say is that they talk about it plays a similar role in their culture as morality plays in our culture. Sorry. In terms of their motivations, I should say. Oh, yeah.

AARON

I think there's plenty of human cultures that are getting at morality. Yeah. So I think especially historically, plenty of human cultures that are getting at the same core concept of morality but just are wrong about it.

DANIEL

Yeah, I think that's right.

AARON

Fundamentalist religious communities or whatever, you can't just appeal to like, oh, we're like they have some sort of weird it's kind of similar but very different thing called morality.

DANIEL

Although, I don't know, I actually think that okay, backing up. All I'm saying is that beings have to care about something, and we ended up caring about morality. And I don't think, like I don't know, I don't think that's super surprising or coincidental or whatever. A side point I want to make is that I think if you get super into being religious, you might actually start referring to a different concept by morality. How familiar are you with classical theism?

AARON

That's not a term that I recognize, although I took a couple of theology classes, so maybe more of them if I hadn't done that.

DANIEL

Yeah, so classical theism, it's a view about the nature of God, which is that I'm going to do a bad job of describing it. Yeah, I'm not a classical theist, so you shouldn't take classical theist doctrine from me. But it's basically that God is like sort of God's the being whose attributes are like his existence or something like that. It's weird. But anyway, there's like some school of philosophical where they're like, yeah, there's this transcendent thing called God. We can know God exists from first principles and in particular their account of goodness. So how do you get around the Euphyro dilemma, right? Instead of something like divine command theory, what they say is that when we talk about things being good, good just refers to the nature of God. And if you really internalize that, then I think you might end up referring to something different than actual goodness. Although I think it's probably there's no such being as God in the article. Theist sense.

AARON

Yeah. So they argue what we mean by good is this other.

DANIEL

Concept. They would say that when everyone talks about good, what they actually mean is pertaining to the divine nature, but we just didn't really know that we meant that the same way that when we talked about water, we always meant H 20, but we didn't used to know that.

AARON

I'm actually not sure if this is I'm very unconfident, but I kind of want to bite the bullet and say, like, okay, fine, in that case, yeah, I'm talking about the divine nature, but we just have radically different understandings of what the divine nature is.

DANIEL

You think you're talking about the divine nature.

AARON

Right?

DANIEL

Why do you think that?

AARON

Sorry, I think I very slightly was not quite pedantic enough. Sorry, bad cell phone or whatever. Once again, not very confident at all.

DANIEL

But.

AARON

Think think that I'm willing to I'm so I think that I'm referring to the divine nature, but what I mean by the divine nature is that which these fundamentalist people are referring to. So I want to get around the term and say like, okay, whatever these fundamentalists are referring to, I am also referring to them.

DANIEL

Yeah, I should say classical theism is not slightly a different when people say fundamentalists, they often mean like a different corner of Christian space than classical theists. Classical. Theists think like Ed Fesser esoteric Catholics or something. Yeah, they're super into it.

AARON

Okay, anyway yes, just to put it all together, I think that when I say morality, I am referring to the same thing that these people are referring to by the divine nature. That's what it took me like five minutes to actually say.

DANIEL

Oh yeah, so I don't think you are. So when they refer to the divine nature, what they at least think they mean is they think that the divine is sort of defined by the fact that its existence is logically necessary. Its existence is in some sense attributes it couldn't conceivably not have its various attributes. The fact that it is like the primary cause of the world and sustainer of all things. And I just really doubt that the nature of that thing is what you mean by morality.

AARON

No, those are properties that they assert, but I feel like tell me if I'm wrong. But my guess is that if one such person were to just suddenly come to believe that actually all of that's right. Except it's not actually logically necessary that the divine nature exists. It happens to be true, but it's not logically necessary. They would still be sort of pointing to the same concept. And I just think, yeah, it's like that, except all those lists of properties are wrong.

DANIEL

I think if that were true, then classical theism would be false.

AARON

Okay.

DANIEL

So maybe in fact you're referring to the same thing that they actually mean by the divine nature, but what they think they mean is this classical theistic thing. Right. And it seems plausible to me that some people get into it enough that what they actually are trying to get at when they say good is different than what normal people are trying to get at when they say good.

AARON

Yeah, I don't think that's true. Okay, let's set aside the word morality because especially I feel like in circles that we're in, it has a strong connotation with a sort of like modern ish analytics philosophy, maybe like some other things that are in that category.

DANIEL

Your video is worsen, but your sound is back.

AARON

Okay, well, okay, I'll just keep talking. All right, so you have the divine nature and morality and maybe other things that are like those two things but still apart from them. So in that class of things and then there's the question of like, okay, maybe everybody necessarily anybody who thinks that there's any true statements about something broadly in their vicinity of goodness in the idea space is pointing to the meta level of that or whichever one of those is truly correct or something. This is pretty speculative. I have not thought about this. I'm not super confident.

DANIEL

Yeah, I think I broadly believe this, but I think this is right about most people when they talk. But you could imagine even with utilitarianism, right? Imagine somebody getting super into the weeds of utilitarianism. They lived utilitarianism twenty four, seven. And then maybe at some point they just substitute in utilitarianism for morality. Now when they say morality, they actually just mean utilitarianism and they're just discarding the latter of the broad concepts and intuitions behind them. Such a person might just I don't know, I think that's the kind of thing that can happen. And then you might just want a.

AARON

Different thing by the word. I don't know if it's a bad thing, but I feel like I do this when I say, oh, x is moral to do or morally good to do. It's like, what's the real semantic relationship between that and it's correct on utilitarianism to do? I feel like they're not defined as the same, but they happen to be the same or something. Now we're just talking about how people use words.

DANIEL

Yeah, they're definitely going to happen to be the same in the case that utilitarianism is like the right theory of morality. But you could imagine that. You could imagine even in the case where utilitarianism was the wrong theory, you might still just mean utilitarianism by the word good because you just forgot the intuitions from which you were building theory of morality and you're just like, okay, look, I'm just going to talk about utilitarianism now.

AARON

Yeah, I think this is like, yeah, this could happen. I feel like this is a cop out and like a non answer, but I feel like getting into the weeds of the philosophy of language and what people mean by concepts and words and true the true nature of concepts. It's just not actually that useful. Or maybe it's just not as interesting to me as I'm glad that somebody thought about that ever.

DANIEL

I think this can happen, though. I think this is actually a practical concern. Right. Okay. Utilitarianism might be wrong, right? Does that strike you as right? Yeah, I think it's possible for you to use language in such a way that if utilitarianism were wrong, what that would mean is that in ordinary language, goodness, the good thing to do is not always the utilitarian thing to do, right? Yes, but I think it's possible to get down an ideological rabbit hole. This is not specific to utilitarianism. Right. I think this can happen to tons of things where when you say goodness, you just mean utilitarianism and you don't have a word for what everyone else meant by goodness, then I think that's really hard to recover from. And I think that's the kind of thing that can conceivably happen and maybe sometimes actually happens.

AARON

Yeah, I guess as an empirical matter and like an empirical psychological matter and yes. Do people's brains ever operate this way? Yes. I don't really know where that leaves that leaves us. Maybe we should move on to a different topic or whatever.

DANIEL

Can I just say one more thing?

AARON

Yeah, totally.

DANIEL

First, I should just give this broad disclaimer that I'm not a philosopher and I don't really know what I'm talking about. But the second thing is that particular final point. I was sort of inspired by a paper I read. I think it's called, like, do Christians and Muslims worship the same god? Which is actually a paper about the philosophy of naming and what it means for proper names to refer to the same thing. And it's pretty interesting, and it has a footnote about why you would want to discourage blasphemy, which is sort of about this. Anyway.

AARON

No, I personally don't find this super interesting. I can sort of see how somebody would and I also think it's potentially important, but I think it's maybe yeah.

DANIEL

Actually it's actually kind of funny. Can I tell you a thing that I'm a little bit confused about?

AARON

Yeah, sure.

DANIEL

So philosophers just there's this branch of philosophy that's the philosophy of language, and in particular the philosophy of right. Like, what does it mean when we say a word refers to something in the real world? And some subsection of this is the philosophy of proper names. Right. So when I say Aaron is going to the like, what do I mean by know who is like, if it turned out that these interactions that I'd been having with an online like, all of them were faked, but there was a real human named Bergman, would that count as making that send is true or whatever? Anyway, there's some philosophy on this topic, and apparently we didn't need it to build a really smart AI. No AI person has studied this. Essentially, these theories are not really baked into the way we do AI these days.

AARON

What do you think that implies or suggests?

DANIEL

I think it's a bit confusing. I think naively, you might have thought that AIS would have to refer to things, and naively, you might have thought that in order for us to make that happen, we would have had to understand the philosophy of reference or of naming, at least on some sort of basic level. But apparently we just didn't have to. Apparently we could just like I don't have that.

AARON

In fact, just hearing your description, my initial intuition is like, man, this does not matter for anything.

DANIEL

Okay. Can I try and convince you that it should matter? Yeah, tell me how I fail to convince you.

AARON

Yeah, all right.

DANIEL

Humans are pretty smart, right? We're like the prototypical smart thing. How are humans smart? I think one of the main ingredients of that is that we have language. Right?

AARON

Yes. Oh, and by the way, this gets to the unpublished episode with Nathan Barnard.

DANIEL

Coming out an UN I think I've seen an episode with him.

AARON

Oh, yeah. This is the second one because he's.

DANIEL

Been very oh, exciting. All right, well well, maybe all this will be superseded by this unpublished episode.

AARON

I don't think so. We'll see.

DANIEL

But okay, we have language, right. Why is language useful? Well, I think it's probably useful in part because it refers to stuff. When I say stuff, I'm talking about the real world, right?

AARON

Yes.

DANIEL

Now, you might think that in order to build a machine that was smart and wielded the language usefully, it would also have to have language. We would have to build it such that its language referred to the real world. Right. And you might further think that in order to build something that use languages that actually succeeds at doing reference, we would have to understand what reference was.

AARON

Yes. I don't think that's right. Because insofar as we can get what we call useful is language in, language out without any direct interaction, without the AIS directly manipulating the world, or maybe not directly, but without using language understanders or beings that do have this reference property, that's what their language means to them, then this would be right. But because we have Chat GPT, what the use comes from is like giving language to humans, and the humans have reference to the real world. But if the humans you need some connection to your reference, but it doesn't have to be at every level or something like that.

DANIEL

Okay, so do you think that suppose we had something that was like Chat GPT, but we gave it access to some robot limbs and it could pick up mice. Maybe it could pick up apples and throw the apples into the power furnace powering its data center. We give it these limbs and these actuators sort of analogous to how humans interact with the world. Do you think in order to make a thing like that that worked, we would need to understand the philosophy of reference?

AARON

No. I'm not sure why.

DANIEL

I also don't know why.

AARON

Okay, well, evolution didn't understand the philosophy of reference. I don't know what that tells us.

DANIEL

I actually think this is, like, my lead answer of, like, we're just making AIS by just randomly tweaking them until they work. That's my rough summary of Scastic gradient descent. In some sense, this does not require you to have a strong sense of how to implement your AIS. Maybe that's why we don't need to.

AARON

Understand philosophy or the SDD process is doing the philosophy. In some sense, that's kind of how I think about it or how I think about it now. I guess during the SDD process, you're, like, tweaking basically the algorithm, and at the end of the day, probably in order to, say, pick up marbles or something, reference to a particular marble or the concept of marble, not only the concept, but both the concept and probably a particular marble is going to be encoded. Well, I guess the concept of marble will be if that's how it was trained, that will be encoded in the weights themselves, you know what I mean? But then maybe a particular marble will be encoded vision to see that marble be encoded in a particular layers activation.

DANIEL

Or something, something like that, maybe. Yeah, I think this is like yeah, I guess what we're getting at is something like look, meaning is like a thing you need in order to make something work, but if you can just directly have a thing that gradually gets itself to work, that will automatically produce meaning, and therefore we don't have to think about it.

AARON

It will have needed to figure out meaning along the way.

DANIEL

Yeah, but we won't have needed to figure it out. That'll just happen in the training process.

AARON

Yeah. I mean, in the same way that everything happens in the training process. Yeah, that's where all the magic happens.

DANIEL

All right, so do you want to hear my new philosophy of language proposal?

AARON

Yes.

DANIEL

Yeah. So here's the new proposal. I think the theory of reference is not totally solved to everyone's satisfaction. So what we're going to do is we're going to train Chat GPT to manipulate objects in the physical world, right? And then we're going to give the weights to the philosophers. We're also going to give it like, a bunch of the training checkpoints, right?

AARON

And then they're going to look at.

DANIEL

This, and then they're going to figure out the philosophy of meaning.

AARON

What are training checkpoints?

DANIEL

Oh, just like the weights at various points during training.

AARON

Okay, and your proposal is that the philosophers are going to well, we haven't solved neck interpretability anyway, right? Yeah. I feel like this is empirically not possible, but conceptually, maybe the outcome won't be like solving meeting, but either solving meeting or deciding that it was a confused question or something. There was no answer, but something like resolvative.

DANIEL

Yeah. I don't know. I brought this up as like a reductiod absurdum or something or sort of to troll. But actually, if we get good enough at mechanical interpretability, maybe this does just shine light on the correct theory of reference.

AARON

I mean, I'm just skeptical that we need a theory of reference. I don't know, it seems like kind of like philosopher word games to me or something like that. I mean, I can be convinced otherwise. It's like haven't seen that.

DANIEL

I'm not sure that we need it. Right. I think I get fine without an explicit one, but I don't think you can tell.

AARON

Yes. Okay.

DANIEL

Can I tell you my favorite? It's sort of like a joke. It's a sentence that yeah. All right, so here's the sentence. You know Homer, right? Like the Greek poet who wrote the Iliad and the.

AARON

Oh, is that the.

DANIEL

No, this is the setup, by the way, do you know anything else about Homer?

AARON

Male? I don't know that I think that.

DANIEL

Yeah, okay, all right. This is not going to be funny as a joke, but it's meant to be a brain tickler, right? So the Iliad and the Odyssey, they weren't actually written by Homer? They were written by a different Greek name, by a different Greek man who.

AARON

Was also named I thought I saw somebody tweet this.

DANIEL

I think she got it from me.

AARON

That'S my okay, cool.

DANIEL

She might have got it from the lecture that I watched.

AARON

Maybe you can explain to me. Other people are saying, oh yeah, I don't think they were rolling on the ground laughing or whatever, but they were like, oh, ha, this is actually very funny after you explain it. And I did not have that intuition at I'm like, okay, so there's two guys named the where's the brain tickly part?

DANIEL

Oh, the brain tickly part is this. How could that sentence possibly be true when all you knew about Homer was that he was a Greek guy who wrote The Iliad and The Odyssey and that he was named.

AARON

How could that sentence okay, so I feel like the sentence on its own doesn't have a truth value, but what it implies. If I just heard that in normal conversation in fact, when I heard it just now, and if I were to hear it in normal conversation, what I would take it to mean is the famous guy who all the academics talk about, turns out, yes, that is Person A. And there was also this other person who is not somebody else has a better, more solid understanding of Homer beyond defining him as the author of The Iliad, The Odyssey, even though that's really all I know about him. I trust there's other people for whom this is not the case. And implicitly, I'm thinking, okay, so there's some philosophy or history dudes or whatever, who they know where he was born, they know his middle name or whatever, and so we're just going to call him Person A. And in fact, there was another guy named Domer, and there's no contradiction there or whatever.

DANIEL

What if nobody alive? What if everything that so I think this is actually plausible, I think, in terms of what living people know about Homer, I think it's just that he was a guy named Homer, he was Greek, he wrote The Iliad and The Odyssey, or at least is reputed to have. And maybe we know something about the period in which he lived, and maybe you can figure out the part of Greece in which he lived from the language, but I think that's probably all humanity currently knows about.

AARON

So maybe, maybe the statement can be it feels like it can be false. And the way it could be false is if we took a census of just suppose we had a census of everybody who ever lived in that period and there was only one Homer, well, then we would know that statement is false.

DANIEL

What do you mean, only one Homer?

AARON

I mean, there was not two individuals in the census, this hypothetical census named.

DANIEL

Homer who were given the name Homer. Gotcha yeah, that would make it.

AARON

And so it seems to be carrying substantive information that, in fact, we have historical evidence of two different individuals, and we have reason to believe there were two different individuals who went by the name Homer, and one of them wrote The Iliad and The Odyssey. And given those two facts, then the statement is true.

DANIEL

Okay, if the statement were so in the past, there are two different people named Homer, and only one of them wrote The Iliad of The Odyssey. But then why would we not say that The Iliad of The Odyssey were written by Homer? Why would we say they weren't written by Homer if they were written by a different guy who was also named Homer?

AARON

Yeah, so this gets back to the difference between the statement per se and my interpretation. So the statement per se, it sounds like there's no difference there. Or the phrase, like, some other guy named where it's, like, redundant, maybe not wrong, but like redundant or something they may have even wrong. I don't know. The information carried in the statement would be equivalent if you just said, we have good reason to believe there was not merely one Homer, but two. And indeed, one of these people wrote The Odyssey and the same statement, basically.

DANIEL

All right, so here's the thing I'm going to hit you up with. I think usually people have, like most people have names that other people also have, right?

AARON

Yes.

DANIEL

Like, there's more than one person named Daniel. There's more than one person named Aaron.

AARON

Right.

DANIEL

There was probably more than one person named Homer around the time when Homer was supposed to have all right, so so, yeah, homer didn't write Thalia and The Odyssey. They were written by some other guy who was also named Homer.

AARON

Yeah, I think that's a true statement.

DANIEL

Oh, I think it's a false. Can I try and convince you that you're wrong to say that's a true statement?

AARON

Yeah.

DANIEL

All right, here's one statement. Homer wrote the iliad and the odyssey. Right?

AARON

Yes.

DANIEL

Do you think that's true?

AARON

Okay, so think it is both true and false, depending on the reference of Homer.

DANIEL

Oh, yeah. So what is the reference?

AARON

Something like, yeah, maybe I'm willing to take back the thing that I previously said because this feels like more normal language or something when I say I'm talking to Daniel, right, that feels like a true statement. But maybe my sister has a friend named Daniel, and if I told that to her right, like, she would be right to say that it's false because you know what? I keep getting back to the fact that who gives a shit? You know? What I mean. I still struggle to see. You can dig down into the true, whether a particular proposition is true or false or indeterminate or something. But in normal language, we have a million psychological and maybe not psychological, but we have a million ways to figure out what is meant by a particular proposition beyond the information contained in its words. Okay. I don't know. This is not an answer or whatever, but it still seems like it's all fine, even if we never figure out.

DANIEL

I guess sorry, I'm going to do a little bit of sweeping. Your audience doesn't want to hear that. I'm going to sweep them, then.

AARON

No, that's totally cool. We're pro sweeping.

DANIEL

All right. Finish. All right.

AARON

Yeah.

DANIEL

I'm inclined to agree that it's fine. So when you say there's a million opportunities to understand the content of a sentence other than just the information contained the words, or understand what somebody means beyond just the statement info containing the words, you might still want to know what the info contained the words actually is. I should say, broadly, the way I relate to this is as an interesting puzzle.

AARON

Yeah, no, I kind of agree. Maybe I'm just like more. Yeah, I think it's like I can see why somebody was lined. It interesting.

DANIEL

Yeah. It gets to a thing where when you try to think of what we mean by something like Homer or what we mean by something like Daniel Filon, at least when other people say it, often it will be you'll come up with a candidate definition. And then there'll be some example which you hadn't anticipated, which I think is part of what makes this interesting. So, for instance, you might think that Daniel Phylan is the person named Daniel Filon, but here's a sentence, daniel Filon could have been named Sam. Or actually, here's a better one. Daniel Filon could have been named Patrick. Like, my dad actually sort of wanted to for a while. My dad was thinking of calling me Patrick. Right.

AARON

I was almost, yeah.

DANIEL

Yeah. So if you think about the sentence, daniel Filon could have been named Patrick. If Daniel Filon just means, like, a person named Daniel Filon, then that's.

AARON

I mean yeah, but that shouldn't.

DANIEL

So then you might say, like, oh, what Daniel Filon means is it's actually just an abbreviation of a bunch of things you might know about me. Right. Like, Daniel Filon is this guy who is Australian, but now lives in Berkeley and hosts this podcast and a few other things. And then the trouble is, you could imagine a parallel world right, where I didn't do any of those things.

AARON

Well, I feel like that's a bad definition. It would be, Janopilot is a human being who was both psychologically and genetically continuous with the being who existed before he was named, or something like that.

DANIEL

Okay. But you still have to nail down which being Daniel Fallon is supposed to be psychologically and genetically continuous with wait, what? Sorry. When you say, like, Daniel Phylan means just beings that are like, human beings that are psychologically and genetically continuous with the being before they were named. I think that's what you said.

AARON

Which angry definition. Yeah, well, I'm talking about you. Yeah, beyond that, I don't think there's any other verbal mishmash I can say will point to that. There's, like, a human being there's, like, a human being that's, like, the atoms aren't the same. Plus not all the memories are the same. There's personal identity issues, but there's a human being with basically your genetics, like, whatever, how many your age, plus a couple months. And that is also Daniel filon.

DANIEL

Yeah. Can you try and say that without using the word you imagine it's somebody who you're not talking to, and so you don't get to wait, what?

AARON

I don't even know wait, what am I supposed to be trying to gesture towards what I'm trying to say?

DANIEL

Yeah, give a definition of what you mean by Daniel Filon in a way that's valid. Like, I would still be Daniel Filon in a way where imagine a counterfactual world where, like, grown up to hate EA or something. You would want to still call that guy Daniel Filon. Daniel but but you're not allowed to use the word you okay?

AARON

Yeah. Daniel Filin is the human being who is currently not. I feel like I kind of mean two different things. Honestly, I don't think there's one definition. One is, like, the actual is the current instantiation and the current and actual instantiation of a particular human being. And the more and the other definition or meaning I have is all like, all human all human beings who either were or will be. I don't know about could be, honestly, or I think could be. I don't know about could have been. Yeah, maybe could have been. Yes. Let's go with what could have been. So throughout the multiverse, if that's a thing, all those beings who either were will be, could have been or could be psychologically and genetically continuous with a human being who was conceived or, like I guess I guess I guess this being started existing when he was a genetic entity or, like, had his full genome or something, which is hard.

DANIEL

Which beings are.

AARON

The counterfactual alternatives of the current being named Daniel Phylan? And this being, in turn, is defined as the current instantiation of an original past self. And that original past self can be delineated in time by the moment that a particular human being had the had all the genes or whatever.

DANIEL

So it's things that branch off the current being that is named Daniel Filon right.

AARON

Or things that branch off the yeah, branch off, but ultra retrospectively, I guess, but yeah.

DANIEL

Okay. And the current being suppose, like so I haven't actually told you this, but my legal name is actually Steve Schmuckson, not Daniel Phylan. Is there anything that the name Daniel Filon refers to?

AARON

Like, there's no factor of the matter.

DANIEL

You think there's no factor of the.

AARON

Here'S my concern. Where is the fact of the matter located? Or something like that. Is it in my neuro? Yeah. Is it, like, moral truth? What is it like, referential truth? Is there anything referential truth is like.

DANIEL

Oh, I don't know. I guess probably not.

AARON

Okay.

DANIEL

But I guess when you say the person names Daniel Filon, I think there's still a question of, like, wait, who is the like, how do you figure out who the person named Daniel Filon is? Like, I think that gets back to.

AARON

The probably it's probably multiple people. Wait, hold on. Pause. Okay, I'll cut this part out. Lindsay, I'm in the middle of a sorry. Sorry. Bye. Okay, I'm back.

DANIEL

Yeah, but when you say, like, the person names Daniel Filon and you're using that in your definition of what do I mean by Daniel filon? That strikes me as kind of circular because how do we know which person is the one who's named Daniel Filon?

AARON

Yeah, I agree. That's a poor definition. I feel like I very weekly think that I could come up with a more rigorous definition that would be, like, really annoying non intuitive.

DANIEL

Okay.

AARON

Not super sure about that.

DANIEL

You should try and then read some phil articles because it's all totally doesn't.

AARON

Matter and it's like a fake question. Oh, yeah, it doesn't matter.

DANIEL

I just think it's a fun puzzle.

AARON

Yeah, but it feels like it's not even yeah, so there's, like, a lot of things I feel like there's mathematical questions that don't matter but are more meaningful in some sense than even this feels kind of like maybe not. How many angels dance in the head of a pin? Yeah, actually kind of like that. Yeah. How many angels can dance in the head of a pin?

DANIEL

I think that question is meaningful.

AARON

What's the answer?

DANIEL

What's the answer? I guess it depends what you mean by angel. Normally in the Christian tradition, I think angels are supposed to not be material.

AARON

I think maybe, like, tradition. I'm asking about the actual answer.

DANIEL

Yeah, I mean, the actual answer to how many angels can dance on the yeah, I think when you use the word angel okay, the tricky thing here is when you use the word angel, you might be primarily referring to angels in the Jewish tradition about which no.

AARON

I'm referring to real angels.

DANIEL

There aren't any real angels.

AARON

Okay, well, then how many angels can dance in the head of a pen?

DANIEL

Zero. Because there aren't any.

AARON

I'm kind of joking sort of adopting your stance when I came to whatever, the aliens with the weird word.

DANIEL

I gave you an answer. What do you want?

AARON

Yeah, I'm also going to give you, like, a series of answers. I mean, I'm not actually going through I think it'll be annoying, but I could give you a series of answers like that or whatever, like I'm referring.

DANIEL

To I'm not sure you could give me another question. That's my answer.

AARON

Oh, okay.

DANIEL

As for how many actual angels, could.

AARON

I feel like I might be trapped here because I thought that was going to trip you up, and it's just like, yeah, it sounds like the right answer. Honestly.

DANIEL

Well, I guess you might think that. Suppose all dogs suddenly died, right. And then later I asked you how many dogs could fit in this room, there would still be an answer to that question that was like greater than zero. Yeah. I think the word angels just, like it just depends on what the word angels refers to. And I'm like, well, if it has to refer to actual angels, then there aren't any actual angels. If we're referring to angels as conceived of in the Christian tradition, then I think infinitely many. If we're referring to angels as conceived of in other traditions, then I think that I don't know the answer.

AARON

Yes, that sounds right. I'm glad you find this sorry. That was like an hour, so that was an annoying way of putting it.

DANIEL

I liked it. That was a fine thing to say.

AARON

At the metal level. At the metal level. I find it interesting that some people find this interesting.

DANIEL

Yeah. Okay, before you go away and try and figure out theory of naming, can I add some side constraints? Some constraints that you might not have thought of?

AARON

Sure.

DANIEL

Okay, so here's a sentence. Like, harry Potter is a wizard. Right.

AARON

There are no wizards.

DANIEL

You think it's false that Harry Potter is a wizard?

AARON

Yes.

DANIEL

All right, but let's just take the okay, like like you kind of know what that means, right?

AARON

Yes.

DANIEL

Let's take another sentence. Like, thor is the god of lightning, right?

AARON

Yes.

DANIEL

Now, I take it you don't believe in the literal existence of Thor or of Harry Potter. Right?

AARON

Yeah. Right.

DANIEL

But when I talk about I'm, I'm wielding the name Harry Potter, and I'm doing a sort of similar thing as when I wield the name Aaron Bergman. Right.

AARON

Similar. Not the same, but similar.

DANIEL

Yeah. Okay, cool. So Harry Potter the thing about Harry Potter is it's like an empty name, right? It's a name that doesn't refer to anything that actually exists. Right.

AARON

Doesn't refer to any configuration of actually existed molecules. It refers to some abstractions, and it refers to a common set of a grouping of properties in various people's minds.

DANIEL

Oh, you think it refers to the grouping of properties rather than so if I said, like, Thor actually exists, that would be true, according to you?

AARON

No, I'm trying to figure out why. I think I figured why I totally.

DANIEL

Think this is a solvable problem, by the way.

AARON

Okay.

DANIEL

I'm not trying to say this is some sort of deepity, like, you will never know. I think this is conceivable. Anyway, the point is, Harry Potter and Thor are examples of names that don't refer to actual humans or gods or whatever, but they're different, right?

AARON

Yes. So that's interesting.

DANIEL

You might have thought that names were nailed down by the sets of things they referred to.

AARON

Hold on. I think something can refer to something without or sorry, there are things besides maybe you don't have a good word, but there are thingy like things, for lack of a better term, that exist in some meaningful sense of exist that are not configurations of quarks or identifiable configuration, or like, yeah, let's go to configurations.

DANIEL

Quarks and leptons. Sure. And you don't just mean like, the Em field. You mean like, things can refer to non physical stuff.

AARON

I think physical is in a useful category. This is also a hot take in some.

DANIEL

Like, wait, do you think that Harry Potter is like, this non physical being that flies around on a broomstick, or do you think that Harry Potter is like, the concept?

AARON

So I think there's multiple things that that term means, and the way it's actually used is depends on Bergman.

DANIEL

Do you think Aaron Bergman means multiple?

AARON

No.

DANIEL

What's the difference?

AARON

Well, I can in fact, Harry Potter might only refer to exactly two things.

DANIEL

What are the two things that Harry Potter refers to?

AARON

Sorry, wait, maybe I'm wrong about that. Okay, hold on. So, like, if I use the not, I don't know, because what I want to say is harry refers to what you think it refers to in two different contexts. And one context is where we pretend that he exists, and the other context is when we recognize or pretend that he doesn't. And now you're going to say, oh, who's you referring to. Am I right?

DANIEL

Yeah.

AARON

Okay, that sounds like what I'm going to say. Okay. No, I feel like there's, like an er Harry Potter, which is like a cluster of traits, like a cluster of things. There's no hardened, well defined thing in the same way there's no well defined notion of what is a bottle of wine. You can keep adding weird tidbits to.

DANIEL

The bottle of wine, but the er Harry Potter is like a bundle of traits.

AARON

Characteristics. Traits. Okay.

DANIEL

Rishi Sunak a bundle of traits.

AARON

I think there's, like two levels. There's, like, the metal Rishi Sunak and the thing that people normally refer to when they refer to Rishi Sunak, which is not merely which is not a bundle of traits. It is distinguished from other from like it is a physical or like a biological mind like thing that is individuated or pointed out in person space by the bundle of traits or something like that.

DANIEL

Yeah, he is that. But I think that when people say Rishi Sunak, I don't think they ever mean the bundle of traits. I think they mean, like, the guy. I think the guy has the bundle of traits, but they don't mean the. Traits, they mean the guy.

AARON

Yeah, I think that's right. I think the way that they, with their mind brain lands on that actual meaning is, like, in some sense, recognizing those letters as pointing to characteristics, as pointing to things, to maybe things or characteristics such as the Prime Minister of Britain or UK or whatever, like things.

DANIEL

That embody the they don't they don't refer to the characteristics themselves. They refer to the things that embody the characteristics. Right.

AARON

I think as an empirical matter, this is true. I can imagine a world in which it's sometimes the first of the bundle of characteristics.

DANIEL

Yeah, I guess I think that would be people speaking a different language. Right. Like, there are all sorts of different languages. Some of them might have the word Rishi sunak. That happens to mean, like, the property of being the Prime Minister of Great Britain and Northern Ireland.

AARON

Well, like, okay, so let's say in a thousand years or whatever, and there's still humans or whatever, there's like a mythology about some being. And in the same way that there's mythology about Thor, there's mythology about this being who's in various myths plays the role of the not plays the role, but is the role in the myths of the Prime Minister of the UK. Which is like some ancient society and has these various traits, then it would behave kind of thought. But yeah, this is like a conceivable thing, in which case there is a reference I wouldn't say that means that the language people speak is in English anymore because they use rishi sunak in that way.

DANIEL

But when they said rishi sunak, they were actually referring to the traits not like some sort of being.

AARON

Well, maybe there were historians in that society who were referring to the being, but most normal people weren't or something.

DANIEL

I guess I think they would be referring to, like I guess to them I would call rishi sunak. Like, sorry, what kinds of things do these people believe about rishi sunak? But how are they using sentences involving rishi sunak?

AARON

So somebody might say, oh, you know, rishi sunak isn't actually a lie. That would be a true statement. It would also be a true sorry. Sorry, or like, wait, yeah.

DANIEL

Sorry is the idea that these people have myths about. Right, all right, sorry. That's the question I was asking. Okay, all right, cool. I guess this would be sort of similar to the case of Santa Claus. The phrase Santa Claus comes from St. Nicholas, who was probably a real guy from Turkey named okay, I like, vaguely.

AARON

Knew that, I think.

DANIEL

Yeah, but I guess this gets us back to where we started with when we say Santa Claus, do we mean like, the bundle of ideas around Santa Claus or do we mean like a guy who dispenses a bunch of presents.

AARON

On I mean, I want to step back.

DANIEL

Anyway.

AARON

Yeah, I feel like maybe insofar as I feel like maybe it does matter, or like, yeah, the question of meaning or sorry, it can matter, but it just has a different answer in particular different cases. And so the right way to go about it is to just discuss reference in the case of morality, for example, the case of Santa Claus and another. And there's no general answer. Or maybe there is a general answer, but it's so abstract that it's not.

DANIEL

Useful in any way that might be. Well, I think even abstract answers can be pretty yeah, I think you might have some hope that there's a general answer for the case of proper names to be even concrete. I think you might think that there's some theory that's sort of specific that unifies the names aaron Bergman, Santa Claus and Zeus.

AARON

Yeah. And I guess I think, oh, it'll be a lot easier and quicker just to actually disambiguate case by case. Maybe I'm wrong. Maybe I'm wrong. So if some tenured philosophers at whatever university want to work on this, people.

DANIEL

Can do that, I should say. I've read theories that purport to explain all of these three naming practices that I found somewhat convincing. When I say papers, I mean one paper. It's actually the paper I cited earlier.

AARON

Okay, you can send it to me or, like, send me a link or whatever, if you want.

DANIEL

Yeah, really, what's happening in this conversation is I read one paper and now I'm trolling you about it. I hope it's a good kind of trolling.

AARON

Yeah, it feels like benevolent trolling. But I actually do think this is kind of meaningful in the context of morality, or at least it's actually kind of non obvious in that case, whereas it generally is obvious, like, what a particular person in real life is referring to. In the case of Santa Claus, just depending on and morality happens to be important. Right. So maybe there's other cases like that. Or I could see legal battles over, like, what does a law refer to? There's, like, two different people. It's like the guy, the state, there's the name itself. Yes, sure. I don't know.

DANIEL

Yeah. This reminds me of various formulations of originalism, which is you've heard of originalism, I guess constitutional.

AARON

Yeah.

DANIEL

So original it's this theory that when you're interpreting laws, you should interpret the original thing going on there, rather than what we currently want it to be, or whatever. And there's this question of, like, wait, what thing that was originally going on? Should we interpret? And sometimes you occasionally hear people say that it's about the original intent. I think this is definitely false, but more often people will say, oh, they mean the original public meaning. But sometimes people say, oh, no, it's the original meaning. In a legal context, people try to get at what exactly they mean by originalism, and it has some of its flavor.

AARON

Yeah, I could talk about object level or at the level we've been talking. I don't think it's, like a fact of the matter, but object level. If you convince me that originalism was true, maybe you couldn't what I want to say is because those people weren't playing by the rules or whatever, we just got to norm it out or something sorry. People writing the Constitution weren't doing it under the pretext of originalism. I don't know. It I could be wrong about this.

DANIEL

Okay. Why do you think it.

AARON

Maybe looks pretty plausible that I'm wrong? I vaguely feel like this is a thing that was, like, developed in, like, the 20th century by, like, legal scholars.

DANIEL

I think that's sort of right. So they had this notion of strict constructionism in the 19th century that I think is kind of analogous to originalism. I think when people talk about originalism, they mean simple enough concepts that it seems plausible to me that people could have been to me. I don't know, maybe this is my bias, but it seems very intuitive to me that when people were writing the Constitution, maybe they were thinking, hey, I want this law to mean what it means right now.

AARON

Yeah. There's a question. Okay. What is me and me. Yeah.

DANIEL

I guess everybody thinks yeah, all right. There's one game which is, like, what did the framers think they were doing when they wrote the Constitution? There's a potentially different question, which is, like, what were they actually doing? They could have been wrong about legal theory. Right. That's conceivable. And then there's a third game, which I think is maybe the best game, which is, like, what's the best way to sort of found a system of laws? Should we hope that all the courts do originalism, or should we hope that all the courts do like, I'm not exactly sure what the alternative is supposed to be, but like, yeah, but what.

AARON

Should we ask from an alternative?

DANIEL

Is like, sorry.

AARON

Yeah, I agree. I assume you mean, like, what actually in 2023 should be the answer, or how should judges interpret the Constitution?

DANIEL

That's the game whereby should I hear means something like, what would cause the most clarity about the laws? And something like that.

AARON

I don't mean that exact same thing. I think I mean something like, more in some sense, ultimately moral, not like clarity is not I don't know. There's other values besides clarity.

DANIEL

Yeah, sure. We might want to limit scope a little bit to make it easier to think about. Right.

AARON

Yeah.

DANIEL

When I'm building a house, if I'm building a house, I probably want to think, like, how will this house not fall down?

AARON

I don't know.

DANIEL

I'm going to have a bunch of concrete requirements, and it's probably going to be better to think about that rather than, like, what should I build? Because I don't want to solve philosophy before building my house.

AARON

Yeah, it's not as obvious what those requirements are for. Possible that just because you can have just, like, two statements issued by the federal court, or you can imagine that the last two judgments by the Supreme Court include unambiguous propositions that are just opposites of one another. And I don't think this would mean that the United States of America has fallen. You know what? Okay, like, nobody knows. What should we do? I don't.

DANIEL

Mean yeah. I would tend to take that as saying that legal judgments don't follow the inference rules of classical logic. Seems fine to me.

AARON

Sure. Also, I think I'm going to have to wrap this up in him pretty soon. Sorry.

DANIEL

Yeah, we can go for ages.

AARON

Do this again. Yeah, this will be the longest one yet.

DANIEL

I feel a bit guilty for just trolling. I don't even properly understand.

AARON

Especially I do think the morality thing is interesting because I think there's definitely, like, a strain of rationalist thought that it's directionally like you were at least in terms of vibes, like where you were coming from. That's pretty influential, at least in some circles.

DANIEL

Yeah, I guess I'm not sure if I did a good job of articulating it. And also, I've sort of changed my mind a little bit about I don't know, I feel like when I talk about morality, I want to get caught in the weird weeds or the semantics rather than, like I think an important fact about morality is it's not a weird contingent fact that humans evolved to care about it. I don't know. To me, it's really interesting that evolutionary accounts of why we care about morality, they don't rely on really fine grained features. They rely on very broad. People talk to each other, and we have common projects, and there's not one guy who's stronger than every other human. I don't know. Yeah, I feel like that's somehow more real and more important than just the weird semantics of it. Anyway, before we close up, can I plug some of my stuff?

AARON

Yes, plug everything that you want.

DANIEL

All right. I have two podcasts. One of my podcasts is called Axrp. It's the AI X Risk Research podcast, and you can listen to me interview AI X Risk researchers about their work and why they do it. I have another podcast called The Phylan Cabinet, where I just talk to whoever about whatever I want. I think if you want to hear some people who strongly who I guess the audience of this podcast is mostly EA's, like young atheist kind of EA types, if you want to hear people who are kind of not like that. I have a few episodes on religion and one three and a half hour conversation with my local Presbyterian pastor about what he thinks about God. And I have another episode with an objectivist about just I don't know, I guess everything Ayn Rand thinks the culmination.

AARON

Oh, no, you cut out at the word objectivist. Sorry, wait, you cut out at the word objectivist.

DANIEL

Oh, yeah, I'll try to say it again. I have one episode where I talk to this objectivist just about a bunch of objectivist thought. So I think we cover objectivists, like, ethics, metaphysics, and a bit of objectivist aesthetics as well. And I don't know, the thing objectivists are most famous for is they're really against altruism. And I ended up thinking that I thought the body of thought was more persuasive than I expected it to be. So maybe I recommend those two episodes to.

AARON

Have been sort of actually haven't listened to it in, like, a week, but was listening to your one with Oliver habrica. But after I finish that, I will look at the objectivist one. Yeah. Everybody should follow those podcasts. Like me.

DANIEL

Everyone. Even if you don't speak English.

AARON

Everyone. In fact, even if you're not a human, like Santa Claus, including yeah. Okay. So anything else to plug?

DANIEL

If you're considering building AGI don't.

AARON

Hear that. I know. Sam, you're listening okay. I know you're listening to Pigeonhoue.

DANIEL

Okay, yeah, I guess that's not very persuasive of me to just say, but I think AI could kill everyone, and that would be really bad.

AARON

Yeah, I actually agree with this. All right, well, yeah, there's more people we can cover this in more nuance next time you come on pigeonholer. Okay, cool.

DANIEL

I'm glad we have a harmonious ending.

AARON

Yeah. Of conflict. Disagreement is good. I'm pro discourse. Cool. All right, take care. See ya. Bye.

Get full access to Aaron's Blog at www.aaronbergman.net/subscribe

Very imperfect transcript: bit.ly/3QhFgEJ

Summary from Clong:

• The discussion centers around the concept of a unitary general intelligence or cognitive ability. Whether this exists as a real and distinct thing.


• Nathan argues against it, citing evidence from cognitive science about highly specialized and localized brain functions that can be damaged independently. Losing linguistic ability does not harm spatial reasoning ability.


• He also cites evidence from AI, like systems excelling at specific tasks without general competency, and tasks easy for AI but hard for humans. This suggests human cognition isn’t defined by some unitary general ability.


• Aaron is more open to the idea, appealing to an intuitive sense of a qualitative difference between human and animal cognition - using symbolic reasoning in new domains. But he acknowledges the concept is fuzzy.


• They discuss whether language necessitates this general ability in humans, or is just associated. Nathan leans toward specialized language modules in the brain.


• They debate whether strong future AI systems could learn complex motor skills just from textual descriptions, without analogous motor control data. Nathan is highly skeptical.


• Aaron makes an analogy to the universe arising from simple physical laws. Nathan finds this irrelevant to the debate.


• Overall, Nathan seems to push Aaron towards a more skeptical view of a unitary general cognitive ability as a scientifically coherent concept. But Aaron retains some sympathy for related intuitions about human vs animal cognition.

Get full access to Aaron's Blog at www.aaronbergman.net/subscribe

Summary (by Claude.ai)

This informal podcast covers a wide-ranging conversation between two speakers aligned in the effective altruism (EA) community. They have a similar background coming to EA from interests in philosophy, rationality, and reducing suffering. The main topic explored is reducing s-risks, or risks of extreme suffering in the future.

Winston works for the Center for Reducing Suffering (CRS), focused on spreading concern for suffering, prioritizing interventions, and specifically reducing s-risks. He outlines CRS's focus on research and writing to build a moral philosophy foundation for reducing suffering. Aaron is skeptical s-risk reduction is tractable currently, seeing the research as abstract without a clear theory of change.

They discuss how CRS and a similar group CLR are trying to influence AI alignment and digital sentience to reduce potential future s-risks. But Aaron worries about identifying and affecting the "digital neural correlates of suffering." Winston responds these efforts aim to have a positive impact even if unlikely to succeed, and there are potential lock-in scenarios that could be influenced.

Aaron explains his hesitancy to donate based on tractability concerns. He outlines his EA independent research, which includes an archive project around nuclear war. More broadly, the two find they largely ethically agree, including on a suffering-focused ethics and "lexical negative utilitarianism within total utilitarianism.

Some disagreements arise around the nature of consciousness, with Aaron arguing rejecting qualia implies nihilism while Winston disagrees. They also diverge on moral realism, with Aaron defending it and Winston leaning anti-realist.

As they wrap up the wide-ranging conversation, they joke about convincing each other and make predictions on podcast listens. They thank each other for the thought-provoking discussion, aligned in ethics but with some disagreements on consciousness and metaethics. The conversation provides an insider perspective on efforts to reduce s-risks through research and outreach.

EA Archive: https://forum.effectivealtruism.org/posts/DndmvDGStD3gTfhXk

Get full access to Aaron's Blog at www.aaronbergman.net/subscribe

Summary/specific topics:

- Stress Tests and AI Regulation: Nathan elaborates on the concept of stress tests conducted by central banks. These tests assess the resilience of banks to severe economic downturns and the potential for a domino effect if one bank fails. They believe that lessons from this process can be applied to AI regulation. Aaron agrees, but also highlights the need for a proactive approach to AI regulation, as opposed to the reactive measures often seen in banking regulation.

- The Role of Central Banks in AI Regulation: Nathan suggests that institutions structured like central banks, staffed with technical experts and independent from government, could be beneficial for AI regulation. They believe such institutions could respond quickly and effectively to crises. However, they acknowledge that this approach may not be effective if AI development leads to rapid, uncontrollable self-improvement.

- Compute Governance: The conversation then shifts to compute governance, which Nathan sees as a promising area for AI regulation due to the obviousness of someone using large amounts of compute. They believe that this could provide governments with a control lever over cutting-edge AI labs, similar to how central banks control banking loans and affairs.

- AI Regulation and the Role of Public Actors: Nathan acknowledges that the leaders of major AI labs seem sensible and aligned with AI safety principles. However, they argue that regulation and public actors can play a crucial role in creating common knowledge between labs and preventing a race to the bottom. They also discuss the potential benefits and drawbacks of different regulatory approaches.

- Financial Regulation as a Model for AI Regulation: Nathan believes that post-crisis financial regulation, such as the Dodd-Frank Act, has generally been effective. They suggest that AI regulation could follow a similar path, especially if AI becomes a significant part of the economy. However, Aaron expresses skepticism about the ability of political processes to produce effective AI regulation.

- Regulation Before and After Crises: The speakers agree that pre-crisis regulation has generally been less effective than post-crisis regulation. They discuss the potential for AI regulation to follow a similar pattern, with effective regulation emerging in response to a crisis.

- Regulatory Arbitrage: The conversation concludes with a discussion on regulatory arbitrage, where banks shift activities to where it's cheapest to do business. Despite evidence of this behavior, Nathan notes that there was no race to the bottom in terms of regulation during the financial crisis.

Get full access to Aaron's Blog at www.aaronbergman.net/subscribe

- Follow Arjun on Twitter: https://twitter.com/panickssery

- Read and subscribe to his blog: https://arjunpanickssery.substack.com

- A mediocre transcription can be found at https://www.assemblyai.com/playground/transcript/6x5h1mcemt-bff1-40fc-a676-9b59c66985f0

🦆 👍

Get full access to Aaron's Blog at www.aaronbergman.net/subscribe