Mutual Understanding

Robin talks about why he thinks developments in AI will be on a continuum with civilizational progress in general, and Ronny, who is mostly trying to understand Robin, talks some about why he thinks many of the things he values about humanity aren't on track to being preserved, and that he cares about that.

Video version available at: https://www.youtube.com/watch?v=G-fBdPnwFrI

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mutualunderstanding.substack.com

In this episode we discuss several ‘frames’ - ways of processing and orienting to information - that are appearing in public discussions around AI, and in particular reflect on how grounded they are in longstanding debates within the AI X-Risk community.

"It’s sort of like the democratic ideal where everybody's sort of out there at the public square arguing with each other, talking about the things… I don't think there are a lot taboos about what to say yet either. It hasn't gotten super corrupted.

Like it's very rare that in the real world, I'm encountering trolleys running over people right? And you can set up these fake scenarios that mess up my moral intuitions…  but maybe it’s not always virtuous to endorse the repugnant conclusion that these very limited, decoupled thought experiment bring you to."

Links:

* MusicLM from Google

* TV’s War With the Robots Is Already Here (Writer’s Strike connection with AI)

* Ronny Fernandez and Robin Hanson Discuss AI Futures

* Roko and Alexandros on AI Risk

* Universal Fire

* Spandrels

Timestamps

[00:01:00] Updates on Politics

[00:07:00] The State of the AI Discourse

[00:13:00] The Yudkowskian Foomer vs Hansonian Continualist

[00:18:00] Mood Affiliations

[00:22:00] Biting Bullets vs Rejecting Fake Hypotheticals

[00:28:00] Divide between Game Theorists and Engineers

[00:36:00] The complexity of predictions about the future

[00:47:00] How similar are the values of optimizing systems?

[00:54:00] The rupture of the GMU Rationalist Alliance

[01:00:00] Public Perception that AI has Moral Weight

[01:03:00] AI Veganism/Freeganism

[01:06:00] Overton Window Shifts

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mutualunderstanding.substack.com

No transcript for this episode yet, but I wanted to get it out there anyway, since it can be hard to listen on Twitter Spaces.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mutualunderstanding.substack.com

Ben Weinstein-Raun

And so my guess is that that fairness seems like a plausible candidate for something like an alien species might have… kind of universal, like given some sort of assumptions about how evolution worked or works.

Like there is something happening that results in us making choices. And if your philosophical determinism denies that, like you're wrong. And, I think it makes about as much sense to talk about will and free will in making choices as it does to talk about a glass of water.

Ben Weinstein-Raun’s Twitter and Website.

Timestamps

[00:02:00] - Meta-Ethical Stance

[00:11:00] - Contrasting with Moral Realism

[00:26:00] - What EA misses

[00:47:00] - Fanaticism and Outside of Distribution Sampling

[01:28:00] - Building new Tools for Forecasting and Thinking

[01:44:00] - Actually using Tooling for Better Decisions

[01:53:00] - Guided by the Aesthetics of our Distributed Systems

This transcript is machine generated and contains errors

Ben Goldhaber: [00:00:00] And hi I'm going to maybe make a quick introduction to our guest here, and then we'll just dive into some conversation. Ben - who has a great first name - Ben Weinstein-Raun has worked at a number of top tech companies including crews and council, number of innovative tech research organizations such as MIRI and Redwood Research, and Secure DNA.

Ben is currently building a tool for forecasting estimation, and dare I say, thinking. Having known Ben for some time now, I'd describe Ben as a careful, discerning thinker on issues of not just technology, but also philosophy.

And so, yes, welcome to the pod, Ben. Yeah, thanks so much. Yeah, we're excited to have you. I was saying to Divia, one of the reasons I was excited to invite you on[00:01:00] was because of a really great conversation that you and I had a few weeks ago about. Naturalism and ethics and how it's put into practice and like all good conversations about philosophy, it was like 3:00 AM around a kitchen table.

And I felt like you really captured something important about I don't know, naturalism and ethics, and I really just wanna see if we could recapture some of that magic in our conversation today. Cool.

Ben WR: Yeah, totally. Great. Yeah, I'm really excited to talk about it. Awesome.

Divia Eden: I'm excited. I haven't heard most of this yet, so I'll get to hear it for the first time.

Ben WR: Yeah. Nice.

Ben Goldhaber: Well, lemme just ask for the start. How would you describe your moral stance?

Ben WR: Yeah, I guess, I guess at the moment, I would describe my, like best I don't know, my, my best stab or something. My best stab at understanding ethics as more like a pretty strong stance [00:02:00] on meta ethics. And not as strong of a stance on like, sort of object level ethics.

Where meta ethics is sort of questions about like where ethics comes from. Like why do we say moral sentence or like sentences about morality, sentences about good and bad and should whereas like ethics, like object level ethics would be sort of like, what sentences do we say about or like should we say about good and bad?

And like sort of what's true on the, on the direct object level there? I guess the, yes, so the meta ethical stance that I really wanna take is basically derived from an observation, which is that MetaEthical questions are under I think some pretty reasonable assumptions entirely empirical.

So whatever the truth of the object level, ethical facts insofar as they exist. [00:03:00] There's some reason that people go around talking about shoulds and, and good and bad and so on. And, and that reason is like entirely inside of physics. It's not mysterious. It's the kind of thing that we could figure out by like, looking carefully at humans and at human societies and at maybe, maybe game theory and math.

We don't need fundamentally different tools to answer that kind of question. If we're interested in answering like, the straightforward question, why do people say sentences like that? What is sort of the driver for humans having these intuitions and having these sort of discussions and, and and so on.

And so that's sort of like the, the, the meta ethical stance. And then I think it has some kind of interesting implications for taking this stance has an interesting implications for the more object level stuff. So I think the most obvious one to me is that it, it does [00:04:00] not seem like it lends itself well to sort of very simplified rule-based systems.

So I think it, it pushes me pretty far away from like a sort of total utilitarian kind of a view where you're aiming to make your object level ethical system very simple and make that sort of like this very beautiful sort of object that you know, is unassailable.

Ben Goldhaber: I'd love to ask a question there about that because I could see some people actually like in intuition that I feel like I have, when I hear that from a meta ethical point of view, everything is within this same kind of system and I suppose all derived from physics on down it. Lends itself to thinking about things in a very maybe not very, but like I could see it applying in like a simplified manner.

Like thinking okay, well because it's physics and because it's something knowable, we can construct theories about [00:05:00] it that are simple and derive into this kind of utilitarian calculus. But it sounds like you're actually pushing in the opposite direction and saying, no, it's much more complicated.

Ben WR: Yeah, I think basically my sense that like you get something complicated is not like solely based on the idea that it's from that meta ethics is like empirically addressable. It's also based on sort of like observing what is going on when people talk about good and bad and should.

I think it seems to me that if you want to have, like, if you wanna have your meta ethical stance basically look correct or like consistent with the evidence I think you need it to include some things, which to me do not. Seem like they would come from an approximation to utilitarianism.

You need it to include things like I don't know. So, so Jonathan Hate has this [00:06:00] book oh shoot. I'm gonna forget what it's called cuz I'm being recorded. But it's the Jonathan Hate book. I feel like there's one Jonathan hate book people quote in this situation. No, it's definitely the out one.

Ben Goldhaber: There's,

Ben WR: let's look out and then we can, and then you can go back and say that. Yeah. Yeah. Ok. Is it the Righteous Mind? I think that is it. Yes. Ok. Okay. So, so Jonathan Ha has has this book, the Righteous Mind where he sort of goes into like doing, excuse me, a lot of the sort of like empirical like analysis of what is going in, what's going on in people's minds when they're having this sort of like set of ethical intuitions and conversations.

And he comes out with like several factors like I think, I think it has like five sort of like key key things that, that are sort of factors of people's morality. And one of them like, like an especially important one and one that like, I think is especially important to me is [00:07:00] sort of harm focused morality.

Which is I mean, it's quite widespread, but it's not like the only thing going on for almost anyone when they're thinking about sort of like what's ethical. So,

Ben Goldhaber: So there are like multiple different things going on when people are calculating, thinking about what's ethical and like a certain re reductionist point of view.

That's just looking at the harm's point of view is probably missing a lot of things. Is this one way to put this?

Ben WR:Yeah. Yeah. I think that's basically,

Divia Eden: yeah. Can I, I just, I just pulled up what the different ax axis, the five one s he has can, do you mind if I say them? Yeah, yeah, totally. Okay. So he has care, harm, fairness, cheating, loyalty, betrayal, authority, subversion and sanity degradation.

I have some, at least that's what I found when I looked it up. I have some imperfect memory that he added, like freedom in there later because …

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mutualunderstanding.substack.com

Divia and Ben discuss politics, reminisce about elections gone by, and try and make AI predictions.

This transcript is machine generated and contains errors.

Ben Goldhaber: [00:00:00] so how's it going Divia?

Divia Eden: It's going well. Great to have a chance to talk to you on the podcast.

Ben Goldhaber: Yeah, same back at you and Hi. Who, who else is joining us for the next minutes?

Divia Eden: Yeah, this is my 15 month old son. He'll be here just for a few minutes and and then, then it'll be just the two of us.

He's our, our podcast guest. Yeah. Likely.

Ben Goldhaber: Yeah. One of our easiest to understand guests. I would bet. You know, just very clear and direct. Absolutely. In a straightforward way. This is no slur against our previous guests but we, I feel like we invite people on in some ways for loving the Galaxy Brain takes.

And the beautiful thing about a 15 month year old is just you know, that's right. Simple, clear wants, I imagine I'm speaking as if I have any actual parenting expertise, but,

Divia Eden: It's all good one day. Yes. So we don't, we don't have a guest for today. But we have a plan that sometimes just two of us [00:01:00] are going to talk and catch up about things that are on our mind.

Ben Goldhaber: Yeah. Seems good.

Divia Eden: And before we started recording, we were just exchanging a few predictions about some political stuff, you know, as you do. So I thought, I thought maybe we would, we would talk about

Ben Goldhaber: that. Yeah. As maybe some listeners know and, and Dvia knows. I, I was about to say I don't have many vices, but I don't think that's for me to say.

But one vice that I do have is I very much enjoy betting on politics. And I will say it has been exciting to see that predict it is looks like it's gonna survive, which, yeah, I was actually much more negative about a few months ago. I did not think it was gonna happen.

Divia Eden: Yeah, there wasn't, unfortunately, I think there was no prediction market for predicted unpredicted, so we

Ben Goldhaber: were left not unpredicted.

There was one unlike poly market, which I don't think I ever, I I, I have, the crypto ones have always been just a little, I don't know. I've, I've done some stuff with them, but I've always been like [00:02:00] feel like they're about to fail or about to run away. But yes, poly market had one, but yeah, was not, when I predicted that would've been way cooler.

Divia Eden: Yeah. I'm also a longtime political better. I, one of my at the, this is actually, I don't know if you'll ever listen to this. Hello to my dad. If, if we had and wonderful short conversation with my dad back when, so I bet on Obama on, I guess it was Intrade at the time when he was running the first time.

And I remember I, cuz my dad was, he was a big, like, he, his read was, Obama's gonna win. He was like, this guy's very charismatic, he's gonna win. And so I was talking to him and I was like, all right, well you think he's gonna win? He's trading it like at whatever it was like 57% right before the first debate or something like that.

I was like, well how much do you think I should bet? And he was like, yeah, you should bet like 50 bucks or something. And I was kinda like, well, okay, but if you're so sure. And we talked about it for a while and then he, we hung up. Yeah. And then he called me back maybe an hour later and he was like, wait, how much do you have liquid?

And I was like, yeah, this is the right question. Cause in fact, yeah. Yeah, [00:03:00] it was a big, and I mean I was, I think I was just out of college. I did not, I had very little money at the time, but I think I did put little liquid on Obama and Nice. It paid off. So that was then, you know, and then I was kind of hooked

Ben Goldhaber: after that.

Yeah, it's, it's compelling. It's, we don't need to get into all the very good reasons. Prediction markets are, are awesome and, and should be more widely used. But yeah, it's just, it's, it's also just fun because I feel like when you're kind of chatting like with your dad, like I, I talk about politics with my, with my dad as well and with friends.

It's really, it's easy to get into the kind of like blather mode and. It's both nice to have as a forcing function, thinking about what your actual bet is gonna turn into or like what this actually means in terms of betting. And also just adds like a nice little, like I, I guess this is for people who are more into sports than I am.

It's the same reason why gambling on sports is probably fun. Just add when, then when you keep following it, you get to like have that additional zing of [00:04:00] excitement of like, oh I've got, got money on this debate.

Divia Eden: Yeah, totally. And yeah, and I also, like, I've been on Trump the first time, but I didn't have a, the thing I was just saying before we started recording was that I, I think like many very online people, underrated Biden last time.

I think I didn't make any big bets. I don't remember. I think I made some small, small ones. I don't remember what they were, but I like, yeah, the people that I personally knew and this was in the primary, weren't that excited about Biden. So I, I guess I didn't really see it, but I think that, I think it's an easy mistake for people like me to make, cuz I think the fundamentals were actually pretty strong.

Totally. And my husband Will kept pointing this out to me.

Ben Goldhaber: Respect. I definitely did not see that happen in the, in the 2021 either. Yeah. As I was, I was saying, I feel like I'm still a little bit surprised that Biden won the 2020 primary, and it's not just because of the fundamentals thing, me ignoring them, but rather the Democratic party.

The way I remember it was like the [00:05:00] Democratic party elites got together in a smoke-filled room, one imagines and convinced the other candidates to drop out so that instead of Bernie winning in their Super Tuesday, I think it was, mm-hmm. Biden was the one . So I think in part I was surprised that the.

Unity of the Democratic Party to deny it to Bernie and give it to Biden.

Divia Eden: Yeah. So that, I think that's kinda interesting because remember, I forget whose thesis this was, but like the party decides that thesis. Yep, yep. Or like, that was a really good predictor. Like how many of the delegates had pledged a basically like, support within the party was meant to be a very good predictor.

And at least the way I remember it, maybe this theory was around for a long time, but like I was started reading articles about that theory not so long before then Trump won the Republican primary. And so I guess I, at that point I was like, well, maybe the party doesn't decide, cuz that clearly wasn't right, what the party elites were going for.

He didn't have any of those early people pledged to him. So then [00:06:00] I thought the theory was dead. But as you say, then with Biden, it seems like that theory was line shot.

Ben Goldhaber: No, I think split. I, I think the party decides on the Democratic side and I don't think the party decides on the Republican side, and I don't know how long, this does not strike me as a.

The evil equilibrium place. Like it doesn't seem like, like something is not finished adjusting. But that is my read over the past, I guess eight years. . Wow. eight Time is weird.

Divia Eden: Yeah, I think it's right.

Ben Goldhaber: It's right.

I think the ultimate example party not deciding is the Jeb Trump matchup and, yeah, I don't know. I guess in some sense then what, like Trump went on to, to do against Clinton. Like that's obviously a different kind of party deciding thing, but certainly some larger narrative of anti-elitism.

Divia Eden: Yeah, and in fact, like the thing I, I most strongly remember from those debates was the, I guess it wasn't Trump, but it was when. Chris Christie called out Rubio, do you remember this? For repeating himself. Oh, I

Ben Goldhaber: remember that, [00:07:00] yeah. Oh, that was great. Watching it with

Divia Eden: Will and, and in real t…

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mutualunderstanding.substack.com

As I’ve said often in the past, utilitarianism is a hell of a drug. And it can get you to do incredibly horrible things while you’re high on it. Utilitarianism not even once, just say no to utilitarian.

All of these discussions are old discussions. Instead of being had among 200 people, they are being had in public among vast numbers.

Perry Metzger writes on Substack at Diminished Capacity and on Twitter.

* [00:02:00] Jupiter Brains and Extropians

* [00:07:00] Life Extension

* [00:11:00] Ethical Systems

* [00:19:00] Rejection of Utilitarianism

* [00:26:00] Anarcho-Capitalism

* [00:30:00] SVB Collapse

* [00:40:00] Finanical System Structuring

* [00:54:00] AI

* [01:00:00] The history of futurist discussion of AI

* [01:03:00] AI Safety

* [01:07:00] Fishing in the sea of AI Minds

* [01:17:00] Mask on the Shoggoth

* [01:22:00] Future Shock

* [01:25:00] Competition in FDA Regulation

* [01:38:00] Kardashev Scale Civilizations

* [01:42:00] Estimates of X-Risk

* [01:52:00] Cultural Effects of AI Developments

* [01:56:00] Nanotechnology

* [02:15:00] Sociology of the Nanotechnology field

* [02:27:00] Engineering vs Theorizing

* [02:43:00] Back to the future; accuracy of early extropian discussion

* [02:52:00] Grabby Aliens

* [02:56:00] Astrophysics

This transcript was machine generated and contains errors.

Perry Metzger: [00:00:00] to try to make our lives better, which is, I'm gonna quit my Firefox instance with 8,500 tabs in it.

Ben Goldhaber: Can't hurt. And I do think that the recording is working now. Cool. 

Divia Eden: Yeah. Okay. So, and do you wanna introduce Perry or should I?

Ben Goldhaber: No, Divia. I think why don't you do the introduction. 

Perry Metzger: Okay. 

Divia Eden: Perry Metzger is a computer scientist who's done academic research, who's also had a bunch of different programming jobs, including startups and consulting. He knows more than anyone else I talked to about nanotech.

I originally met Perry through my husband, who met him through an ancap meetup (Anarcho-Capitalism). And from my perspective, he's one of those guys who's been around the futurist scene forever. He was an early member of the Cypherpunks mailing list, started the original cryptography list, and started the Extropians mailing list.

He says that he claims no ownership or originality of any transhumanist ideas, except that he did coin the term Jupiter Brain. So, Perry thanks so much for coming on our podcast. 

Perry Metzger: Hello. Actually no one hires me to write software [00:01:00] anymore. People, people hire me to be a horrifying management consultant.

Or what have you. Secretly I still write software here and there, and it horrifies people really, really badly. I have scarred a number of people working for me by having to deal with my software. They, you know, we, but we don't talk about that mostly, you know, and I've, I've gotten rid of most of the bodies over the years successfully.

So provided, no one finds them. We should be okay. Yes. Yeah I’ve done all sorts of stuff. I have no idea actually what I'm going to be when I grow up, but I've been told that if you don't actually figure that out by the time you're 60 or 70, you don't have to grow up.

So I might actually just have to opt for that. 

Divia Eden: Sounds good to me. Well, we have a lot of questions, but I was wondering if you could first start by telling our listeners what a Jupiter Brain is, in case they don't already know. 

Perry Metzger: Oh, okay. So, I should set a little context, which is for those that, [00:02:00] have no idea what Divia was mentioning when she said, I started the Extropians mailing list about 723,000 years ago.

In, you know, slightly before Homo Sapien showed up I was hanging out with a friend of mine. And I'll cut the story really short by saying that we discovered this zine, this was back in the era when people would decide, you know, they'd get access to a photocopier and they'll start publishing a magazine.

There was this Zine called Extropy: Vaccine for Future Shock put out by a gentleman who was then Max O'Connor, but is now Max Moore. And, you know, Harry and I were reading this thing and it looked like, oh, you know, these guys came down on the same spaceship as us. And I got in touch with Max and I said, Hey, I'd like to set up a mailing list to talk about these ideas.

And the ideas like the Magazine was roughly centered on anarcho-capitalism, radical life extension, transhumanism, uploading artificial intelligence you know neurotropic drugs, the usual stuff that young people were interested in, back then. So, I got in touch with Max and I said, Hey, max, I want to set up a mailing list for subscribers of this thing.

And he said, well, what's that? And I said, well, you know why I set up this thing and a bunch of people then can have an online conversation. And he said, okay. And I sent out invites on something that was called Libernet back then, which was for Crazy Libertarians and the Cryonics mailing list for crazy people who want to freeze their heads to save their asses.

And there's

Divia Eden: some crazy people on this call. Many of us are such people.

Perry Metzger: And I put out a call on a few Usenet news groups for those that don't remember Usenet, it's okay. You know, you don't want to hear how grandpa used to have to walk both ways up the hill to go to school anyway.

And the next thing you know, we have a few hundred people, many of whom become pretty famous, you know, like arguing about all of these topics. And at one point, we were talking about what the future, what the post-human future would look like.

And I did a back of the envelope calculation and I said, well, you know, the largest practical computer I could imagine making would be something like the size of Jupiter, you know, so if youhad an ai, the size of Jupiter, what's roughly the ratio between its cognition levels and the cognition levels of the average human being?

This looks pretty dismal. Right. You know, it's a lot worse than sense. It's a lot worse than the ratio between human beings and ants. And I haven't done the calculation in a while, so, you know, I invite people to go off and figure it out. And so this is where the Jupiter brain meme came in.

My buddy [00:05:00] Harry being, you can figure out what he's like from his initial comment was, well, if your brain's the size of Jupiter, how large is your penis? But anyway the jupiter brain meme died very early on because a bunch of people figured out that the cooling problems of having a computer that large would be bad. And, and the design that all the people who want to build a kishev you know, type two civilization these days are interested in, is the so-called Matryoshka brain, where you take all of the matter in the solar system and turn it into concentric shells of, of, of, you know, photovoltaic and computronium and have them communicate with each other and fly in swarms around the sun.

And of course, completely blot out the sun because why would you want to let any of that precious solar radiation escape when you could use it for computation? By the way, if you ever note there was a tweet that I think I forwarded a while ago, which said, you know, something to the effect of, I'm a conservative, I think that we should leave a hole [00:06:00] in the matryoshka swarm so that sunlight can get to the earth.

Right. You know, and then there are, you know, the more radical types who think that we should just disassemble the earth, because why would you leave all of that precious material being mostly wasted, you know? But nevermind that. So, yeah, I believe that I was the first person to coin the Jupiter brain meme, but it's a dead meme, so who cares?

Is this around obsolete technology? This was the very early 1990s, early nineties. Yeah, I still have somewhere the original invite for people to join the extropians mailing list. I can probably even find the date because thanks to text to the modern world, I can, no.

I would've thought that just doing a spotlight search on my desktop would find it pretty easily, but it didn't. But yeah,…

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit mutualunderstanding.substack.com