Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Doing 1-on-1s Better - EAG Tips Part II, published by kuhanj on March 24, 2022 on The Effective Altruism Forum.

Context

Many EAGx and EAG conferences are coming up (EAGx Oxford March 26-27, EAGx Boston April 1-3, and EAG London April 15-17), and attendees are often encouraged to spend most of their time at the conference in 1-on-1 meetings. However, the case for why 1-on-1s are so valuable is not often clear to attendees, nor is the process for setting up 1-on-1s.

We (Kuhan and Akash) thought it would be useful to share some thoughts on why we think 1-on-1s are so valuable, who to set up meetings with, and how to get the most out of them.

Why 1-on-1s/meetings can be one of the best uses of time

Here are a few reasons why we think 1-on-1s are often valuable:

1-on-1s offer a unique opportunity to get personalized feedback and information that is helpful for your top priority questions, uncertainties, career plans, and more, unlike talks and workshops that aren’t specifically catered to you.

People often think better when talking to others.

Perhaps most importantly, you get very quick feedback loops on your thinking, which are harder to get in other formats - mistakes, alternatives, and potential improvements to your ideas and their communication can be pointed out quickly

They offer a uniquely good setting for asking and being asked/answering really good questions.

It can be easier to stay focused when talking to someone else and being regularly required to actively contribute

Conversations can force you to refine your thinking and communication of ideas, since your conversation partner needs to understand what you’re saying.

You can incorporate new perspectives/information from your conversation partner quickly

Easy accountability for ideas that come up - ideas can turn into concrete plans, which your conversation partner can provide accountability for.

1-on-1s can turn into long-lasting friendships, professional connections, references/referrals for job/project/internship opportunities

1-on-1s provide data for how to communicate better moving forward, not just in future conversations but also for writing and presentations

We’ve unsurprisingly learned the most about how to talk, write, and present about EA, longtermism, AI safety, and more from many individual conversations, more so than feedback on writing/blogposts and presentations.

Find the right people to have 1-on-1s with

Given the ways 1-on-1s can be valuable, think about who to set up 1-on-1s with. Conferences like EAG/(x) are a great time to meet lots of people, but most community members I know are happy to have calls/in-person meetings outside of conferences as well.

Here are a few tips to brainstorm good people to reach out to:

Considering your areas of expertise and background, reach out to people who might benefit a lot from talking to you. Examples might include:

People interested in doing something similar to your current (or a previous) work who have less experience (e.g. junior AI safety researchers or group organizers).

People with smaller networks who would benefit from being introduced to people in yours

People from your city/university/country if you’re a group organizer (remember to look up your city/university/country on Swapcard and see if there are people you should reach out to/catch up with)

People who would benefit from learning how you think about important topics you have expertise in (both people who have different opinions/models and people who haven’t thought much about the topic but would benefit a lot from learning more about it).

Given your most important uncertainties/questions, reach out to the people best suited to address your most important questions/uncertainties. It can be good to reach out to people who you expect will have complementary knowledge and perspectives (e.g. tal...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Awards for the Future Fund’s Project Ideas Competition, published by Nick Beckstead on March 24, 2022 on The Effective Altruism Forum.

This post announces the winners of the Future Fund’s Project Ideas Competition, and reflects on the process of running the competition.

We got an overwhelming response to the competition, receiving close to 1000 submissions. The blog post received more comments than any other post on the Forum; in fact, more than double the second-most-commented post. We were thrilled at the level of excitement that the competition generated. So that we can appropriately reward the submissions, we’ve decided to include a category of “honorable mentions”: these will not go onto the website, but will receive an award of $1000. We will be contacting the winners individually about how to receive their prizes.

Winners

The winners - which will soon go up on our project ideas page (often in modified form) - are as follows:

gavintaylor - Infrastructure to support independent researchers

Epistemic Institutions, Empowering Exceptional People

The EA and Longtermist communities appear to contain a relatively large proportion of independent researchers compared to traditional academia. While working independently can provide the freedom to address impactful topics by liberating researchers from the perverse incentives, bureaucracy, and other constraints imposed on academics, the lack of institutional support can impose other difficulties that range from routine (e.g. difficulties accessing pay-walled publications) to restrictive (e.g. lack of mentorship, limited opportunities for professional development). Virtual independent scholarship institutes have recently emerged to provide institutional support (e.g. affiliation for submitting journal articles, grant management) for academic researchers working independently.

We expect that facilitating additional and more productive independent EA and Longtermist research will increase the demographic diversity and expand the geographical inclusivity of these communities of researchers. Initially, we would like to determine the main needs and limitations independent researchers in these areas face and then support the creation of a virtual institute focussed on addressing those points.

Konstantin Pilz - EA content translation service

Effective Altruism, Movement-Building

EA-related texts are often using academic language needed to convey complex concepts. For non-native speakers reading and understanding those texts takes a lot more time than reading about the same topic in their native language would. Furthermore, today many educated people in important positions, especially in non-western countries, do not speak or only poorly speak English. (This is likely part of the reason that EA currently mainly exists in English speaking countries and almost exclusively consists of people speaking English well.)

To make EA widely known and easy to understand there needs to be a translation service enabling e.g. 80k, important Forum posts or the Precipice to be read in different languages. This would not only make EA easier to understand - and thus spread ideas further - but also likely increase epistemic diversity of the community by making EA more international.

Mackenzie Arnold - A regulatory failsafe for catastrophic or existential biorisks

Biorisk and Recovery from Catastrophes

Currently, many government regulators (like the FDA in the US) apply a static set of criteria when evaluating countermeasures used to fight disease or other public harms. While these criteria may operate relatively well during normal times, during catastrophic events, they would likely impose overly cautious limitations on response efforts and, in some cases, may even prohibit the development or deployment of countermeasures with relatively minor risk p...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cyberwar escalation, published by ChristianKl on March 23, 2022 on LessWrong.

From arstechnica article Sabotage: Code added to popular NPM package wiped files in Russia and Belarus:

Two weeks ago, the node-ipc author pushed a new version of the library that sabotaged computers in Russia and Belarus, the countries invading Ukraine and providing support for the invasion, respectively. The new release added a function that checked the IP address of developers who used the node-ipc in their own projects. When an IP address geolocated to either Russia or Belarus, the new version wiped files from the machine and replaced them with a heart emoji.

I would expect that there are currently officers at the FSB and also non-government actors in Russia who are thinking about how to retaliate against this attack.

If you don't have a setup for data backup and would lose important data if someone wiped your hard drive, now is the time to fix it.

If you have the keys for any package on which others depend it also makes sense to make sure that you aren't an easy target for the FSB and other Russian hackers who want to retaliate in kind.

Packaging your cyberweapon in a peace-not-war module is like it's 1984. It would be good if the FBI reacts fast and charges RIAEvangelist for hacking to prevent escalation of the conflict.

The Biden administration also warns for increased cyber attacks:

This is a critical moment to accelerate our work to improve domestic cybersecurity and bolster our national resilience. I have previously warned about the potential that Russia could conduct malicious cyber activity against the United States, including as a response to the unprecedented economic costs we’ve imposed on Russia alongside our allies and partners. It’s part of Russia’s playbook. Today, my Administration is reiterating those warnings based on evolving intelligence that the Russian Government is exploring options for potential cyberattacks.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paul Farmer (1959 – 2022), published by Gavin on March 24, 2022 on The Effective Altruism Forum.

One of the most prominent people in global health died last month.

Wikipedia makes it hard to see what Paul Farmer actually did, behind the mountain of honors and appointments he received for it (Harvard prof, MacArthur grant, UN envoy, etc). He cofounded Partners In Health, now a $100m+ per year organisation. PIH works in Haiti, Peru, Mexico, Russia, Lesotho, Kazakhstan, Rwanda, Malawi, Sierra Leone, India and Liberia, greatly upgrading the quality of healthcare near their operations, and doing a huge amount of welfare programmes besides.

They class themselves as a humanitarian organisation, though most of their work is not the crisis work you'd associate with that (except incidentally, as when an earthquake happened next to the hospital they were building).

They do a huge, huge variety of things, under the root-cause theory of public health, where stuff like good shelter and food is treated as part of healthcare. They started out doing HIV treatment - and once you're doing that it only makes sense to do HIV prevention - and after that you're kinda hosed. An incomplete list: "HIV treatment, tuberculosis treatment... food baskets, transportation, lodging... dirty water... from in-home consultations to cancer treatments... job skills training, small business loans... a university in Rwanda... the world's largest TB research study... biocontainment unit in Lima... allowing [children] to attend school and receive food".

Farmer's most notable work is being the single most powerful advocate for Haitian public health; PIH staff serve nearly half the entire country, and he was mates with Bills Clinton and Gates. Haiti will need another Farmer:

He was an anthropologist by trade. In the past I've gotten annoyed with anthropology for bad epistemics (or for conflating good epistemics and good activism). But it's hard to fault Farmer as exemplar of the general approach "don't just watch, do something".

I take his lifework to amount to the importance of operations. He didn't develop any vaccines or pills, he didn't make a pile of cash and give it away, and his research wasn't the main feature. Instead (as per Wikipedia) his org "created specific initiatives", "improved medical infrastructure" and provided "accompaniment rather than charity". (As with all large development organisations, they have their own homebrew intellectual framework, "Supervision-Partners-Incentives-Choice-Education".) Suitably unglamorous terms for good unglamorous things.

He was extremely good at getting powerful people to care. There's a whole book about him, and he has been widely elegised. This marks out his strategy as the prestige route, using the system. Near the ceiling of that approach, perhaps.

Achievements

As usual in global health, most discussion of PIH's impact is actually about their inputs: number of staff, number of programmes, etc. (Not their budget though, which is both good and bad.)

GiveWell gave them an ultra-tentative recommendation in 2007, but this was largely on priors about good healthcare in countries with a shortage of it. ("We would guess that it is improving health outcomes") When evaluated properly in 2010, GW found that PIH weren't formally evaluating their own work and mostly wouldn't share their programme budgets, and so they couldn't give them the internalist stamp of approval. ("We would guess that they are outside – though not necessarily far outside – what we consider to be a reasonable range [of cost-effectiveness].")

"focused on AIDS prevention during the HIV crisis and successfully decreased HIV transmission rates by 4% from mothers to babies"

Built and run several hospitals, including 40% of Haiti's medical system. One nice feature: they're big on videoconferencing for ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The BEAHR: Dust off your CVs for the Big EA Hiring Round!, published by Sanjay on March 24, 2022 on The Effective Altruism Forum.

[epistemic status: I wrote this post fairly quickly. I wouldn't be surprised if I’ve missed some relevant points, feel free to correct me in the comments]

Over the coming two weeks, the FTX foundation will be making its decisions and likely disbursing $100m, possibly more.

Let’s assume that those funds are going to be spent on salaries (imprudent assumption) and that those salaries average $100k (probably a prudent assumption, I’d guess), and that no more than $100m is donated (prudent assumption)...

. then there could be 1,000 EA jobs up for grabs in the coming months.

As if I don’t have enough eccentricities, I’m going to refer to the Big EA Hiring Round as the BEAHR.

Has the EA community ever had this many new jobs created in such a short space of time?

Not as far as I’m aware.

As far as I know, it’s as if a BEAHR from deepest darkest Peru has suddenly and magically turned up at a tube station.

What does this mean for you, a potential applicant?

Are you someone who has been thinking of getting a job in an EA org, but not gotten round to it?

Now is a good time to get ready for the BEAHR!

If you want to take action right now, you might want to prepare your CV. When I have hired for EA jobs in the past, I have requested a CV, but I have not been very prone to reject people for a bad CV – I’ve mostly used it as an indication of intent to apply, plus a useful reference point before the interview.

This means that if you’re the sort of person who agonises over whether your CV is good enough, you can relax. At least, assuming that other organisations take the same approach as me!

If anyone wants to include further thoughts or advice on putting together your CV in the comments, feel free.

Vaidehi provided some useful more general advice encouraging people to apply for jobs here.

We will need the right infrastructure, including jobs boards.

I am happy to report that I have used the 80,000 hours jobs board and found it to be a hugely valuable resource. It has led to a very high calibre of applicant, and a good quantity too. I’m hugely grateful to 80,000 hours for providing this great resource.

However I have heard reports that people have had an inconsistent experience of using the 80,000 jobs board, and suggestions that it is underresourced.

I also understand that 80,000 hours has been hiring for someone to manage the jobs board, so I very much hope that that person can hit the ground running and will relish the challenge of meeting the substantial needs for this resource.

Even assuming that the 80,000 hours jobs board is functioning perfectly in the coming months, I still think that duplication/competition is valuable.

I would love it if someone decided to quickly create and boldly market a new EA jobs board.

This will be useful for the forthcoming BEAHR, and possibly for beyond.

. and don’t forget the Effective Altruism Job Postings facebook group!

If you’re not on that facebook group, I would encourage you to join it!

And if you’re on it, please check in on it. I predict that the BEAHR may make that space a bit more frenzied in the coming months!

Does this mean the end of “it’s really really hard to get hired by an EA organisation”

Three years ago, a plaintive, heartfelt post struck a deep chord within the community, and became a hit on the EA forum.

Will the BEAHR bring about an end to it being so hard to get hired?

Sadly, not necessarily.

I predict that some people will feel the great joy of getting a new job, but others will still experience a series of rejections without getting hired.

And that this will happen for some very talented people.

Why?

Firstly, there is an idiosyncratic need that organisations have. It is really normal for me to...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is part four of: Christiano, Cotra, and Yudkowsky on AI progress, by Eliezer Yudkowsky and Ajeya Cotra

9.12. Self-duplicating factories, AI spending, and Turing test variants

[Yudkowsky][17:05]

so, here's an idea that is less epistemically virtuous than our making Nicely Resolvable Bets

what if we, like, talked a bunch about our off-the-cuff senses of where various AI things are going in the next 3 years

and then 3 years later, somebody actually reviewed that

[Christiano][17:06]

I do think just saying a bunch of stuff about what we expect will happen so that we can look back on it would have a significant amount of the value

[Yudkowsky][17:06]

and any time the other person put a thumbs-up on the other's prediction, that prediction coming true was not taken to distinguish them

[Cotra][17:06]

i'd suggest doing this in a format other than discord for posterity

[Yudkowsky][17:06]

even if the originator was like HOW IS THAT ALSO A PREDICTION OF YOUR THEORY

well, Discord has worked better than some formats

[Cotra][17:07]

something like a spreadsheet seems easier for people to look back on and score and stuff

discord transcripts are pretty annoying to read

[Yudkowsky][17:08]

something like a spreadsheet seems liable to be high-cost and not actually happen

[Christiano][17:08]

I think a conversation is probably easier and about as good for our purposes though?

[Cotra][17:08]

ok fair

[Yudkowsky][17:08]

I think money can be inserted into humans in order to turn Discord into spreadsheets

[Christiano][17:08]

and it's possible we will both think we are right in retrospect

and that will also be revealing

[Yudkowsky][17:09]

but, besides that, I do want to boop on the point that I feel like Paul should be able to predict intuitively, rather than with necessity, things that should not happen before the world economy doubled in 4 years

[Christiano][17:09]

it may also turn up some quantitative differences of view

there are lots of things I think won't happen before the world economy has doubled in 4 years

[Yudkowsky][17:09]

because on my model, as we approach the end times, AI was still pretty partial and also the world economy was lolnoping most of the inputs a sensible person would accept from it and prototypes weren't being commercialized and stuff was generally slow and messy

[Christiano][17:09]

prototypes of factories building factories in <2 years

[Yudkowsky][17:10]

"AI was still pretty partial" leads it to not do interesting stuff that Paul can rule out

[Christiano][17:10]

like I guess I think tesla will try, and I doubt it will be just tesla

[Yudkowsky][17:10]

but the other parts of that permit AI to do interesting stuff that Paul can rule out

[Christiano][17:10]

automated researchers who can do ML experiments from 2020 without human input

[Yudkowsky][17:10]

okay, see, that whole "factories building factories" thing just seems so very much after the End Times to me

[Christiano][17:10]

yeah, we should probably only talk about cognitive work

since you think physical work will be very slow

[Yudkowsky][17:11]

okay but not just that, it's a falsifiable prediction

it is something that lets Eliezer be wrong in advance of the End Times

[Christiano][17:11]

what's a falsifiable prediction?

[Yudkowsky][17:11]

if we're in a world where Tesla is excitingly gearing up to build a fully self-duplicating factory including its mining inputs and chips and solar panels and so on, we're clearly in the Paulverse and not in the Eliezerverse!

[Christiano][17:12]

yeah

I do think we'll see that before the end times

just not before 4 year doublings

[Yudkowsky][17:12]

this unfortunately only allows you to be right, and not for me to be right, but I think there are also things you legit only see in the Eliezerverse!

[Christiano][17:12]

I mean, I don't think they will be doing mining for a long time because it's...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is part three of: Christiano, Cotra, and Yudkowsky on AI progress, by Eliezer Yudkowsky and Ajeya Cotra

9.9. Prediction disagreements and bets

[Christiano][16:19]

anyway, I'm pretty unpersuaded by the kind of track record appeal you are making here

[Yudkowsky][16:20]

if the future goes the way I predict and yet anybody somehow survives, perhaps somebody will draw a hyperbolic trendline on some particular chart where the trendline is retroactively fitted to events including those that occurred in only the last 3 years, and say with a great sage nod, ah, yes, that was all according to trend, nor did anything depart from trend

trend lines permit anything

[Christiano][16:20]

like from my perspective the fundamental question is whether I would do better or worse by following the kind of reasoning you'd advocate, and it just looks to me like I'd do worse, and I'd love to make any predictions about anything to help make that more clear and hindsight-proof in advance

[Yudkowsky][16:20]

you just look into the past and find a line you can draw that ended up where reality went

[Christiano][16:21]

it feels to me like you really just waffle on almost any prediction about the before-end-of-days

[Yudkowsky][16:21]

I don't think I know a lot about the before-end-of-days

[Christiano][16:21]

like if you make a prediction I'm happy to trade into it, or you can pick a topic and I can make a prediction and you can trade into mine

[Cotra][16:21]

but you know enough to have strong timing predictions, e.g. your bet with caplan

[Yudkowsky][16:21]

it's daring enough that I claim to know anything about the Future at all!

[Cotra][16:21]

surely with that difference of timelines there should be some pre-2030 difference as well

[Christiano][16:21]

but you are the one making the track record argument against my way of reasoning about things!

how does that not correspond to believing that your predictions are better!

what does that mean?

[Yudkowsky][16:22]

yes and if you say something narrow enough or something that my model does at least vaguely push against, we should bet

[Christiano][16:22]

my point is that I'm willing to make a prediction about any old thing, you can name your topic

I think the way I'm reasoning about the future is just better in general

and I'm going to beat you on whatever thing you want to bet on

[Yudkowsky][16:22]

but if you say, "well, Moore's Law on trend, next 3 years", then I'm like, "well, yeah, sure, since I don't feel like I know anything special about that, that would be my prediction too"

[Christiano][16:22]

sure

you can pick the topic

pick a quantity

or a yes/no question

or whatever

[Yudkowsky][16:23]

you may know better than I would where your Way of Thought makes strong, narrow, or unusual predictions

[Christiano][16:23]

I'm going to trend extrapolation everywhere

spoiler

[Yudkowsky][16:23]

okay but any superforecaster could do that and I could do the same by asking a superforecaster

[Cotra][16:24]

but there must be places where you'd strongly disagree w the superforecaster

since you disagree with them eventually, e.g. >2/3 doom by 2030

[Bensinger][18:40] (Nov. 25 follow-up comment)

">2/3 doom by 2030" isn't an actual Eliezer-prediction, and is based on a misunderstanding of something Eliezer said. See Eliezer's comment on LessWrong.

[Yudkowsky][16:24]

in the terminal phase, sure

[Cotra][16:24]

right, but there are no disagreements before jan 1 2030?

no places where you'd strongly defy the superforecasters/trend extrap?

[Yudkowsky][16:24]

superforecasters were claiming that AlphaGo had a 20% chance of beating Lee Se-dol and I didn't disagree with that at the time, though as the final days approached I became nervous and suggested to a friend that they buy out of a bet about that

[Cotra][16:25]

what about like whether we get some kind of AI ability (e.g. coding better...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is part two of: Christiano, Cotra, and Yudkowsky on AI progress, by Eliezer Yudkowsky and Ajeya Cotra

9.5. Brain size and evolutionary history

[Cotra][14:24]

On software beating hardware, how much of your view is dependent on your belief that the chimp -> human transition was probably not mainly about brain size because if it were about brain size it would have happened faster? My understanding is that you think the main change is a small software innovation which increased returns to having a bigger brain. If you changed your mind and thought that the chimp -> human transition was probably mostly about raw brain size, what (if anything) about your AI takeoff views would change?

[Yudkowsky][14:25]

I think that's a pretty different world in a lot of ways!

but yes it hits AI takeoff views too

[Christiano][14:25]

regarding software vs hardware, here is an example of asking this question for imagenet classification ("how much compute to train a model to do the task?"), with a bit over 1 year doubling times (/). I guess my view is that we can make a similar graph for "compute required to make your AI FOOM" and that it will be falling significantly slower than 2x/year. And my prediction for other tasks is that the analogous graphs will also tend to be falling slower than 2x/year.

[Yudkowsky][14:26]

to the extent that I modeled hominid evolution as having been "dutifully schlep more of the same stuff, get predictably more of the same returns" that would correspond to a world in which intelligence was less scary, different, dangerous-by-default

[Cotra][14:27]

thanks, that's helpful. I looked around in IEM and other places for a calculation of how quickly we should have evolved to humans if it were mainly about brain size, but I only found qualitative statements. If there's a calculation somewhere I would appreciate a pointer to it, because currently it seems to me that a story like "selection pressure toward general intelligence was weak-to-moderate because it wasn't actually that important for fitness, and this degree of selection pressure is consistent with brain size being the main deal and just taking a few million years to happen" is very plausible

[Yudkowsky][14:29]

well, for one thing, the prefrontal cortex expanded twice as fast as the rest

and iirc there's evidence of a lot of recent genetic adaptation... though I'm not as sure you could pinpoint it as being about brain-stuff or that the brain-stuff was about cognition rather than rapidly shifting motivations or something.

elephant brains are 3-4 times larger by weight than human brains (just looked up)

if it's that easy to get returns on scaling, seems like it shouldn't have taken that long for evolution to go there

[Cotra][14:31]

but they have fewer synapses (would compute to less FLOP/s by the standard conversion)

how long do you think it should have taken?

[Yudkowsky][14:31]

early dinosaurs should've hopped onto the predictable returns train

[Cotra][14:31]

is there a calculation?

you said in IEM that evolution increases organ sizes quickly but there wasn't a citation to easily follow up on there

[Yudkowsky][14:33]

I mean, you could produce a graph of smooth fitness returns to intelligence, smooth cognitive returns on brain size/activity, linear metabolic costs for brain activity, fit that to humans and hominids, then show that obviously if hominids went down that pathway, large dinosaurs should've gone down it first because they had larger bodies and the relative metabolic costs of increased intelligence would've been lower at every point along the way

I do not have a citation for that ready, if I'd known at the time you'd want one I'd have asked Luke M for it while he still worked at MIRI 😐

[Cotra][14:35]

cool thanks, will think about the dinosaur thing (my first reaction is that this should depend on the actual fi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is part one of: Christiano, Cotra, and Yudkowsky on AI progress, by Eliezer Yudkowsky and Ajeya Cotra

This post is a transcript of a discussion between Paul Christiano, Ajeya Cotra, and Eliezer Yudkowsky on AGI forecasting, following up on Paul and Eliezer's "Takeoff Speeds" discussion.

Color key:

Chat by Paul and Eliezer Chat by Ajeya Inline comments

8. September 20 conversation

8.1. Chess and Evergrande

[Christiano][15:28]

I still feel like you are overestimating how big a jump alphago is, or something. Do you have a mental prediction of how the graph of (chess engine quality) vs (time) looks, and whether neural net value functions are a noticeable jump in that graph?

Like, people investing in "Better Software" doesn't predict that you won't be able to make progress at playing go. The reason you can make a lot of progress at go is that there was extremely little investment in playing better go.

So then your work is being done by the claim "People won't be working on the problem of acquiring a decisive strategic advantage," not that people won't be looking in quite the right place and that someone just had a cleverer idea

[Yudkowsky][16:35]

I think I'd expect something like... chess engine slope jumps a bit for Deep Blue, then levels off with increasing excitement, then jumps for the Alpha series? Albeit it's worth noting that Deepmind's efforts there were going towards generality rather than raw power; chess was solved to the point of being uninteresting, so they tried to solve chess with simpler code that did more things. I don't think I do have strong opinions about what the chess trend should look like, vs. the Go trend; I have no memories of people saying the chess trend was breaking upwards or that there was a surprise there.

Incidentally, the highly well-traded financial markets are currently experiencing sharp dips surrounding the Chinese firm of Evergrande, which I was reading about several weeks before this.

I don't see the basic difference in the kind of reasoning that says "Surely foresightful firms must produce investments well in advance into earlier weaker applications of AGI that will double the economy", and the reasoning that says "Surely world economic markets and particular Chinese stocks should experience smooth declines as news about Evergrande becomes better-known and foresightful financial firms start to remove that stock from their portfolio or short-sell it", except that in the latter case there are many more actors with lower barriers to entry than presently exist in the auto industry or semiconductor industry never mind AI.

or if not smooth because of bandwagoning and rational fast actors, then at least the markets should (arguendo) be reacting earlier than they're reacting now, given that I heard about Evergrande earlier; and they should have options-priced Covid earlier; and they should have reacted to the mortgage market earlier. If even markets there can exhibit seemingly late wild swings, how is the economic impact of AI - which isn't even an asset market! - forced to be earlier and smoother than that, as a result of wise investing?

There's just such a vast gap between hopeful reasoning about how various agents and actors should all do the things the speaker finds very reasonable, thereby yielding smooth behavior of the Earth, versus reality.

9. September 21 conversation

9.1. AlphaZero, innovation vs. industry, the Wright Flyer, and the Manhattan Project

[Christiano][10:18]

(For benefit of readers, the market is down 1.5% from friday close -> tuesday open, after having drifted down 2.5% over the preceding two weeks. Draw whatever lesson you want from that.)

Also for the benefit of readers, here is the SSDF list of computer chess performance by year. I think the last datapoint is with the first version of neural net evaluations, though I ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is part two of: More Christiano, Cotra, and Yudkowsky on AI progress, by Eliezer Yudkowsky

11. September 24 conversation

11.1. Predictions and betting (continued 2)

[Christiano][13:15]

I think RH is not that surprising, it's not at all clear to me where "do formal math" sits on the "useful stuff AI could do" spectrum, I guess naively I'd put it somewhere "in the middle" (though the analogy to board games makes it seem a bit lower, and there is a kind of obvious approach to doing this that seems to be working reasonably well so that also makes it seem lower), and 7% GDP growth is relatively close to the end (ETA: by "close to the end" I don't mean super close to the end, just far enough along that there's plenty of time for RH first)

I do think that performance jumps are maybe more dispositive, but I'm afraid that it's basically going to go like this: there won't be metrics that people are tracking that jump up, but you'll point to new applications that people hadn't considered before, and I'll say "but those new applications aren't that valuable" whereas to you they will look more analogous to a world-ending AGI coming out from the blue

like for AGZ I'll be like "well it's not really above the deep learning trend if you run it backwards" and you'll be like "but no one was measuring it before! you can't make up the trend in retrospect!" and I'll be like "OK, but the reason no one was measuring it before was that it was worse than traditional go algorithms until like 2 years ago and the upside is not large enough that you should expect a huge development effort for a small edge"

[Yudkowsky][13:43]

"factor of 4 jump from some clever innovation" - can you say more about that part?

[Christiano][13:53]

like I'm surprised if a clever innovation does more good than spending 4x more compute

[Yudkowsky][15:04]

I worry that I'm misunderstanding this assertion because, as it stands, it sounds extremely likely that I'd win. Would transformers vs. CNNs/RNNs have won this the year that the transformers paper came out?

[Christiano][15:07]

I'm saying that it gets harder over time, don't expect wins as big as transformers

I think even transformers probably wouldn't make this cut though?

certainly not vs CNNs

vs RNNs I think the comparison I'd be using to operationalize it is translation, as measured in the original paper

they do make this cut for translation, looks like the number is like 100 4

100x for english-german, more like 10x for english-french, those are the two benchmarks they cite

but both more than 4x

I'm saying I don't expect ongoing wins that big

I think the key ambiguity is probably going to be about what makes a measurement established/hard-to-improve

[Yudkowsky][15:21]

this sounds like a potentially important point of differentiation; I do expect more wins that big.

the main thing that I imagine might make a big difference to your worldview, but not mine, is if the first demo of the big win only works slightly better (although that might also be because they were able to afford much less compute than the big players, which I think your worldview would see as a redeeming factor for my worldview?) but a couple of years later might be 4x or 10x as effective per unit compute (albeit that other innovations would've been added on by then to make the first innovation work properly, which I think on your worldview is like The Point or something)

clarification: by "transformers vs CNNs" I don't mean transformers on ImageNet, I mean transformers vs. contemporary CNNs, RNNs, or both, being used on text problems.

I'm also feeling a bit confused because eg Standard Naive Kurzweilian Accelerationism makes a big deal about the graphs keeping on track because technologies hop new modes as needed. what distinguishes your worldview from saying that no further innovations are needed for AGI ...