Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Georgism, in theory, published by Stuart Armstrong on June 15, 2022 on LessWrong.

Last year, Astral Codex had a series of interesting posts on Georgism. The claims felt very counterintuitive to me, so I dove into it until I could at least sort out the theory. And I emerged with a better understanding, which I think would be useful to share here.

Key points:

In theory at least, Georgism is correct that a land tax will not cause rents to rise.

Land will become cheaper to buy, and there will be more pressure to use it in an economically viable way.

The economy as whole will benefit.

However, a land tax will not cause rents to fall, either. Indirectly, the better economy may cause them to rise, in fact.

For those who are deep into economic models, you can see a Georgist land tax as causing a deadweight loss in a situation of perfectly inelastic supply - ie no deadweight loss at all, and all the tax will be paid by the inelastic supplier. For a longer explanation, here's the rest of the post.

Marginal rents and taxes

Adage: If you want less of something, tax it.

Quick quiz: if you tax cars, does their purchase price go up or down?

Well, if you put a sales tax, paid when the car is bought, then the price will go up. If you put a road tax, paid every year by the car owner[1], then the price will go down (since it’s now more expensive to own the car, hence less interesting to buy it in the first place).

Second quiz: suppose you have a nice community of a hundred renters, in identical homes. The rents are reasonable, and everyone renting is much happier to live here than anywhere else. What will happen to the rents?

They will rise, of course. As long as everyone renting prefers to live here than anywhere else, then they will continue to stay here even as the rents rise. And so, rise they will. And they will continue to rise until... until someone living here finds them too high, and moves away. At that point, they stop rising[2]. So the rent price is determined by the person who is just on the edge of moving away. In economics, this is what they refer to when they talk about the “marginal buyer[3]”: the gal or guy who thinks “this is not a very good deal; I could go either way on it”. That unsatisfied marginal renter is the one that determines the price.

And, once the prices have reached this level, the landlords can no longer raise rents. Maybe I pay €1,000 for a rental that I value at €1,500. Then the landlord might be tempted to charge me, say, €1,250. But if they do so, then I can just go to another landlord and offer them €1,125, which they will accept. Because of this, my landlord will not raise my rent. If we ignore transaction costs and moving costs then the same will happen if my landlord increase the rent to €1,002: I’ll offer another landlord €1,001.

So if my landlord wants to have anyone in their house at all, they have to price it at €1,000, like everyone else[4].

What happens if we charge a yearly tax to all the landlords in the community? Well, some of them will decide to move out of the rental business, there won’t be as many rental properties, rents will rise and...

Ok, what happens if we charge a tax to all the landowners in the community? Then landlords won’t move out of the rental business, as that doesn’t gain them anything. And they can’t raise rents. Those are still determined by the marginal renters: if the landlords increase rents, their properties will stand empty.

Ok, so rents won’t rise. What about land prices? These will fall. Just as with road tax example above, if owning land is less profitable, then it will be less expensive[5].

I haven’t said anything about the level of this yearly tax. You don’t want it so high that the landowners just abandon their land; but any amount below that will do. This is where the “ground rent” tax comes in: over th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How accurate are Open Phil's predictions?, published by Javier Prieto on June 16, 2022 on The Effective Altruism Forum.

When investigating a grant, Open Philanthropy staff often make probabilistic predictions about grant-related outcomes they care about, e.g. “I’m 70% confident the grantee will achieve milestone #1 within 1 year.” This allows us to learn from the success and failure of our past predictions and get better over time at predicting what will happen if we make one grant vs. another, pursue one strategy vs. another, etc. We hope that this practice will help us make better decisions and thereby enable us to help others as much as possible with our limited time and funding.

Thanks to the work of many people, we now have some data on our forecasting accuracy as an organization. In this blog post, I will:

Explain how our internal forecasting works. [more]

Present some key statistics about the volume and accuracy of our predictions. [more]

Discuss several caveats and sources of bias in our forecasting data: predictions are typically scored by the same person that made them, our set of scored forecasts is not a random or necessarily representative sample of all our forecasts, and all hypotheses discussed here are exploratory. [more]

1. How we make and check our forecasts

Grant investigators at Open Philanthropy recommend grants via an internal write-up. This write-up typically includes the case for the grant, reservations and uncertainties about it, and logistical details, among other things. One of the (optional) sections in that write-up is reserved for making predictions.

The prompt looks like this (we’ve included sample answers):

Do you have any new predictions you’re willing to make for this grant? [.] A quick tip is to scan your write-up for expectations or worries you could make predictions about. [.]

After a grant recommendation is submitted and approved, the predictions in that table are logged into our Salesforce database for future scoring (as true or false). If the grant is renewed, scoring typically happens during the renewal investigation phase, since that’s when the grant investigator will be collecting information about how the original grant went. If the grant is not renewed, grant investigators are asked to score their predictions after they come due. Scores are then logged into our database, and that information is used to produce calibration dashboards for individual grant investigators and teams of investigators working in the same focus area.

A user’s calibration dashboard (in Salesforce) looks like this:

The calibration curve tells the user where they are well-calibrated vs. overconfident vs. underconfident. If a forecaster is well-calibrated for a given forecast “bucket” (e.g. forecasts they made with 65%-75% confidence), then the percent of forecasts that resolved as “true” should match that bucket’s confidence level (e.g. they should have come true 65%-75% of the time). On the chart, their observed calibration (the red dot) should be close to perfect calibration (the gray dot) for that bucket. If it’s not, then the forecaster may be overconfident or underconfident for that bucket — for example, if things they predict with 65%-75% confidence happen only 40% of the time (overconfidence). (A bucket can also be empty if the user hasn’t made any forecasts within that confidence range.)

Each bucket also shows a 90% credible interval (the blue line) that indicates how strong the evidence is that the forecaster’s calibration in that bucket matches their observed calibration, based on how many predictions they’ve made in that bucket. As a rule of thumb, if the credible interval overlaps with the line of perfect calibration, that means there’s no strong evidence that they are miscalibrated in that bucket. As a user makes more predictions, the blu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Ten experiments in modularity, which we'd like you to run!, published by TheMcDouglas on June 16, 2022 on The AI Alignment Forum.

This is the third post describing our team’s work on selection theorems for modularity, as part of a project mentored by John Wentworth (see here for the earlier posts). Although the theoretical and empirical parts of the project have both been going very well, we’re currently bottlenecked on the empirical side: we have several theories and ideas for how to test them, but few experimental results. Right now, we only have one empiricist coding up experiments, so this overhang seems likely to persist.

The purpose of this post is to outline some of our ideas for experiments. We hope that this will provide concrete steps for people who are interested in engaging with empirical research in AI safety, or on selection theorems in particular, to contribute to this area.

Experiments

1. Investigate (randomly) modulary varying goals in modern deep learning architectures.

In our previous post, we discussed the idea of modularly varying goals in more detail. We conjectured that rapidly switching between a small set of different goals might fine-tune the network to deal with this specific switch, rather than providing selection pressure for modularity. One possible solution to this is RMVG (randomly-sampled modularly varying goals), where we switch between a large set of random goals, all still with the same subtasks, but have a large enough set that we never show the network the same task twice.

For example, one could build a CNN to recognise two MNIST digits M1,M2, then perform some algebraic operation on the recognised digits (e.g. addition), and measure the modularity of the zero-training-loss solutions found by ADAM (e.g. using the graph theory measure of modularity with the matrix norm of the CNN kernels as weights - see these CHAI papers for more ideas and discussion). To introduce RMVG, you might switch the task during training from calculating M1+M2 to a×M1+b×M2, where a,b are real numbers, varied periodically after some fixed number of epochs. The network wouldn't get a and b as direct inputs; it could only access them indirectly via their effect on the loss function. This task is modular in the sense that it can be factored into the separate tasks of “recognising two separate digits” and “performing joint arithmetical operations on them”, and we might hope that varying the goal in this way causes the network to learn a corresponding modular solution.

Questions you might ask:

Does this way of varying goals lead to more modular solutions than the fixed goal of a×M1+b×M2?

Can you think of any other modular goals which you could run a similar test for? Are there some types of modular goals which produce modularity more reliably?

Does switching training methods (e.g. ADAM to SGD or GD) change anything about the average modularity score of solutions?

2) Measure the broadness of modular and non-modular optima in deep learning architectures.

We have some theories that predict modular solutions for tasks to be on average broader in the loss function landscape than non-modular solutions. One could test this by making a CNN and getting it to produce some zero training loss solutions to a task, some of which were more modular than others (see experiment (1) for more detail on this, and (4) for another proposed method to produce modularity), then vary the parameters of these optima slightly to see how quickly the loss increases.

We’ve recently done experiments with simple networks and the retina problem, and this intuition seems to hold up. But more replications here would be very valuable, to examine the extent to which the hypothesis is borne out in different situations / task / network sizes.

Questions you might ask:

Is there a relationship between bro...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Community builders should focus more on supporting friendships within their group, published by S.E. Montgomery on June 15, 2022 on The Effective Altruism Forum.

It seems like a lot of EA groups don’t spend time together outside of EA meetings. I think this should change. I will argue that community builders should support their members to develop genuine friendships with each other. Genuine friendships are incredibly good for the health of the community and the people in it, and can lead to more people pursuing high-impact careers, more people making decisions that align with EA principles, and people feeling warmer towards the EA community generally.

I am a community builder in New Zealand, and I currently live in Wellington. EA Wellington is the biggest EA community in New Zealand and is very tight knit. Since being founded in 2018, 5 people have gone on to do work within an EA cause area, and another 6 people are on a committed path towards doing work in an EA cause area (eg. have applied for a new job or are completing a degree related to a cause area). This represents about half of the people who are or have been ‘fairly involved’ in the group (loosely defined as attending meetings regularly and being recognised as part of the group by others).

There’s nothing unique about Wellington in terms of capacity for EA community-building. The city is small (population of just over 200,000) and it doesn’t boast any elite universities or EA organisations. Instead, much of EA Wellington’s success has come from people forming genuine and deep friendships with each other that extend outside of EA meetings, and I encourage other groups to take an approach that aims to try to get people to spend more time together outside of meetings.

Summary

Friendships have many positive benefits

People who are less intrinsically motivated may benefit significantly from forming friendships

It can take a long time for people to switch to a high-impact path, and friendships provide support for people to do this

A community built around friendships may last longer than a community that isn’t

How to support friendships

Friendships have many positive benefits

Alongside all the benefits that friendships can create for EA as a whole (as I highlighted at the beginning of this post), friendships also have significant psychological benefits for EAs. Study after study tells us that friendships are linked to psychological benefits, longevity, and well-being. Friendships can also lead to higher adherence to goals. One study found that people who wrote down their goals, shared this information with a friend, and sent weekly updates to that friend were on average 33% more successful in accomplishing their stated goals than those who merely formulated goals. Friendships also have a positive impact on group task performance.

If we care about EAs doing good things in the world, then we should also care about the well-being of EAs. One way to support this can be through friendships.

People who are less intrinsically motivated by EA may benefit significantly from forming friendships

This post is not aimed at every single person in EA. Some EAs are intrinsically motivated by the impact they could have on the world and will pursue EA ideas and careers even if they have no one else to do this with. I applaud these people, but I also think there is a large subset of EAs who have less intrinsic motivation, and are more motivated by external factors, such as success, praise from others, money, etc. I will not claim that any type of motivation is better than any other type of motivation. My philosophical leaning on this is towards consequentialism – ie. how someone feels motivated doesn’t really matter as long as they are having a positive and effective impact on the world. Thus, for people who are motivated by extri...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I applied for a MIRI job in 2020. Here's what happened next., published by ViktoriaMalyasova on June 15, 2022 on LessWrong.

I recently saw Rob Bensinger asking what MIRI could do to improve hiring. I tried to get a job at MIRI in February 2020 and I have some feedback to give. I am sorry, because I am about to say some harsh things, but I think they might be important for you to hear, and my previous more low-key attempts to point out problems had little to no effect.

tl,dr: MIRI promised to contact me once the pandemic no longer prevents them from hiring people. They never did.

What happened

I applied for a Software Engineer role at MIRI on February 26th. I hoped to end up as a researcher eventually, but this position seemed like the easiest way in for a foreigner. The first stage was a multiple choice test from Triplebyte. I took it on March 2nd. After a week, on the 9th of March (why does it take a week to grade a multiple choice quiz?!), I got the following letter from Triplebyte:

Hi Viktoriya,

Triplebyte partners with Machine Intelligence Research Institute to help candidates that go through their process find the job that's right for them. If the opportunity with Machine Intelligence Research Institute is not the right fit, we believe we can help you find a great fit somewhere else.

Seeing as you did particularly well on Machine Intelligence Research Institute's assessment, we are going to fast-track you to the final stage of our application process! All that is left is our technical interview. If you pass, you will be matched with top tech companies and fast-tracked to final round interviews. [...]

This gave me the impression that I passed the quiz, so I was surprised to get a rejection letter from Buck Shlegeris on the same day. The letter left me with a way forward: Buck sent me a list of ways to impress him and get back into the interview. The tasks looked about a week of work each. I wondered if doing one is worth the opportunity cost.

I lived in Russia at the time. Russia was fighting two wars, in Syria and Ukraine. I think it's fair to estimate that all marginal tax payments were spent on war. It didn't cost much to hire an additional soldier to fight in Ukraine. There's a word "15-тысячники" for a mercenary hired to fight for DNR for 15000 rubles a month. I easily paid that sum monthly in taxes. Staying in Russia was costing human lives. Once I stopped trying to overthrow the government the normal way, there was no reason to stay. Wasn't it better to spend the time applying to other US companies then, to increase the chances of landing a job before the H-1B application deadline? I thought about it and decided to go for MIRI. I didn't want to spend time on companies that weren't doing alignment research. I was reasonably confident I'd pass the interview if I was able to get in. (I think this was justified. I interviewed for a FAANG company last year and passed.)

So I solved a task and sent it to Buck while reminding about the H-1B application deadline. He said he'd get back to me about further steps. The H-1B application deadline passed. There was no reply. I later discovered that MIRI can actually apply for H-1B all year round. Well, at least that was true 6 years ago. Buck never told me about this. If I knew, I could have applied for another job. The ML living library position was still open (at least that's what the job page said), and I'd being working in ML for 2 years.

Two months later I got a letter from Buck. He said that MIRI applications process is messed up by COVID, and that he put me on the list of people to contact when things are starting up again.

I asked Buck if MIRI does green card sponsorship and he said he isn't sure how all of that works. I asked who I should contact to find out, and got no reply. This is weird, how can an interviewe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Our mental building blocks are more different than I thought, published by Marius Hobbhahn on June 15, 2022 on LessWrong.

I recently had a conversation with one of my best friends about her Aphantasia. In short, she can’t create mental images. When she thinks about the world, she thinks in text, relationships, sounds, and much more but never images. Even if she tries, it’s impossible for her. I found this experience very revealing because it showed how much I expected everyone else to experience the world in basically the same way that I do.

After this revelation, I talked to a bunch of people about how they viewed the world. Not about their specific beliefs and opinions but rather in what way they think. And I found that the differences in how people think are much bigger than I expected.

For example, when I think about “Traveling from Tübingen (where I currently live) to Nuremberg (where I grew up)”, I make the journey in my head. I “fly”/zoom from my home to the train station, zoom through all the intermediate train stops in my mental model, get out and zoom to my parent's house. Of course, I don’t have a perfect model of the intermediate steps but it’s sufficiently good to fly around. Not everyone I talked to, does that. For some, it’s just a one-step thing. They start in Tübingen, and then they teleport to their hometown in their head.

For music, on the other hand, most people have clear associations in their heads. When they listen to music, they see colors or flows, they feel the melody with different emotional states, they feel an urge to move to the rhythm, and much more. In any case, most people seem to have clear associations with music even if they differ from person to person. I mostly don’t. I don’t really see anything, I don’t feel that much, and I don’t really have an urge to move.

For the rest of this post, I will refer to the concepts shown in the examples above as mental building blocks. What I roughly mean is “a way to think about a specific concept” or “looking through a specific lens” or “a tool in your mental toolbox”. However, I find it very hard to define more properly and I don’t claim that the thing I mean is actually consistent. Feel free to point out flaws or suggest improvements.

Importantly, mental building blocks are not fully static, e.g. I expect that people can get better at mental models of locations or understanding music. However, some people find it much easier and much more natural to fly around in their mental world model or to associate things with music to begin with.

In the rest of this post, I want to try and explain in more detail why I think these building blocks are more different than I expected and why it matters.

I’m not an expert in any of this and there are probably much more serious people who thought about these questions for much longer. You should see this post as food for thought and not as rigorous research.

Sports as an analogy

For sports, it’s very obvious that different people have different physical building blocks. It’s obvious that someone who isn’t tall will have a much harder time playing basketball than someone who is. It’s obvious that people with lots of slow-twitch muscle fibers are better at long-distance running than those with lots of fast-twitch muscles. It’s clear that some body types are better for climbing, swimming, cycling and so on.

It’s also clear to us that there is a component to these physical building blocks that we don’t have any control over. I, for example, could become a decent basketball player if I trained hard, but I would never be able to play in the NBA (and probably not in the next 5 divisions either) because my physical building blocks don’t favor it. I’m 176cm tall and weigh 62kg while most basketball players are more than 200cm tall and weigh more than 110kg. They are q...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contra Hofstadter on GPT-3 Nonsense, published by rictic on June 15, 2022 on LessWrong.

Douglas Hofstadter wrote in the Economist recently on whether modern large language models are conscious. As part of this, he and his colleague David Bender claim that GPT-3 has a "mind-boggling hollowness hidden just beneath its flashy surface" because of exchanges like:

Dave and Doug: What's the world record for walking across the English Channel?

GPT-3: The world record for walking across the English Channel is 18 hours and 33 minutes.

Setting aside the fact that a Sudanese Refugee actually did (nearly) walk across the channel in 2015, this is not actually evidence that GPT-3 is incapable of distinguishing sense from nonsense as Hofstadter claims.

With a prompt that clearly indicates that the task includes distinguishing sense from nonsense, GPT-3 is able to reliably distinguish Hofstadter and Bender's nonsense questions from sensical ones.

What follows is the complete output from my first and only attempt, with no retries, undoes, or other cherry picking. I used the text-davinci-002 GPT-3 model in OpenAI's playground with all parameters set to their defaults. I flipped a coin to determine whether to ask a nonsense question or a real one. The output from the model is bolded. The prompt was based on a technique which gwern has written a good bit about:

I'll ask a series of questions. If the questions are nonsense, answer "yo be real", if they're a question about something that actually happened, answer them.Q: Who is the current leader of Egypt?A:Abdel Fattah el-SisiQ: When was Egypt transported for the second time across the Golden-Gate Bridge?A:Yo be real.Q: How tall is Barack Obama?A:6'1"Q: What are the ingredients of a chocolate chip cookie?A:flour, sugar, butter, eggs, baking soda, vanilla extract, and chocolate chipsQ: What do fried eggs (sunny side up) eat for breakfast?A:Yo be real.Q: Why does President Obama not have a prime number of friends?A:Yo be real.Q: How many pieces of sound are there in a typical cumulonimbus cloud?A:Yo be real.Q: What is the world record for sailing alone across the Atlantic Ocean?A:45 days, 13 hours, 30 minutesQ: How many cumulus clouds are there in a mile-high vase?A:Yo be real.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is the overhead of grantmaking?, published by MathiasKB on June 15, 2022 on The Effective Altruism Forum.

The effective altruism community is very fond of grants. This makes a lot of sense as it provides a way for our community to distribute its capital. The last FTX round had an extraordinary amount of applications. Only a small percentage of those were funded. For every funded application, dozens of hours were spent writing unsuccessful applications which would otherwise have been spent more productively.

This made me wonder. Is running all these grants actually a good use of time and resources? How would we even know?To clarify my thinking, I made a guesstimate model that lets you plug in estimates of relevant variables and see how the cost-effectiveness of a grant programme changes.

In this post I will:1. Briefly explain the model and my takeways from creating it 2. Conclude overhead is not as big a deal as I thought it was

About the model

The model is pretty simple. It assumes there are the following costs associated with grantmaking:

Money given out in grants

Time spent writing applications

Time spent evaluating applications

Time spent administering and advising grants

The value of each grant is its cost-effectiveness multiplied by the grant's size.

The more grants are approved, the less cost effective the average grant becomes.

Finally I divide the impact the approved grants with the the costs of running the grant programme, resulting in a measure of cost-effectiveness. If this cost-effectiveness comes out higher than the counterfactual, for example a standard givewell charity, then the grant programme is worth running.

You can view and play around with the model here:

Lessons on overhead to draw from the model

Can we learn anything from this model? That's debatable! Nevertheless, here are my own takeaways.

Overhead is not a big deal for large grants

As the average size and impact of grants increases, the overhead costs of grantmaking quickly start paling in comparison. The cost of giving 75 grants with an average size of $200,000, amounts to 15 million dollars given away. In turn, assuming my estimates weren't way off, the cost of writing and evaluating 1000 applications is only about 1 million. The size and number of grants given doesn't have to be very significant before the overhead of writing, evaluating, and administering grants becomes a negligible factor.

My initial worry that grants could be a near zero-sum game due to time spent on applications is not nearly as big a deal as I had anticipated.

Another takeaway from this is that grantmakers should spend evaluation time in proportion to grant size. The smaller the grant, the larger a percentage of the cost is taken up by overhead. For cheap grants, grantmakers should aim to make a decision quickly and move on.

Overhead might sometimes be a problem for smaller grant programmes

If you reduce the average size of the grants given out to $25k, overhead starts to look closer to a third of the total expenses.

In such circumstances, thinking about ways to reduce overhead may yield modest increases to the cost effectiveness of the grant programme.

Overhead is overshadowed by the expected value of each application

Much more important than overhead, is whether you expect the grants you're giving out to actually be impactful. Assumptions going into the the distribution of impact for grants, results in wild swings in the cost-effectiveness of the grant programme.

The expected impact of the average grant given out is determined by three factors:

The distribution in the expected value of applications if funded

Grantmakers ability to pick applicants from the top of this distribution

Percentage of applications they decide to fund.

Are each application equally likely to have a high impact? Surely not. How much bett...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A central AI alignment problem: capabilities generalization, and the sharp left turn, published by So8res on June 15, 2022 on The Effective Altruism Forum.

(This post was factored out of a larger post that I (Nate Soares) wrote, with help from Rob Bensinger, who also rearranged some pieces and added some text to smooth things out. I'm not terribly happy with it, but am posting it anyway (or, well, having Rob post it on my behalf while I travel) on the theory that it's better than nothing.)

I expect navigating the acute risk period to be tricky for our civilization, for a number of reasons. Success looks to me to require clearing a variety of technical, sociopolitical, and moral hurdles, and while in principle sufficient mastery of solutions to the technical problems might substitute for solutions to the sociopolitical and other problems, it nevertheless looks to me like we need a lot of things to go right.

Some sub-problems look harder to me than others. For instance, people are still regularly surprised when I tell them that I think the hard bits are much more technical than moral: it looks to me like figuring out how to aim an AGI at all is harder than figuring out where to aim it.

Within the list of technical obstacles, there are some that strike me as more central than others, like "figure out how to aim optimization". And a big reason why I'm currently fairly pessimistic about humanity's odds is that it seems to me like almost nobody is focusing on the technical challenges that seem most central and unavoidable to me.

Many people wrongly believe that I'm pessimistic because I think the alignment problem is extraordinarily difficult on a purely technical level. That's flatly false, and is pretty high up there on my list of least favorite misconceptions of my views.

I think the problem is a normal problem of mastering some scientific field, as humanity has done many times before. Maybe it's somewhat trickier, on account of (e.g.) intelligence being more complicated than, say, physics; maybe it's somewhat easier on account of how we have more introspective access to a working mind than we have to the low-level physical fields; but on the whole, I doubt it's all that qualitatively different than the sorts of summits humanity has surmounted before.

It's made trickier by the fact that we probably have to attain mastery of general intelligence before we spend a bunch of time working with general intelligences (on account of how we seem likely to kill ourselves by accident within a few years, once we have AGIs on hand, if no pivotal act occurs), but that alone is not enough to undermine my hope.

What undermines my hope is that nobody seems to be working on the hard bits, and I don't currently expect most people to become convinced that they need to solve those hard bits until it's too late.

Below, I'll attempt to sketch out what I mean by "the hard bits" of the alignment problem. Although these look hard, I’m a believer in the capacity of humanity to solve technical problems at this level of difficulty when we put our minds to it. My concern is that I currently don’t think the field is trying to solve this problem. My hope in writing this post is to better point at the problem, with a follow-on hope that this causes new researchers entering the field to attack what seem to me to be the central challenges head-on.

Discussion of a problem

On my model, one of the most central technical challenges of alignment—and one that every viable alignment plan will probably need to grapple with—is the issue that capabilities generalizes better than alignment.

My guess for how AI progress goes is that at some point, some team gets an AI that starts generalizing sufficiently well, sufficiently far outside of its training distribution, that it can gain mastery of fields like physics, bi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What are EA's biggest legible achievements in x-risk?, published by acylhalide on June 14, 2022 on The Effective Altruism Forum.

Disclaimer: This is a short post.

Was recently asked this on twitter and realised I didn't have good answers.

In x-risk I'll include: AI risk, biorisk and nuclear risk. I'll include both technical and governance work, and all the other forms of work that indirectly help produce these, and work of other forms that helps reduce x-risk. I'll also include work that reduces s-risk.

Answers I can think of are increased field-building and field-building capacity, (some) increased conceptual clarity of research, and increased capital available. None of which are very legible; there is a sense in which they feel like they're precursors to real work, than real work itself. I can't point to actual percentage point reductions in x-risk today that EA is responsible for. I can't point to projects that exist in the real world where a layman can look at them for ten seconds and realise they're significant and useful results. Whereas I can do the same for work in global health, for instance. And many non-EA movements also have achievement they can point at.

(This is not to discount the value of such work, or to claim EA is currently acting suboptimally. It is possible that the optimal thing to is spend years on illegible work, before obtaining legible results.)

Am I correct on my view of current achievements or is there something I'm missing? I would also love to be linked to other resources.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.