Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Linkpost: Dwarkesh Patel interviewing Carl Shulman, published by Stefan Schubert on June 14, 2023 on The Effective Altruism Forum.

We ended up talking for 8 hours, so I'm splitting this episode into 2 parts.

This part is about Carl’s model of an intelligence explosion, which integrates everything from:

how fast algorithmic progress & hardware improvements in AI are happening,

what primate evolution suggests about the scaling hypothesis,

how soon before AIs could do large parts of AI research themselves, and whether there would be faster and faster doublings of AI researchers,

how quickly robots produced from existing factories could take over the economy.

We also discuss the odds of a takeover based on whether the AI is aligned before the intelligence explosion happens, and Carl explains why he’s more optimistic than Eliezer.

The next part, which I’ll release next week, is about all the specific mechanisms of an AI takeover, plus a whole bunch of other galaxy brain stuff.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Common Sense View on Meat Implies Some Pretty Radical Conclusions, published by Omnizoid on June 14, 2023 on The Effective Altruism Forum.

Crosspost of this from my Substack.

Somewhere, right now, a cat is probably being harmed. Some psychopathic child is stomping on it, or beating it, or burying it alive, or dousing it in gasoline before he sets it on fire. This is, sadly, not uncommon. And almost everyone agrees that it’s bad.

Almost everyone agrees that, though animals don’t have the same rights as people, we should not hurt them for trivial reasons. Even if this child and a few friends are having a good time hurting a cat, this is still immoral. And it’s wrong not just because of what it does to the character of the boys—it’s partly wrong because of what is being done to the cat. Torturing a cat is worse than torturing a robot that one believes to be a cat, even though they’d have the same effects on one’s character because one causes actual torment to an animal and the other does not.

The common sense view around animals seems to be “it’s okay to eat them, but we should try our hardest not to mistreat them.” The philosophical merits of such a position can be debated, but this seems like something that almost everyone agrees with. But if this is true, then common sense condemns our current ghastly mistreatment of animals.

People act like the vegan position that our current meat eating is seriously wrong is a radical position—a position that requires believing extreme views. But it’s not—it’s about as moderate as you can get.

It’s not radical to be opposed to eating eggs from chickens that were stuffed in a cage, covered in the falling, acidic feces of those above them, that burns their flesh and enters their nose, making it impossible to breathe and making their eyes constantly burn. It’s not radical to not want to buy eggs from chickens stuffed in a tiny wire cage that causes them to develop foot conditions, constantly rubbing against sharp metal, too small for them to ever be able to turn around or lie down comfortably. It’s not radical not to want to buy eggs from suppliers when every second of every day, there are at least 6 million hens being systematically starved, because there’s a way to trick the bodies of the hens into thinking it’s egg-laying season by starving them, resulting in more eggs being laid. When this is the way that 5-10% of hens die, it’s not radical not to want to pay to exacerbate their torment.

When the egg industry grinds up billions of bouncy baby male chicks, just because they can’t lay eggs, it isn’t radical to say that we shouldn’t be paying them to grind up more. When the conditions are so bad that the hens go crazy and throw themselves against the sides of the cages, every natural behavior thwarted, everything that might bring them joy snuffed out in the dark, filthy, disgusting juggernauts of death, torment, and despair, it’s not radical to not want to fund that.

It’s not radical to think that it’s wrong to buy chickens when they have been artificially engineered to be in constant pain—their entire bodies twisted and warped into maximally efficient machines for generating flesh. When thousands of chickens are stuffed into crates in transport every hour in extreme weather, killing 5-10% of them, it’s not wrong to say that we won’t pay for that until they stop their systematic abuse. Chickens live in windowless sheds, constantly sleep-deprived from artificial lighting. When animals have their beaks, tails, and testicles cut off with a sharp knife, with no anesthetic, when they’re given third-degree burns because it makes it easier to identify them, it isn’t wrong to refuse to pay for that until the people systematically torturing them can get their act together and stop the torture.

And yes, it is torture. This is not hyperbole...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We are Hiring: The Knowledge & Research Team @ Power for Democracies, published by Stephan Schwahlen on June 14, 2023 on The Effective Altruism Forum.

Power for Democracies is a new non-profit democracy charity evaluator based in Berlin, Germany, and operating globally. It was founded by Markus N. Beeko, currently secretary general of Amnesty International (Germany), together with Stephan Schwahlen and Stefan Shaw, founders of legacies.now, an EA-aligned philanthropic advisory in Munich, as well as co-founders (Stefan) and advisors (Stephan) at effektiv-spenden.org, the German language EA-aligned donation platform based in Berlin.Power for Democracies is looking to hire 5-6 democracy enthusiasts to form their ‘Knowledge & Research Team’. The objective of the team is twofold: To build and execute a ‘knowledge building roadmap’ that will lead to a growing set of methodologies for identifying highly effective pro-democracy interventions and promising NGOs to apply them. And to use these methodologies to generate giving recommendations for the international community of democracy-focused, effectiveness-driven donors.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA organizations should have a transparent scope, published by Joey on June 14, 2023 on The Effective Altruism Forum.

Executive summary

One of the biggest challenges of being in a community that really cares about counterfactuals is knowing where the most important gaps are and which areas are already effectively covered. This can be even more complex with meta organizations and funders that often have broad scopes that change over time. However, I think it is really important for every meta organization to clearly establish what they cover and thus where these gaps are; there is a substantial negative flowthrough effect when a community thinks an area is covered when it is not.

Why this matters

The topic of having a transparent scope recently came up at a conference as one of the top concerns with many EA meta orgs. Some negative effects that have been felt by the community are in large part due to unclear scopes, including:

Organizations leaving a space thinking it's covered when it's not.

Funders reducing funding in an area due to an assumption that someone else is covering it when there are still major gaps.

Two organizations working on the same thing without knowledge of each other, due to both having a broad mandate, but simultaneously putting resources into an overlapping subcomponent of this mandate.

Talent being turned off or feeling misled by EA when they think an org misportrays itself.

Talent ‘dropping out of the funnel’ when they go to what they believe is the primary organization covering an area and finding that what they care about isn’t covered, due to the organization claiming too broad a mandate.

There can also be a significant amount of general frustration caused when people think an organization will cover, or is covering, an area and then an organization fails to deliver (often on something they did not even plan on doing).

What do I mean when I say that organizations should have a transparent scope:

Broadly, I mean organizations being publicly clear and specific about what they are planning to cover both in terms of action and cause area.

In a relevant timeframe: I think this is most important in the short term (e.g., there is a ton of value in an organization saying what they are going to cover over the next 12 months, and what they have covered over the last months).

For the most important questions: This clarity needs to both be in priorities (e.g., cause prioritization) and planned actions (e.g., working with student chapters). This can include things the organization might like or think is impactful to do but are not doing due to capacity constraints or its current strategic direction.

For the areas most likely for people to confuse: It is particularly important to provide clarity about things that people think one might be doing (for example, Charity Entrepreneurship probably doesn’t need to clarify that it doesn’t sell flowers, but should really be transparent over whether it plans to incubate projects in a certain cause area or not).

How to do this

When I have talked to organizations about this, I sometimes think that the “perfect” becomes the enemy of the good and they do not want to share a scope that is not set in stone. All prioritizations can change, and it can sometimes even be hard internally to have a sense of where the majority of your resources are going. However, given the importance of counterfactuals and the number of aspects that can help proxy these factors, I think a pretty solid template can be created. Given that CE is also often asked this question I made a quick template below that I think gives a lot of transparency if answered clearly and can give people a pretty clear sense of an organization's focus. It's worth noting that what I am suggesting is more about clarity rather than justification. While an org can choo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Aptitudes for AI governance work, published by Sam Clarke on June 14, 2023 on The Effective Altruism Forum.

I outline 8 “aptitudes” for AI governance work. For each, I give examples of existing work that draws on the aptitude, and a more detailed breakdown of the skills I think are useful for excelling at the aptitude.

How this might be helpful:

For orienting to the kinds of work you might be best suited to

For thinking through your skill gaps for those kinds of work

Offering an abstraction which might help those thinking about field-building/talent pipeline strategy

Epistemic status:

I've spent ~3 years doing full-time AI governance work. Of that, I spent ~6 months FTE working on questions related to the AI governance talent pipeline, with GovAI.

My work has mostly been fairly foundational research—so my views about aptitudes for research-y work (i.e. the first four aptitudes in this post) are more confident than for more applied or practical work (i.e. the latter three aptitudes in this post).

I've spent ~5 hours talking with people hiring in AI governance about the talent needs they have. See this post for a write-up of that work. I've spent many more hours talking with AI governance researchers about their work (not focused specifically on talent needs).

This post should be read as just one framework that might help you orient to AI governance work, rather than as making strong claims about which skills are most useful.

Some AI governance-relevant aptitudes

Macrostrategy

What this is: investigating foundational topics that bear on more applied or concrete AI governance questions. Some key characteristics of this kind of work include:

The questions are often not neatly scoped, such that generating or clarifying questions is part of the work.

It involves balancing an unusually wide or open-ended range of considerations.

A high level of abstraction is involved in reasoning.

The methodology is often not very clear, such that you can’t just plug-and-play with some standard methodology from a particular field.

Examples:

Descriptive work on estimating certain ‘key variables’

E.g. reports on AI timelines and takeoff speeds.

Prescriptive work on what ‘intermediate goals’ to aim for

E.g. analysis of the impact of US govt 2022 export controls.

Conceptual work on developing frameworks, taxonomies, models, etc. that could be useful for structuring future analysis

E.g. The Vulnerable World Hypothesis.

Useful skills:

Generating, structuring, and weighing considerations. Being able to generate lots of different considerations for a given question and weigh up these considerations appropriately.

For example, there are a lot of considerations that bear on the question “Would it reduce AI risk if the US government enacted antitrust regulation that prevents big tech companies from buying AI startups?”

Some examples of considerations are: “How much could this accelerate or slow down AI progress?”, “How much could this increase or decrease Western AI leadership relative to China?”, “How much harder or easier would this make it for the US government to enact safety-focused regulations?” “How would this affect the likelihood that a given company (e.g., Alphabet) plays a leading role in transformative AI development?” etc.

Each of these considerations is also linked to various other considerations. For instance, the consideration about the pace of AI progress links to the higher-level consideration “How does the pace of AI progress affect the level of AI risk?” and the lower-level consideration “How does market structure affect the pace of AI progress?” That lower-level consideration can then be linked to even lower levels, like “What are the respective roles of compute-scaling and new ideas in driving AI progress?” and “Would spreading researchers out across a larger number of startups ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contingency: A Conceptual Tool from Evolutionary Biology for Alignment, published by clem acs on June 12, 2023 on LessWrong.

1. Motivation

As AI systems become increasingly powerful, the chance of them developing dangerous features becomes increasingly likely. A key concern of AI alignment researchers is that there are dangerous features, such as power-seeking or situational awareness, which are convergent. This would mean that these capabilities would likely arise regardless of the particular trajectory AI development took. In a recent tweet, Eliezer Yudkowsky argues that this is the key question for AI alignment research:

> From my perspective, we are trying to evaluate a picture of convergent ruin rather than any pictures of contingent ruin. That is, I'm relatively less interested in contingent-doom scenarios like "What if superintelligence is easily alignable, but Greenpeace is first to obtain superintelligence and they align It entirely to protect Earth's environment and then the SI wipes out humanity to do that?" because this is not a case for almost-all paths leading to ruin. I worry that contingent scenarios like "Age of Ultron" can fascinate humans (particularly contingent scenarios where some Other Monkey Faction Gets AGI First oh noes), and derail all possibility of a discussion about whether, in fact, the superintelligence kills everyone regardless of who 'gets' It first.

> Similarly, if somebody is considering only the contingent-early-ruin scenario of "a prototype of a somewhat advanced AGI starts to consider how to wipe out humanity, and nobody spots any signs of that thought", that contingency can mask a convergent-ruin continuation of "even if they spot the early signs, everyone is still in an arms-race scenario, so they optimize away the visible signs of doom and keep going into ruin".

> So I think the key question is "Why would or wouldn't humanity be convergently doomed?" rather than "How might humanity be contingently doomed?" Convergent doom isn't the same as inevitable doom, but ought to be the sort of doom that defeats easy solution / historically characteristic levels of competence for Earth / the sort of efforts we've seen already.

> A doom that can only be averted by an effective multinational coalition shutting down all AI runs larger than GPT-4 for 6 years, to make time for a crash project on human intelligence augmentation that has to complete within 6 years, may not be an inevitable doom; but it was probably a pretty convergent doom if that was your best path for stopping it.

> Conversely, "What if all the AI companies and venture capitalists just decide together to not pursue the rewards of AI models stronger than GPT-4, and high-impact journals and conferences stop accepting papers about it" is a sufficiently unlikely contingency that dooms avertable in that way may still be said to be convergent. It seems reasonably convergent that, if there isn't a drastic effective global shutdown enforced by state-level actors that will enforce that treaty even on non-signatory states, somebody tries.

This framing introduces “contingency” as the opposite of convergence - features of systems which are convergent are, by definition, those which are not contingent. In this piece I suggest an alternative framing, in which convergent features are themselves often contingent on some other feature. This is an initially unintuitive framing which, once you’ve seen it once, comes up everywhere. In the case of alignment, it would allow us to take some dangerous feature of AI systems which we expect to be convergent, and ask “what is the feature contingent on?”. My hope is that this shift in perspective can reveal implicit assumptions and thus allow us to spot new potential solutions for AI alignment.

The field where this shift in perspective is most salient ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Multiple stages of fallacy - justifications and non-justifications for the multiple stage fallacy, published by AronT on June 13, 2023 on LessWrong.

[Epistemic status: I am reasonably confident (75%) that these claims are largely correct.]

To estimate the probability of something happening, e.g. AI takeover, one approach is to break this down into stages and multiply the (conditional) probabilities together. I claim this approach yields systematically low answers. This is often referred to as the multiple stage fallacy - however - the term ‘fallacy’ in its label shouldn’t be viewed as an indisputable verdict. Smart people disagree with whether it truly is a fallacy, and it is the default method for even very good forecasters - so what’s going on? I haven’t been able to find a source that does this justice, and my impression is that consensus opinion on this overestimates the importance of disjunctive alternatives whilst underestimating the importance of correlated conditional probabilities. On a meta/ Bayesian level, I’ve seen it claimed several times that this is a fallacy because, empirically, it produces systematically low probabilities, but haven’t seen good evidence of this.

Disjunctive alternatives

What I see cited most often is that multiplying conditional probabilities fails to account for disjunctive routes. If x1,x2,x3 are events, and you are trying to estimate P(x3), then P(x3|x2)P(x2|x1)P(x1) doesn’t give the probability P(x3), it gives the probability P(x1∧x2∧x3).

If you want to compute the probability P(x3), you have to sum over all 4 possible combinations of x2 and x3 being true/ false. As the number of events grows, the number of combinations grows exponentially. In theory, this sounds like a problem, but in practice I believe the effect is smaller than claimed. In most situations where this fallacy appears to apply, I think it’s often reasonable to claim that the conclusion does actually imply the premises with reasonably high probability.

To take a concrete example, consider Carlsmith’s report on the risk of AI takeover. He lists 5 premises:

Timelines are short enough to build AGI by 2070.

There will be sufficient incentives to build AGI.

Alignment will be hard.

Systems will be deployed in ways that expose them to inputs that lead to them trying to seek power.

This will lead to AI takeover.

He computes the conditional probabilities of each premise, and then finally a conditional probability of an existential catastrophe conditional on these premises occurring.

In theory, there are 25=32 disjunctive routes to existential catastrophe here, but in practice I believe it’s reasonable to place fairly small probability mass on routes to catastrophe where the premises do not all occur.

Conditional probabilities are more correlated than you expect

Where the real fallacy lies is in incorrectly estimating conditional probabilities. The central claim is that in almost all cases, you should expect to underestimate how correlated the conditional probabilities you assign to each stage are.

As an example, consider the sample correlation matrix of the logits assigned to each stage by each of the Carlsmith report reviewers:

[Of the 15 nontrivial coefficients, 13 are positive. What is the probability of this occurring by chance? If X∼Bin(15,0.5), then P(X≤2)=0.4% which is significant at the 5% significance level. How much should you trust this? Obviously, almost not at all, but it does illustrate my point.]

There are two ways through which this happens, which roughly correspond to a failure of inside and outside views respectively:

Inside view failures

When we make predictions about the world, our predictions stem from an interplay of heuristics, biases, instincts, and in some cases formal models. These heuristics will often influence predictions of various seemingly ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lightcone Infrastructure is looking for funding, published by habryka on June 14, 2023 on LessWrong.

Lightcone Infrastructure is looking for funding and are working on the following projects:

We run LessWrong, the AI Alignment Forum, and have written a lot of the code behind the Effective Altruism Forum.

During 2022 and early 2023 we ran the Lightcone Offices, and are now building out a campus at the Rose Garden Inn in Berkeley, where we've been doing repairs and renovations for the past few months.

We've also been substantially involved in the Survival and Flourishing Fund's S-Process (having written the app that runs the process) and are now running Lightspeed Grants.

We also pursue a wide range of other smaller projects in the space of "community infrastructure" and "community crisis management". This includes running events, investigating harm caused by community institutions and actors, supporting programs like SERI MATS, and maintaining various small pieces of software infrastructure.

If you are interested in funding us, please shoot me an email at [email protected] (or if you want to give smaller amounts, you can donate directly via PayPal here).

Funding is quite tight since the collapse of FTX, and I do think we work on projects that have a decent chance of reducing existential risk and generally making humanity's future go a lot better, though this kind of stuff sure is hard to tell. We are looking to raise around $3M to $6M for our operations in the next 12 months.

Also feel free to ask any questions in the comments.

Two draft readers of this post expressed confusion that Lightcone needs money, given that we just announced a funding process that is promising to give away $5M in the next two months. The answer to that is that we do not own the money moved via Lightspeed Grants and are only providing grant recommendations to Jaan Tallinn and other funders.

We do separately apply for funding from the Survival and Flourishing Fund, through which Jaan has been our second biggest funder. We also continue to actively fundraise from both SFF and Open Philanthropy (our largest funder).

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: MetaAI: less is less for alignment., published by Cleo Nardo on June 13, 2023 on LessWrong.

Summary

In May 2023, MetaAI submitted a paper to arxiv called LIMA: Less Is More for Alignment. It's a pretty bad paper and (in my opinion) straightforwardly misleading. Let's get into it.

The Superficial Alignment Hypothesis

The authors present an interesting hypothesis about LLMs

We define the Superficial Alignment Hypothesis: A model’s knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used when interacting with users.

If this hypothesis is correct, and alignment is largely about learning style, then a corollary of the Superficial Alignment Hypothesis is that one could sufficiently tune a pretrained language model with a rather small set of examples.

We hypothesize that alignment can be a simple process where the model learns the style or format for interacting with users, to expose the knowledge and capabilities that were already acquired during pretraining.

(1) This hypothesis would have profound implications for AI x-risk

It suggests that we could build a safe competent oracle by pretraining an LLM on the entire internet corpus, and then finetuning the LLM on a curated dataset of safe competent responses.

It suggests that we could build an alignment researcher by pretraining an LLM on the entire internet corpus, and then finetuning the LLM on a curated dataset of alignment research.

(2) Moreover, as by Ulisse Mini writes in their review of the LIMA paper,

Along with TinyStories and QLoRA I'm becoming increasingly convinced that data quality is all you need, definitely seems to be the case for finetuning, and may be the case for base-model training as well. Better scaling laws through higher-quality corpus? Also for who haven't updated, it seems very likely that GPT-4 equivalents will be essentially free to self-host and tune within a year. Plan for this!

(3) Finally, the hypothesis would've supported many of the intuitions in the Simulators sequence by Janus, and I share these intuitions.

So I was pretty excited to read the paper! Unfortunately, the LIMA results were unimpressive upon inspection.

MetaAI's experiment

The authors finetune MetaAI's 65B parameter LLaMa language model on 1000 curated prompts and responses (mostly from StackExchange, wikiHow, and Reddit), and then compare it to five other LLMs (Alpaca 65B, DaVinci003, Bard, Claude, GPT4).

Method:

To compare LIMA to other models, we generate a single response for each test prompt. We then ask crowd workers to compare LIMA outputs to each of the baselines and label which one they prefer. We repeat this experiment, replacing human crowd workers with GPT-4, finding similar agreement levels.

Results:

In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback.

Conclusion:

The fact that simple fine-tuning over so few examples is enough to compete with the state of the art strongly supports the Superficial Alignment Hypothesis, as it demonstrates the power of pretraining and its relative importance over large-scale instruction tuning and reinforcement learning approaches.

Problems with their experiment

(1) Human evaluators

To compare two chatbots A and B, you could ask humans whether they prefer A's response to B's response across 300 test prompts. But this is pretty bad proxy, because here's what users actually care about:

What's the chatbots' accuracy on benchmark tests, e.g. BigBench, MMLU?

Can the chatbot pass a law exam, or a medical exam?

Can the chatbot write Python code that actually matches the specification?

Can the chatbot perform worthwhi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My guess for why I was wrong about US housing, published by romeostevensit on June 14, 2023 on LessWrong.

A few years ago I wrote a post highlighting what I felt were underappreciated considerations in home buying, most of which cut against the standard advice of homes as a major part of your investment portfolio. It garnered a lot of argument, most of which, I felt, boiled down to reference class forecasting over which mean housing would regress to for price trajectory: a more current one, or a more long term one.

Since then, housing has gone up 60+%. And the reason I think I got the reference class wrong is that I missed something very important about the US housing market. Since discovering it I find most people I mention it to are surprised by it.

I was very surprised that the US had the second lowest house price to income ratio in the world. It's less surprising when you consider that the income side of the equation is quite high, but still. Since then, the US has started dropping down the list and is currently in 6th. But this means hundreds of other countries have more expensive houses relative to local incomes. I was so busy looking at reference classes that are local price trends that I missed the major reference class of the US having started out as an outlier.

Were the US to continue to regress towards the world median of around 12, prices would need to appreciate another 2-2.5x from where they are at currently, which would put the median house at $900k-1.2MM. I failed to appreciate that most of the world has a much sharper property owner class divide and it wouldn't be all that surprising to see a relatively younger nation move in that direction over time. This also renders it much less surprising that investment firms are buying huge swathes of housing stock, even at the new higher rates.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.