Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meta AI announces Cicero: Human-Level Diplomacy play (with dialogue), published by Jacy on November 22, 2022 on The Effective Altruism Forum.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Short Research Summary: Can insects feel pain? A review of the neural and behavioural evidence by Gibbons et al. 2022, published by Meghan Barrett on November 22, 2022 on The Effective Altruism Forum.

This short research summary briefly highlights the major results of a new publication on the scientific evidence for insect pain in Advances in Insect Physiology by Gibbons et al. (2022). This EA Forum post was prepared by Meghan Barrett, Lars Chittka, Andrew Crump, Matilda Gibbons, and Sajedeh Sarlak.

The 75-page publication summarizes over 350 scientific studies to assess the scientific evidence for pain across six orders of insects at, minimally, two developmental time points (juvenile, adult). In addition, the paper discusses the use and management of insects in farmed, wild, and research contexts. The publication in its entirety can be reviewed here. The original publication was authored by Matilda Gibbons, Andrew Crump, Meghan Barrett, Sajedeh Sarlak, Jonathan Birch, and Lars Chittka.

Major Takeaway

We find strong evidence for pain in adult insects of two orders (Blattodea: cockroaches and termites; Diptera: flies and mosquitoes). We find substantial evidence for pain in adult insects of three additional orders, as well as some juveniles. For several criteria, evidence was distributed across the insect phylogeny, providing some reason to believe that certain kinds of evidence for pain will be found in other taxa. Trillions of insects are directly impacted by humans each year (farmed, managed, killed, etc.). Significant welfare concerns have been identified as the result of human activities. Insect welfare is both completely unregulated and infrequently researched.

Given the evidence reviewed in Gibbons et al. (2022), insect welfare is both important and highly neglected.

Research Summary

The Birch et al. (2021) framework, which the UK government has applied to assess evidence for animal pain, uses eight neural and behavioral criteria to assess the likelihood for sentience in invertebrates: 1) nociception; 2) sensory integration; 3) integrated nociception; 4) analgesia; 5) motivational trade-offs; 6) flexible self-protection; 7) associative learning; and 8) analgesia preference.

Definitions of these criteria can be found on pages 4 & 5 of the publication's main text.

Gibbons et al. (2022) applies the framework to six orders of insects at, minimally, two developmental time points per order (juvenile, adult).

Insect orders assessed: Blattodea (cockroaches, termites), Coleoptera (beetles), Diptera (flies, mosquitoes), Hymenoptera (bees, ants, wasps, sawflies), Lepidoptera (butterflies, moths), Orthoptera (crickets, katydids, grasshoppers).

Adult Blattodea and Diptera meet 6/8 criteria to a high or very high level of confidence, constituting strong evidence for pain (see Table 1, below). This is stronger evidence for pain than Birch et al. (2021) found for decapod crustaceans (5/8), which are currently protected via the UK Animal Welfare (Sentience) Act 2022.

Adults of the remaining orders (except Coleoptera) and some juveniles (Blattodea, Diptera, and last juvenile stage Lepidoptera) satisfy 3 or 4 criteria, constituting substantial evidence for pain (see Tables 1 + 2).

We found no good evidence that any insect failed a criterion.

For several criteria, evidence was distributed across the insect phylogeny (Figure 1), including across the major split between the hemimetabolous (incomplete metamorphosis) and holometabolous (complete metamorphosis) insects. This provides some reason to believe that certain kinds of evidence for pain (e.g., integrated nociception in adults) will be found in other taxa.

Our review demonstrates that there are many areas of insect pain research that have been completely unexplored. Research gaps are particularly substantial for juveniles, hig...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Walkthrough of In-Context Learning and Induction Heads (w/ Charles Frye) Part 1 of 2, published by Neel Nanda on November 22, 2022 on The AI Alignment Forum.

New paper walkthrough: In-Context Learning and Induction Heads. This is the second paper in Anthropic's Transformer Circuits thread, a series of papers trying to reverse engineer transformer language models. I read through it with Charles Frye (from Full-Stack Deep Learning), and we discuss the paper, and give takes and intuitions. See the original paper and a Twitter thread of my paper takeaways

This is pitched so that it's hopefully accessible to people who haven't read the paper (very interested in feedback on this!), but I expect you to get more out of it if you understand transformers, and especially if you've read A Mathematical Framework. We only got partway through the paper, so there's a more in-the-weeds Part 2 in the works where we finish it off - let me know if you're interested in seeing it!

Disclaimer: I worked on this paper, along with Catherine Olsson, Nelson Elhage and Chris Olah, when I was at Anthropic, but I have since left and everything in this video is purely my own takes!

If you find this useful, check out my previous walkthroughs: A Mathematical Framework for Transformer Circuits and Interpretability in the Wild

And I'd be excited to see other researchers do these kinds of walkthroughs! The effort to usefulness ratio is way better than writing papers (and to my tastes, it's much more fun!)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Toby Ord's new report on lessons from the development of the atomic bomb, published by Ishan Mukherjee on November 22, 2022 on The Effective Altruism Forum.

Toby Ord has written a new report with GovAI on lessons from the development of the atomic bomb relevant to emerging technologies like artificial intelligence, synthetic biology, and nanotechnology.

The creation of the atomic bomb is one of the most famous and well-studied examples of developing a transformative technology — one that changes the shape of human affairs. There is much we don’t know about the future development of these technologies. This makes it much more difficult to reason about the strategic landscape that surrounds them. Which, in turn, makes it more difficult to help make sure the development is safe and beneficial for humanity. It is thus very useful to have a case study of developing a transformative technology.The making of the atomic bomb provides such a reference case. This report summarises the most important aspects of the development of atomic weapons and draws out a number of important insights for the development of similarly important technologies.One should treat the development of the atomic bomb not as a map to one’s destination, but as a detailed account of another traveller’s journey in a nearby land.

Something that provides valuable hints to important dangers or strategies we might not have considered, and which we neglect at our own peril.

Read the report here.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LessWrong readers are invited to apply to the Lurkshop, published by Jonas Vollmer on November 22, 2022 on LessWrong.

Summary: We’re running a free retreat for LessWrong users (lurkers) who aren’t yet working on existential risk mitigation but might be interested, in the hopes of catalyzing some of them into people who are. It’s called the Lurkshop. Apply here by December 1st, ideally sooner! If you refer a friend and we accept them, we’ll give you $100. We cover all costs, including flights. Please apply even if you don't think you have skills that seem directly relevant to reducing x-risk!

Do you read LessWrong?

(Presumably!)

Are you doing anything about the dire existential warnings you're hearing about?

(Probably not!)

If you want to be, we're offering an all-expenses paid workshop, Dec 16 (noon) to Dec 18 (10pm) in Berkeley, CA to come meet community members, talk with each other, and otherwise engage with the main plot of the world. If this goes well we might offer more programs like this, so please apply even if the dates don’t work well for you and we might invite you to those future programs!

What? Why?

LessWrong is one of the places in the world where words mean things and people think openly about the hardest problems facing the world, and we'd like to get to know the people who read it a bit better and help them make progress on the world’s problems.The rationality community mostly exists on online forums. Forums are great, but it’s really hard to replicate some important aspects of in-person conversations.

You can have different kinds of interactions and conversations in face-to-face interactions, which are useful for building trust, context, and deeper relationships

It’s helpful to be able to have conversations that aren't mostly in public. There's a lot that people want to talk about that's sensitive or personal or private

It’s helpful to be able to have long conversations with lots of context that are laser focused on you and your situation

You probably can't find a cofounder just by writing LessWrong comments.

Forums make the community feel remote and actions feel impossible

The rationality community is not some distant mythical group—it’s a bunch of people!

Many people quite like you are doing big ambitious risky projects, and seeing that helps people realize they can do those kinds of projects too

There really isn’t someone on top of all the important stuff. Come see for yourself how disorganized and incompetent everyone is!

Notably, this retreat probably won’t be a great way to

Learn a bunch of deep rationality stuff (or to learn other stuff)

We might have some cool talks, but mostly for tone setting, etc.

Get you plugged into some kind of exclusive in-group

Purely have fun and chill

Look, we’re not against fun

But the organizers have a mission, and it is to lash you together into an engine of grit and spite, with which we might kill the machine god.

Um, so what, concretely, will happen?

We will likely run some of the following:

Sessions on How To Actually Do Things

How do you go from an idea for a project, to getting funding and setting up a corporation? Which sounds hard, but isn’t?

And then how do you make the thing actually happen, which sounds easy, but isn’t?

LessWrong post writing day

Co-founder speed-dating

Someone Talks You Through How You Could Maybe Do More Useful Things With Your Life

Lightning talks

One-on-ones

Attending winter solstice

Who should apply?

If you like the kinds of ideas explored on LessWrong, you should apply. It’s ok if you haven’t read a ton of the material yet, though we suspect this workshop will be most useful for those who have read many of the Sequences already. What we’re looking for in the application is mostly intellectual vibrancy, agency, and the ability to build your own models of the world, more than famil...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Utilitarianism Meets Egalitarianism, published by Scott Garrabrant on November 21, 2022 on LessWrong.

This post is mostly propaganda for the Nash Bargaining solution, but also sets up some useful philosophical orientation. This post is also the first post in my geometric rationality sequence.

Utilitarianism

Let's pretend that you are a utilitarian. You want to satisfy everyone's goals, and so you go behind the veil of ignorance. You forget who you are. Now, you could be anybody. You now want to maximize expected expected utility. The outer (first) expectation is over your uncertainty about who you are. The inner (second) expectation is over your uncertainty about the world, as well as any probabilities that comes from you choosing to include randomness in your action.

There is a problem. Actually, there are two problems, but they disguise themselves as one problem. The first problem is that it is not clear where you should get your distribution over your identity from. It does not make sense to just take the uniform distribution; there are many people you can be, and they exist to different extents, especially if you include potential future people whose existences are uncertain.

The second problem is that interpersonal utility comparisons don't make sense. Utility functions are not a real thing. Instead, there are preferences over uncertain worlds. If a person's preferences satisfy the VNM axioms, then we can treat that person as having a utility function, but the real thing is more like their preference ordering. When we get utility functions this way, they are only defined up to affine transformation. If you add a constant to a utility function, or multiply a utility function by a positive constant, you get the same preferences. Before you can talk about maximizing the expectation over your uncertainty about who you are, you need to put all the different possible utility functions into comparable units. This involves making a two dimensional choice. You have to choose a zero point for each person, together with a scaling factor for how much their utility goes up as their preferences are satisfied.

Luckily, to implement the procedure of maximizing expected expected utility, you don't actually need to know the zero points, since these only shift expected expected utility by a constant. You do, however need to know the scaling factors. This is not an easy task. You cannot just say something like "Make all the scaling factors 1." You don't actually start with utility functions, you start with equivalence classes of utility functions.

Thus, to implement utilitarianism, we need to know two things: What is the distribution on people, and how do you scale each person's utilities? This gets disguised as one problem, since the thing you do with these numbers is just multiply them together to get a single weight, but it is actually two things you need to decide. What can we do?

Egalitarianism

Now, let's pretend you are an egalitarian. You still want to satisfy everyone's goals, and so you go behind the veil of ignorance, and forget who you are. The difference is that now you are not trying to maximize expected expected utility, and instead are trying to maximize worst-case expected utility. Again, the expectation contains uncertainty about the world as well as any randomness in your action. The "worst-case" part is about your uncertainty about who you are. You would like to have reasonably high expected utility, regardless of who you might be.

When I say maximize worst-case expected utility, I am sweeping some details under the rug about what to do if you manage to max out someone's utility. The actual proposal is to maximize the minimum utility over all people. Then if there are multiple ways to do this, consider the set of all people for which it is still possible to incre...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A socialist view on liberal progressive criticisms of EA, published by freedomandutility on November 21, 2022 on The Effective Altruism Forum.

As someone who identifies as a socialist, politically to the left of many progressive critics of EA, I wanted to outline where I disagree with some liberal progressives criticisms of EA.

Obviously, I am just one socialist and my views probably differ a fair amount from the average socialist.

My view of "liberal progressive criticisms of EA" is mostly based on tweets, but I think this article is a good example of what I categorise as a liberal progressive criticism of EA:/.

Explicit and Implicit Prioritisation

Some liberal progressive criticisms of EA treat opportunity costs and prioritisation as things that EA has created which unfairly pit good causes against one another.

All political movements have limited resources and must prioritise certain issues. Often, individuals aligned with political movements do not realise that they are prioritising certain issues over others, or do not want to recognise this reality, because deeming some good causes less worthy than other good causes is deeply emotionally uncomfortable for most people.

EAs are less uncomfortable with this because they are used to prioritisation, and prioritise between causes explicitly rather than implicitly.

I think many liberal progressives focused on issues like student loan in Western countries do not recognise, or do not want to recognise, that they are prioritising this issue over vaccinating the world's poorest children against disease. However, EAs prioritising existential risks generally explicitly recognise and state that they are prioritising this over global health.

This leads some critics to a double standard of criticising those who explicitly prioritise issues over global health in favour of those who implicitly prioritise other issues over global health, despite the impacts on global health being the same.

Scope Insensitivity

EAs are generally aware of scope insensitivity.

If asked to compare issues, it is likely that liberal progressives would at least agree that global health should in theory be prioritised over issues such as student loans or better healthcare for poorer Westerners. However, liberal progressives will likely still not prioritise pandemic preparedness and global health sufficiently because of scope insensitivity and failures to understand the gulf in impact between initiatives focused on poorer Westerners vs the global poor.

Differences in Priorities and Neglectedness

Liberal progressivism is a much more successful ideology / movement than EA, in terms of reach and influence. EA claims to do the most good, but adherents of many ideologies and movements expect that they are doing the most good too, and are likely surprised when EA's cause priorities do not reflect their own. They may also simply critique EA out of disagreement between their priorities and EA's priorities.

But EA priorities differing from the priorities of mainstream political movements is a systematic feature of EA, because EA focuses on neglected issues. If liberal progressivism or conservatism sufficiently prioritised AI safety, pandemic preparedness, global health or factory farming, there is a good chance that EA would switch to new priorities.

EA does not focus on the "most important" issues - it focuses on the issues where additional time or money will have the largest social impact.

Localism, Internationalism and Xenophobia

My view of liberal progressivism is that it largely embraces localism with a preference for local actors solving local problems using local resources.

However, the world is very unequal. The areas with the best resources and the most skilled actors to deploy them are also the areas with the least severe problems. The areas with the most...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Review: What We Owe The Future, published by Kelsey Piper on November 21, 2022 on The Effective Altruism Forum.

For the inaugural edition of Asterisk, I wrote about What We Owe The Future. Some highlights:

What is the longtermist worldview? First — that humanity’s potential future is vast beyond comprehension, that trillions of lives may lie ahead of us, and that we should try to secure and shape that future if possible.

Here there’s little disagreement among effective altruists. The catch is the qualifier: “if possible.” When I talk to people working on cash transfers or clean water or accelerating vaccine timelines, their reason for prioritizing those projects over long-term-future ones is approximately never “because future people aren’t of moral importance”; it’s usually “because I don’t think we can predictably affect the lives of future people in the desired direction.”

As it happens, I think we can — but not through the pathways outlined in What We Owe the Future.

The stakes are as high as MacAskill says — but when you start trying to figure out what to do about it, you end up face-to-face with problems that are deeply unclear and solutions that are deeply technical.

I think we’re in a dangerous world, one with perils ahead for which we’re not at all prepared, one where we’re likely to make an irrecoverable mistake and all die. Most of the obligation I feel toward the future is an obligation to not screw up so badly that it never exists. Most longtermists are scared, and the absence of that sentiment from What We Owe the Future feels glaring.

If we grant MacAskill’s premise that values change matters, though, the value I would want to impart is this one: an appetite for these details, however tedious they may seem.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pre-Announcing the 2023 Open Philanthropy AI Worldviews Contest, published by Jason Schukraft on November 21, 2022 on The Effective Altruism Forum.

At Open Philanthropy we believe that future developments in AI could be extremely important, but the timing, pathways, and implications of those developments are uncertain. We want to continually test our arguments about AI and work to surface new considerations that could inform our thinking.

We were pleased when the Future Fund announced a competition earlier this year to challenge their fundamental assumptions about AI. We believe this sort of openness to criticism is good for the AI, longtermist, and EA communities. Given recent developments, it seems likely that competition is no longer moving forward.

We recognize that many people have already invested significant time and thought into their contest entries. We don’t want that effort to be wasted, and we want to incentivize further work in the same vein. For these reasons, Open Phil will run its own AI Worldviews Contest in early 2023.

To be clear, this is a new contest, not a continuation of the Future Fund competition. There will be substantial differences, including:

A smaller overall prize pool

A different panel of judges

Changes to the operationalization of winning entries

The spirit and purpose of the two competitions, however, remains the same. We expect it will be easy to adapt Future Fund submissions for the Open Phil contest.

More details will be published when we formally announce the competition in early 2023. We are releasing this post now to try to alleviate some of the fear, uncertainty, and doubt surrounding the old Future Fund competition and also to capture some of the value that has already been generated by the Future Fund competition before it dissipates.

We are still figuring out the logistics of the competition, and as such we are not yet in a position to answer many concrete questions (e.g., about deadlines or prize amounts). Nonetheless, if you have questions about the contest you think we might be able to answer, you can leave them as comments below, and we will do our best to answer them over the next few weeks.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Hebbian Natural Abstractions] Introduction, published by Samuel Nellessen on November 21, 2022 on LessWrong.

With this sequence, we (Sam + Jan) want to provide a principled derivation of the natural abstractions hypothesis (which we will introduce in-depth in later posts) by motivating it with insights from computational neuroscience.

Goals for this sequence are:

show why we expect natural abstractions to emerge in biological brains,

provide empirical evidence and a mechanistic explanation for the emergence of natural abstractions in biology, and

spelling out implications of the emergence of natural abstractions in biology.

Author’s note: This is currently my (Sam’s) main research project, but my first nonetheless. Happy to receive any feedback! Some of the original ideas and guidance come from Jan. I don’t expect you, the reader, to have solid background knowledge in any of the discussed topics. So, whenever you get lost, I will try to get you back on board by providing a more high-level summary of what I said.

Why are ‘abstractions’ relevant?

As alignment researchers, the higher-level problem we are trying to solve is: ‘How do we teach an AI what we value?’. To simplify the question, we assume that we already know what we value. Now, we’ve got to teach an AI that we value “things out there in the world”, e.g. trees. Specifying “trees” should be easy, right?

. o no .

You: Yea, so. we value trees. These for example. gestures wildly AGI: Of course! So you only value these trees in your backyard? Okay, I will just quickly measure the constellation of molecules in— You: No, no, no! Stop, that’s not what I meant! Not only those, but you know. the idea behind the trees? I also value trees that aren’t present here and certainly not their constituting molecules. Look, let’s go over to my neighbors backyard. These are trees too. AGI: Huh, this tree seems to be made out of papier-mâché. You: Ahhh uups. Yea, humans are fallible. Don’t treat my best guess estimations as unfailing sources of truth. Please note that I care about the territory - not just the map in my head that might be mistaken. AGI: Wh- What? How should I know about the relation between your map and the territory?

In a similar vein, John Wentworth spells out the pointers problem: ‘An AI should optimize for the real-world things I value, not just my estimates of those things’. He formalises the problem as follows:

“What functions of what variables (if any) in the environment and/or another world model correspond to the latent variables in the agent’s world model”.

Let’s apply JW’s definition to the trees example from above.

The concept of a ‘tree’ (that’s what JW calls a ‘latent variable’) is not directly observable and is only clear in your head.

To make the concept observable, you have to specify the correct relationship between the concept in your head and the observable objects in the world (that’s what JW calls ‘functions of [.] variables (if any) in the environment and/or another world model’).

Eventually, you want to transfer something (the concept of a tree) in your map to the AGI’s map (that’s what JW calls ‘agents world model’).

There is also an analogous view on the pointers problem from classic philosophy, called the ‘problem of universals’. Similar to our story, we face issues when pointing towards the ‘universal of a tree’. We will explain the problem of universals using an example:

Let’s consider the thought process of the divine mind when creating trees. Trees come in very different shapes or differing branching patterns. Now, who imposed on them that they are ‘trees’? Did the divine mind already have a category in mind and created trees from an existing idea of trees, universalia ante rem (‘universals before the thing’)? Is the idea of a tree realized in trees, universalia in re (’universals in ...