Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Operations Slack, published by AmritSidhu-Brar on October 25, 2022 on The Effective Altruism Forum.

The EA Operations Slack workspace is an online community of people doing operations work at effective altruism aligned organisations. The purpose of the space is to facilitate the sharing of operations-related insights, best practices and advice, and thereby improve the quality of operations work done on impactful projects. Discussion covers areas including HR, recruitment, accounting, event planning, personal assistance and office management. The space was founded around five years ago, and as of October 2022 has around 550 members.

We’re writing this post as an invitation: if you work on operations in an EA-aligned organisation, you’re very welcome to apply to be a member of the space! You can read our membership criteria here and, if you’re eligible to join, apply using this application form.

If you have questions, the current active admins of the space are Amrit Sidhu-Brar and Sawyer Bernath. You can contact us at [email protected].

Membership criteria

The workspace has somewhat restrictive membership criteria, with the aim of fostering a high-trust environment where operations staff can share questions openly.

You can read our full criteria here. In short, we accept anyone engaged in paid work for an organisation that passes a bar of EA-alignedness. This includes contractors/freelancers, paid local group organisers, and non-ops employees (though workspace discussion should be about operations). We also accept unpaid board members. We do make occasional exceptions to these criteria on a case-by-case basis, for example for volunteers doing unusually complex/committed operations work.

If you don’t yet meet our membership criteria, the EA Operations Facebook group may suit your needs as a more open community. The Operations tag here on the EA Forum also contains many resources you may find useful.

Our thanks to Vaidehi Agarwalla for encouraging us to post this publicly, and drafting the post for us!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Search, subforums, and other Forum updates (October 2022), published by Lizka on October 25, 2022 on The Effective Altruism Forum.

It’s only been around a month since our last update post, but we have a bunch of new features for everyone. A quick summary of the updates:

We’ve significantly updated the Forum search feature ⬇️

Two pilot subforums have launched ⬇️

The site is now ~25% faster ⬇️

September was a month of record usage on the Forum ⬇️

Some other changes and updates ⬇️

As a reminder, if you have any feature requests or suggestions, you’re very welcome to share them on the feature suggestion thread (or to get in touch). If you have comments or questions about anything here, you can also just leave a comment on this post.

An updated search feature

We have a new search UI! This is hopefully easier to use, and it also lets you filter by date, topic, and a couple of other things. You can also exclude terms by writing phrases like “Forum -update” (to search for things that include the word “Forum” but exclude the word “update”), and search for specific phrases by using quotation marks.

For now, filtering is only possible on desktop. You can access the search feature by typing something into the search bar that appears at the top right corner of your Forum window and hitting enter, or just going to this link.

This update doesn’t incorporate all the suggestions we’ve received, but it’s a start. If you have other suggestions – particularly if there are instances where the old UI was better for you – please let us know! We might not implement your suggestions right away, as we’re focusing on some unrelated projects right now, but we’d love to hear them and may prioritize them.

Two pilot subforums: bioethics and software engineering

As the amount of content on the Forum grows, we want to help people to engage with the content that is most relevant and helpful for them.

One approach we are exploring is subforums, and we launched our first pilot subforum (the bioethics subforum) last month. We later launched a software engineering subforum and will likely be rolling out more subforums over time. The direction we take will depend on how these pilots go.

We are continually adding features to the subforums, and your feedback is always appreciated!

Performance improvements

We’ve made a number of technical changes that improve the site’s performance. For instance, the Frontpage should now load around 25% faster.

We’ve also had a couple of brief outages recently, which caused some distress. We fixed an issue with our deployment process which should prevent this from happening in the future.

Record usage

The Forum has continued its rapid growth, and September hit records in every single engagement metric we track; total views, number of logged-in users, number of comments, number of accounts created with at least 5 posts viewed, number of posts with at least 2 upvotes, number of unique voters, total votes, and the number of monthly active users. The final day of the EA Criticism Contest also led to the highest single engagement day in Forum history.

Thanks all for using the Forum!

Other changes and updates

We are changing the Forum’s license

Sharang Phadke joined the forum team as a product manager

Our test “ask everyone anything” thread seems to be a success – 211 comments at the moment

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] A Response to Rebecca Ackermann’s “Inside Effective Altruism” Editorial, published by stecas on October 25, 2022 on The Effective Altruism Forum.

Link to the Google doc.

On October 17, 2022, Rebecca Ackermann published an MIT Technology Review editorial entitled “Inside effective altruism, where the far future counts a lot more than the present.” It is a piece on Effective Altruism that contains a number of untrue and misleading claims. In response, I offer a response to it in the form of annotations to the original text and figures. I’m open to corrections/criticism. I also thank Ackermann for the engagement and effort that went into the editorial. I believe it was written in good faith. By my count, there are 13 false and 23 misleading claims. Each is explained with sources in the annotations.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consider trying Vivek Hebbar's alignment exercises, published by Akash on October 24, 2022 on LessWrong.

Vivek Hebbar recently developed a list of alignment problems. I think more people should try them. I'm impressed with how well they (a) get people to focus on core problems, (b) encourage people to come up with their own ideas, (c) encourage people to notice and articulate confusions, and (d) accomplish a-c while also providing a fair amount of structure and guidance.

You can see the problems in this google doc or pasted below.

Note that Vivek is also a mentor for SERI-MATS. These exercises are also the questions that people need to answer to apply to work with him and Nate Soares. Applications are due today.

Problems:

These problems are basically research questions — we expect them to be difficult, and good responses will be valuable as research in their own right. MIRI will award prizes (likely on the order of $5000 for excellent submissions).

Instructions: We recommend that you focus on 1 or 2 of the hard questions and leave the other hard questions blank. It is mandatory to attempt either #1a-c or #2.

Note on word counts: These are guidelines for how long we think a typical good response will be, but feel free to write more. If you have lots of ideas, it’s great to write them all, and don’t bother trying to shorten them.

Hard / time-consuming questions (contest problems):

Problem 1

Pick an alignment proposal[footnote 1] and specific task for the AI.[footnote 2]

First explain, in as much concrete detail as possible, what the training process looks like. Then go through Eliezer’s doom list. Pick 2 or 3 of those arguments which seem most important or interesting in the context of the proposal.

(~250 words per argument) For each of those arguments:

What do they concretely mean about the proposal?

Does the argument seem valid?

If so, spell out in as much detail as possible what will go wrong when the training process is carried out

What flaws and loopholes do you see in the doom argument? What kinds of setups make the argument invalid?

(~200 words) For one argument only, suggest a modification to the proposal, or a research direction, to get around the doom argument.

Do you think your proposed modification/direction will work?

If so, explain why.

If not, do you think your modification is in a space where continued brainstorming and iteration will lead to a correct solution, or is there a fundamental obstacle which stops all things in that space from working? What is this obstacle?

(Optional; ~200 words) Overall, how promising or doomed does the alignment proposal seem to you (where ‘promising’ includes proposals which fail as currently written, but seem possibly fixable).

If promising, summarize how it circumvents the usual reasons for doom.

If not promising, what is the most fatal and unfixable issue?

If there are multiple fatal issues, is there a deeper generator for all of them?

What is the broadest class of alignment proposals which is completely ruled out by the issues you found?

Footnote 1: Examples of alignment proposals you could consider. We recommend you pick either a proposal you're familiar with, or something from 11 Proposals so you don't waste time parsing difficult writeups.

Safety via debate

Safety via market-making

Retargeting the search (Note: Not sure if it’s concrete/developed enough as written)

ELK (Note that ELK is a problem statement and not a proposal -- you’d have to specify an alignment proposal which uses a solution to ELK as a component. There’s a sketch of this in the ELK appendix. We don’t know if this is a good idea to attempt for this question.)

Relaxed adversarial training

Recursive reward modeling

Externalized reasoning oversight

Anything from 11 proposals (or pure HCH + IDA, which is just the “imitative amplification” part of prop...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Walkthrough of A Mathematical Framework for Transformer Circuits, published by Neel Nanda on October 25, 2022 on The AI Alignment Forum.

A Mathematical Framework for Transformer Circuits is, in my opinion, the coolest paper I've ever had the privilege of working on. But it's also very long and dense and at times confusing, and this makes me sad! So I've run an experiment, where I recorded myself reading through the paper and narrated a stream of conscious as I go - which bits are particularly cool but under-appreciated, which bits are a bit of a waste of time, which bits do I think do or do not replicate, attempting to explain the parts I think are particularly confusing, etc. You can watch it here. Sadly, it turns out I have a lot of things to say about Transformer Circuits and this turned into a 3 hour monologue, but I hope it's still useful! This is an experimental format for me for good research communication, and I'd love to hear feedback on how well it works for you! This was much easier to make than writing an entire paper, but could easily be a total waste of time if it's not clear enough to be useful!

Disclaimer: The views in this video are entirely my personal takes - the paper was a team effort from everyone at Anthropic, especially Chris Olah, Nelson Elhage and Catherine Olsson, and I am no longer employed by Anthropic. I do not necessarily expect that any of the other authors would agree with any specific thing that I've said, but hope an unfiltered series of takes is useful!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Furry Rationalists & Effective Anthropomorphism both exist, published by agentydragon on October 25, 2022 on LessWrong.

(cross-posted on EA Forum and my website)

Hi!

I’m Rai and I’m a furry (specifically, dragon). The last couple years, I’ve been running a Furry Rationalists Telegram group. It looks like we exist, and not everyone who should know we exist does yet, so I wanted to just write this to advertise that this furry+EA/rationality corner exists, and if you’re furry-adjacent & rationality-adjacent and nice, you’re invited to join us :)

Here’s the invite link for the Furry Rationalists group:/+KTmU37JO3Wk0OTQ0

There’s ~50 of us and we’re chill - we have self-improvement, science, and cute animal GIFs. If you’d like a preview, here’s the guidelines + meta doc. We’re 18+, but we’re not adult-oriented - we’re 18+ just so that we can talk about adult stuff if it does come up. If you happen to be <18 and wanna join, let me know, we might update this.

If you’re reading this a while later, and the link expired, contact me (via some method on my website, agentydragon.com), or look us up on/, a search for “rationality” should find us.

There’s also a smaller Effective Anthropomorphism Discord server, run by bird:

Come say hi, and feel free to share if you know anyone who’d be interested!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Consider Applying to the Future Fellowship at MIT, published by Jeff Kaufman on October 25, 2022 on The Effective Altruism Forum.

Are you interested in doing a PhD or Masters on something altruistic? The MIT Media Lab has a

new fellowship funded by the

FTX

Future Fund for "ambitiously benevolent projects that could not be accomplished anywhere else". They give some examples:

creating works that expand our moral circle,

designing to cultivate wisdom,

developing new technologies to promote or safeguard wellbeing, or

directly improving the lives of current beings in ways

anticipated to benefit future generations

If you're interested you would apply for admission to the MIT Media

Lab for Fall 2023 (details) and also to the fellowship (details), deadline 2022-12-01. If accepted to both, the fellowship would cover your tuition and health insurance, pay you a stipend (~$45k/y), and cover $25k/y in additional research costs.

While the application requires you to say which research groups you're interested in joining, what it doesn't make clear is that this is, unusually for the US, a direct-admission system: each faculty member

chooses individually who they would like to admit. This means it's definitely worth reaching out specifically to someone whose group you're interested in joining (though they may not respond—they get tons of email) and also thinking carefully about how your research interests intersect.

Unlike many programs, you can work on a very wide range of things: art, science, engineering, law, governance, economics, etc, and research that would be hard to do in a more traditional graduate program is especially welcome. If you have questions about what the

Media Lab is like I'm happy to try to answer, though I've only been here a few months.

(Disclosure: I work at the NAO which is part of Kevin Esvelt's Sculpting Evolution group at the Media Lab, and Kevin asked if I might want to write something about this opportunity.)

Comment via: lesswrong

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Maps and Blueprint; the Two Sides of the Alignment Equation, published by Nora Ammann on October 25, 2022 on The AI Alignment Forum.

Tl;dr

In this post, I will introduce two conceptual tools I found useful when thinking about (or introducing people to) AI alignment. Namely, I introduce the distinction between Maps and Blueprints (part 1), describe (what I call) the Two Sides of the “Alignment Equation” (part 2), and discuss how all these things relate to each other (part 3).

Part 1: Map and Blueprint

Most people reading this will already be familiar with the Map–Territory distinction. In this metaphor, the territory represents reality and the map represents your current best understanding of reality. The corollaries: “the map is not the territory”; “maps are never true, but they can be more or less useful”, etc.

I want to introduce one more idea into this picture: the blueprint. If the map is what we construct from investigating the word, a blueprint is what allows us to render a (new) bit of territory (e.g., building some artefact). Roughly speaking, map-making is closer to what science is trying to do, while blueprint-making is closer to what engineering is about.

While maps are causally posterior to the territory (they are drawn based on what we know about existing territory by looking at it), blueprints are causally prior (they are used to create new bits of territory). In other words, the red arrows in the diagram below depict the authorial relationship between these concepts.

Figure 1: The relationship between territory, maps, and blueprints; arrows represent an authorial relationship.

(There is an interesting question as to what sort of epistemic object blueprints are (e.g., compared to maps). For those curious, I can recommend Herbert Simon’s "The Sciences of the Artificial" which discusses how (if at all) we can make "the artificial"/design/blueprints our subject of study.

Part 2: Two sides of the “Alignment Equation”

Here is a simple model that intends to capture the basic structure of the AI alignment problem. I will call it the “Alignment Equation”:

Let us assume there are two (intelligent) agents that stand in relationship R to one another. Ri can take different shapes, among which is Raligned, which is a relationship that guarantees alignment between the two agents. Let us name our two agents O and S, referring to the object and subject of this alignment endeavour, respectively. Specifically, we're trying to align O with S, i.e., Raligned, OS.

Figure 2: The “alignment equation”, with the object of alignment on one side and the subject of alignment on the other.

In other words, when we talk about AI alignment, we talk about how we want one agent—in this case, the AI system(s), i.e., the object of alignment—to stand in a particular relationship to the subject of alignment (e.g., human intents, human values, etc.)—namely, an aligned relationship.

Of course, this wee toy model fails to capture a lot of relevant nuance. And yet, it lets us point at and talk about some critical subparts of the problem in fairly simple terms. For example:

1. Finding the right specification for O and S

In the toy model, O and S are placeholders. For example, O might represent a single centralised AI system, or multiple centralised AI systems, or a distributed network of AI services, etc. S might represent the intents or preferences of an individual human, or the (~aggregate) values of a collective of humans, or of sentient beings more generally, or something else yet. Finding the right ways to fill in the placeholders matters, and is itself subject to disagreement and study.

2. Finding mechanisms that can shape R

Different mechanisms can define or shape the relationship R. We can understand work on AI alignment as exploring what different mechanisms we have at hand and how to a...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Beyond Kolmogorov and Shannon, published by Alexander Gietelink Oldenziel on October 25, 2022 on The AI Alignment Forum.

This post is the first in a sequence that will describe James Crutchfield's Computational Mechanics framework. We feel this is one of the most theoretically sound and promising approaches towards understanding Transformers in particular and interpretability more generally. As a heads up: Crutchfield's framework will take many posts to fully go through, but even if you don't make it all the way through there are still many deep insights we hope you will pick up along the way.

EDIT: since there was some confusion about this in the comments: These initial posts are supposed to be an introductionary and won't get into the actually novel aspects of Crutchfield's framework yet. It's also not a dunk on existing information- theoretic measures - rather an ode!

To better understand the capability and limitations of large language models it is crucial to understand the inherent structure and uncertainty ('entropy') of language data. It is natural to quantify this structure with complexity measures. We can then compare the performance of transformers to the theoretically optimal limits achieved by minimal circuits. This will be key to interpreting transformers.

The two most well-known complexity measures are the Shannon entropy and the Kolmogorov complexity. We will describe why these measures are not sufficient to understand the inherent structure of language. This will serve as a motivation for more sophisticated complexity measures that better probe the intrinsic structure of language data. We will describe these new complexity measures in subsequent posts. Later in this sequence we will discuss some directions for transformer interpretability work.

Compression is the path to understanding

Imagine you are an agent coming across some natural system. You stick an appendage into the system, effectively measuring its states. You measure for a million timepoints and get mysterious data that looks like this:

...00110100100100110110110100110100100100110110110100100110110100...

You want to gain an understanding of how this system generates this data, so that you can predict its output, so you can take advantage of the system to your own ends, and because gaining understanding is an intrinsic joy. In reality the data was generated in the following way: output 0, then 1, then you flip a fair coin, and then repeat. Is there some kind of framework or algorithm where we can reliably come to this understanding?

As others have noted, understanding is related to abstraction, prediction, and compression. We operationalize understanding by saying an agent has an understanding of a dataset if it possesses a compressed generative model: i.e. a program that is able to generate samples that (approximately) simulate the hidden structure, both deterministic and random, in the data.

Note that pure prediction is not understanding. As a simple example take the case of predicting the outcomes of 100 fair coin tosses. Predicting tails every flip will give you maximum expected predictive accuracy (50%), but it is not the correct generative model for the data. Over the course of this sequence, we will come to formally understand why this is the case.

Standard measures of information theory do not work

To start let's consider the Kolmogorov Complexity and Shannon Entropy as measures of compression, and see why they don't quite work for what we want.

Kolmogorov Complexity

Recall that the Kolmogorov(-Chaitin-Solomonoff) complexity K(x) of a bit string x is defined as the length of the shortest programme outputting x [given a blank output on a chosen universal Turing machine]

One often discussed downside of the K complexity is that it is incomputable. But there is another more conceptual do...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What does it take to defend the world against out-of-control AGIs?, published by Steve Byrnes on October 25, 2022 on The AI Alignment Forum.

Intended audience: People very familiar with AGI safety / alignment discourse. Lots of jargon, lots of unspoken & unjustified background assumptions.

Confidence level: What I currently believe and why. I mainly work on technical alignment and don’t consider myself an expert on AGI deployment / governance issues. Interested in feedback and pushback. Pretty please with cherries on top do not make important real-world decisions based on this post.

Tl;dr:

Almost all of my posts are about technical aspects of the alignment problem. This post is instead assuming for the sake of argument that some group will manage to sculpt an AGI’s motivations such that it’s either under control / corrigible / docile, and/or has prosocial motivations, or is safe for some other reason. But this post is also assuming that it’s possible to mess up and make an AGI that seeks power and escapes control. (This post is focused on out-of-control AGI accidents; I’ll generally ignore bad actor / misuse and other problems.) The point of this post is to think through whether and how under-control “good” AGIs can defend the world against omnicidal out-of-control “bad” AGIs (including by preventing them from coming into existence in the first place), or if not, what can be done about that problem.

This is a topic of ongoing debate in the community:

An example pessimist would be Eliezer Yudkowsky, who thinks that we’re basically doomed unless one of the first groups with AGI performs a so-called “pivotal act” (more on which in Section 3.5.1) that aggressively prevents any other groups on Earth from making misaligned AGIs.

An example (relative) optimist would be Paul Christiano, who argues in The strategy-stealing assumption that, if a big tech company with a giant compute cluster trains a friendly aligned powerful AGI in year Y, we probably have little cause for global concern if it happens that, in year Y+2, some small group in an office park somewhere messes up and makes a misaligned power-seeking AGI, because whatever power- or resource-grabbing strategies that the latter can come up with and execute, the former probably would have come up with and executed those same strategies already—or even better strategies.

This post explains why I put myself in the pessimistic camp on this issue. I think Paul’s “strategy-stealing assumption” is a very bad assumption, and I will argue more generally that we’re pretty much doomed even if the first groups able to develop AGI manage to keep it under control. And this is a major contributor to how my overall current P(AGI doom) winds up so high, like 90%+.

Other underlying assumptions: I have lots of beliefs about future AI, including (1) takeoff speed is measured in years (or less), as opposed to decades or centuries; (2) Transformative AI will look like one or more AGI agents with motivations, able to figure things out and get stuff done. I have argued for these assumptions (and others) in various other posts, but here I’m mostly trying to avoid relying on those assumptions in the first place. I’m not sure how well I succeeded; these worldview assumptions are in the back of my mind and probably bleeding through.

1. Ten example scenarios where powerful good AGIs would fail to defend against out-of-control bad AGIs

(This section is intended as a series of grimly-amusing vignettes to help set the stage for the somewhat-more-careful analysis in the rest of this post, not as slam-dunk irrefutable arguments.)

1. The tech company has a powerful AI and knows how to keep it under human control. The tech company CEO goes up to the General of USSTRATCOM [the branch of the US military that deals with nuclear war] and says “We know how to...