Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hessian and Basin volume, published by Vivek Hebbar on July 10, 2022 on The AI Alignment Forum.

Thanks to Thomas Kwa for the question which prompted this post.

Note: This is mostly a primer / introductory reference, not a research post. However, the details should be interesting even to those familiar with the area.

When discussing “broad basins” in the loss landscape of a DNN, the Hessian of loss is often referenced. This post explains a simple theoretical approximation of basin volume which uses the Hessian of loss.

Suppose our minimum has loss=0. Define the basin as the region of parameter space draining to our minimum where loss < threshold T.

Simplest model

If all eigenvalues of the Hessian are positive and non trivial, we can approximate the loss as a paraboloid centered on our minimum:

The vertical axis is loss, and the horizontal plane is parameter space. The shape of the basin in parameter space is the shadow of this paraboloid, which is an ellipsoid.

The principal directions of curvature of the paraboloid are given by the eigenvectors of the Hessian. The curvatures (second derivative) in each of those directions is given by the corresponding eigenvalue.

Radii of the ellipsoid: If we start at our minimum and walk away in a principal direction, the loss as a function of distance traveled is L(x)=12λix2, where λi is the Hessian eigenvalue for that direction. So given our loss threshold T, we will hit that threshold at a distance of x=√2Tλi. This is the radius of the loss-basin ellipsoid in that direction.

The volume of the ellipsoid is Vbasin=Vn∏i√2Tλi, where the constant Vn is the volume of the unit n-ball in n dimensions. Since the product of the eigenvalues is the determinant of the Hessian, we can write this as:

So the basin volume is inversely proportional to the square root of the determinant of the Hessian. Everything in the numerator is a constant, so only the determinant of the Hessian matters in this model.

The problem with this model is that the determinant of the Hessian is usually zero, due to zero eigenvalues.

Fixing the model

If we don't include a regularization term in the loss, the basin as we defined it earlier can actually be infinitely big (it's not just a problem with the paraboloid model). However, we don't really care about volume that is so far from the origin that it is never reached.

A somewhat principled way to fix the model is to look at volume weighted by the initialization distribution. This is easiest to work with if the initialization is Gaussian. To make the math tractable, we can replace our ellipsoid with a "fuzzy ellipsoid" -- i.e. a multivariate Gaussian. Now we just have to integrate the product of two Gaussians, which should be easy. There are also somewhat principled reasons for using a "fuzzy ellipsoid", which I won't explain here.

However, this is only somewhat principled; if you think about it further, it starts to become unclear: Should we use the initialization Gaussian, or one based on the expected final L2 norm? What about cases where the norm peaks in the middle of training, and is smaller at the start and finish?

If we have an L2 regularization term in the loss, then the infinite volume problem usually goes away; the L2 term makes all the eigenvalues positive, so the formula is fine. If we have weight decay, we can interpret this as L2 regularization and add it to the loss.

For a relatively simple approximation, I recommend the formula:

Where:

Loss is the unregularized loss

λ is the amount of weight decay (or L2 regularization 12λθ2)

c=k/σ2, where σ is the standard deviation of the initialization Gaussian, and k is a constant on the order of unity. I have not calculated the theoretically most appropriate value of k. For a crude model, k=1 is probably good enough.

T is the loss threshold. If you really care about...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Case for Making Professional Degrees Undergraduate Degrees, published by ColdButtonIssues on July 8, 2022 on The Effective Altruism Forum.

Cross-posted from Cold Button Issues

In the United States, there are many professions that are generally impossible to enter without both a graduate professional degree and an undergraduate degree, which may be in a completely unrelated field. Examples of these professions are physicians, attorneys, pharmacists, and physical therapists. Such requirements are unnecessary, as demonstrated by many developed countries that operate successfully without such a two-degree requirement. I estimate that these superfluous undergraduate degree requirements cost 730,000 person-years of work annually and foregone wages and salaries equal to 0.8% of all U.S. wages and salaries.

In the United States, occupational licensing is a state-level responsibility. These legal requirements for superfluous undergraduate degrees could be abolished in many states using ballot initiatives.

Using data on costs from recent statewide ballot initiatives, I estimate that this intervention would meet Open Philanthropy’s threshold for cost-effectiveness in some but not all states.

Why It’s Important

Research generally shows that adding additional educational requirements to enter professions produces no benefits.

Barrios (2021) showed that requiring accountants to achieve 150 semester hours instead of the traditional 120 hours is linked to fewer new accountants but not to higher ability in new accountants. Timmons and Mills (2018) showed that introducing optician licensing in the United States raised the salaries of opticians, but did not improve quality. In 2015, the Obama Administration’s Council of Economic Advisors concluded that:

Licensing laws also lead to higher prices for goods and services, with research showing effects on prices of between 3 and 16 percent. Moreover, in a number of other studies, licensing did not increase the quality of goods and services, suggesting that consumers are sometimes paying higher prices without getting improved goods or services.

More qualitatively, we can observe that these education requirements were considered unneeded in years past in the United States and are still considered unneeded in many countries around the world. For instance, a pharmacy degree used to be an undergraduate degree but has now been replaced by a professional doctorate in the United States and some other countries. In the United Kingdom, medical school is an undergraduate program. The American practice of requiring an undergraduate degree before entering many professional schools generates tremendous waste.

To estimate the lost years of work caused by required undergraduate degrees prior to professional education across the United States, I obtained the number of professional degrees granted in the 2018-2019 academic year from the National Center for Education Statistics that are generally used to enter licensed professions and that follow undergraduate degrees. I then multiplied the number of degrees granted by four, which represents the normative length of an undergraduate degree in the United States. This is not a precise estimate as some students take more or less than four years to graduate, some schools offer joint undergraduate-professional degrees to slightly speed the total time in education, some earn professional degrees but do not enter the profession, and some licenses require a certain number of credit hours rather than a specific degree.

To estimate the lost wages/salary, for each profession, I multiplied the estimated lost years of work by the median salary for that profession. I estimate that $67 billion in wages and salary is lost each year, which is roughly 0.8% of total U.S wages and salary. This also does not include any benefi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Grouped Loss may disfavor discontinuous capabilities, published by Adam Jermyn on July 9, 2022 on The AI Alignment Forum.

Thanks to Evan Hubinger and Beth Barnes for comments on these ideas.

Language models exhibit scaling laws, where the loss is a power-law in model size. This offers a lot of predictive power, and seems like a useful thing to know. By contrast, individual capabilities can exhibit sharp discontinuities in performance as a function of model size and training time.

It would be great if individual capabilities just gradually improved like the broader loss. Then we wouldn’t have to worry quite so much about surprising new capabilities emerging suddenly during training.

Is there a way to change the loss function so that it incentivizes more gradually capability improvements?

Grouped Loss

Imagine grouping training examples by the kind of capability they exhibit. For instance arithmetic problems go in one group, “parse json” could go in another, and so on. With these groups, we could define a new loss function

where ℓ is the loss function we originally used (e.g. cross-entropy loss) and ℓg means to compute the sum of ℓ over examples from group g, e.g.

which may be estimated by using random examples drawn from g.

Because we have squared the group losses, the overall loss is dominated by the worst group. As a result, the model is incentivized to develop capabilities in each group at comparable rates, and so has little incentive to e.g. finely hone its poetry skills while being unable to multiply numbers.

Challenge: Defining Groups

It’s possible that using grouped loss results in smooth development of capabilities that aren’t represented in the groups. For instance, it seems plausible that if “adding arabic numerals” and “translating words into arabic numerals” are two groups but “adding numbers written as words” is not, performance on the latter could nonetheless develop smoothly as the model gets better at the others. It would certainly be weird if performance ”adding numbers written as words” advanced as a sudden leap in this case.

This points to a general problem though, which is that if we have to define the groups manually we have to foresee the capabilities we’re worried about. That seems bad.

Gradient Cluster Grouping

If we could automatically group examples we wouldn’t need to do it manually. How could we do this?

I think the key feature of a group is that when the model updates, the loss of most examples in a group changes in a similar way. When that happens, it seems intuitive to say that there’s a discrete capability somewhere in the model and that those examples all depend on it.

This suggests looking for examples where the loss has similar gradients, because these probably make use of similar machinery in the model.

Concretely, I’m imagining the following procedure:

Draw N examples from the training set.

Evaluate the gradient of ℓ for each example.

Group the examples by clustering their gradients, evaluate the grouped loss, and perform gradient descent on that.

As a technical note: In this approach, the grouped loss is a moving target. As the model learns and capabilities form the groups shift. This means that SGD is no longer minimizing a constant loss. I don’t think that’s a problem, in part because all versions of the loss agree when the model has reached zero-loss, so the different iterations of the loss function all point towards better capabilities.

Challenge: How many groups?

I don’t know of a principled way to pick the number of groups to cluster examples into, and that seems like a problem. Guessing too many groups loses the advantage of grouping because each group reflects an extremely narrow task. Guessing too few groups also loses the advantage of grouping, because then the capabilities that show gradual improvements will be very broa...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Train first VS prune first in neural networks., published by Donald Hobson on July 9, 2022 on The AI Alignment Forum.

This post aims to answer a simple question about neural nets, at least on a small toy dataset. Does it matter if you train a network, and then prune some nodes, or if you prune the network, and then train the smaller net.

What exactly is pruning.

The simplest way to remove a node from a neural net is to just delete it. Let yj=f(∑ixiwij+bj) be the function from one layer of the network to the next.

Given I and J as the set of indicies that aren't being pruned, this method is just

~wij={wiji∈I and j∈Jnothingelse

~bj={bjj∈Jnothingelse

however, a slightly more sophisticated pruning algorithm adjusts the biases based on the mean value of xi in the training data. This means that removing any node carrying a constant value doesn't change the networks behavior.

The formula for bias with this approach is

~bj={bj+∑i∉I¯xiwijj∈Jnothingelse

This approach to network pruning will be used throughout the rest of this post.

Random Pruning

What does random pruning do to a network. Well here is a plot showing the behavior of a toy net trained on spiral data. The architecture is

And this produces an image like

In this image, points are colored based on the network output. The training data is also shown. This shows the network making confidant correct predictions for almost all points. If you want to watch what this looks like during training, look here

When half of the nodes are pruned from both intermediate layers, adjusting the bias appropriately, the result looks like this.

If you fine tune those images to the training data, it looks like this.

If you take the untrained network, and train it, the result looks like this.

Ok. Well this shows that pruning and training don't commute with random pruning. This is kind of as expected. The pruned then trained networks are functional high scoring nets. The others just aren't. If you prune half the nodes at random, often a large chunk of space is categorized wrongly. This shows that the networks aren't that similar. This is kind of to be expected. However, these networks do have some interesting correlations. Taking the main 50x50 weight matrixes from each network and plotting them against each other reveals.

Same plot, but zoomed in to show detail.

Notice the dashed diagonal line. The middle piece of this line is on the diagonal to machine precision. It consists of the points that were never updated during training at all. The uniform distribution with sharp cutoffs is simply due to that being the initialization distribution. The upper and lower sections consist of points moved about ±0.59 by the training process. For some reason, the training process likes to change values by about that amount.

Nonrandom Pruning

A reasonable hypothesis about neural networks is that a significant fraction of neurons aren't doing much, so that if those neurons are removed then the network will have much the same structure with or without training. Lets test that by pruning the nodes with the smallest standard deviation.

This pruning left an image visually indistinguishable from the original. entirely consistent with the hypothesis that these nodes weren't doing anything.

When those same nodes are removed first, and the model is then trained, the result looks like this.

Similar to the trained and then pruned (see top of document), but slightly different.

Plotting the kernels against each other reveals

This shows a significant correlation, but still some difference in results. This suggests that some neurons in a neural net aren't doing anything. No small change will make them helpful, so the best they can do is keep out the way.

It also suggests that if you remove those unhelpful neurons from the start, and train without them, the remaining neur...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Visualizing Neural networks, how to blame the bias, published by Donald Hobson on July 9, 2022 on The AI Alignment Forum.

Background

This post is strongly based on this paper, which it calls the LRP algorithm,

I later learned of the existence of this paper, which is even more similar to the ideas discussed here.

In it, I examine 2 of the methods for neural network visualization, and show that they have structural similarities. I show that these algorithms only differ by a difference in how they treat the biases, and (possibly a difference in getting started)

The second algorithm obeys conservation laws, it tries to parcel credit and blame for a decision up to the input neurons, even when credit and blame

Intro

The task we want is to assign importance to different inputs of a neural network in production of an output. So for example, in the case of a trained image classifier, the visualization method would take in a particular image, and highlight the parts of the image that the network thought were important.

A general method for visualizing neural networks is back-propagation. First evaluate the network forwards. Then work backwards through the network by using some rule about how to reverse each individual layer.

One example of this is differentiation. Finding the rate of change of the output, with respect to each input. But there are others.

Firstly, lets pretend biases in the network don't exist. We are allowed non-linearities, so long as they satisfy the equation f(0)=0 .

Lets look at various layer types and the different back-propagation rules.

Maximum

Most often found in the form of max pooling.

Gradient

The rule used in gradient descent, and I think the only rule used in the paper above for back propagating maximum. (Notation note. R here isn't exactly a function. Its more like ddx , its output is related to the context in which the input occurs. Think of every number in the forward net having an associated number )

Ignoring the case of an exact tie. Exact ties, and what to do on that kind of singular point will be ignored in general, because they are fiddly and unimportant.

Radial.

Treats 0 as a special point.

Where the plot above shows si and the si are non-negitive and sum to 1.

Case. ai<0, b>0⟹si=0 . Relative to 0, a negative value contributes nothing to a positive maximum.

Case, ai>0⟹si=ai∑ai>0ai (In particular si=1 if i is the only value with positive ai

Case all negative. si=1ai∑i1ai .

Matrix multiplication

A common operation in neural networks. Even convolutions can be expressed as a matrix multiplication. The matrix just happens to be sparse, and contain repetitions.

Gradient

Yet again a popular choice.

Normalized

What the LRP algorithm uses is

(This is deduced from equation 6 in )

Where blip is this function

A straight line of gradient 1, except for a little vertical jump to avoid 0.

Why are they using this blip? Because they want to divide by bj here to get an interesting theoretical property (conservation of total) but if they don't add this little jump, they get numerical instability caused by dividing by something too close to 0.

Nonlinearity

There are several mechanisms proposed to deal with an arbitrary potentially non-linear function b=f(a), applied elementwise. We will impose the condition f(0)=0 for now.

Ignore

Very simple R(a)=R(b)

Gradient

R(a)=f′(a)R(b) standard rule of calculus.

Slope

R(a)=baR(b)

Repeat

R(a)=f(R(b))

Downside: Is nonlinear in R(b), unlike every other method here.

Consistency rules

You can't just pick any option from each of these lists. Well you could. But there are some nice mathematical consistency properties it would be nice to have.

Scaling equivalence

Suppose you want to multiply all values in the network by a constant α, there are 2 ways you could do this. You could see the constant as a scaled identity matrix, and...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How STVs may make it harder for an AGI to "trick" us, published by Tor Økland Barstad on July 9, 2022 on The AI Alignment Forum.

Summary / Preamble

AI Alignment has various sub-areas. The area I focus on here is ways we might use a superintelligent AGI-system to help with creating an aligned AGI-system, even if the AGI we start out with isn’t fully aligned.

Imagine a superintelligence that “pretends” to be aligned. Such an AI may give output that seems to us like what we want. But for some types of requests, it’s very hard to give output that seems to us like what we want without it actually being what we want (even for a superintelligence). Can we obtain new capabilities by making such requests, in such a way that the scope of things we can ask for in a safe way (without being “tricked” or manipulated) is increased? And if so, is it possible to eventually end up with an aligned AGI-system?

One reason for exploring such strategies is contingency planning (what if we haven’t solved alignment by the time the first superintelligent AGI-system arrives?). Another reason is that additional layers of assurance could be beneficial (even if we think we have solved alignment, are there ways to relatively quickly add additional layers of alignment-assurance?).

When dealing with a genie/oracle, we may not want to ask it to provide some direct solution/answer to what you want (and have it argue why it’s a good idea). Instead we obtain what we want more indirectly, by having different instances of the genie/oracle aid us in the construction of more narrow/specialized programs (that score high in terms of verifiability).

We could call such programs STVs, where STV is an abbreviation for Specialized Transparent Verifiable Program.

STV is a blurry concept (much like how AI and AGI are blurry concepts).

An STV could be (but would not have to be) a narrow AI. More colloquially, STVs could also be referred to as “generators”.

In this post I explore principles/techniques/strategies for using STVs in ways that enable us to get things we want from an AGI without being “tricked”.

I hope and think this article reads well in isolation, even though it's part 2 of a series.

Future posts in this series will include discussion of:

AI-generated computational proofs/argument-trees, and methods/strategies for verifying such proofs/arguments.

Ideas for how formalisms for computational proofs could try to incorporate human cluster-like concepts, concept vagueness, formalism-to-reality mappings, references to code and results from running code, etc (from within themselves).

How different “pieces” outlined in this series might be put together. Can we get from unaligned AGI to aligned AGI without being "tricked"/hacked/manipulated along the way, and without being stopped by chicken-or-egg problems?

Ideas for how we might want to make a system of sub-systems, consisting of various "siloed" AGI-systems (with different settings, architectures, and alignment methodologies).

Strategic and moral considerations, and possible first steps when starting to use superintelligent AGI-systems to do things outside of the digital realm.

Are any of the ideas in this series new? See here for a discussion of that.

Me: I think it may be helpful to start out with a simplistic example.Me: So let’s imagine that we are interested in finding good chess-moves, and want an AGI to help us with this.

Imaginary friend: Ok.

Me: We could ask it directly for good chess-moves, along with explanations for why these chess-moves seem like good chess-moves.

Me: But if we want to make it hard for the AGI to “trick” us (to pretend that it’s giving us what we want, but not actually doing so). One way could be to ask for a program that searches for good chess-moves (instead of asking directly for the moves themselves).

Me: This could be benefici...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Senior EA 'ops' roles: if you want to undo the bottleneck, hire differently, published by AnonymousThrowAway on July 9, 2022 on The Effective Altruism Forum.

I've decided to submit this anonymously so I can be as honest as possible without 'outing' the orgs I've engaged with, or to accidentally poison the well for myself with prospective employers. But if anyone reading this would like to learn more, happy to give more specific advice to you but with strict confidentiality - so feel free to DM.

I suspect it shouldn't be too hard to 'out' me. Please don't - I think it's important to be frank about where I've seen hiring not go well, and these reflections could hurt those who have inspired them because feedback is something you need to be ready to receive. I do not want this to happen, so I am making more general criticisms and I would not have written this if I thought it would inadvertently target some orgs / people.

Summary

EA is bottlenecked by 'good senior ops people'; we hear this a lot. But my experience hiring, building and running teams myself and engaging with EA organisations, has made me think that there is a lot of room for improvement when it comes to attracting and hiring these people. So these are personal reflections on what I think a lot of EA orgs get wrong when it comes to hiring these roles and what I think can change to unblock the bottleneck.

The issues:

Many EA org 'operations specialists' jobs will not appeal to the calibre of candidate you want because:

the priority tasks are often mundane, not challenging and not the best use of their skills

the role is mostly positioned as "enabling the existing leadership team" to the extent that it seems like "do all the tasks that I / we don't like"

hiring managers give the impression that they will likely discount the perspectives and skills the new hire could bring

I have a sense of excessive scepticism towards people who are not 'EA / long-termist enough' when comes to hiring, but this will prevent you getting high calibre talent who can bring diversity of experience and new ideas. Not 'EA enough' is either:

not having worked in a recognised EA / long-termist org before

not being bought into what are considered almost doctrinal EA / long-termist ideas which a reasonable person could have well-grounded objections to

Solutions are given in the final section.

Where am I coming from, and what type of roles am I referring to?

I've worked as a generalist in management positions for about nine years, mostly project roles starting up new teams to build / make something. I've run teams of 20+ people on complex multi-stakeholder projects, making key strategic decisions and reporting to very senior stakeholders.

In the last few months, I've been approached directly many times to apply for fairly senior 'ops roles'. The type of roles I was engaged over, and that I am talking about in the rest of this post, would be things like...:

Managing Director / Chief Operating Officer / Chief of Staff

Director / Head of Operations

Director / Head of Special Projects

Why this matters?

Having looked at a lot of job roles, a lot of them are unappealing in how they are written or a less than positive impression gets made by the hiring team. So I didn't apply to many of them.

My assumption is that if I'm feeling this, other good candidates might feel the same.

The skills and attributes that a senior 'ops person' will have

Someone qualified to do any of the roles above will have the following skillsets, and will consider their comparative advantage to lie in...:

Strategic thinking, understanding how the organisation should tackle its goals, what its comparative advantage is and what it should leave to others to do

Importantly, the ability to brutally prioritise, challenging other senior team members to 'kill their darlings' in ord...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: MATS Models, published by johnswentworth on July 9, 2022 on LessWrong.

I’ve been using the summer 2022 SERI MATS program as an opportunity to test out my current best guesses at how to produce strong researchers. This post is an overview of the methods I’ve been testing, and the models behind them.

The Team Model

My MATS participants are in three-person teams with specialized roles for each person: theorist, experimentalist, and distillitator (“distiller” + “facilitator”). This team setup isn’t just a random guess at what will work; the parts are nailed down by multiple different models. Below, I’ll walk through the main models.

Jason Crawford’s Model & Bits-of-Search

In Shuttling Between Science and Invention, Jason Crawford talks about the invention of the transistor. It involved multiple iterations of noticing some weird phenomenon with semiconductors, coming up with a theory to explain the weirdness, prototyping a device based on the theory, seeing the device not work quite like the theory predicted (and therefore noticing some weird phenomenon with semiconductors), and then going back to the theorizing. The key interplay is the back-and-forth between theory and invention/experiment.

The back-and-forth between theory and experiment ties closely to one of the central metaresearch questions I think about: where do we get our bits from? The space of theories is exponentially huge, and an exponentially small fraction are true/useful. In order to find a true/useful theory, we need to get a lot of bits of information from somewhere in order to narrow down the possibility space. So where do those bits come from?

Well, one readily available source of tons and tons of bits is the physical world. By running experiments, we can get lots of bits. (Though this model suggests that most value comes from a somewhat different kind of experiment than we usually think about - more on that later.)

Conversely, blind empiricism runs into the high-dimensional worlds problem: brute-force modeling our high-dimensional world would require exponentially many experiments. In order to efficiently leverage data from the real world, we need to know what questions to ask and what to look at in order to eliminate giant swaths of the possibility space simultaneously. Thus the role of theory.

In order to efficiently build correct and useful models, we want empiricism and theory coupled together. And since it’s a lot easier to find people specialized in one or the other than people good at both, it makes sense to partner a theorist and an experimentalist together.

Nate Soares’ Model

The hard step of theory work typically involves developing some vague intuitive concept/story, then operationalizing it in such a way that an intuitive argument turns into a rigorous mathematical derivation/proof. Claim which I got from Nate Soares: this process involves at most two people, and the second person is in a facilitator role.

(Note that the description here is mine; I haven’t asked Nate whether he endorses it as a description of what he had in mind.)

What does that facilitator role involve? A bare-minimum version is the programmer’s “Rubber Duck”: the rubber duck sits there and listens while the programmer explains the problem, and hopefully the process of explanation causes the programmer to understand their problem better. A skilled facilitator can add a lot of value beyond that bare minimum: they do things like ask for examples, try to summarize their current understanding back to the explainer, try to restate the core concept/argument, etc.

What the facilitator does not do is actively steer the conversation, suggest strategies or solutions or failure modes, or drop their own ideas in the mix. Vague intuitive concepts/stories are brittle, easy to accidentally smash and replace with some other story; you d...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some research questions that you may want to tackle, published by Aaron Bergman on July 9, 2022 on The Effective Altruism Forum.

Formerly titled "Write up my research ideas for someone else to tackle? Fine - you asked for it!"

Unrelatedly, thanks to Jessica McCurdy for telling me to write down some of my research ideas and questions in case someone else wants to tackle one (or a few).

The list

Cause prio but for earning to give

As far as I know, SBF relied on his personal knowledge and intuition when deciding to try building FTX.

It doesn’t have to be this way! I can imagine a more systematic effort to identify and describe which earning to give opportunities are most promising. Is there a $100B idea with a 1% chance of working? A $1T idea with a 0.1% chance? I think we can and should find out.

Are there cheap and easy ways to kill fish quickly?

Right now, I estimate 250 million fish years are spent in agony each year as wild fish are killed by asphyxiation or being gutted alive, which takes a surprisingly long time to cause death. There must be a better way.

Related: can we just raise (farm) a ton of fish ourselves, but using humane practices, with donations subsidizing the cost difference relative to standard aquaculture

From my red teaming project on extinction risk reduction:

Unpacking which particular biorisk prevention activities seem robust to a set of plausible empirical and ethical assumptions and which do not; and

Seeking to identify any AI alignment research programs that would reduce s-risks by a greater magnitude than "mainstream" x-risk-oriented alignment research.

From my “half baked ideas comment” on the Forum:

Figure out how to put to good use some greater proportion of the approximately 1 Billion recent college grads who want to work at an "EA org"

This might look like a collective of independent-ish researchers?

There should be way more all-things-considered, direct comparisons between cause areas.

So I guess the research question is: what is the most important cause area to work on and/or donate to, all things considered?

No more “agreeing to disagree” - I want an (intellectual) fight to the death. Liberal-spending longtermists should make an affirmative case that this ethos is the best way to spend money on the margin, and objectors should argue that it isn’t.

In particular, I don't think a complete case has been made (even from a total utilitarian, longtermist perspective) that at the current funding margin, it makes sense to spend marginal dollars on longtermism-motivated projects instead of animal welfare projects. I'd be very interested to see this comparison in particular

[Related to above] Is anyone actually arguing that neartermist, human-centric interventions are the most ethical way to spend time or money?

That’s not a rhetorical question! The hundreds of millions of dollars being directed to AMF et al. instead of some other charity or cause area should be more seriously justified or defended, IMO.

For anyone who does think that improving human welfare in the developing world is the best thing to do: do AMF-type charities actually increase the number of human life-years lived?

(As I asked on Twitter) What jobs/tasks/roles are high impact (by normal EA standards) but relatively low status within EA?

I think one of the big ways EA could screw up is by having intra-EA status incongruent (at least ordinally) with expected impact.

What would an animal welfare movement with the ambition, epistemic quality, and enthusiasm (and maybe funding) of the longtermist movement look like?

[I might tackle this] What can AI safety learn from human brains’ bilateral asymmetry

The whole “brain hemisphere difference” thing is surrounded by plenty pop science myths, surrounding it, but there really are some quite profound differences as described in Ian McGil...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Doom Circles, published by Amy Labenz on July 8, 2022 on The Effective Altruism Forum.

History of Doom Circles

Doom Circles were first attempted at the Boston Hamming Workshop in April 2015, which was co-taught by Anna Salamon, Michael / Valentine Smith, and Kenzi (Amodei) Ashkie. This was a workshop where the organizers wanted to push people harder than usual - the name “Doom Circle” came from the fact that getting people in a circle to say what’s wrong with each other is, in general, a terrible idea for an activity! But by lampshading the doominess of the activity, and making it super clear what to expect, Doom Circles can avoid some of the emotional fallout of group critique while still getting the benefit. It’s just really valuable to know why things are most likely, or at least seem most likely, to go wrong!

Doom Circle Procedure

The Doom Circle is an activity for a group of people, each of which has at least one thing they’re working on. The thing can be professional, personal, whatever. It can be as broad as simply existing well within a shared community, or as narrow as a specific project. Each person has a turn being the focus of the circle.

When it’s your turn as the focus of the circle, there’s first a disclaimer. It goes something like (to quote Ducan Sabien’s version):

We are gathered here this hour for the Doom of Steve. Steve is a human being who wishes to improve, to be the best possible version of himself, and he has asked us to help him “see the truth” more clearly, to offer perspective where we might be able to see things he does not. The things we have to say may be painful, but it is a pain in the service of a Greater Good. We offer our truths out of love and respect, hoping to see Steve achieve his goals, and he may take our gifts or leave them, as he sees fit.

Then everyone takes turns, as advertised, sharing whatever they think the focal person might fail at their efforts — called, in this case, their “doom”. Everyone has to participate, though offerings can be as short as 10 seconds or as long as 90. Nobody should go on for several minutes, though, and all critiques should be offered in good faith to help the focal person improve. The focal person can only say “thank you” to each doom.

After someone’s turn is complete there’s an outro. Again, to use Duncan’s version:

The Doom of Steve has come, and soon we will send it on its way. Like the calm after a storm, it is fitting that we hold the silence for a few minutes, in acknowledgement of what we've shared. Steve, thank you for hearing what we had to offer.

There are a few additional rules:

If you have to leave partway, you can’t come back. Ideally, everyone stays the whole time, and so everyone gives everyone else doom.

People can request aftercare if they expect they’ll need it – for example, they could request hugs, compliments, or a moment of silence for processing; these things happen after the doom, however, rather than in the middle.

People should make an effort to be as kind as possible, while still honoring the spirit of sharing doom.

Variants

Mirror: to help handle the common issue of projection, pass a mirror around when offering dooms. After each person offers their doom to the focal person in this variation, they then look in the mirror and offer the same doom up to themselves to see if it sticks.

Gentle: to help soften the doom experience, follow the doom with things they admire. Also, consider cutting the intro/outro.

Fly-on-the-wall: to help deal with awkwardness around looking participants in the eye while getting brutal feedback, ask the person who's the focus to leave the circle entirely, and sit with their back to the circle, listening in silence as everybody in the circle just talks for 5-10min.

Blindsight: in cases where participants feel they don't know the focus person ...