Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Two tentative concerns about OpenPhil's Macroeconomic Stabilization Policy work, published by remmelt on January 3, 2022 on The Effective Altruism Forum.

Here’s something I’ve been pondering on-and-off for about two years now, but frankly know little about. I'm an amateur, and just found 3 hours to write up my speculations. Would love to get your vigorous input and corrections, particularly on economic considerations where you have some expertise as an academic or practician.

OpenPhil argues: "monetary policymakers currently face political pressure to over-emphasize risks of inflation, relative to the suffering and lost output caused by unemployment."

Holden Karnofsky elaborates more on it here:...

"We've come to the view that there's an institutional bias in a particular direction. We believe that there is more inflation aversion than is consistent with a "most good for everyone" attitude. We think some of that bias reflects the politics and pressures around the Federal Reserve. We've been interested in macroeconomic stabilization for a while. There's this not very well-known institution, which is not very well understood and makes esoteric decisions. It's not a big political issue, but it may have a bigger impact on the world economy and on the working class than basically anything else the government is doing. Maybe even bigger than anything else that anyone is doing.

I think it's kind of a twofer. We haven't tried to do the calculations on both axes, but certainly, it seems like it could provide broad-based growth and lower unemployment. There are a lot of reasons to think thoseat might lead to better societal outcomes. Outcomes such as better broad-based values, which are then reflected in the kinds of policies we enact and the kinds of people we elect.

I also think that if the economy is growing, and especially if that growth is benefiting everyone across the economy: if labor markets are tighter, and if workers have better bargaining power, better lives, better prospects in the future, then global catastrophic risk might decrease in some way. I haven't totally decided, how does the magnitude of that compare to everything else? But I think if we had the opportunity to go bigger on that cause, we would be thinking harder about it."

And Alexander Berger here:

"We’ve also done a bunch of work on U.S. policy causes, including ... macroeconomic stabilization. We’re not currently planning to grow that work as much because we think we probably can find some better opportunities in the future. ...

So we’re not totally sure about the future of that program. We’re not actively winding it down, but we haven’t been doing a lot more. We have been thinking about pivoting a little bit more to work in Europe, where if you just compare the E.U. policy response to the Great Recession to the American one, I think there’s a huge gap. And also frankly, the recoveries to the Great Recession — as much as I complained about the U.S. policy response, the degree of self-inflicted wounds by European monetary policymakers is I think genuinely somewhat astonishing.

Obviously there are concerns. We’re an American funder. We don’t know as much about policy in Europe as we do about the U.S., and so there’s risks there, and we try to be cognizant of those. But I think we might continue to do a little bit more in that space and focus more on Europe. Or at some point we might say like. I don’t know if it would be literally declaring victory, but we might say like, we’re not sure there’s a ton more that we need to do here. The case doesn’t look as good as it did before. Why don’t we just step back?

Rob Wiblin: I guess in the U.S. they’re slightly worried that possibly the pendulum has swung too far in the other direction. People always respond to the last thing that went wrong, and now we’ve over...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Feature Announcement] Rich Text Editor Footnotes, published by Jonathan Mustin on January 4, 2022 on The Effective Altruism Forum.

I'm excited to announce that the EA Forum now has footnote support in our default rich text editor.

You can add footnotes in two ways:

1. Using the tooltip menu when text is selected, under the [] label:

2. Using markdown footnote syntax. To use this:

Type [^n], where n is the index of the footnote to be inserted.

To insert a new footnote, use n = # of existing footnotes + 1. To reference an existing footnote, use n = the index of the footnote in question.

Both of these create a footnote reference at the current position, which looks like this , while also inserting a corresponding footnote at the bottom of the page (unless referencing an existing footnote).

A few points:

Footnotes are automatically kept in order by their first reference.

Deleting a footnote removes all of its references and renumbers subsequent footnotes.

Hovering over a footnote reference displays the content of the footnote in a tooltip. This functionality is backward-compatible with preexisting footnotes created using the markdown editor.

We hope you enjoy them! If you run into any issues, feel free to let us know in the comments of this post.

Thanks, and happy new year!

Sample footnote.

A tradeoff of this approach is that footnotes might reorder without you noticing in the midst of writing a post, for instance when a footnote gets inserted out of order or deleted. If you're adding an additional reference for an existing footnote, it's probably a good idea to check that it still has the index you expect.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More Is Different for AI, published by Jacob Steinhardt on January 4, 2022 on The AI Alignment Forum.

Machine learning is touching increasingly many aspects of our society, and its effect will only continue to grow. Given this, I and many others care about risks from future ML systems and how to mitigate them.

When thinking about safety risks from ML, there are two common approaches, which I'll call the Engineering approach and the Philosophy approach:

The Engineering approach tends to be empirically-driven, drawing experience from existing or past ML systems and looking at issues that either: (1) are already major problems, or (2) are minor problems, but can be expected to get worse in the future. Engineering tends to be bottom-up and tends to be both in touch with and anchored on current state-of-the-art systems.

The Philosophy approach tends to think more about the limit of very advanced systems. It is willing to entertain thought experiments that would be implausible with current state-of-the-art systems (such as Nick Bostrom's paperclip maximizer) and is open to considering abstractions without knowing many details. It often sounds more "sci-fi like" and more like philosophy than like computer science. It draws some inspiration from current ML systems, but often only in broad strokes.

I'll discuss these approaches mainly in the context of ML safety, but the same distinction applies in other areas. For instance, an Engineering approach to AI + Law might focus on how to regulate self-driving cars, while Philosophy might ask whether using AI in judicial decision-making could undermine liberal democracy.

While Engineering and Philosophy agree on some things, for the most part they make wildly different predictions both about what the key safety risks from ML will be and how we should address them:

Both Engineering and Philosophy would agree on some high-level points: they would agree that misaligned objectives are an important problem with ML systems that is likely to get worse. Engineering believes this because of examples like the Facebook recommender system, while Philosophy believes this based on conceptual arguments like those in Superintelligence. Philosophy is more confident that misaligned objectives are a big problem and thinks they could pose an existential threat to humanity if not addressed.

Engineering and Philosophy would both agree that out-of-distribution robustness is an important issue. However, Philosophy might view most engineering-robustness problems (such as those faced by self-driving cars) as temporary issues that will get fixed once we train on more data. Philosophy is more worried about whether systems can generalize from settings where humans can provide data, to settings where they cannot provide data even in principle.

Engineering tends to focus on tasks where current ML systems don't work well, weighted by their impact and representativeness. Philosophy focuses on tasks that have a certain abstract property that seems important, such as imitative deception.

In my experience, people who strongly subscribe to the Engineering worldview tend to think of Philosophy as fundamentally confused and ungrounded, while those who strongly subscribe to Philosophy think of most Engineering work as misguided and orthogonal (at best) to the long-term safety of ML. Given this sharp contrast and the importance of the problem, I've thought a lot about which—if either—is the "right" approach.

Coming in, I was mostly on the Engineering side, although I had more sympathy for Philosophy than the median ML researcher (who has ~0% sympathy for Philosophy). However, I now feel that:

Philosophy is significantly underrated by most ML researchers.

The Engineering worldview, taken seriously, actually implies assigning significant weight to thought experiments.

On the...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The phrase “hard-core EAs” does more harm than good, published by Helen on January 4, 2022 on The Effective Altruism Forum.

In casual discussions in EA spaces, it’s often useful to be able to refer separately to:

The smaller group of people who are 100% bought into EA, in the thick of things, relying on EA ideas and the EA community to steer many of their decisions

The larger group of people who are interested in EA, perhaps have engaged with it some, perhaps have been to a few meetups or donated to GiveWell charities, or maybe just agree with EA principles but haven’t engaged with the community much

I often hear the former category referred to as "hard-core EAs." I think that this is harmful and exclusive, that it reflects and contributes to sloppy thinking, and that we should stop doing it. This post explains why and offers a couple alternatives.

Why?

My issues with the phrase “hard-core EAs” mostly boil down to the implicit hierarchy it creates. I think implying a hierarchy like this is unwelcoming and distorts our thinking.

Unwelcoming: This one should hopefully be obvious. If this phrase is in common use, anyone who doesn't think they count as hard-core, or isn't sure, is likely to feel like they don't belong or are doing something wrong. “Hard-core” is also condescending — anyone describing themselves or their friends this way naturally sounds like they're looking down their nose at others.

Distorts our thinking: Talking in terms of how “hard-core” someone is conflates together a bunch of different dimensions that are importantly different. Saying someone is hard-core could be taken to mean that they:

Are actually having a significant positive impact on the world

Are deeply committed to overarching EA principles, e.g. impartiality, cost effectiveness, cause prioritization, etc.

Are deeply embedded in the current EA community, e.g. buying into the community’s specific priorities/frameworks, spending lots of time with other EAs, etc.

Work themselves extremely hard to try to maximize impact

...among other things.

In my mind, #1 and #2 are the traits that we as a community should promote and think highly of. But in practice, I think “hard-core EAs” is usually used to point to people whose distinguishing features are #3 & #4. I worry that this reflects and reinforces distorted beliefs the community has about itself, by making it sound like #3 & #4 are the more desirable traits. This worries me for two reasons:

First, current EA ideas, paradigms, and conclusions are bound to be wrong in important ways, even if they’re importantly right in some other ways.[1] According to me, the community is currently too far in the direction of encouraging conformity and convergence, and should move substantially towards encouraging exploration, questioning, and paradigm-busting. Glorifying “hard-core” EAs who buy into all the existing EA frameworks shuts us off from being curious about what we've gotten wrong and where other people or communities might be right. We should be looking out for people trying to figure this out in a serious way and getting excited when they reach different conclusions from us, not deriding them for not being fully part of our club.

Second, planning to work yourself to the bone for the rest of your life is unlikely to be the highest impact strategy. Comments that really hard-core people don’t think it’s important to pace yourself or have a life outside EA (like this tweet, quoting a speaker at EA Global 2021) make me despair for our ability to make the world a better place.

It really shouldn’t be controversial to say that (a) some people can work incredibly hard and be productive the whole time, (b) many others need different rhythms, and (c) everyone should figure out what works for them—including sometimes pushing yourself to test and expand your limit...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Omicron Post #12, published by Zvi on January 4, 2022 on LessWrong.

It involved a backlog, but the USA reported over 1 million cases yesterday. That’s a lot of cases, and we’re missing a ton more of them. Schools, hospitals and all businesses face massive disruptions from people unable to work, even with the new CDC isolation guidelines.

So far, we’re holding up well, better than I expected even after previous severity estimates were adjusted. Things are going to be severely disrupted for a while, but not as severely as we might have feared, and even if we get a median outcome that is much better than the mean outcome given the tail risk of true disaster.

Still, brace yourself, and choose how you want to handle the next month or two. If you’re serious about trying to avoid Omicron, now is the time to show it.

Vaccine Effectiveness

A Danish study from earlier found large negative vaccine effectiveness. Large negative effectiveness is not a thing, we would know about it and the world would look very different, but anti-vax people are of course jumping on it and the attempt to explain isn’t going all that well. Which it mostly shouldn’t, in my view, because when you get nonsense results that are clearly nonsense results but you don’t control properly and so you publish anyway, what else do you expect?

I looked at the paper. They control for the basics, including geographical region.

VE was calculated as 1-HR with HR (hazard ratio) estimated in a Cox regression model adjusted for age, sex and geographical region, and using calendar time as the underlying time scale.

I’ve heard reasons that make it possible for vaccine effectiveness to go negative, but those reasons don’t seem compatible with large positive effectiveness shortly after two doses, and large positive effectiveness again after a booster shot.

More than that, check the sections on severity and hospitalization below. They make it very clear that being vaccinated offers a lot of effective protection and is very much a good idea, and getting boosted is an even better idea.

Treatment

CDC lifts the pause on anti-Delta monoclonal antibodies, says they can be used if Delta is still in your region and there aren’t other options available. My guess is there are still some places in America where this makes sense, since even a small chance of getting the old effectiveness is better than doing nothing, but this window is ending rapidly.

Quarantine, Isolation and Travel

Instead of ending all travel-based quarantines because they no longer make any sense, many new travel restrictions are being imposed.

This is deeply stupid. It makes sense to make someone quarantine if they are at much higher risk than the surrounding population, and there’s some hope of stopping the spread. Two places having mutual travel restrictions usually means both are making a mistake. These quarantines are quite expensive (as are the required PCR tests, which given the need for reliable results often cost hundreds).

What prompted this was that I noticed that France added the United States to their ‘red list’, including the first comment as a ‘same energy’ to indicate the mindset.

France has twice as many cases per capita as the United States right now, and about two thirds of the death rate.

At this point, quarantines even for people with large known exposures seem questionable. If you learn you’ve been exposed to Covid-19, you have not received that many bits of information, and it’s not clear how much behavior change is justified. Not zero, especially if you’re going to potentially see vulnerable people, but if you used the old thresholds for quarantine you would (quite literally) never leave the house.

At least it’s better than universal isolation, such as the curfews in Quebec:

Whereas Princeton is telling its students they can’t leave the coun...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Promising posts on AF that have fallen through the cracks, published by Evan R. Murphy on January 4, 2022 on The AI Alignment Forum.

Yesterday on the Weekly Alignment Research Coffee Time call as people were sharing updates on their recent work, Vanessa Kosoy brought up her and Diffractor's post Infra-Bayesian physicalism: a formal theory of naturalized induction which she was interested in getting feedback on.

Vanessa seemed a bit disappointed/frustrated that this post had received no comments yet. I had to agree after learning that it proposes the first formal decision theory realizing naturalized induction, which has been an open problem in alignment for years. It's also been over 5 weeks since she posted it.

So why hasn't anyone commented? Well it's possible folks have just been busy since it's been the end of the year and the holidays. It also may have to do with that the post includes accompanying proofs and invokes some complex-looking maths; such papers take more time to digest and it's only a subset of people on this forum and LessWrong who are capable of doing so.

But it's also possible that the post was forgotten. Even though it may be important, since it's no longer on the front page of the Alignment Forum and hasn't been referenced anywhere else, nobody would think to look at it again unless it happened to come up in one of their searches or they were prompted about it. In other words, it may have just "fallen through the cracks".

This got me wondering about what other posts on the Alignment Forum might have fallen through the cracks. I did a quick search on the forum to see if I could find out. My search criteria was the following:

A post received substantial upvotes (arbitrarily 20+ points), suggesting that the community thought it was somewhat interesting or valuable

But it received no comments or pingbacks on AF - It may still have received comments on LessWrong though or comments/pingbacks by the original authors

It was posted more than 2 weeks ago

It's an original "researchey" post - I filtered out posts that were event announcements, linkposts, etc.

Here are the posts that came up in my search. I only went through about ~200 posts for this so consider it exemplary rather than exhaustive:

Infra-Bayesian physicalism: a formal theory of naturalized induction

Jitters No Evidence of Stupidity in RL

Sources of intuitions and data on AGI

How DeepMind's Generally Capable Agents Were Trained

To be sure, I don't expect all of these posts to necessarily be super important and in dire need of comments. There could be other reasons a post met these criteria, for example that people liked it but it was relatively straightforward and self-contained.

But if there's even a 10% chance that a post was forgotten and that its author is blocked or slowed in their research progress by not receiving any feedback, it seems like an easy win for the community for us to periodically resurface such posts. Authors may be too shy or concerned about looking self-promotional to repost their own work.

Do you know of other posts that you think are important and could benefit from feedback but didn't receive any or enough? This can include your own work. Mention them in the comments below.

I expect this problem to only get worse as the Alignment Forum grows. So I wanted to raise this issue and start a conversation about it. I or someone else could periodically do a post like this to help resurface posts that have fallen through the cracks. Or perhaps there are other more systematic ways we could address this problem as the community scales.

Note: This may remind you of a project that's underway to provide peer review for posts on the Alignment Forum. I consider this to be related but distinct from what I'm talking about here. In that project, Adam, Joe and Jérémy are providing in-de...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: D𝜋's Spiking Network, published by lsusr on January 4, 2022 on LessWrong.

Update: The algorithm performs poorly when applied to data even slightly more complciated than MNIST.

D𝜋 posted an interesting article full of original research Self-Organised Neural Networks: A simple, natural and efficient way to intelligence. D𝜋 describes his invention as a "message in a bottle".

Is it real? I don't know yet but I have publicly registered my belief that there exist simple algorithms which are vastly superior to the multilayer perception. D𝜋's algorithm lit up my heuristics as something worth looking into. D𝜋's open source code compiles, run and claims[1] to perform well on the standard permutation invariant dataset PI-MINST. D𝜋's work is precise enough to be falsifiable.

[A]t least a wild goose chase gives you some exercise.

―In the Beginning was the Command Line by Neal Stephenson

This article is my attempt to communicate D𝜋's theory in my own words. I have applied my own names to many of D𝜋's ideas. These names are not yet fixed. I may change them.

Spiking Networks

D𝜋 calls his invention a Self-Organized Neural Network (SONN). I prefer to call it a Spiking Neural Network (SNN) or spiking network for short. In this article, I will use the term "spiking network" to refer exclusively to D𝜋's invention. (I ignore the fact that other people have been working on SNNs for decades.)

Most artificial neural networks (ANNs) in use today are based on the multilayer perceptron. The multilayer perceptron is designed to handle continuous data. You can feed discrete data into a multilayer perceptron by embedding it into a vector space. A spiking network works the opposite way. It accepts only binary values. If you want to feed continuous data into a spiking network you need to discretize it into binary values first. D𝜋 buckets MINST's greyscale training data into four shades of grey per pixel. Our system thus has four receptors per pixel. Exactly one receptor activates per pixel per image.

MINST's dataset contains images of handwritten digits. There are ten digits 0123456789. D𝜋 creates ten groups of neurons. He calls them "groups". I will call them columns instead. D𝜋 creates one column of neurons for each digit.

D𝜋's spiking network as presented utilizes no hidden layers. Neurons do not read each other. D𝜋's ten columns of neurons are connected directly to the receptors. It may be possible to improve D𝜋's spiking network by adding hidden layers. We will not do so in this article. D𝜋's spiking network performs adequately on MINST without hidden layers.

Each neuron is connected to "a few" receptors of input data. Each connection between a neuron and a receptor has a weight. When a receptor is on, the weight associated with its connections are added to the potential of its respective neuron. Since receptors are binary, no multiplication is necessary.

The potential of a neuron is a non-negative number representing the neuron's internal state. Neuron potentials start at zero. When a neuron's potential passes a threshold value, the neuron spikes and the neuron's potential is reset to zero. Our system counts spikes.

To classify a digit, we first preprocess the pixels into receptor activations. We use the receptor activations to determine which weights to add to the neuron potentials. Some of the neurons spike[2]. We count the spikes in each column. Whichever column has the most spikes is our prediction.

Learning

The spiking network learns in two ways.

The spiking network can adjust the weights of the active connections. We reinforce a column by adding a fixed value to all of its active connections. We negatively reinforce a column by subtracting a fixed value from all of tis active connections. We may, optionally, add a maximum absolute value of each connection beyond which it cannot increase....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [New Feature] Support for Footnotes!!, published by Ruby on January 4, 2022 on LessWrong.

It is with great excitement that I am pleased to announce that the main LessWrong text editor now has support for footnotes! A huge thanks to our friends over at the Effective Altruism Forum who coded this one up.

You can insert footnotes via:

1. Manually selecting text in the text box and selecting insert footnote from the footnotes menu icon.

2. Using Markdown syntax

Type [^n] where is the number of the footnote you wish to insert.

To insert a new footnote, use n that is ; to reuse an existing footnote, set n to be whichever footnote you are reusing.

Footnotes will automatically renumber as you add and delete them!

What's more, footnotes will render with hover-over previews once published:

That's it. Go forth and create scholarly works!

I mean it, really. I've looked forward to us adding this support for years.

That is, the LW Docs editor, as distinct from the Markdown editor and legacy Draft-JS editor.

The Markdown editor already had support for footnotes using Markdown footnote syntax.

Yes, footnotes can have footnotes. And those footnotes can reference themselves.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Observation of Vavilov Day, published by Elizabeth on January 3, 2022 on LessWrong.

I aspire to be a person who does good things, and who is capable of doing hard things in service of that. This is a plan to test that capacity.

I haven’t been in a battle, but if you gave me the choice between dying in battle and slowly starving to death, I would immediately choose battle. Battles are scary but they are short and then they are over.

If you gave me a chance to starve to death to generate some sufficiently good outcome, like saving millions of people from starvation, I think I would do it, and I would be glad to have the opportunity. It would hurt, but only for a few weeks, and in that time I could comfort myself with the warm glow of how good this was for other people.

If you gave me a chance to save millions of people by starving, and then put food in front of me, I don’t think I could do it. I would do okay for a few days, maybe a week, but I worry that eventually hunger would incapacitate the part of my brain that allows me to make moral trade-offs at my own expense, and I would wake up to find I’d eaten half the food. I want to think I’d manage it, but if the thought experiment gods didn’t let me skip the hard part with more proactive measures, I’m not confident I could.

During the siege of Leningrad, scientists and other staff of the Institute of Plant Study faced the above choice, and to the best of our knowledge, all of them chose hunger. 12 of them died for it, the rest merely got close (English language sources list 9 deaths, which is the number of scientists who died in service of the seed bank but not the total number of people). They couldn’t kill themselves because they were needed to protect the food from rats and starving citizens. Those survival odds are better than the certain death of my hypothetical, but they didn’t have the same certainty of impact either, so I think it balances out.

That’s heroism enough, but a fraction of what’s present in this story. Those scientists worked at an institute founded by Nikolai Vavilov, a Soviet botanist who has the misfortune to be right on issues inconvenient to Joseph Stalin. Vavilov’s (correct) insistence that his theories could feed Russians and those of Stalin’s favored scientist couldn’t got him arrested, tortured, and sent to a gulag, where he eventually starved to death.

The seeds Vavilov and his staff protected now cover 80% of the cropland of Russia. Credit for scientific revolutions is hard to apportion, but as I reckon it Valilov is responsible for, at a minimum, tens of millions people living when they would have starved or never born, and the number could be closer to a billion.

Nikolai Vavilov is my hero.

In honor of Nikolai Vavilov, I’m doing a ~36 hour calorie fast from dinner on 1/25 (the day before Vavilov died in the gulag) to breakfast on 1/27 (the end of the siege of Leningrad). Those of you who know me know this is an extremely big deal for me, I do not handle being hungry well, and 36 hours is a long time. This might be one of the hardest things I could do while still being physically possible. Moreover, I’m not going to allow myself to just lie in bed for this: I’m committing to at least one physical activity that day (default is outdoor elliptical, unless it’s raining), and attempting to work a normal schedule. I expect this to be very hard. But I need to demonstrate to myself that I can do things that are at least this hard, before I’m called on to do so for something that matters.

If this story strikes a chord with you to the point you also want to observe Valilov + associates’ sacrifice, I’d enjoy hearing how. I have enough interest locally (bay area California) that there’s likely to be a kick-off dinner + reading the night of the 25th. It would also be traditional for a fasting hol...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Apply for research internships at ARC!, published by Paul Christiano on January 3, 2022 on The AI Alignment Forum.

In addition to hiring full-time researchers, ARC is also hiring for 1-3 month internships. We’re open to interns for summer 2022 in particular, but if a different time works for you that’s fine too.

Internships are appropriate for undergraduates and graduate students, or for anyone considering a career change.

To apply for an internship, you can use this application and check the “internship” box. If you submitted an application previously there’s no need to resubmit; we’ll clarify if there is ambiguity.

Salary for interns is $15k/month. We encourage interns to work from our Berkeley office especially at the start of their internship.

(The rest of this post is copied from our previous hiring post.)

What is ARC?

ARC is a non-profit organization focused on theoretical research to align future machine learning systems with human interests. We are aiming to develop alignment strategies that would continue to work regardless of how far we scaled up ML or how ML models end up working internally.

Probably the best way to understand our work is to read Eliciting Latent Knowledge, a report describing some recent and upcoming research, which illustrates our general methodology.

We currently have 2 research staff (Paul Christiano and Mark Xu). We’re aiming to hire another 1-2 researchers in early 2022. ARC is a new organization and is hoping to grow significantly over the next few years, so early hires will play a key role in helping define and scale up our research.

Who should apply?

Most of all, you should send in an application if you feel excited about proposing the kinds of algorithms and counterexamples described in our report on ELK.

We’re open to anyone who is excited about working on alignment even if you don't yet have any research background (or your research is in another field). You may be an especially good fit if you:

Are creative and generative (e.g. you may already have some ideas for potential strategies or counterexamples for ELK, even if they don't work).

Have experience designing algorithms, proving theorems, or formalizing concepts.

Have a broad base of knowledge in mathematics and computer science (we often draw test cases and counterexamples from these fields).

Have thought a lot about the AI alignment problem, especially in the limit of very powerful AI systems.

Hiring will be a priority for us in early 2022 and we don't mind reading a lot of applications, so please err on the side of sending in an application even if you’re not sure you’ll be a fit!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.