Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Fish Welfare Initiative: The 2.5 Year Retrospective, published by haven on December 18, 2021 on The Effective Altruism Forum.

In the time-honored tradition of EA organizations writing EA Forum updates, we wanted to share such an update on Fish Welfare Initiative (FWI), 2.5 years after launching.

FWI is a product of the Effective Altruism (EA) community: Our founders were inspired by EA ideas in their university groups, we were incubated by Charity Entrepreneurship, and most of our support since, financial and advisory, has come from people in the EA community. For these we are immensely grateful, and we thus want to update the community on the project your support has built and the impact it has enabled.

Below we walk through a fairly comprehensive overview of what Fish Welfare Initiative is doing, what we’ve accomplished, what’s holding us back, and what we’ve learned. As the post is fairly long, you may want to skim to the sections that most interest you.

We believe this post will be most useful to the following people:

Those interested in charity entrepreneurship, particularly in animal welfare or in lower and middle income countries

Those interested in keeping up to date with FWI

Those interested in seeing what sort of projects the EA community can create and enable

If you have feedback or questions about our work, please leave them below!

Note: This post does not discuss in detail the welfare issues associated with fish farming, nor does it discuss why we think this is a particularly important problem area. For more on these, see our initial Forum post and our India Scoping Report.

Summary

Fish Welfare Initiative launched in 2019, via the Charity Entrepreneurship Incubation Program. Our focus is improving the lives of farmed fish.

Our primary country of operation is India, with smaller projects in China, the Philippines, Portugal, and the UK.

In India, our current main program is working with farmers to implement welfare improvements. We are also conducting earlier stage corporate outreach and policy work.

We estimate we have improved the lives of 214,000 fish thus far.

Our key bottleneck right now is improving our welfare standard. Based on our water quality data collection, our current welfare standard does not seem sufficiently impactful to warrant scaling it up significantly. Improving our welfare standard, along with early stage corporate outreach and governmental work, is our primary focus for 2022.

The main ways you can support FWI are by advising us, connecting us with relevant people, and donating.

Current Status of FWI’s Work

In Brief

Main country of operation: India

Current staff size: 13 full-time-equivalent (9 employed or contracted in India)

Launched in: 2019

Launched via: Charity Entrepreneurship

Registered entities in: USA, India

2020 annual budget: $150K

2021 annual budget: $300K

2022 annual budget: $600K

Other documents: See transparency page.

Work in India

Why India? The following are the reasons we believe that India is a particularly promising country for fish welfare work:

Scale: India has the second largest population of farmed fish in the world (after China).

Hireability: Compared to the other countries we visited, we found it easier to find talented hires in India (although this is still a bottleneck—see Key Bottlenecks below).

Animal movement presence: Though little work has been done for fish in India, India does have one of the most well-established animal protection movements in the world. This means there are many experienced people we can consult on our work.

English speaking: English is widespread.

Our work in India began in the fall of 2020, with the hiring of our Managing Director Karthik Pulugurtha.

FWI engages in three program areas in India. At this stage, all are somewhat exploratory, and we expect in the next few years to c...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Occupational Infohazards, published by jessicata on December 18, 2021 on LessWrong.

[content warning: discussion of severe mental health problems and terrifying thought experiments]

This is a follow-up to my recent post discussing my experience at and around MIRI and CFAR. It is in part a response to criticism of the post, especially Scott Alexander's comment, which claimed to offer important information I'd left out about what actually caused my mental health problems, specifically that they were caused by Michael Vassar. Before Scott's comment, the post was above +200; at the time of writing it's at +61 and Scott's comment is at +382. So it seems like people felt like Scott's comment discredited me and was more valuable than the original post. People including Eliezer Yudkowsky said it was a large oversight, to the point of being misleading, for my post not to include this information. If I apply the principle of charity to these comments and reactions to them, I infer that people think that the actual causes of my psychosis are important.

I hope that at least some people who expressed concern about the causes of people's psychoses will act on that concern by, among other things, reading and thinking about witness accounts like this one.

Summary of core claims

Since many people won't read the whole post, and to make the rest of the post easier to read, I'll summarize the core claims:

As a MIRI employee I was coerced into a frame where I was extremely powerful and likely to by-default cause immense damage with this power, and therefore potentially responsible for astronomical amounts of harm. I was discouraged from engaging with people who had criticisms of this frame, and had reason to fear for my life if I published some criticisms of it. Because of this and other important contributing factors, I took this frame more seriously than I ought to have and eventually developed psychotic delusions of, among other things, starting World War 3 and creating hell. Later, I discovered that others in similar situations killed themselves and that there were distributed attempts to cover up the causes of their deaths.

In more detail:

Multiple people in the communities I am describing have died of suicide in the past few years. Many others have worked to conceal the circumstances of their deaths due to infohazard concerns. I am concerned that in my case as well, people will not really investigate the circumstances that made my death more likely, and will discourage others from investigating, but will continue to make strong moral judgments about the situation anyway.

My official job responsibilities as a researcher at MIRI included thinking seriously about hypothetical scenarios, including the possibility that someone might cause a future artificial intelligence to torture astronomical numbers of people. While we considered such a scenario unlikely, it was considered bad enough if it happened to be relevant to our decision-making framework. My psychotic break in which I imagined myself creating hell was a natural extension of this line of thought.

Scott asserts that Michael Vassar thinks "regular society is infinitely corrupt and conformist and traumatizing". This is hyperbolic (infinite corruption would leave nothing to steal) but Michael and I do believe that the problems I experienced at MIRI and CFAR were not unique or unusually severe for people in the professional-managerial class. By the law of excluded middle, the only possible alternative hypothesis is that the problems I experienced at MIRI and CFAR were unique or at least unusually severe, significantly worse than companies like Google for employees' mental well-being.

Scott asserts that Michael Vassar thinks people need to "jailbreak" themselves using psychedelics and tough conversations. Michael does not often use ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: In Defense of Attempting Hard Things, and my story of the Leverage ecosystem, published by Cathleen on December 17, 2021 on LessWrong.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Disentangling Perspectives On Strategy-Stealing in AI Safety, published by shawnghu on December 18, 2021 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

Additional thanks to Ameya Prabhu and Callum McDougall for their thoughts and feedback on this post.

Introduction

I’ve seen that in various posts people will make an offhanded reference to “strategy-stealing” or “the strategy-stealing assumption” without very clearly defining what they mean by “strategy-stealing”. Part of the trouble with this is that these posts are often of wildly different flavors, and it’s often not clear what their connection to each other is, or perhaps it’s unclear under what conditions it might be feasible to “steal the strategy” of an AI, or sometimes it’s unclear where there is any notion of doing any stealing of strategies at any point.

Spoiler upfront, part of the reason for this is that the term comes from a relatively esoteric game-theoretic concept that basically never applies in a direct sense to the scenarios in which it is name-dropped. In this post, I’ll try to explain the common use of this seemingly-inappropriate term and why people are using it. In order to do this, I’ll first attempt to develop a more general framework for thinking about strategy-stealing, and then I’ll try to extend the intuition that develops to bridge the gap between a couple of seemingly highly disparate posts. Along the way I’ll attempt to clarify the connection to a few other ideas in alignment, and clarify what implications the strategy-stealing framework might have for the direction of other alignment research.

I’ll declare up front that I think that in an AI safety context, the strategy-stealing framework is best thought of as a tool for managing certain intuitions about competition; rhetorically the goal of my post is to distill out those aspects of “game-theoretic” strategy-stealing that are broadly applicable to alignment so that we can mostly throw out the rest.

As an aside, I think that due to the (very relatable) confusion I’ve seen elsewhere, strategy-stealing is not actually a good name for the concept in an AI safety context, but the term is already in use and I don’t have any ideas for any better ones, so I’ll continue to just use the term strategy-stealing even in contexts where no strategies are ever stolen. My hope is that I do a good enough job of unifying the existing uses of the term to avoid a proliferation of definitions.

Strategy Stealing in Game Theory

The strategy-stealing argument page on Wikipedia has a very good short explanation of the term, reproduced here: In combinatorial game theory, the strategy-stealing argument is a general argument that shows, for many two-player games, that the second player cannot have a guaranteed winning strategy. The strategy-stealing argument applies to any symmetric game (one in which either player has the same set of available moves with the same results, so that the first player can "use" the second player's strategy) in which an extra move can never be a disadvantage.

The argument works by obtaining a contradiction. A winning strategy is assumed to exist for the second player, who is using it. But then, roughly speaking, after making an arbitrary first move – which by the conditions above is not a disadvantage – the first player may then also play according to this winning strategy. The result is that both players are guaranteed to win – which is absurd, thus contradicting the assumption that such a strategy exists.

Although they were made pretty clear, note explicitly the following assumptions and limitations of the argument:

The game is a two-player, turn-based game [1].

The game is symmetr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Six Takeaways from EA Global and EA Retreats, published by Akash on December 16, 2021 on The Effective Altruism Forum.

I attended EAG London, as well as four retreats for student group organizers (these were focused mostly on community building & how to make strong university groups).

In this post, I summarize some of my takeaways and lessons learned. I also offer a few suggestions, heuristics, and action items for those who want to put some of these ideas into practice.

Here are my takeaways:

Think Bigger

Take More [Calculated] Risks

Be More Entrepreneurial and Agentic (Do Things)

Apply for Funding

Value Your Time

Reach out to EAs

Takeaway #1: Think bigger

In the early 2010s, EA was an extremely young movement with some idealistic thinkers. In 2021, it is still a very young movement, and it still has idealistic thinkers, but it also has a lot more energy and funding.

We can think bigger, grander, and more ambitiously about how to improve the present

and the future. There is a particular interest in megaprojects— highly impactful projects that could scale widely, even if they require large amounts of funding.

Okay, thinking bigger and grander sounds good in theory, but how do we actually do it? Here are some heuristics that I’ve been using & that others have recommended:

For any project idea, ask yourself “how could this be 10X or 100X bigger or more impactful?”

For any project idea, ask yourself “how could this be better if I had 10X or 100X more funding?

Imagine that this project or career went really, really well. What happened? What does the 90th or 99th percentile version of this look like? (Or, more simply, what does the best-case scenario look like?)

Spend 10 minutes brainstorming ideas– including ideas that seem crazy or wild– and then spend 10 minutes thinking about how each idea can be implemented (see also these LessWrong posts about babbling/pruning).

Ask yourself “if something came along that solved this entire problem, what would it be?”

Example: Sydney V founded Uncommon Sense, a summer program in which high school students learn about rationality and longtermism. The program reached about 20 students in its first year. Sydney’s first reaction was to think “ah, great, next year, we can do a modified version of Uncommon Sense and expand to about 30 students.”

Then, after being prompted by other EAs, she began to think more ambitiously about her plan to scale, and she asked herself how the project could be much more impactful. She realized that she could work with other EAs to turn Uncommon Sense into a large program that could potentially receive thousands of applications and hundreds of attendees. She’s now working on this (highly ambitious) high school outreach project.

Takeaway #2: Take more (calculated) risks

TLDR: Consider upsides and don’t be overly afraid of [specific types of] failure. (see also this 80k post about ambition & risks). Note: Many points in this section come from, or are inspired by, a talk by Jonas Vollmer.

Consider Upsides

Here’s a conversation that I have heard in my head, and I’ve also heard from other EAs:

Me: Here’s an idea!

Risk-averse me: That’s a cool idea. But here are a few ways it can go wrong. Also, you haven’t done something like [idea] before. Also, base rates for [idea] succeeding are low. And [idea] could even cause harm—especially if [bad thing #1] happens or [bad thing #2] happens.

Me: Hm, yeah. Good points. I should probably think of a different idea.

The problem is that there’s a crucial step that’s missing. Many people find it more

natural to think of downsides (how could this go wrong?) than upsides (how could this go right?). But when estimating the expected value of a project, it’s important to weigh the upsides against the downsides.

This idea is deceptively simple and seems obvious when spelled out in wri...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reviews of "Is power-seeking AI an existential risk?", published by Joe Carlsmith on December 16, 2021 on The Effective Altruism Forum.

Open Philanthropy solicited reviews of my draft report “Is power-seeking AI an existential risk?” from various sources. Where the reviewers allowed us to make their comments public in this format, links to these comments are below, along with some responses from me in blue.

Leopold Aschenbrenner

Ben Garfinkel

Daniel Kokotajlo

Neel Nanda

Nate Soares

Christian Tarsney

David Thorstad

David Wallace

Anonymous 1 (software engineer at AI research team)

Anonymous 2 (academic computer scientist)

The table below (spreadsheet link here) summarizes each reviewer’s probabilities and key objections.

An academic economist focused on AI also provided a review, but they declined to make it public in this format.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing the Principles of Intelligent Behaviour in Biological and Social Systems (PIBBSS) Fellowship, published by Adam Shimi on December 18, 2021 on The AI Alignment Forum.

Cross-posted to EA Forum

Introduction

How would you go about scientifically studying aliens? Arik Kershenbaum’s The Zoologist Guide to The Galaxy proposes to use evolutionary thinking to uncover constraints on how alien species could evolve. One of his most interesting points is that evolution constrains function far more than form, because function depends significantly less on the details of the environment. Hence we should expect crisper answers to “How would aliens behave?” than “What would aliens look like?”. And in the course of his book, he gives the best answer he can find to the former question.

So when confronted with the question of how to study something he couldn’t gather data on, Kershenbaum leveraged analogies to biological systems he could and had studied, and the underlying constraints brought on by the mechanisms of natural selection.

On a completely unrelated note, the new summer fellowship Principles of Intelligent Behavior in Biological and Social Systems (PIBBSS) (funded by the LTFF) aims at creating valuable AI alignment research through studying analogies to many complex systems (evolution, brains, language, social structures.). Fellows will have graduate research experience in fields studying such systems, working on a concrete alignment project in collaboration with an established alignment researcher. The fellowship will run during all of Summer 2022.

The point of this post is to introduce this fellowship, explain the reasoning behind it and give more concrete details about how it will go. Note that I’m not an organizer of this fellowship, I’m just assisting with the writing of this post; credits for the ideas and arguments should go to Nora Ammann and TJ, the organizers of the fellowship.

Analogies as General Epistemic Strategies for Alignment

As I’ve written elsewhere, alignment cannot directly leverage most epistemic strategies and approaches used in Science and Engineering, because it’s about solving a problem that doesn’t exist yet on a technology we still have to invent.

One epistemic strategy that survives this major problem is the leveraging of analogies with existing biological or social systems that implement complex or intelligent behavior. Consider how a wide variety of such systems (from biology, physics, linguistics and other fields) exhibit similar properties: adaptation, robustness, goal-directed behavior, learning, embeddedness, modularity, phase transitions, and more. Since AI research focuses on mechanisms that lead to complex and intelligent behavior with many of these properties, careful analogies with these complex systems may allow us to transfer knowledge about all these behaviors and properties to the study of alignment and AGI.

Also note that the other main epistemic strategy used in alignment, figuring things out from first principles, can and often does take inspiration from other existing systems like evolution, brains and languages.

Examples of Successful Analogies in Alignment

If analogical thinking is such a valuable epistemic approach to alignment, we should find ample examples of valuable alignment research using such analogies. And that’s indeed what we see.

Risks from Learned Optimization by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant explicitly uses biological evolution of humans as an example of/inspiration for mesa-optimizers. Evolution here is the search process, with a base-objective of increasing fitness and propagation of genes, but the learned models (humans) end up doing search for different goals.

What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs) by ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Case for Radical Optimism about Interpretability, published by Quintin Pope on December 16, 2021 on LessWrong.

Summary:

I argue we can probably build ML systems that are both high-performance and much more interpretable than current systems.

I discuss some reasons why current ML interpretability lags behind what should be possible.

I highlight some interesting interpretability work on current systems.

I propose a metric for estimating the interpretability of current systems and discuss how we might improve interpretability.

Definitional note: for this essay, "interpretable" serves as a catch-all for things like "we can abstractly understand the algorithms a system implements" and "we can modify internal states to usefully influence behavior in a predictable manner". Systems are more interpretable when it's easier to do things like that to them.

1: Strong, interpretable systems are possible

Imagine an alien gave you a collection of AI agents. The agents vary greatly in capability, from about as smart as a rat to human level (but not beyond). You don't know any details about the AI's internal algorithms, only that they use something sort of like neural nets. The alien follows completely inhuman design principles, and has put no effort whatsoever into making the AIs interpretable.

You are part of a larger community studying the alien AIs, and you have a few decades to tackle the problem. However, you operate under several additional constraints:

Computation can run both forwards and backwards through the AIs (or in loops). I.e., there's no neat sequential dependency of future layers on past layers.

Each AI is unique, and was individually trained on different datasets (which are now unavailable).

The AIs are always running. You can't copy, backup, pause or re-run them on different inputs.

Each AI is physically instantiated in its own hardware, with no pre-existing digital interface to their internal states.

If you want access to internal states, you'll need to physically open the AIs up and insert electrodes directly into their hardware. This is delicate, and you also risk permanently breaking or damaging the AI with each "operation".

Your electrodes are much larger than the AI circuits, so you can only ever get or influence a sort of average activation of the surrounding neurons/circuits.

You are limited to 10 or fewer probe electrodes per AI.

You are limited to the technology of the 1960s. This includes total ignorance of all advances in machine learning and statistics made since then.

Please take a moment to predict how far your interpretability research would progress.

Would you predict any of the following?

That you'd be able to find locations in the AI's hardware which robustly correspond to the AI's internal reward signal.

That these locations would be fairly consistent across different AIs.

That running a current through these regions would directly increase the AIs' perceived reward.

That the location of the electrodes would influence the nature and intensity of the reward-seeking behavior produced.

If you base your predictions on how little progress we've made in ML interpretability, these advances seem very unlikely, especially given the constraints described.

And yet, in 1953, James Olds and Peter Milner discovered that running a current through certain regions in rats' brains was rewarding and influenced their behavior. In 1963, Robert Heath showed similar results in humans. Brain reward stimulation works in every vertebrate so far tested. Since the 1960s, we've had more advances in brain interpretability:

A recent paper was able to reversibly disable conditioned fear responses from mice.

We’re able to cure some instances of treatment resistant depression by stimulating select neurons.

We can (badly) reconstruct images from brain activity.

fMRI-based lie...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Perishable Knowledge, published by lsusr on December 18, 2021 on LessWrong.

Some information matters. Most information doesn't. Learning things that matter makes you smarter. Knowledge that doesn't matter is called trivia.

There is no absolute reference point for intelligence. Being smart is a competition against your age cohort. Trivia is the junk food of information. If you fill your head with trivia that means you're not filling your head with useful information.

Knowledge has a shelf life. Useful knowledge becomes trivia by going out-of-date. In theory, there is a spectrum between trivia and enduring wisdom. In practice, the distinction between trivia and enduring wisdom is binary because knowledge is distributed on a log scale. The shelf live of knowledge tends to be either longer than a human lifetime or much shorter than a human lifetime.

History tends to matter because history is curated. Hearing about something that happened 200 years ago is evidence that the the importance of the event has stood the test of time. Facts with a shelf life much shorter than a human lifetime constitute trivia.

How do you determine the shelf life of a fact?

Humans are mortals. The older we get the more likely to die[1]. According to Lindy's Law, immortals (like vampires) become less likely to die (in any particular year) the older they are. A country or a language that has been around for a thousand years is more likely to persist for for another thousand years than a country or language that was invented yesterday. The life-expectancy of a non-perishable thing is proportional to its age.

I like Linux/Unix because it so rarely changes. Unix was invented in 1969. It is 52 years old. I started learning Linux nearly ten years ago. Everything I have ever learned about Linux remains useful to this day. My knowledge of Linux grows monotonically. I expect my knowledge of Linux to remain relevant for as long as I can program computers.

I dislike Android/iOS development because it changes frequently. I first programmed an iPhone in 2015, around the time Apple released the Swift programming language. Swift is a superior programming language to Objective-C for programming most software on a modern iPhone. My older co-workers who learned to program in Objective-C had decades of accumulated knowledge rendered obsolete.

Apple changes their development toolchain so often that half my time was spent keeping my app up-to-date with Apple's rapidly-changing development environment. My knowledge of Apple software went obsolete as fast as I could learn it. The only durable fact I learned was "don't write native code for iOS".

Big companies changing development environments under their programmers feet goes all the way back to when Microsoft dominated the software industry.

The companies that do well are the ones who rely least on big companies and don’t have to spend all their cycles catching up and re-implementing and fixing bugs that crop up only on Windows XP.

―Fire and Motion by Joel Spolsky, published in 2002

Want to get off the treadmill of learning a new IDE every decade? Code in Vim (or emacs).

Junk Media

New information tends to have a short shelf life because of Lindy's Law. Avoiding transient information like news, trends, intellectual fashions and so on makes you smarter. But it's also important to respond rapidly to major events like COVID-19. How do you respond to events like COVID if you're ignoring the news?

I was aware of the threat a global pandemic for many years before COVID-19 existed. I started paying attention to COVID-19 when Wuhan was quarantined. I was keeping tabs on COVID-19 for months before it was a major news story in the USA.

Mass media has a record of misleading the public about where things are going. These were not failures of the system. These are examples of a pr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why do you find the Repugnant Conclusion repugnant?, published by willbradshaw on December 17, 2021 on The Effective Altruism Forum.

The Repugnant Conclusion has always seemed straightforwardly and unobjectionably true to me. I've always been confused by its alleged repugnance, or why such an anodyne-seeming conclusion merits such a dramatic name.

This isn't like the other standard objections to utilitarianism. I'm not persuaded by concerns about utility monsters or trolley problems, but I feel the sting of those objections – they feel like bullets I need to bite. Whereas the Repugnant Conclusion just seems like a non-problem to me.

I say all this not to argue against concerns about the Repugnant Conclusion, but to motivate my question here. I'd like to have a better understanding of the intuitions that lead people to seeing this as such a serious problem, and whether I'm missing something that might cause me to put more weight on these sorts of concerns. I'm less interested in technical philosophical arguments here than in intuition pumps – simple thought experiments, or real-world scenarios, or related problems that might help me feel the sting of the objections a bit more.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.