Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Linkpost for various recent essays on suffering-focused ethics, priorities, and more, published by Magnus Vinding on September 28, 2022 on The Effective Altruism Forum.

The following are (links to) various essays that I have published over the last few months. Some of the essays have been published on the website of the Center for Reducing Suffering (CRS), and some of them have been published on my own blog.

CRS essays

A phenomenological argument against a positive counterpart to suffering

Various views deny that suffering has a positive counterpart. Proponents of such views often pursue a line of argument that focuses on the prevalence of subtle frustrations and bothersome sensations. That is, when we typically think that we are in a neutral state, and we claim that some pleasure takes us above that neutral state, what we are experiencing is really a subtly bothered and unsatisfied state that becomes (somewhat) relieved of its commonly overlooked unpleasant features (see e.g. Sherman, 2017, pp. 103-107; Gloor, 2017, sec. 2.1; Knutsson, 2022, sec. 6).

This essay will pursue a different line of argument. Rather than focusing on unpleasant states, and arguing for their subtle omnipresence, my aim here is instead to zoom in on the purportedly positive side. I will argue that purportedly positive experiences do not possess any property that renders them genuine opposites of painful and uncomfortable experiences, neither in phenomenological nor axiological terms.

Reply to the “evolutionary asymmetry objection” against suffering-focused ethics

An objection that is sometimes raised against suffering-focused ethics is that our intuitions about the relative value of suffering and happiness are skewed toward the negative for evolutionary reasons, and hence we cannot trust our intuition that says that the reduction of suffering is more valuable and more morally important than the creation of happiness. My aim in this post is to reply to this objection.

Reply to the scope neglect objection against value lexicality

Some views hold that no amount of mild discomfort can be worse than a single instance of extreme suffering (i.e. they endorse value lexicality between extreme suffering and mild discomfort). An objection to such views is that they are biased by scope neglect — our tendency to disregard the number of affected beings in our evaluations of a problem. Since we cannot comprehend the badness of a vast amount of mild discomfort, the objection goes, we cannot trust our intuitive assessment that extreme suffering is worse than any amount of mild discomfort. My aim in this brief post is to reply to this objection.

Comments on Mogensen’s “The weight of suffering”

Andreas Mogensen’s paper “The weight of suffering” presents an interesting argument in favor of the axiological position that “there exists some depth of suffering that cannot be compensated for by any measure of well-being” — a position he calls “LTNU” (Mogensen, 2022, abstract). Mogensen then proceeds to explore how one might respond to that argument and thereby reject LTNU.

My aim in this post is to raise some critical points in response to this paper. As a preliminary note, I should say that I commend Mogensen for taking up this crucial issue regarding the weight of suffering, and for exploring it in an open-ended manner.

Reply to Chappell’s “Rethinking the Asymmetry”

My aim in this post is to respond to the arguments presented in Richard Yetter Chappell’s “Rethinking the Asymmetry”. Chappell argues against the Asymmetry in population ethics, which roughly holds that the addition of bad lives makes the world worse, whereas the addition of good lives does not make the world better (other things being equal).

A thought experiment that questions the moral importance of creating happy lives

Many people have the intuition...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I think strong general AI is coming soon, published by porby on September 28, 2022 on LessWrong.

I think there is little time left before someone builds AGI (median ~2030). Once upon a time, I didn't think this.

This post attempts to walk through some of the observations and insights that collapsed my estimates.

The core ideas are as follows:

We've already captured way too much of intelligence with way too little effort.

Everything points towards us capturing way more of intelligence with very little additional effort.

Trying to create a self-consistent worldview that handles all available evidence seems to force very weird conclusions.

Some notes up front

I wrote this post in response to the Future Fund's AI Worldview Prize. Financial incentives work, apparently! I wrote it with a slightly wider audience in mind and supply some background for people who aren't quite as familiar with the standard arguments.

I make a few predictions in this post. Unless otherwise noted, the predictions and their associated probabilities should be assumed to be conditioned on "the world remains at least remotely normal for the term of the prediction; the gameboard remains unflipped."

For the purposes of this post, when I use the term AGI, I mean the kind of AI with sufficient capability to make it a genuine threat to humanity's future or survival if it is misused or misaligned. This is slightly more strict than the definition in the Future Fund post, but I expect the difference between the two definitions to be small chronologically.

For the purposes of this post, when I refer to "intelligence," I mean stuff like complex problem solving that's useful for achieving goals. Consciousness, emotions, and qualia are not required for me to call a system "intelligent" here; I am defining it only in terms of capability.

Is the algorithm of intelligence easy?

A single invocation of GPT-3, or any large transformer, cannot run any algorithm internally that does not run in constant time complexity, because the model itself runs in constant time. It's a very large constant, but it is still a constant.

They don't have any learnable memory about their internal state from previous invocations. They just have the input stream. Despite all their capability, transformers are fundamentally limited.

This is part of the reason why asking GPT-3 to do integer division on large numbers in one shot doesn't work. GPT-3 is big enough to memorize a number of results, so adding small numbers isn't too hard even without fine tuning. And GPT-3 is big enough to encode a finite number of unrolled steps for more complex algorithms, so in principle, fine tuning it on a bunch of arithmetic could get you better performance on somewhat more complex tasks.

But no matter how much retraining you do, so long as you keep GPT-3's architecture the same, you will be able to find some arithmetic problem it can't do in one step because the numbers involved would require too many internal steps.

So, with that kind of limitation, obviously transformers fail to do basic tasks like checking whether a set of parentheses are balanced... Oh wait, GPT-3 was just writing dialogue for a character that didn't know how to balance parentheses, and then wrote the human's side of the dialogue correcting that character's error. And it writes stories with a little assistance with long-run consistency. And it can generate functioning code. And a bunch more. That's just GPT-3, from 2020.

Some of this is already productized.

This is an architecture that is provably incapable of internally dividing large integers, and it can handle a variety of difficult tasks that come uncomfortably close to human intuition.

Could the kind of intelligence we care about be algorithmically simpler than integer division?

This can't be literally true, if we want to inclu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 7 traps that (we think) new alignment researchers often fall into, published by Akash on September 27, 2022 on The Effective Altruism Forum.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why we're not founding a human-data-for-alignment org, published by LRudL on September 27, 2022 on LessWrong.

TL;DR

One-paragraph summary: we (two recent graduates) spent about half of the summer exploring the idea of starting an organisation producing custom human-generated datasets for AI alignment research. Most of our time was spent on customer interviews with alignment researchers to determine if they have a pressing need for such a service. We decided not to continue with this idea, because there doesn’t seem to be a human-generated data niche (unfilled by existing services like Surge) that alignment teams would want outsourced.

In more detail: The idea of a human datasets organisation was one of the winners of the Future Fund project ideas competition, still figures on their list of project ideas, and had been advocated before then by some people, including Beth Barnes. Even though we ended up deciding against, we think this was a reasonable and high-expected-value idea for these groups to advocate at the time.

Human-generated data is often needed for ML projects or benchmarks if a suitable dataset cannot be e.g. scraped from the web, or if human feedback is required. Alignment researchers conduct such ML experiments, but sometimes have different data requirements than standard capabilities researchers. As a result, it seemed plausible that there was some niche unfilled by the market to help alignment researchers solve problems related to human-generated datasets. In particular, we thought - and to some extent confirmed - that the most likely such niche is human data generation that requires particularly competent or high-skill humans. We will refer to this as high-skill (human) data.

We (Matt & Rudolf) went through an informal co-founder matching process along with four other people and were chosen as the co-founder pair to explore this idea. In line with standard startup advice, our first step was to explore whether or not there is a concrete current need for this product by conducting interviews with potential customers. We talked to about 15 alignment researchers, most of them selected on the basis of doing work that requires human data. A secondary goal of these interviews was to build better models for the future importance and role of human feedback in alignment.

Getting human-generated data does indeed cost many of these researchers significant time and effort. However, we think to a large extent this is because dealing with humans is inherently messy, rather than existing providers doing a bad job. Surge AI in particular seems to offer a pretty good and likely improving service. Furthermore, many companies have in-house data-gathering teams or are in the process of building them.

Hence we have decided to not further pursue this idea.

Other projects in the human data generation space may still be valuable, especially if the importance of human feedback in ML continues to increase, as we expect. This might include people specializing on human data as a career.

The types of factors that are most important for doing human dataset provision well include: high-skill contractors, fast iteration, and high bandwidth communication and shared understanding between the research team, the provider organisation and the contractors.

We are keen to hear other people’s thoughts, and would be happy to talk or to share more notes and thoughts with anyone interested in working on this idea or a similar one in the future.

Theory of Change

A major part of AI alignment research requires doing machine learning (ML) research, and ML research in turn requires training ML models. This involves expertise and execution ability in three broad categories: algorithms, compute, and data, the last of which is very neglected by EAs.

We expect training on data from human feedback to bec...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Book review: “The Heart of the Brain: The Hypothalamus and Its Hormones”, published by Steven Byrnes on September 27, 2022 on LessWrong.

1. Introduction

1.1 Hypothalamus as “business logic”

In software jargon, there’s a nice term “business logic”, for code like the following (made-up) excerpt from corporate tax filing software (based on here):

file.attachDocument("SR008-04X") file.attachDocument("SR008-04XI") if file.ledgerAmnt >= 500_000: # Ledger of 500K or more requires AUTHLDG-1A

file.attachDocument("AUTHLDG-1A")

When you think of “business logic”, think of stuff like that—i.e., parts of source code that more-or-less directly implement specific, real-world, functional requirements.

By contrast, things that are NOT business logic include infrastructure & subroutines & plumbing that are generally useful in many contexts—e.g. code for initializing a database, or code for memory management, or code for performing stochastic gradient descent.

If genomes are the “source code” of brains, then they need to encode “business logic” too—specific calculations to do specific things that help an animal thrive and reproduce in its particular biological niche. For example:

(We could also call these things “innate reactions”.)

Machine Learning people might interject here: Why does that have to be in the genome? Why can’t the brain derive those kinds of rules via a within-lifetime learning algorithm instead? Well, often it does! But:

Some things can’t be learned within a lifetime, because their evolutionary benefits are only apparent in hindsight, perhaps even many generations hence.

Other things can’t be learned within a lifetime, because not doing them, even once, is potentially fatal.

Still other bits of “business logic” involves setting up the animal’s within-lifetime learning algorithms. A particularly important example in this category is that animals learn (in part) by reinforcement learning, and reinforcement learning needs a reward function—the thing that says eating yummy food is good and touching a hot stove is bad, as opposed to the other way around. Calculating that reward function involves a lot of “business logic”.

Anyway, the genome puts much of the brain’s “business logic” into a part of the brain called the hypothalamus, the subject of this post. (I’ll walk through an example in Section 3 below.) The “business logic” doesn’t all go into the hypothalamus—the brainstem gets some too. But I believe that, apart from the hypothalamus and brainstem, the other 96% (!) of human brain volume has essentially no “business logic” at all, but rather is dedicated to running within-lifetime learning algorithms—see my earlier post “Learning from scratch” in the brain.

In his book, Gareth Leng poetically gives some examples of what the hypothalamus does for us:

When we are dehydrated, [the hypothalamus] makes us thirsty and tells our kidney not to waste water but to concentrate our urine. When we are ill, it raises our body temperature, generating a fever that kills off infections. When our blood sugar is low, it tells us to eat, and when our stomach is full, to stop. It determines the shape of our bodies, how tall we will grow, how fat we will be, and where our fat and muscle will grow. When we are frightened or anxious or stressed it determines whether we will set our teeth, stiffen our sinews, freeze, or fight—or flee. The rhythms of days and of seasons are beaten by its drums, and we grow and attain puberty under its tutelage. And, led by the hypothalamus, we preen and woo, and the sap rises in our loins; a man produces sperm, and in a woman the ovarian cycle turns; we court and mate and bond.

A woman conceives, and her body changes to the needs of the child within, and she gives birth and produces milk, and she loves and nurtures her children.

1.2 Neuroanatomy

Here’s the hyp...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why we're not founding a human-data-for-alignment org, published by LRudL on September 27, 2022 on The Effective Altruism Forum.

TL;DR

One-paragraph summary: we (two recent graduates) spent about half of the summer exploring the idea of starting an organisation producing custom human-generated datasets for AI alignment research. Most of our time was spent on customer interviews with alignment researchers to determine if they have a pressing need for such a service. We decided not to continue with this idea, because there doesn’t seem to be a human-generated data niche (unfilled by existing services like Surge) that alignment teams would want outsourced.

In more detail: The idea of a human datasets organisation was one of the winners of the Future Fund project ideas competition, still figures on their list of project ideas, and had been advocated before then by some people, including Beth Barnes. Even though we ended up deciding against, we think this was a reasonable and high-expected-value idea for these groups to advocate at the time.

Human-generated data is often needed for ML projects or benchmarks if a suitable dataset cannot be e.g. scraped from the web, or if human feedback is required. Alignment researchers conduct such ML experiments, but sometimes have different data requirements than standard capabilities researchers. As a result, it seemed plausible that there was some niche unfilled by the market to help alignment researchers solve problems related to human-generated datasets. In particular, we thought - and to some extent confirmed - that the most likely such niche is human data generation that requires particularly competent or high-skill humans. We will refer to this as high-skill (human) data.

We (Matt & Rudolf) went through an informal co-founder matching process along with four other people and were chosen as the co-founder pair to explore this idea. In line with standard startup advice, our first step was to explore whether or not there is a concrete current need for this product by conducting interviews with potential customers. We talked to about 15 alignment researchers, most of them selected on the basis of doing work that requires human data. A secondary goal of these interviews was to build better models for the future importance and role of human feedback in alignment.

Getting human-generated data does indeed cost many of these researchers significant time and effort. However, we think to a large extent this is because dealing with humans is inherently messy, rather than existing providers doing a bad job. Surge AI in particular seems to offer a pretty good and likely improving service. Furthermore, many companies have in-house data-gathering teams or are in the process of building them.

Hence we have decided to not further pursue this idea.

Other projects in the human data generation space may still be valuable, especially if the importance of human feedback in ML continues to increase, as we expect. This might include people specializing on human data as a career.

The types of factors that are most important for doing human dataset provision well include: high-skill contractors, fast iteration, and high bandwidth communication and shared understanding between the research team, the provider organisation and the contractors.

We are keen to hear other people’s thoughts, and would be happy to talk or to share more notes and thoughts with anyone interested in working on this idea or a similar one in the future.

Theory of Change

A major part of AI alignment research requires doing machine learning (ML) research, and ML research in turn requires training ML models. This involves expertise and execution ability in three broad categories: algorithms, compute, and data, the last of which is very neglected by EAs.

We expect training on data from hu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Onion Test for Integrity, published by chanamessinger on September 27, 2022 on LessWrong.

You (or your organization or your mission or your family or etc) pass the “onion test” for integrity if each layer hides but does not mislead about the information hidden within.

When people get to know you better, or rise higher in your organization, they may find out new things, but should not be shocked by the types of information that were hidden. If they are, you messed up in creating the outer layers to describe appropriately the kind-of-thing that might be inside.

Examples

Positive Example:

Outer layer says "I usually treat my health information as private."

Next layer in says: "Here are the specific health problems I have: Gout, diabetes."

Negative example:

Outer layer says: "I usually treat my health info as private."

Next layer in: "I operate a cocaine dealership. Sorry I didn't warn you that I was also private about my illegal activities."

Negative example:

Outer layer says: "Is it ok if I take notes on our conversation?"

Next layer in: "Here’s the group chat where I mocked each point you made to 12 people, some of whom know you”

Positive Example:

Outer layer says "Is it ok if I take notes on our conversation? Also, I’d like to share my unfiltered thoughts about it with some colleagues later."

Next layer in says: "Jake thinks the new emphasis on wood-built buildings won’t last. Seems overconfident."

Passing the test is a function both of what you conveyed (explicitly and implicitly) and the expectations of others. If it’s normal to start mocking group chats, then it doesn’t need to be said to avoid shock and surprise. The illusion of transparency comes to bite here.

Andrew:

I originated this concept for teaching people better honesty practices.

Social friction minimization is the default trend that shapes the outer layers of a person or institution, by eroding away the bits of information that might cause offence, leaving layers of more pungent information underneath. The “onion model” of honesty or integrity is that each layer of your personality or institution should hide but not mislead about the layer underneath it. This usually involves each layer sharing something about the kinds of information that are in the next layer in, like “I generally keep my health information private”, so people won’t assume that a lack of info about your health means you’re doing just fine health-wise.It takes a bit of work to put sign-posts on your outer layer about what kinds of information is inside, and it takes more work to present those sign-posts in a socially smooth way that doesn’t raise unnecessary fears or alarms.

However, if you put in that work, you can safely get to know people without them starting to wonder, “What else is this person or institution hiding from me?” And, if everyone puts in that work, society in general becomes more trustworthy and navigable.

I started using the onion model in 2008, and since then, I’ve never told a lie. It’s surprisingly workable once you get the hang of it. Some people think privacy is worse than lies, but I believe the opposite is true, and I think it’s worth putting in the effort to eliminate lying entirely if you’re up to the challenge. Going a bit further, you can add an outer layer of communications that basically tells people what kinds of things you’re keeping private, so not only have you not lied, you’ve also avoided misleading them. That’s the whole onion model.

Chana:

I have found this model extremely useful in the last few months talking about organizational strategy as a way of carving between “not everyone gets to know everything” and “actively pointing people in the wrong direction about what’s true lacks integrity” and avoiding “I didn’t lie but I knowingly misled.”

So far I have thought about it as a backwards ref...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Soft skills for meetups, published by mingyuan on September 27, 2022 on LessWrong.

This post is adapted from a memo I wrote for the 2022 global meetup organizer retreat, and owes some of its content to the organizers who attended.

'Soft skills' is a broad and somewhat fuzzy term, but the OED defines it as "personal attributes that enable someone to interact effectively and harmoniously with other people", and that seems close enough to me. Being able to interact effectively and harmoniously with other people is obviously an important part of running a community, and it can be hard to work on, especially because everyone is starting out with very different strengths and weaknesses.

The most important thing I have to say about this topic is:

If there's something that you don't know how to do, acknowledge that and find a way to work around it, rather than ignoring it / pretending it's not a problem.

For example, a lot of 'soft skills' involve reading non-verbal cues. If this is hard for you, maybe you can work on it, but in the meantime, you still need to be aware of conflicts, tension, and discontentment in your community.

One way to solve this would be to enlist a trusted friend / member of your community who's better at reading people, and make it their job to notice potential problems and bring them to your attention. This way you can still address problems despite your trouble with non-verbal cues. Similarly, if you find it hard to talk to people or ask people for things, you might have a friend do the actual talking in your stead.

Welcoming new members

Engage with first-time attendees

When a new person shows up, speak to them (ask their name, etc) and help them get oriented. Ask if they have any questions, and set expectations for what's going to happen at this event. You might also ask them what brought them there, what they're hoping to get out of it, etc, but just conversing at all is good. When you inevitably have to leave them, make sure you've introduced them into another conversation, rather than just leaving them alone.

Note: This doesn't necessarily just apply to in-person events! The Miami ACX organizer DMs every new person who joins his Facebook group.

Appoint Welcomers

A Welcomer is someone whose job it is to attend to the experience of new members, as described above. By default you as the organizer are in this role, but if your community is large, or if you struggle with shyness or awkwardness, you can delegate it to one or more other people. (If you're lucky, you may have enough people naturally filling this role that you don't need to appoint anyone.)

Recognize that, if you take on the role of Welcomer, you need to be attending to it constantly. I've seen large-ish public events where the organizers just got absorbed in conversations and no one took responsibility for welcoming. New people who showed up would just stand around looking lost :(

Make points of contact legible

If you're gathering in a large group, the Welcomer(s) or organizer(s) should be visually identifiable, so that anyone in the crowd can find someone to help if they need something. This is the reason why events often give their volunteers brightly-colored T-shirts.

Examples:

At a 100+ person meetup in NYC, one of the organizers made a tall flag that he carried around so he would always be findable despite moving.

If meeting somewhere with only one entrance, you can have a Welcomer stationed at the door.

The East Bay organizer always announces beforehand to look for his distinctive hat, so when people show up for the first time they not only know they're in the right place but also have a default person to talk to.

Retaining members

In order to retain people, you need to give them reasons to stay! Here are some ways you might do that:

Make your events pleasant to go to

This is the ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Does Economic Growth Meaningfully Improve Well-being? An Optimistic Re-Analysis of Easterlin’s Research: Founders Pledge, published by Vadim Albinsky on September 27, 2022 on The Effective Altruism Forum.

Acknowledgments

I would like to thank Michael Plant, Matt Lerner and Rosie Bettle for their helpful comments and advice.

Summary

Understanding the relationship between wellbeing and economic growth is a topic that is of key importance to Effective Altruism (e.g. see Hillebrandt and Hallstead, Clare and Goth). In particular, a key disagreement regards the Easterlin Paradox; the finding that happiness[1] varies with income across countries and between individuals, but does not seem to vary significantly with a country’s income as it changes over time. Michael Plant recently wrote an excellent post summarizing this research. He ends up mostly agreeing with Richard Easterlin’s latest paper arguing that the Easterlin Paradox still holds; suggesting that we should look to approaches other than economic growth to boost happiness. I agree with Michael Plant that life satisfaction is a valid and reliable measure, that it should be a key goal of policy and philanthropy, and that boosting income does not increase it as much as we might naively expect.

In fact, we at Founders Pledge highly value and regularly use Michael Plant’s and Happier Lives Institute’s (HLI) research; and we believe income is only a small part of what interventions should aim at. However, my interpretation of the practical implications of Easterlin’s research differ from Easterlin’s in three ways which I argue in this post:

Easterlin finds small coefficients in his preferred regressions of changes in countries’ happiness on changes in GDP. He concludes that these coefficients have low “economic significance” and that increasing economic growth is not a good way to make people happier. However, even if we take these coefficients at face value, they still represent a very meaningful increase in wellbeing within the effective altruism framework, consistent with the impacts of unconditional cash transfers on individuals. The benefits become very large when aggregated across all the people in a country for many years.

We also have reason to doubt Easterlin’s results, in that they are highly sensitive to small changes in methodology. We perform two variations on his regression that fully accept his methodology of only including “full cycle” countries, but update it slightly, reversing the result. If we replicate his results counting one more country as a “transition” economy, the Easterlin paradox largely disappears. If we repeat his analysis with new data from 2020 instead of 2019, the paradox also seems to largely disappear.

It may be difficult to find things we can influence whose change over time will have a higher correlation to a country’s change in happiness than changes in GDP. Even if we accept that boosting GDP does not meaningfully increase happiness, other potential means of boosting national happiness may increase it even less. If we rerun Easterlin’s analysis using three interventions Easterlin and Plant suggest (health, pollution, and a comprehensive welfare state), their implied impacts on national happiness are much smaller than the impacts for GDP or negative. However, I have low confidence in this conclusion, and think it is a very valuable project to identify the interventions that are most likely to have an impact on happiness.

1. Taking Easterlin’s results at face value and estimating impact

Easterlin and O’Connor (2022) rely on two regressions for their conclusions, both comparing annual changes in a country’s happiness to annual changes in per capita GDP. The first measures happiness using a “life satisfaction” survey question on a smaller set of countries from 1981-2019 and the second u...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Thoughts on the ML Safety Course, published by zeshen on September 27, 2022 on The AI Alignment Forum.

This summary was written as part of Refine. The ML Safety Course is created by Dan Hendrycks at the Center for AI Safety. Thanks to Adam Shimi and Thomas Woodside for helpful feedback.

Overview

Background

I recently completed the ML Safety Course by watching the videos and browsing through the review questions, and subsequently writing a short summary. As an engineer in the upstream oil and gas industry with some experience in dealing with engineering safety, I find the approach of the course of thinking in this framework especially valuable.

This post is meant to be a (perhaps brutally) honest review of the course despite me having no prior working experience in ML. It may end up reflecting my ignorance of the field more than anything else, but I would still consider it as a productive mistake. In many cases, if my review seems to be along the lines of ‘this doesn’t seem to be right’, it should be read as ‘this is how a course participant may misinterpret the course contents’. I am also well aware that it is much easier to criticize something useful than actually doing something useful.

For each section of the course, I will give a short summary, describe what I liked, and what I didn’t like. I may be especially brief with the parts about what I liked, and the brevity is no way a reflection about how much I liked it.

Thomas Woodside, who helped with creating parts of the course, has kindly provided feedback to this post. His comments are formatted in italics.

My Initial Expectations of the Course

Having engaged with AI Safety as an outsider, my general impression of the field were:

Predominantly based with AI FOOM scenarios and primarily concerned with abstract concepts like agency.

Even among prosaic AI alignment, it appears that the general research modus operandi is that people would (somewhat randomly) generate ideas that could be applicable to certain classes of AI safety problems. Although the ideas may be interesting and valuable, they tend to be rather narrow and specific problems and may not be scalable.

One of the more common arguments for advocating AI alignment is that failure to align AI systems lead to existential scenarios. From this perspective, a pragmatic approach towards AI safety with the aim of minimizing risks by reducing AI misalignments may not be very useful, since a superintelligent AI will exploit any every slight misalignment and immediately cause human extinction.

Hence, I was pleasantly surprised when I came across the ML Safety Course, which I thought would be a good attempt at tackling the problem of prosaic AI alignment in a holistic, systematic, and practical manner. Although this approach may not directly solve the ‘hard problem’ completely, it would still help by minimizing existential risks and buy us more time to address the ‘hard problem’. (Feedback from Thomas: the creators of the course disagree with the framing of “solving the hard problem” as there are many hard problems that need to be iteratively worked on)

Summary

My overall impression of the course is:

It is grounded on real-world safety principles that uses a systematic framework to reduce ML safety risks.

It (rightly) does not seem to directly tackle the ‘hard problem’, but in my opinion there is nevertheless a lot of value in buying us more time while solving the ‘hard problem’ (Feedback from Thomas: the creators of the course disagree with the framing of “solving the hard problem” as there are many hard problems that need to be iteratively worked on)

It details many approaches that are useful in some specific settings, but it is unclear how it scales towards more powerful AI systems.

It covers several approaches that don’t seem to be very related to the ‘core’ r...