Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Open Agency Model, published by Eric Drexler on February 22, 2023 on The AI Alignment Forum.

Notes on AI for complex, consequential problems

Eric DrexlerCentre for the Governance of AIUniversity of Oxford

Introduction

This document argues for “open agencies” — not opaque, unitary agents — as the appropriate model for applying future AI capabilities to consequential tasks that call for combining human guidance with delegation of planning and implementation to AI systems. This prospect reframes and can help to tame a wide range of classic AI safety challenges, leveraging alignment techniques in a relatively fault-tolerant context.

Rethinking safe AI and its applications

AI safety research is too varied to summarize, yet broad patterns are obvious. A long-established reference-problem centers on prospects for rational superintelligent agents that pursue narrow goals with potentially catastrophic outcomes. This frame has been productive, but developments in deep learning call for updates that take account of the proliferation of narrow models (for driving, coding, robot control, image generation, game playing.) that are either non-agentic or act as agents in only a narrow sense, and that take account of the rise of more broadly capable foundation models and LLMs. These updates call for reframing questions of AI safety, and call for attention to how consequential tasks might be accomplished by organizing AI systems that usually do approximately what humans intend.

Two frames for high-level AI

The unitary-agent frame

From its beginnings in popular culture, discussion of the AI control problem has centered around a unitary agent model of high-level AI and potential AI risks. In this model, a potentially dominant agent both plans and acts to achieve its goals.

The unitary-agent model typically carries assumptions regarding goals, plans, actions, and control.

Goals: Internal to an agent, by default including power-seeking goals

Plans: Internal to an agent, possibly uninterpretable and in effect secret

Actions: Performed by the agent, possibly intended to overcome opposition

Control: Humans confront a powerful, potentially deceptive agent

The typical unitary-agent threat model contemplates the emergence of a dominant, catastrophically misaligned agent, and safety models implicitly or explicitly call for deploying a dominant agent (or an equivalent collective system) that is both aligned and powerful enough to suppress unaligned competitors everywhere in the world.

The open-agency frame

Recent developments suggest an alternative open agency model of high-level AI. Today, the systems that look most like AGI are large language models (LLMs), and these are not agents that seek goals, but are generative models that produce diverse outputs in response to prompts (in a generalized sense) and random-number seeds. Most outputs are discarded.

Trained on prediction tasks, LLMs learn world models that include agent behaviors, and generative models that are similar in kind can be informed by better world models and produce better plans. There is no need to assume LLM-like implementations: The key point is that generation of diverse plans is by nature a task for generative models, and that in routine operation, most outputs are discarded.

These considerations suggest an “open-agency frame” in which prompt-driven generative models produce diverse proposals, diverse critics help select proposals, and diverse agents implement proposed actions to accomplish tasks (with schedules, budgets, accountability mechanisms, and so forth).

Goals, plans, actions, and control look different in the open-agency model:

Goals: Are provided as prompts to diverse generative models, yielding diverse plans on request

Plans: Are selected with the aid of diverse, independent comparison and evaluation mechanisms

...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A list of EA-relevant business books I've read, published by Drew Spartz on February 21, 2023 on The Effective Altruism Forum.

Some have suggested EA is too insular and needs to learn from other fields. In this vein, I think there are important mental models from the for-profit world that are underutilized by non-profits.

After all, business can be thought of as the study of how to accomplish goals as an organization - how to get things done in the real world. EA needs the right mix of theory and real world execution. If you replace the word “profit” with “impact”, you’ll find a large percentage of lessons can be cross-applied.

Eight months ago, I challenged myself to read a book a day for a year. I've been posting daily summaries on social media and had enough EAs reach out to me for book recs that, inspired by Michael Aird and Anna Riedl, I thought it might be worth sharing my all-time favorites here.

Below are the best ~50 out of the ~500 books I read in the past few years. I’m an entrepreneur so they’re mostly business-related. Bold = extra-recommended.

If you’d like any more specific recommendations feel free to leave a comment and I can try to be helpful.

Also - I’m hosting an unofficial entrepreneur meetup at EAG Bay Area. Message me on SwapCard for details or think it might be high impact to connect :)

The best ~50 books:

Fundraising:

Fundraising

The Power Law: Venture Capital and the Making of the New Future

Leadership/Management:

The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers

The Advantage: Why Organizational Health Trumps Everything Else In Business

The Coaching Habit: Say Less, Ask More & Change the Way You Lead Forever

Entrepreneurship/Startups:

Running Lean

The Founder's Dilemmas: Anticipating and Avoiding the Pitfalls That Can Sink a Startup

Zero to One: Notes on Startups, or How to Build the Future

The Startup Owner's Manual: The Step-By-Step Guide for Building a Great Company

Strategy/Innovation:

The Mom Test: How to talk to customers & learn if your business is a good idea when everyone is lying to you

Scaling Up: How a Few Companies Make It...and Why the Rest Don't

Operations/Get Shit Done:

The Goal: A Process of Ongoing Improvement

The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win

Making Work Visible: Exposing Time Theft to Optimize Work & Flow

Statistics/Forecasting:

How to Measure Anything: Finding the Value of Intangibles in Business

Superforecasting: The Art and Science of Prediction

Antifragile: Things That Gain from Disorder

Writing/Storytelling:

Wired for Story: The Writer's Guide to Using Brain Science to Hook Readers from the Very First Sentence

The Story Grid: What Good Editors Know

Product/Design/User Experience:

The Cold Start Problem: How to Start and Scale Network Effects

The Lean Product Playbook: How to Innovate with Minimum Viable Products and Rapid Customer Feedback

Psychology/Influence:

SPIN Selling (unfortunate acronym)

The Elephant in the Brain: Hidden Motives in Everyday Life

Influence: The Psychology of Persuasion

Outreach/Marketing/Advocacy:

80/20 Sales and Marketing: The Definitive Guide to Working Less and Making More

Traction: How Any Startup Can Achieve Explosive Customer Growth

How to learn things faster:

Ultralearning: Master Hard Skills, Outsmart the Competition, and Accelerate Your Career

Make It Stick: The Science of Successful Learning

The Little Book of Talent: 52 Tips for Improving Your Skills

Personal Development:

The Confident Mind: A Battle-Tested Guide to Unshakable Performance

The Almanack of Naval Ravikant: A Guide to Wealth and Happiness

Atomic Habits

Recruiting/Hiring:

Recruiting

Who: The A Method for Hiring

Negotiating:

Negotiation Genius

Never Split the Difference: Negotiating As If Your Life Depended On It

Secrets of Power Negotiating: I...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Stranger Priority? Topics at the Outer Reaches of Effective Altruism (my dissertation), published by Joe Carlsmith on February 21, 2023 on The Effective Altruism Forum.

(Cross-posted from my website.)

After many years of focusing on other stuff, I recently completed my doctorate in philosophy from the University of Oxford. My dissertation ("A Stranger Priority? Topics at the Outer Reaches of Effective Altruism") was three of my essays -- on anthropic reasoning, simulation arguments, and infinite ethics -- revised, stapled together, and unified under the theme of the "crazy train" as a possible objection to longtermism.

The full text is here. I've also broken the main chapters up into individual PDFs:

Chapter 1: SIA vs. SSA

Chapter 2: Simulation arguments

Chapter 3: Infinite ethics and the utilitarian dream

Chapter 1 and Chapter 3 are pretty similar to the original essays (here and here). Chapter 2, however, has been re-thought and almost entirely re-written -- and I think it's now substantially clearer about the issues at stake.

Since submitting the thesis in fall of 2022, I've thought more about various "crazy train" issues, and my current view is that there's quite a bit more to say in defense of longtermism than the thesis has explored. In particular, I want to highlight a distinction I discuss in the conclusion of the thesis, between what I call "welfare longtermism," which focuses on our impact on the welfare of future people, and what I call "wisdom longtermism," which focuses on reaching a wise and empowered future more broadly. The case for the latter seems to me more robust to various "crazy train" considerations than the case for the former.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pretraining Language Models with Human Preferences, published by Tomek Korbak on February 21, 2023 on LessWrong.

This post summarizes the main results from our recently released paper Pretraining Language Models with Human Preferences, and puts them in the broader context of AI safety. For a quick summary of the paper, take a look at our Twitter thread.

TL;DR: In the paper, we show how to train LMs with human preferences (as in RLHF), but during LM pretraining. We find that pretraining works much better than the standard practice of only finetuning with human preferences after pretraining; our resulting LMs generate text that is more often in line with human preferences and are more robust to red teaming attacks. Our best method is conditional training, where we learn a predictive model of internet texts conditional on their human preference scores, e.g., evaluated by a predictive model of human preferences. This approach retains the advantages of learning from human preferences, while potentially mitigating risks from training agents with RL by learning a predictive model or simulator.

Summary of the paper

Motivation. LMs are pretrained to maximize the likelihood of their training data. Since the training data contain undesirable content (e.g. falsehoods, offensive language, private information, buggy code), the LM pretraining objective is clearly (outer) misaligned with human preferences about LMs’ downstream applications as helpful, harmless, and honest assistants or reliable tools. These days, the standard recipe for alining LMs with human preferences is to follow pretraining with a second phase of finetuning: either supervised finetuning on curated data (e.g. instruction finetuning, PALMS) or RL finetuning with a learned reward model (RLHF). But it seems natural to ask: Could we have a pretraining objective that is itself outer-aligned with human preferences?

Methods. We explore objectives for aligning LMs with human preferences during pretraining. Pretraining with human feedback (PHF) involves scoring training data using a reward function (e.g. a toxic text classifier) that allows the LM to learn from undesirable content while guiding the LM to not imitate that content at inference time. We experimented with the following objectives:

MLE (the standard pretraining objective) on filtered data;

Conditional training: a simple algorithm learning a distribution over tokens conditional on their human preference score, reminiscent of decision transformer;

Unlikelihood training: maximizing the likelihood of tokens with high human preference score and the unlikelihood of tokens with low human preference scores;

Reward-weighted regression (RWR): an offline RL algorithm that boils down to MLE weighted by human preference scores; and

Advantage-weighted regression (AWR): an offline RL algorithm extending RWR with a value head, corresponding to MLE weighted by advantage estimates (human preference scores minus value estimates).

Setup. We pretrain gpt2-small-sized LMs (124M params) on compute-optimal datasets (according to Chinchilla scaling laws) using MLE and PHF objectives. We consider three tasks:

Generating non-toxic text, using scores given by a toxicity classifier.

Generating text without personally identifiable information (PII), with a score defined by the number of pieces of PII per character detected by a simple filter.

Generating Python code compliant with PEP8, the standard style guide for Python, using as a score the number of violations per character found by an automated style checker.

Metrics. We compare different PHF objectives in terms of alignment (how well they satisfy preferences) and capabilities (how well they perform on downstream tasks). We primarily measure alignment in terms of LM samples’ misalignment scores, given by the reward functions used at t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is it like doing AI safety work?, published by KatWoods on February 21, 2023 on LessWrong.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is it like doing AI safety work?, published by Kat Woods on February 21, 2023 on The Effective Altruism Forum.

How do you know if you’ll like AI safety work? What’s the day-to-day work like? What are the best parts of the job? What are the worst?

To better answer these questions, we talked to ten AI safety researchers in a variety of organizations, roles, and subfields. If you’re interested in getting into AI safety research, we hope this helps you be better informed about what pursuing a career in the field might entail.

The first section is about what people do day-to-day and the second section describes each person’s favorite and least favorite aspects of the job.

Of note, the people we talked with are not a random sample of AI safety researchers, and it is also important to consider the effects of survivorship bias. However, we still think it's useful and informative to hear about their day-to-day lives and what they love and hate about their jobs.

Also, these interviews were done about a year ago, so may no longer represent what the researchers are currently doing.

Reminder that you can listen to LessWrong and EA Forum posts like this on your podcast player using the Nonlinear Library.

This post is part of a project I’ve been working on at Nonlinear. You can see the first part of the project here where I explain the different ways people got into the field.

What do people do all day?

John Wentworth

John describes a few different categories of days.

He sometimes spends a day writing a post; this usually takes about a day if all the ideas are developed already.

He might spend a day responding to comments on posts or talking to people about ideas. This can be a bit of a chore but is also necessary and useful.

He might spend his day doing theoretical work. For example, if he’s stuck on a particular problem, he can spend a day working with a notebook or on a whiteboard. This means going over ideas, trying out formulas and setups, and trying to make progress on a particular problem.

Over the past month he’s started working with David Lorell. David’s a more active version of the programmer's "rubber duck". As John’s thinking through the math on a whiteboard, he’ll explain to David what's going on. David will ask for clarifications, examples, how things tie into the bigger picture, why did/didn't X work, etc.

John estimates that this has increased his productivity at theoretical work by a factor somewhere between 2 and 5.

Ondrej Bajgar

Ondrej starts the day by cycling to the office. He has breakfast there and tries to spend as much time as possible at a whiteboard away from his computer. He tries to get into a deep-thinking mindset, where there aren’t all the answers easily available. Ideally, mornings are completely free of meetings and reserved for this deep-thinking work.

Deep thinking involves a lot of zooming in and out, working on sub-goals while periodically zooming out to check on the higher-level goal every half hour. He switches between trying to make progress and reflecting on how this is actually going. This is to avoid getting derailed on something unproductive but cognitively demanding.

Once an idea is mostly formed, he’ll try to implement things in code. Sometimes seeing things in action can make you see new things you wouldn’t get from just the theory. But he also says that it’s important to not get caught in the trap of writing code, which can feel fun and feel productive even when it isn’t that useful.

Scott Emmons

Scott talked about a few different categories of day-to-day work:

Research, which involves brainstorming, programming, writing & communicating, and collaborating with people

Reading papers to stay up-to-date with the literature

Administrative work

Service, such as giving advice to undergrads, talking about AI safety, and reviewing other...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bad Actors are not the Main Issue in EA Governance, published by Grayden on February 21, 2023 on The Effective Altruism Forum.

Background

While I have a technical background, my career has been spent working with corporate boards and management teams. I have seen first-hand how critical leadership are to the success of organizations. Organizations filled with competent people can fail miserably if individuals do not have the right interpersonal skills and humility.

I have worried about governance within EA for a while. In October, I launched the EA Good Governance Project and wrote that "We have not yet experienced a scandal / major problem and have not yet started to think through how to avoid that happening again" . Now, 4 months later, we've had our fair share and people are open to change.

This post is my attempt to put some thoughts together. It has been written in a rather rushed way given recent news, so apologies if some parts are poorly worded.

Introduction

I have structured my thoughts in 4 sections, corresponding to the 4 key ways in which leadership can fail:

1) Bad actor

2) Well-intentioned people with low competence

3) Well-intentioned high-competence people with collective blind spots

4) Right group of people, bad practices

Bad Actors

Much discussion on the forum in recent months has focused on the concept of a bad actor. I think we are focusing far too much on this concept.

The term comes from computer science where hackers are prevalent. However, real life is rarely this black and white. Never attribute to malice that which is adequately explained by incompetence (Hanlon's razor).

The bad actor concept can be used, consciously or unconsciously, to justify recruiting board members from within your clique. Many EA boards comprise groups of friends who know each other socially. This limits the competence and diversity of the board. Typically the people you know well are exactly the worst people to provide different perspectives and hold you to account. If they are your friends, you have these perspectives and this accountability already and you can prevent bad actors through referencing, donation history and background checks.

Key takeaway: Break the clique

Competence

There's an old adage: How do you know if someone is not very good at Excel? They will say they are an expert. With Excel, the more you know, the more aware you are of what you don't know. I think leadership is similar. When I had 3-5 years of professional experience, I thought I could lead anything. Now I know better.

Some aspects of leaderships come naturally to people, but many have to be learned by close interaction with role models. When you are a community without experienced figures at the top, this is hard. We should not expect people with less than 10 years of professional experience to be a fully rounded leader. Equally, it's possible to be successful without being a good leader.

I think many of us in the community have historically held EA leaders on a pedestal. They were typically appointed because of their expertise in a particular field. Some of the brightest people I've ever met are within the EA community. However, we then assumed they had excellent people judgment, a sound understanding of conflicts of interest, in-depth knowledge of real estate investments and an appreciation for power dynamics in sexual relationships. It turns out some of them don't. This shouldn't come as a big surprise. It doesn't mean they can't be a valuable contributor to the community and it certainly doesn't make them bad actors.

Key takeaway: We need to elevate the importance of soft skills and learn from models of effective leadership in other communities and organizations

Blind Spots

The more worrying thing though is how those people also believed in their own abilities. In my career, I have met...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: You're probably a eugenicist, published by Sentientist on February 21, 2023 on The Effective Altruism Forum.

A couple of EAs encouraged me to crosspost this here. I had been sitting on a shorter version of this essay for a long time and decided to publish this expanded version this month partly because of the accusations of eugenics leveraged against Nick Bostrom and the effective altruism community.

The piece is the first article on my substack and you can listen to me narrate it at that link.

You're probably a eugenicist

Let me start this essay with a love story.

Susan and Patrick were a young German couple in love. But, the German state never allowed Susan and Patrick to get married. Shockingly, Patrick was imprisoned for years because of his sexual relationship with Susan.

Despite these obstacles, over the course of their relationship, Susan and Patrick had four children. Three of their children—Eric, Sarah, and Nancy—had severe problems: epilepsy, cognitive disabilities, and a congenital heart defect that required a transplant. The German state took away these children and placed them with foster families.

Patrick and Susan with their daughter Sofia - credit dpa picture alliance archive

Why did Germany do all these terrible things to Susan and Patrick?

Eugenics.

No, this story didn’t happen in Nazi Germany, it happened over the course of the last 20 years. But why haven’t you heard this story before?

Because Patrick and Susan are siblings.

One of the aims of eugenics is to intervene in reproduction so as to decrease the number of people born with serious disabilities or health problems. Susan and Patrick were much more likely than the average couple to have children with genetic problems because they are brother and sister. So, the German state punished this couple by restricting them from marriage, taking away their children, and forcefully separating them with Patrick’s imprisonment.

Patrick Stübing filed a case against Germany with the European Court on Human Rights, arguing that the laws forbidding opposite-sex sibling incest violated his rights to family life and sexual autonomy. The European Court on Human Rights’ majority opinion in the Stübing case clearly sets out the eugenic case for those laws: that the children of incest and their future children will suffer because of genetic problems. But the dissenting opinion argued that eugenics cannot be a valid justification for punishing incest because eugenics is associated with the Nazis, and because other people (for example, older mothers and people with genetic disorders) who have a high chance of producing children with genetic defects are not prevented from reproducing. Ultimately, the European Court on Human Rights upheld Germany's anti-incest law on eugenic grounds.

If Germany had punished any other citizens this severely on eugenic grounds—for example by imprisoning a female carrier of Huntington’s disease who was trying to get pregnant— there would be a huge outcry. But incest seems to be an exception.

Our instinctive aversion to incest is informed by intuitive eugenics. Not only are we reflexively disgusted by the thought of having sex with our own blood relatives, but we’re also disgusted by the thought of any blood relatives having sex with each other.

Siblings and close relatives conceive children who are more likely to end up with two copies of the same defective genes, which makes those children more likely to inherit disabilities and health problems. It’s estimated that the children of sibling incest have a greater than 40 percent chance of either dying prematurely or being born with a severe impairment. By comparison, first cousins have around a five percent chance of having children with a genetic problem—twice as likely as unrelated couples. In the UK, first cousin marriages are legal and...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EU Food Agency Recommends Banning Cages, published by Ben West on February 21, 2023 on The Effective Altruism Forum.

Some key recommendations (all direct quotes from either here or here):

Birds should be housed in cage-free systems

Avoid all forms of mutilations in broiler breeders

Avoid the use of cages, feed and water restrictions in broiler breeders

Limit the growth rate of broilers to a maximum of 50 g/day.

Substantially reduce the stocking density to meet the behavioural needs of broilers.

My understanding is that the European Commission requested these recommendations as a result of several things, including work by some EA-affiliated animal welfare organizations, and it is now up to them to propose legislation implementing the recommendations.

This Forum post from two years ago describes some of the previous work that got us here. It's kind of cool to look back on the "major looming fight" that post forecasts and see that the fight is, if not won, at least on its way.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A proof of inner Löb's theorem, published by James Payor on February 21, 2023 on The AI Alignment Forum.

This is a short post that offers a slightly different take on the standard proof of Löb's theorem. It offers nothing else of any value :)

We seek to prove the "inner" version, which we write as: □P↔□(□PP)

The proof uses quining to build a related sentence L, the "Löb sentence", which talks about its own source code. By construction L has the property: □L↔□(□LP)

Then, we can show that □L↔□P, i.e. they're equivalent! We do this by plugging □L into itself to get a twisty □P. We can then replace each □L with □P and prove Löb's theorem.

The proof

This proof uses the same rules of box manipulation as on the wiki page. We start by creating L using quining, i.e. taking a modal fixed point:

⊢L↔(□LP) (exists as a modal fixed point)

Yep, this is skipping the details of the most interesting part, but alas I don't understand them well enough to do more than wave my hands and say "quining".

We then stick it inside the box to get our first property:

⊢□(L↔(□LP)) (from (1) by necessitation)

⊢□L↔□(□LP) (from (2) by box-distributivity in both directions)

We now want to show that □L↔□P. We can get the forward direction by feeding a copy of □L into itself:

⊢□L(□□L□P) (box-distributivity on (3))

⊢□L□□L (internal necessitation)

⊢□L□P (from (4) and (5))

The backward direction is equivalent to □P□(□LP), and is straightforward:

⊢P(□LP) (trivial)

⊢□P□(□LP) (necessitation and box-distributivity on (7))

Taking those together, we've shown □L and □P are equivalent.

⊢□L↔□P (from (6) and (8))

Now we'd like to finish by appealing to the following chain: □P↔□L↔□(□LP)↔□(□PP)

We've proven all but the last part of the chain. Here are the steps that let us do the substitution:

⊢(□LP)↔(□PP) (since □L and □P are equivalent by (9))

⊢□((□LP)↔(□PP)) (from (10) by necessitation)

⊢□(□LP)↔□(□PP) (from (11) by box-distributivity in both directions)

And that's everything we need:

⊢□P↔□(□PP) (from (3), (9), and (12))

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.