Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Beta Feature] Google-Docs-like editing for LessWrong posts, published by Ruby on February 23, 2022 on LessWrong.

TL;DR: LessWrong now has similar features to Google Docs. Warning! Still rough around the edges. To enable the collaborative editor, you must check "Opt into experimental features" in your account settings and then press the green "Share" button that appears when editing your post.

It's been a loooong time coming but at last, we are ready to unveil collaborative editing features for the LessWrong text editor. These features will be familiar to those used to working in Google Docs:

Multiple users can edit a document at once

Fine-grained permission for view/commenting/editing by link or username

Inline comments (only viewable while in edit mode)

Making and accepting suggested edits

Automatic saving

Version history viewer

How to get started

Step 1: Opt in to experimental features in your account settings

Step 2: While editing your post, click the green "Share" button in top right. Enable permissions for some other users. Boom! Your post is now in collaborative mode.

Step 3: Users explicitly shared on your post will receive a notification. Send the url to anyone else you want to grant access.

Step 4: When in collaborative mode, the text of your post is automatically saved. To view how your post will look when published, press the Preview button. To make the current state of your document live, press the Publish button.

The collaborative editor allows you (and others) to continue editing and commenting on a post even once it's been published. Edits aren't automatically published to the live version. To update the published version to the current state of the document in editing, press "Publish".

Step 5: Leave comments and suggestions.

To enable track changes, use the track changes button in the popup menu (third icon from the right) or set your Mode to Commenting in the header bar.

Step 6: Leave feedback about the feature!

This feature is still under development. Any feedback from early-adopters is hugely helpful. Just leave a comment on this post or message us on Intercom.

Warning! Rough around the edges

We're releasing this feature in beta mode because it still requires a few more finishing touches and might be a little confusing to use in places. As above, any feedback is greatly appreciated.

Why collaborative editing?

When developing features, there's always a question of "does anyone want this?" In the case of collaborative editing features, there's good evidence of demand from Google Docs. There's a common workflow that goes: (1) write a draft in Google Docs, (2) invite some close friends or collaborators to give feedback, (3) incorporate feedback, (4) copy to LessWrong, (5) publish.

Drawbacks of this workflow are (a) overhead of copying and reformatting the post, (b) enforcing a hard break between the feedback stage and the publication stage, (c) Google Docs does not support LaTeX, (d) valuable comments left in the feedback stage never get published to the wider world.

By introducing collaborative editing to LessWrong, we address (a), (b), and (c). We haven't yet made it so in-line comments in editing mode can be published in the final version, but we'll look into ways to allow for that. I also expect that we'll add additional features to our editor that Google Docs doesn't have, such that users will benefit from the possibility of doing all their writing on LessWrong.

Having collaborative editing features on LessWrong also lets us start to build programs that rely on easy ways to give people feedback on their drafts. For example, I'd like to run a writing and research workshop for students that involves peer and mentor feedback. With collaborative editing on LessWrong, that will now be much more convenient to do.

What's your writing workflow?

If you're...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How satisfied should you expect to be with your partner?, published by Vaniver on February 22, 2022 on LessWrong.

I got married today, to the particular fellow mentioned in my Turning 30 post. In a sort of 'inverse cat tax' for a sappy announcement, here's a mathematical model of whether you should expect to like your partner more than yourself. I don't mean this in a moralistic way ('thou shalt love thy parents'), tho that might be another post for another time, or necessarily in a utilitarian way ("I would rather they get this ice cream than me"), but as a matter of raw respect ("I think they're a better person than I am, according to my values").

For simplicity's sake, let's consider everyone as having a 'stat' vector and a 'preferences' vector with the same dimensionality, and giving a candidate partner a 'score' based on the dot product of those two vectors. We'll assume that all of the stats are universally good (no one ever prefers an uglier partner over a smarter one, tho they might not care about physical attractiveness much). The preferences vector we'll normalize to have unit magnitude (so it's just an angle in N-dimensional space, basically, defined as a point on the positive sector of the N-spherical shell). For reasons, I'll run simulations with the stat vector as 6 dimensions with 3d6 per stat, leading to a discrete distribution a bit like a truncated normal, with no correlation between the stats.

Let's start by considering the heterosexual version of the stable marriage problem, in which people partner up using the well-known Gale-Shapley algorithm, and a simulation with 1,000 each of randomly sampled men and women. Mating is highly assortative; a correlation between total stats of 85.7%, with 83.3% of people have an average total stat difference of less than 6 (the dimensionality).

The interesting result is that 91% of people like their partner at least as much as they like themselves, with an average net satisfaction of 1.24. Note that we haven't baked in any correlation between one's stats and one's preferences, and so this result is, in some sense, not very surprising. The preferences are exerting pressure on the partner (thru who you can stably match with) and not exerting pressure on the self, and so you should expect that pressure to result in higher other-satisfaction.

So let's add an adjustment to the preferences with a scalable parameter corr, so that now it's the (renormalized) sum of the stat vector (times corr) and the previous preference vector (times abs(1 - corr)). As we smoothly vary this from 0 to 1, the average net satisfaction decreases to -0.2 (liking themselves more than their partner) and the fraction happier decreases to 21%.

While the change in net satisfaction is relatively smooth, the change in fraction happier looks much more sigmoidal, with the main drop between corr = 0.4 and corr = 0.8. The main change here is in self-satisfaction, which increases by about 5 points while other-satisfaction increases by only about 3 points.

You can also imagine situations where people specifically want their complements, rather than their mirror. Negative correlations between your stats and your preferences seem unlikely; a more appropriate model seems to be something like relationship satisfaction being a function of the minimum stat between the two partners (or the minimum plus half the maximum, or so on).

The 'marriage' situation with full bisexuality is typically called the stable roommate problem, solved with a similar algorithm. I'll leave it as an exercise for the reader how that impacts the results.

Anyway, my sense is that when people talk about their 'better half', they're mostly being serious, and this is something that can easily be symmetric.

On Imgur, it's common for cat owners to end posts that collect images of use for som...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New grantmaking program: supporting the effective altruism community around Global Health and Wellbeing, published by Zachary Robinson on February 22, 2022 on The Effective Altruism Forum.

We are searching for a program officer to help us launch a new grantmaking program. The program would support projects and organizations in the effective altruism community (EA) with a focus on improving global health and wellbeing (GHW).

Background

We have an existing program in effective altruism community growth. Like our new program, the existing program is focused on community building — increasing the number of people working to do as much good as possible with their time and resources, and helping them to work more effectively. However, the existing program evaluates grants through the lens of longtermism, focusing on projects that aim to raise the chance of a very long-lasting and positive future (including by reducing risks from existential catastrophes).

By contrast, the new program will focus on projects related to areas in our GHW portfolio, which is focused on improving health and wellbeing for humans and animals around the world. This portfolio currently includes our work on global health and development, farm animal welfare, global aid advocacy, South Asian air quality, and scientific research; our cause prioritization team is also actively working to identify additional cause areas.

Why we’re looking for a program officer

We view the EA community as pursuing a similar endeavor to Open Philanthropy; people in the community aim to do as much good as possible, and consider a broad range of approaches. We think there are many projects within this community that don’t fit the longtermist focus of our existing program, but do have an expected impact above the “bar” we use to evaluate GHW grants.

We’ve already made some grants to EA projects that were highly promising from a GHW perspective, including Charity Entrepreneurship (supporting the creation of new animal welfare charities) and Founders Pledge (increasing donations from entrepreneurs to outstanding charities).

However, hiring a program officer to focus on this category will give us the capacity to significantly expand our grantmaking, develop our knowledge and strategy in this area, and evaluate the progress of our grantees.

We’re looking to hire someone who is very familiar with the EA community, has ideas about how to grow and develop it, and is passionate about supporting projects in global health and wellbeing. To see more details and apply, visit our job description.

How the program will operate

Our founding program officer will have at least $10 million in available funds to allocate in their first year, and funding could grow significantly from there depending on the volume of good opportunities they find.

The new program will not impact our funding for EA community projects with a longtermist focus, and we don’t expect to reduce our grantmaking in that area.

While both EA programs will operate independently, they may co-fund grants when there is a strong case for impact as assessed through both our GHW and longtermist frameworks.

Note that these projects don’t have to focus exclusively on global health and wellbeing. For example, Founders Pledge raises money for charities in many areas, including longtermist areas, but its impact on GHW-focused charities alone still made it a strong opportunity.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Retrospective on Shall the Religious Inherit The Earth, published by isabel on February 22, 2022 on The Effective Altruism Forum.

Thanks to Leopold Aschenbrenner, Max Daniel and Will MacAskill for feedback on this post, and for the Forethought Foundation for providing research support during the creation of this post.

Introduction

This piece is a retrospective on "Shall the Religious Inherit the Earth", a book by Eric Kaufmann published in 2010 which goes into one possible way that the future might go based on past demographic trends: in societies in which it is common for some couples to choose to have small families, there is a substantial gap between the fertility rates of the most religious, and everyone else. The book focuses especially on the growth of fundamentalist religious groups. In some cases, like the Amish, these groups have a near total rejection of modern technology. The book makes a number of predictions, some of which should have materialized by now, while others are still decades out. I will evaluate the claims about the specific high fertility religious groups the author brings up, as well as discussing overall global trends in religion and fertility. Future posts will delve further into arguments for why fertility rates matter and what influences them.

TL;DR

Book focuses on high growth rates of various religious groups, but for the most part they are experiencing substantial fertility decline in parallel with more secular groups, as well as secularization

In the US:

Identification with Protestantism, both mainline and Evangelical, is in decline, and there is little evidence of high growth from a committed core

Hispanic immigration did not deliver a boost as large as predicted to Catholicism, both because of lower fertility and because of growing secularism

Mormon fertility has declined

In the Middle East, North Africa and Asian countries with high Muslim populations, fertility has fallen substantially, with many places now close to or below replacement fertility

While the Muslim share of Europe will grow, it is likely that it will not grow fast enough to prevent population decline in the medium-term future (i.e. by 2050) without substantial further migration

Of the groups discussed in the book, the only substantial exceptions in terms of declining fertility are:

Old Order Anabaptists (Amish, Hutterite, Old Order Mennonite)

Though future growth may be difficult to maintain as farmland is growing less affordable for them, and farming is associated with higher fertility within these groups

Haredi (Ultra-Orthodox) Jews

Though there are signs of increasing acceptance of the usage of technology and of higher labor force participation rates among both men and women, this has not had a clear impact on fertility or retention

But I’m interested in hearing arguments for other groups!

Globally, the dynamic will be driven by declining populations in highly secular Asian countries and rapidly growing highly religious countries in sub-Saharan Africa, which was not discussed in any detail in the book

Much of the uncertainty about the future global share of the religious stems from uncertainty about the future religiosity of sub-Saharan Africa, and whether the current ongoing fertility decline in Africa will continue at the current pace, accelerate, or slow down

Birth rates matter for future economic growth, technological development and political power. Immigration might help in the short run for countries facing population decline in the immediate future, but is often domestically unpopular and does not help with global fertility rates falling below replacement.

Central claim of book

The book argues that

Religious people have higher fertility rates than non-religious people, and within a religion, fundamentalist groups have higher fertility rates than moderate ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Learning By Writing, published by Holden Karnofsky on February 22, 2022 on The Effective Altruism Forum.

Cross-posted from Cold Takes

I have very detailed opinions on lots of topics. I sometimes get asked how I do this, which might just be people making fun of me, but I choose to interpret it as a real question, and I’m going to sketch an answer here.

You can think of this as a sort of sequel to Minimal-Trust Investigations. That piece talked about how investigating things in depth can be valuable; this piece will try to give a sense of how to get an in-depth investigation off the ground, going from “I’ve never heard of this topic before” to “Let me tell you all my thoughts on that.”

The rough basic idea is that I organize my learning around writing rather than reading. This doesn’t mean I don’t read - just that the reading is always in service of the writing.

Here’s an outline:

Step 1 Pick a topic Step 2 Read and/or discuss with others (a bit) Step 3 Explain and defend my current, incredibly premature hypothesis, in writing (or conversation) Step 4 Find and list weaknesses in my case Step 5 Pick a subquestion and do more reading/discussing Step 6 Revise my claim / switch sides Step 7 Repeat steps 3-6 a bunch Step 8 Get feedback on a draft from others, and use this to keep repeating steps 3-6

The “traditionally” hard parts of this process are steps 4 and 6: spotting weaknesses in arguments, trying to resist the temptation to “stick to my guns” when my original hypothesis isn’t looking so good, etc.

But step 3 is a different kind of challenge: trying to “always have a hypothesis” and re-articulating it whenever it changes. By doing this, I try to continually focus my reading on the goal of forming a bottom-line view, rather than just “gathering information.” I think this makes my investigations more focused and directed, and the results easier to retain. I consider this approach to be probably the single biggest difference-maker between "reading a ton about lots of things, but retaining little" and "efficiently developing a set of views on key topics and retaining the reasoning behind them."

Below I'll give more detail on each step, then some brief notes (to be expanded on later) on why this process is challenging.

My process for learning by writing

Step 1: pick a topic. First, I decide what I want to form an opinion about. My basic approach here is: “Find claims that are important if true, and might be true.”

This doesn’t take creativity. We live in an ocean of takes, pundits, advocates, etc. I usually cheat by paying special attention to claims by people who seem particularly smart, interesting, unconventionally minded (not repeating the same stuff I hear everywhere), and interested in the things I’m interested in (such as the long-run future of humanity).

But I also tend to be at least curious about any claim that is both “important if true” and “not obviously wrong according to some concrete reason I can voice,” even if it’s coming from a very random source (Youtube commenter, whatever).

For a concrete example throughout this piece, I’ll use this hypothesis, which I examined pretty recently: “Human history is a story of life getting gradually, consistently better.”

(Other, more complicated examples are the Collapsing Civilizational Competence Hypothesis; the Most Important Century hypothesis; and my attempt to summarize history in one table.)

Step 2: read and/or discuss (a bit). I usually start by trying to read the most prominent 1-3 pieces that (a) defend the claim or (b) attack the claim or (c) set out to comprehensively review the evidence on both sides. I try to understand the major reasons they’re giving for the side they come down on. I also chat about the topic with people who know more about it than I do, and who aren’t too high-stakes to chat with.

In th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reward Good Bets That Had Bad Outcomes, published by Neel Nanda on February 22, 2022 on LessWrong.

Introduction

I am a very anxious person. One of the most damaging ways this manifests is that I am pretty risk-averse and afraid of failure. In situations of uncertainty, I often want to freeze up, and know I’ll feel safer doing nothing.

This is a really bad problem! If I let this dictate my life, I lose a ton of value. In particular, there are a lot of areas in my life that are hits-based, where the best way to be successful is to persevere through many failures and seek the upside risk of things occasionally going super well. I want to be someone who can be a great researcher, find really awesome friends, and generally be ambitious about all areas of my life going well. And to achieve this, it is important that I be the kind of person who can take actions with high expected value, and persevere through failures without feeling paralysed. The key thing going wrong here is that I beat myself up over bad outcomes even when I had no way of knowing it wouldn’t work out, given the information at the time. And anxiety gives me a negative prior and causes suboptimal outcomes to stick in my mind. Failures feel painful in a way that missed opportunities do not.

My solution to this is to think in bets, not outcomes. To clearly notice all of the good bets that I take, the actions that I endorse given what I knew at the time, and to be happy about each of those. And to think of my life in terms of this, rather than the concrete outcomes of the bet, and whether that was a success or failure. At the end of the day, the only thing I can control is the bets that I make, and the policies I follow, and there will always be uncertainty on the outcomes. And if I make a good bet with a bad outcome, I should be happy about this, not sad! I refuse to let my negative emotions be tied to things fundamentally outside my control.

My Underlying Model

I first formed this view when I did a trading internship a few years ago. In settings like financial trading or poker, the fundamental skill is about engaging well with uncertainty, and getting past anxieties is a key part of that! And noticing all of the expected value I was missing when I froze up did a lot for helping me notice this failure mode and learning how to solve it. And though the lessons generalise, real life is often a much harder learning environment than these settings - I make fewer bets and so get fewer data points, and it’s much harder to explicitly calculate what’s going on.

I find it easiest to understand what’s going on here and how to fix it when thinking of myself as having a reinforcement learning system inside my head, shaping my actions. I take actions in the world, get feedback from my environment, and use this to update the policies that I follow. Within this framing, there are two clear problems with learning to make good bets with high upside, while being anxious.

The first problem arises because reality is noisy! Even if I had zero anxiety, fundamentally reality has unknowns and I must make decisions under uncertainty. But, by default, I only learn about my actions from their outcomes. And this makes it really hard to learn strategies around pursuing occasional major upsides! It’s obviously worth it to go on 99 unsuccessful dates if the hundredth results in marriage. But by default, my reinforcement learner will likely be discouraged and stop after 99 failures. While, if I can reframe it as 99 successful bets, then I get much better!

The second problem comes from anxiety, which causes me to over-update on negative feedback, and consider it way more important than positives. This is a fundamental issue with my learning algorithm that means I will learn systematically bad policies. By focusing on the action I took being ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Butterfly Ideas, published by Elizabeth on February 22, 2022 on LessWrong.

Or “How I got my hyperanalytical friends to chill out and vibe on ideas for 5 minutes before testing them to destruction”

Sometimes talking with my friends is like intellectual combat, which is great. I am glad I have such strong cognitive warriors on my side. But not all ideas are ready for intellectual combat. If I don’t get my friend on board with this, some of them will crush an idea before it gets a chance to develop, which feels awful and can kill off promising avenues of investigation. It’s like showing a beautiful, fragile butterfly to your friend to demonstrate the power of flight, only to have them grab it and crush it in their hands, then point to the mangled corpse as proof butterflies not only don’t fly, but can’t fly, look how busted their wings are.

When I’m stuck in a conversation like that, it has been really helpful to explicitly label things as butterfly ideas. This has two purposes. First, it’s a shorthand for labeling what I want (nurturance and encouragement). Second, it explicitly labels the idea as not ready for prime time in ways that make it less threatening to my friends. They can support the exploration of my idea without worrying that support of exploration conveys agreement, or agreement conveys a commitment to act.

This is important because very few ideas start out ready for the rigors of combat. If they’re not given a sheltered period, they will die before they become useful. This cuts us off from a lot of goodness in the world. Examples:

A start-up I used to work for had a keyword that meant “I have a vague worried feeling I want to discuss without justifying”. This let people bring up concerns before they had an ironclad case for them and made statements that could otherwise have felt like intense criticism feel more like information sharing (they’re not asserting this will definitely fail, they’re asserting they have a feeling that might lead to some questions). This in turn meant that problems got brought up and addressed earlier, including problems in the classes “this is definitely gonna fail and we need to make major changes” and “this excellent idea but Bob is missing the information that would help him understand why”.

This keyword was “FUD (fear, uncertainty, doubt)”. It is used in exactly the opposite way in cryptocurrency circles, where it means “you are trying to increase our anxiety with unfounded concerns, and that’s bad”. Words are tricky.

Power Buys You Distance From The Crime started out as a much less defensible seed of an idea with a much worse explanation. I know that had I talked about it in public it would have caused a bunch of unproductive yelling that made it harder to think because I did and it did (but later, when it was ready, intellectual combat with John Wentworth improved the idea further).

The entire genre of “Here’s a cool new emotional tool I’m exploring”

The entire genre of “I’m having a feeling about a thing and I don’t know why yet”

I’ve been on the butterfly crushing end of this myself- I’m thinking of a particular case last year where my friend brought up an idea that, if true, would require costly action on my part. I started arguing with the idea, they snapped at me to stop ruining their dreams. I chilled out, we had a long discussion about their goals, how they interpreted some evidence, and why they thought a particular action might further said goals, etc.

A week later all of my objections to the specific idea were substantiated and we agreed not to do the thing- but thanks to the conversation we had in the meantime, I have a better understanding of them and what kinds of things would be appealing to them in the future. That was really valuable to me and I wouldn’t have learned all that if I’d crushed the butterfly in...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Harms and possibilities of schooling, published by TsviBT on February 22, 2022 on LessWrong.

To explore better possibilities for nurturing new minds, and to care about the problem in the first place, it helps to remember what's wrong with what we do to new minds. John Taylor Gatto speaks about this from experience and insight: Seven Lessons Taught in School, 1991

(If you're going to read the following, at least read the seven lessons (part I) of that essay.)

Here's another list of harms caused by schooling.

1. You aren't a mind, and don't bother trying to behave like one.

Children naturally attend to things until they're done with them:

From Maria Montessori, My System of Education, 1915: IPFS pdf link

A little girl, about three years of age, was deeply absorbed in the work of placing wooden blocks and cylinders in a frame for that purpose. The expression of her face was that of such intense attention, that it was almost a revelation to me. Never before had I seen a child look with such "fixedness" upon an object, and my conviction about the instability of attention which goes incessantly from one thing to another, a fact which is so characteristic in little children, made the phenomenon the more remarkable to me.

I watched the child without interrupting her, and counted how many times she would do her work over and over. It seemed that she was never going to stop. As I saw that it would take a very long time, I took the little armchair on which she was sitting and placed child and chair on the big table. Hastily she put the frame across the chair, gathered blocks and cylinders in her lap, and continued her work undisturbed. I invited the other children to sing, but the little girl went on with her work and continued even after the singing had ceased. I counted forty-four different exercises which she made, and when she finally stopped, and did so absolutely independently from an exterior cause that could disturb her, she looked around with an expression of great satisfaction, as if she were awakening from a deep and restful sleep.

The impression I received from the observation was that of a discovery. The same phenomenon became very common among those children, and it was noticed in every school in every country where my system was introduced; therefore it can be considered as a constant reaction which takes place in connection with certain exterior conditions that can be well established. Each time a similar "polarization" of the attention occurred, the child began to transmute itself completely; it became calmer, more expressive, more intelligent, and evidenced extraordinary interior qualities, which recalled the phenomena of the highest mentality. When the phenomenon of polarization of the attention had occurred, all that was confused and drifting in the conscience of the child seemed to assume a form, the marvelous characters of which were reproduced in each individual.

School usually steamrolls this process. The teacher has dominion over your attention; you have to either listen to what the teacher is saying, or work on the worksheets. Your location and (nominal) topic switches every hour. Even at home you do worksheets and projects. Your attention isn't you or yours; you're at war with your attention, it says no but your teacher--and implicitly your parents and college admissions and society, and "you"--insist.

Not only are you not the authority on what is worth it for you to spend minutes, days, or years on; if you are even consulted, it's superficial ("Which current event to do a report on? There are many options.") and maybe only for Potemkin village purposes ("We offer many electives."). You have to be a receptacle for what the teacher has brought to give you, because you don't have a perspective, you aren't an organized/organizing entity, and therefore ca...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some reflections on testing fit for research, published by rosehadshar on February 21, 2022 on The Effective Altruism Forum.

Thanks to Michael Aird for feedback on the ideas in the post, and to Nora Ammann and Damon Binder for feedback on the post itself.

At the end of last year, I decided that I wanted to pursue some research interests of mine, and test my fit for research in the EA space more generally. I ended up on a 3 month, part time contract as a researcher, working on various aspects of the history of social movements with a mentor at a different organisation.

Looking back over these 3 months, I think I learned some pretty useful things from trying to test my fit for research. In this post, I try to share these learnings, primarily with an audience of people who are interested in doing research, but don’t have much experience yet.

Context

My research background: prior to this three month project, I had done a masters in literary history and some policy research, but no directly truth-seeking work on EA relevant questions. I had also worked in research management for a few years at FHI, so I had a fair bit of context on doing EA research. No quant or technical background.

The research I did: there are lots of different sorts of research, and these learnings won’t port across to all of them. Types of research I did:

Background reading (e.g. reading lots of wikipedia pages on different social movements, reading introductory literature from the field of social movement studies)

A bit of literature review work (e.g. what have EAs already written on the history of social movements and how good is it)

High-level thinking about the space of questions that matter (e.g. a list of research ideas on the history of social movements which I hope to post soon)

Some research analyst style, back of the envelope answers to quantitative-ish questions (e.g. this post on the rate of terrorist attacks for different groups)

Notes on my own credibility:

Research: from this test, I think I have the potential to be an average but not great researcher, within the domain I’m interested and competent in. If you want advice on how to do great research, ask a great researcher.

Testing fit for research: I think I did a good job at testing my fit for research, though there were things that could have been better.

I only drew a tentative and quite mild conclusion from the test (average but not great). However, I think it’s hard to get very conclusive results from this sort of test (loads of confounders), and so for a 3 month test this is par for the course.

I think I learned a lot from the test. You can assess this directly from this post.

I also made some mistakes in the way I set up the test. In particular, I think I should have tried harder to get someone to mentor me on a question they cared about for their own research, and that I should have set my goals more clearly.

Reflection: I think I’m above average at reflection. You can assess this from this post, and the post here.

The below is basically a series of short reflective essays. Rather than offering abstract models, I’m trying to share the texture of my experience, in the hopes that this will make the things I’m writing about easier to really ‘get’. Unfortunately this writing style is also quite wordy. For a quicker read, skim the headings, and then pick ones that sound relevant to you to read in full.

Forming opinions is really useful

It’s (relatively) easy to ask interesting questions and speculate on their answers, but harder to actually come out and say ‘currently x is my best guess’, because then you can be wrong.

Testing my fit for research forced me to form opinions in various ways:

To try to answer a research question, I needed to form some kind of opinion.

I knew my mentor would ask me what my opinion was, and wanted to be ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ELK Proposal: Thinking Via A Human Imitator, published by Alex Turner on February 22, 2022 on The AI Alignment Forum.

I want to use the AI’s intelligence to figure out how to translate into the human ontology. The hope is to route a smart entity's performance through a dumb entity's understanding and thereby get the smart entity to solve interpretability-by-the-dumb-entity. While my proposed current architecture overcomes a class of counterexamples in a manner which I find elegant, it is still broken by several plausible counterexamples.

A previous version of this post linked to a Google doc.

Previously, on ELK

This section summarizes ELK. Skip if you’re already familiar.

In the mythos of Eliciting Latent Knowledge, there exists a forbidden vault locking away an enormous diamond:

In order to fend off thieves, this vault brims with controllable traps and tricks. We've trained a really smart AI to take in a video feed, and consider an action sequence (e.g. "open the trapdoor"), and then predict what the vault camera will show if those actions are taken. At the end of the day, we just want to know whether the diamond is safe. Looking at the camera is not always enough:

But the AI is really smart. It knows what happened to the diamond. Can we elicit its latent knowledge?

The original ELK document tries to figure out how to train a "reporter" to figure out what's going on in a really smart AI predictor. The intended reporter would directly translate from the AI's ontology to the human’s ontology, and then answer whether the diamond is there or not. In the original ELK document, the reporter is wired as follows:

The training questions/answers are generated by a human considering a range of simple, human-comprehensible situations (i.e. (video, action sequence) tuples), querying their model of what happened, and answering e.g. whether the diamond is safe or not:

We never want there to be a situation where the AI predictor (the "figure out what's going on" module) unambiguously knows that the diamond was taken, but the reporter says "the diamond is fine." More precisely, we don't want such a configuration to plausibly minimize training loss.

The direct translator minimizes training loss because it answers all questions correctly on the training distribution. So far, so good. However, the human simulator reporter also minimizes training loss, by extracting the predicted video completion from the AI predictor, and then conditioning the human labeler on the entire video and action sequence, and then outputting whatever the human would think about the situation. On the training distribution, this setup gets all of the questions right (and so minimizes training loss), because it exactly reproduces the labeling process! But in general, the human won't understand many situations, and so the human simulator will output answers which the AI predictor knows to be incorrect.

The direct translator solves ELK for narrowly defined questions (e.g. "is the diamond still in the room?"). How do we get it?

Thinking Via A Human Imitator

I want to structure the problem so that direct translation is instrumentally required for minimizing predictive training loss. To that end: What if the AI had to do some proportion p of its thinking in the human net?

Assume that both the AI and the human reason using Bayes nets. Suppose we have access to a perfect simulation of not only a human Bayes net, but of the human who labels the training data.

The AI receives the first part of a video and a sequence of actions. Suppose the AI can think for n time steps in total, before it has to predict the remainder of the video. At timestep (1−p)n/2, the AI encodes its cognitive work into the human imitator net, where the human’s inference algorithm runs for another pn steps. Lastly, the AI decodes the human’s state of ...