Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How complex are myopic imitators?, published by Vivek Hebbar on February 8, 2022 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

TL;DR: Time-limited myopic agents might be desirable for their safety properties; one such case is that of "myopic imitators", which imitate a non-myopic process like a human, but do so without considering consequences. Since DL systems are biased towards simple functions, I consider the question of whether myopic imitators are simpler or more complex than their deceptively aligned competitors.

For this analysis, we impose a strong assumption which is similar to minimal program complexity, as well as a clean division between a world model, an "internal interpreter", and a main algorithm which is executed. The analysis is structured as follows:

Operationalizations for "myopia" and "simplicity prior"

Methodology for assessing the simplicity of a candidate algorithm

Analysis of candidate algorithms under various sets of further assumptions

Under these very strong assumptions, and out of the algorithms analyzed, myopic imitators are usually favored. One of the two leading myopic imitators is a straightforward human-imitator which seems to have nice properties, but is sometimes disadvantaged by an inability to do a certain Goodhart-style exploit. The other one, though myopic, is a "training set predictor" which generalizes terribly in some cases. That being said, the specific results are to be taken with a grain of salt, since the assumptions are unlikely to hold in reality. The main value going forward is the general approach and methodology.

Assumptions, worldview, and research vision

The analysis in this post applies to a world where a very specific set of assumptions applies. To be clear, I am quite confident that this set of assumptions does not apply to current deep learning systems. Current deep learning systems seem to be characterized more by circuits and interpolation than by the types of programs I consider here. However, the analysis here is interesting as a limiting case, and the methodology might be transferable to more realistic assumption sets.

The main value of this post is to provide a particular frame and methodology for thinking about the question: "How is a system with a strong simplicity prior likely to turn out by default?"

By default, when I say "simple" or "complex", I am referring to program length, not runtime. However, note that program length is not the only possible measure of complexity; in fact, the current inductive bias literature uses a different measure, as will be discussed later.

1 - Program approximation and power: The trained model closely approximates a program in some language, which is allowed to contain such abstractions as function reuse, recursion, interpretation, etc. This may be realized in reality through things like recurrence.

2 - Internal interpreter: The model will contain some sort of "interpreter" internally which can take an algorithm represented in terms of internally defined tokens and execute it. This might follow from minimal program length; a large program in an inefficient language can be refactored as {interpreter for a more suitable language} + {much shorter program}. One effect of this assumption is that the complexity of other parts can depend on the lengths of tokens in the language of this internal interpreter.

3 - Simplicity prior: Training a model to zero loss corresponds to taking the set of all programs below a certain length and runtime which perfectly fit the training set, then choosing one at random, weighted by some simplicity prior. In other words, the probability ratio between two perfect algorithms depends on only th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Experimenting with microsolidarity crews, published by 4thWayWastrel on February 6, 2022 on LessWrong.

In 2020 I was introduced to "microsolidarity", a set of ideas and methodologies on relating to others in groups. Over the last 6 months I've been experimenting with it in the form of "crewing". This post is a casual outline for anyone interested of the steps I took organising crews of rats and EAs to meet weekly and:

help each other with life problems

nerd out over mutual obsessions, from wellbeing to emergence to ambition

just generally connect as human beings

I didn’t set out with a plan, I just did what felt right at the time so take this as less of a recipe and more of a retrospective from which you might want to pick and choose some stuff of your own to try!

What I did:

#1 Collect "free electrons": I'd keep an eye out for people I suspected might be interested in crewing and would be fun to hang out with. I found mine:

In "introductions" slack channels for events (be on the lookout for key interests like "community", "governance", "wellbeing" or "relating")

At virtual conferences for communities like effective altruism, complexity weekend, or radicalXchange

People I know in real life (though never more than 1 per crew)

People recommended by other free electrons

#2 Connect: I'd reach out to these people to see if they'd be interested in a 1-1 chat

Normally this would be a message like "Hey, if you were interested in jumping on a call sometime to connect I'd love to chat about X 😊"

I'd go on a walk, call them up at the agreed time, and just get to know them. The conversations might involve:

swapping backgrounds

discussing ambitions

ending up talking about shared topics of interest

Normally they'd also nerd out about something I think a lot about (community, utilitarianism, public goods, civics, etc) and the conversation would just spark up

#3 Mention Microsolidarity: Inevitably I'd start talking about theories of groups or experiences I'd had crewing, and if they showed an interest I'd mentioned there might be an opportunity to join a time bounded crew coming up. If they seemed keen I'd add them to my list.

Note: I've noticed people are normally a lot more interested in joining something with a time limit, rather than an indefinite commitment.

#4: Form the crew: Once there were 3 free electrons who've expressed interest I thought would vibe I just dumped everyone in a group-chat and we found a time to meet weekly for 6 weeks. I would just go with combinations of people that "felt right", but two things I came to realise I was doing were:

Organising the group around a shared interest like "ai-safety", "wellbeing", or "civics". Basically people who get fired up over the same thing.

Trying to avoid an "EQ" vacuum in a crew. The more people who feel comfortable discussing the way something is making them feel right now, rather than what it makes them think of, the better. Cerebral groups aren't worse than "feely" groups, but I was gunning for the latter.

#5 Have an intro session: We'd get on a video chat and spend 90 minutes getting to know one another. Generally I found everyone gets so high on "new crew energy" that 🔥 topics of conversation will just spontaneously emerge as people introduce themselves. Though if the spirit of emergence doesn't fall on the group you may need to be willing to "stand up on stage" and act as a foil for the group, talking about something you're currently passionate about.

#6 Case clinics: Case clinics are the secret ingredient. Over the next 4 sessions each person would bring some problem they were working on to the group, and receive 90 minutes of undivided attention from their peers. I would ask around before the session to see who was interested in volunteering to bring a case. Examples of some of the topics we've had:

Social anxiety...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Capability Phase Transition Examples, published by gwern on February 8, 2022 on The AI Alignment Forum.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Release of Existential Risk Policy database, published by rumtin on February 7, 2022 on The Effective Altruism Forum.

Today, we are releasing an existential risk policy database:

This database collects and categorises the policy ideas and recommendations put forward by the field of existential risk studies.

If this field wants to maximise its impact, it needs to develop and communicate high quality policy ideas. Many in the field are trying to do so. However, until now, policy ideas are spread across various sources, at varying levels of specificity or quality, for different jurisdictions, and on different aspects of the risks.

The purpose of this database is to enable researchers and other members of the field to quickly identify policy ideas for existential and global catastrophic risks. For example, the database could help researchers to select and package up the policy ideas, such as for a meeting with a policy-maker or researching for a paper. The database may also reveal gaps where more policy work is needed, such as risks that have few or flawed policy ideas.

As at 7 Feb 2022, 257 policy ideas from 37 sources have been included in a full (non-public) database. The publicly accessible version of the database we are releasing today only includes those ideas deemed sufficiently specific, effective or feasible. The judgment is not based on thorough analysis - it is a prima facie assessment based on the views of the database contributors. We’ve cut a reasonably high threshold at this stage to ensure we are not promoting ideas that lack specificity or are not ‘ready for prime time’. We will look to reassess or loosen to have many more ideas included.

This is just the start. We intend to continue improving the database. For now, we hope you find the database a useful resource for your policy endeavours.

If you wish to view the full database, please contact Rumtin on [email protected].

Q&A

Which sources are included?

So far, we have focused on post-2015 works by individuals or organisations that entirely or explicitly focus on existential and global catastrophic risk. In the full database, we seek to cast a wide net. We include ideas from both formal products - such as academic papers, policy papers and submissions, technical reports and books - as well as from less formal settings, such as podcasts, forum posts or unpublished shared documents. In the publicly accessible database, however, we filter out the informal works.

What is a ‘policy idea’?

A policy idea is where the author(s) suggest or recommend a course of action for a national government or international body.

For sake of completeness, vague policy ideas or broad policy goals (‘The government should improve its understanding of existential risk”) are included in the full database, but are filtered out for the publicly available version. We also err on the side of including long-termist ideas even if they are not strictly related to existential risk.

What can I expect next?

This database require more work to make it a more powerful resource. The following steps are being considered:

Mine more sources for policy ideas. (We are roughly 15% of the way through the possible policy-relevant works on existential and global catastrophic risk)

Properly evaluate existing policy ideas to identify and promote the set of high-quality ideas.

Set up a google form so that individuals can submit their own policy ideas.

Develop short policy briefs for specific risks or risk areas that combine and present policy ideas in a digestible manner.

We intend to submit a funding application to hire researchers that can complete this work.

Any interesting insights so far?

We have some very early findings - though take them with a grain of salt given the relatively early state of the database. As more sources are included, we will have b...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why will AI be dangerous?, published by Legionnaire on February 4, 2022 on LessWrong.

LessWrong should offer a short pitch on why AI will be dangerous, and this aims to be that.1

Many people think, "Why would humanity make dangerous AI? Seems like a stupid idea. Can't we just make the safe kind?" No. Humanity will make dangerous AI for the same reason we made every other technology dangerous: it's more useful.

A knife sharp enough to cut fruit can cut your finger. Electrical outlets with enough power to run your refrigerator can stop your heart. A car with enough horsepower to carry your family up a hill can easily kill pedestrians. Useful systems must be dangerous because useful implies they can have large effects on their environment. "Large effects" can be good or bad depending on your perspective.

We'll make AI powerful too: knowledgeable enough to cure diseases, tactical enough to outsmart terrorists, and capable enough to run an economy by itself, all so we can relax! That means our AI will also be knowledgeable enough to invent new diseases, tactical enough to outsmart freedom fighters, and responsible enough to run a military-industrial complex all by itself.

We won't do the latter on purpose any more than we crash cars or cut our fingers on purpose, the problem is that there are more ways for complex systems to go "wrong" than "right" because this universe is psychopathic, even when nobody is being malicious. The asteroid that killed the dinosaurs wasn't malicious, it just didn't care. Nowhere in the behavior of the asteroid was encoded a consideration of life, or even itself. It just followed mathematical laws (of gravity) which don't care about "morality".

Like the asteroid, AI systems just follow mathematical laws (of intelligence) which don't care about "morality". We'll add safety components that can consider moral questions, but mistakes will be made as have been made with all previous technologies.

The odds of a mistake go up a lot when you consider that the whole selling point of AI is to be smarter than us, and that entities smarter than you are unpredictable. They will do things you didn't think of, sometimes things you didn't even know were possible. AI presents a unique and unfamiliar danger to humanity, because unlike other technologies, an AI disaster might not wait around for you to come clean it up. That wouldn't be very intelligent, would it?

Footnotes

1. Most posts are long, detailed, and use esoteric language. Not good for first impressions. Entry level posts should be easier to find, too.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Stop procrastinating on career planning, published by freedomandutility on February 6, 2022 on The Effective Altruism Forum.

After 2+ years of EA involvement, I finally stopped procrastinating and applied to speak to 80000 hours.

I think the thought of discussing my career plan with EAs makes me feel self-conscious - what if they think I'm not being selfless enough, or disagree strongly with my views on global priorities?

I also find thinking about careers stressful and scary in general. I strongly believe that your career is the best way to do good, and that some careers will do a lot more good than others. These are some of my life's biggest decisions, and will always feel scary and stressful to think about, with insecurities such as:

"What if I'm missing an opportunity for much larger impact?"

"What if I sacrifice too much personal happiness?"

"What if I completely change my mind about EA?"

As a result, I often avoid thinking about my career plans.

I had also convinced myself that 80000 hours' advice won't be that useful. But rationally, it's likely that people with experience in EA- aligned careers advising will give useful advice. And it seems very unlikely to be harmful, and obviously, if I don't want to follow their advice, I don't have to.

Rationally, I really should have applied to speak to 80000 hours 2+ years ago!

If I had done this 2 years ago, I probably could have made more progress in whichever direction I'd settled on.

If like me you're procrastinating on career planning, take this as a nudge to apply to 80000 hours.

Applying to 80000 hours also made me think I should write out a more formally reasoned EA career plan to share with them (and with family, friends and other EAs I know).

I've had a career planning Google Doc for a long time, but I was shocked by how much more carefully I was thinking about things when I knew that I'd be sharing the plan with someone else.

I would definitely recommend that other people also write up their career plans with the intention for someone else to look at and give feedback on.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sledding Is Underrated, published by jefftk on February 5, 2022 on LessWrong.

Sledding is pretty great:

No lines. You don't have to wait, just go.

Minimal cost. A basic plastic sled costs $15-$20 at a hardware store, and while they wear out pretty quickly it's still less than

$5/outing. Aside from that, all you need is standard winter gear

(including snowpants).

Close. Around here there are two decent places to sled within a

15min walk, and several within a 15min drive. You can go sledding after work, or as one of several activities on a Saturday.

Good learning curve. There are many hills where all you have to do is sit on your sled, no skill required. As you figure out how to steer, this opens up other hills, and more interesting options for going down existing ones.

Fast. Zooming down a hill is just really fun.

The competing ways to move similarly fast are much more expensive, require more travel, and you spend much more time time waiting around and less actually moving: amusement parks, skiing, etc. I'm not trying to knock those activities, they are definitely fun, but I think sledding more than holds its own here.

This morning the kids and I went with their aunt to a park near our house:

We were there for about an hour and a half, and there were less than ten other people sledding. I'm sure there are more now, since it's a bit later in the day, but still, compared to the number of people in the area I think would enjoy sledding it's surprisingly low.

Why is that? Some guesses:

You have to walk back up the hill. True, but it's good exercise and it beats standing in line.

There isn't any advertising for it, beyond advertising for sleds.

In most places it can't be scheduled. You have to notice that the conditions are right and decide to go, typically on less than a days notice.

We think of it as something for kids. But amusement parks have this too and are still popular with all ages.

In places that have good sledding many times in the winter people think of it as a normal thing, and we get more excited about uncommon activities?

Maybe people don't know where to go? There are sites about the best sledding options, but they aren't great -- some sort of sledding hill review site where people describe hills and give ratings could be nice.

If you're in Boston and have been meaning to give it a try, conditions are good this weekend. I'd be happy to lend you a sled and/or other gear, and give suggestions of places to go!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What's your prior probability that "good things are good" (for the long-term future)?, published by Linch on February 5, 2022 on The Effective Altruism Forum.

Sometimes, as a way for me to get more strategic clarity about what intermediate goals I want my research to accomplish, I try to evaluate whether things that are locally good/bad are good for the long-term future. For example, technological growth, economic growth, forecasting, IIDM, democracy, poverty reduction, non-existential catastrophic risks, and so forth.

My standard argument/frame of thinking goes like this:“Well, you start with a prior of ~50-50 that any given macroscopic change is good for the long-term future, and then you update on the evidence that-”

And if this is done in conversation, my interlocutor often interrupts me with“50-50 is a crazy prior because-”

And often it’s some argument that locally good things should be expected to be globally good. Sometimes people reference flow-through effects.There’s different flavors of this, but the most elegant version I’ve heard is “duh, good things are good.”

And like, I sort of buy this somewhat. I think it’s intuitive that good things are good, and I’ve argued before that we should start with an intuition that first-order effects (for a specific target variable) are higher than second-order effects. While that argument is strongest about local variables, perhaps we should expect that generally there’s a correlation between a thing’s goodness on one metric to its goodness on other metrics (even if the tails come apart and things that are amazing for one metric aren't the best for other metrics).

But when it comes to the long-term future, how much should we buy that things that are considered good by near-term proxies that don’t consider the long-term future are good for long-term stuff?

Put another way, what’s our prior that “an arbitrarily chosen intervention that we believe to be highly likely to be net positive for the experience of sentient beings in the next 0-5 years increases the likelihood of P(utopia)?"

Related post (though I don't exactly agree with the formalisms)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bryan Caplan meets Socrates, published by Sam Enright on February 5, 2022 on LessWrong.

Socrates and Glaucon are walking down from the Acropolis, when they encounter a stranger from a distant land.

Caplan: Greetings, Socrates.

Socrates: Greetings, stranger. From whence do you come?

Caplan: I am from a faraway land.

Socrates: Sparta? Thrace?

Caplan: Much further out than that.

Socrates: Where, then?

Caplan: It is not important right now. I have heard that you are the wisest man in Athens, and I have sought your expertise. Socrates, what is the purpose of education?

Socrates: To refine virtue, of course.

Caplan: And so those with an education are more virtuous than those without?

Socrates: Yes.

Caplan: Is it not true, then, that those with an education will be entrusted with greater responsibilities? That they will be made rulers, put in charge of important military expeditions, and will be respected craftsmen?

Socrates: Of course.

Caplan: After a time, will men not seek out an education just for these good consequences?

Socrates: They surely will. It would be better if they sought education for its own sake. Some men will seek it for its good consequences, but at least some will refine their virtues in doing so.

Caplan: What if sophists took over the academies, and no longer taught virtue at all? Men would learn nothing of import, and only become educated to enter the skilled professions.

Socrates: No one would trust such academies.

Caplan: Perhaps. But what if the academies taught both virtue and sophistry? Would the self-interested man not take lessons so as to give the appearance of virtue, while exerting himself to the minimum extent? And imagine, Socrates, that you are employing a skilled professional. Would you not employ that man with the greatest education?

Socrates: I surely would.

Caplan: Is it not the case, then, that to the professionals looking for workers, it does not matter whether they had a valuable education? It only matters that their education signals them to be good workers, who will show up on time and work to their greatest extent?

Socrates: It appears so.

Glaucon: Your words are indeed convincing, traveller. However, I do not see their import. Athens is the most learned city of them all, and even here boys are educated only for a few years. Boys will not sit around learning sophistry if there are wars to fight, or if there is food to grow.

Caplan: That is no doubt true, Glaucon. However, consider this: a ruler will be popular if he supports education. The people are not trained in philosophy, and they cannot follow the argument I have given you. And if they can, they do not wish to.

Socrates: The purpose of a ruler is not to be popul-

Caplan: Yes, yes! But it is only natural for a ruler to desire to be liked by his citizens.

Socrates: The education of a ruler should rid him of such desires, as I discussed before with Glaucon.

Caplan: Have we not already said that the academies can be infiltrated by sophists?

Socrates: I know of no such academy that philosophers respect.

Caplan: But men in the military and the skilled professions are not philosophers. They must rely on crude appearances, to save time. But let us put this aside for the moment. Rulers will be popular if they support education. They will also have been told from a young age that education instils virtue, even if it does not. Teachers themselves stand to gain a great deal from maintaining the prestige and wealth that rulers grant them. Rulers therefore will give much more wealth and esteem to education than it deserves.

Glaucon: Rubbish!

Socrates: Glaucon, restrain yourself! Our traveller has proved himself to be philosophically learned. But it is getting dark, and Glaucon must return home. I will think this over and we will discuss it in the morning.

The next evening.

Caplan: Socra...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The best $5,800 I’ve ever donated (to pandemic prevention)., published by ASB on February 5, 2022 on The Effective Altruism Forum.

Disclaimer: I lead grantmaking for biosecurity and pandemic preparedness at Open Philanthropy, but this endorsement is written in a purely personal capacity.

Carrick Flynn is running for congress in Oregon’s 6th district as a Democrat. He is also the first person to ever run for US congress on a platform of preventing future pandemics. This campaign provides a rare opportunity for smaller donors to make a large impact, since political contributions are capped at $5,800 per person ($2,900 for the primary and $2,900 for the general, but you can donate both up front).

His campaign website is here, launch video here, and donation link is here. Only US citizens and permanent residents can donate. The campaign also needs volunteers and staff (more details at the bottom of this post).

About Carrick

Carrick is a close personal friend and somebody I have enormous admiration for. I’ve worked with him since 2015, when he came to Oxford and subsequently set up the Centre for the Governance of AI. His wife Kathryn Mecrow-Flynn is also amazing, being the founder and CEO of Magnify Mentoring (formerly WANBAM).

Carrick embodies a number of virtues that I want to briefly highlight, and his story is remarkable. He grew up in poverty in rural Oregon, in an abusive household. A flood left him homeless as a child, and he shared a mattress on the floor with his brother until he was 17.

These experiences drove him to help others in poverty. After graduating from the top law school (Yale), he disregarded lucrative career opportunities to work in international development. In India, he saved potentially thousands of lives by clearing a roadblock to a nationwide vaccination program. He also secured a court decision that reallocated over $1 billion to high impact health programs by manually going through over 1,000 pages of accounting documentation.

This is just one instance of Carrick’s determination and work ethic. At Oxford he worked harder than almost anybody else I knew—in fact, out of everybody I’ve worked with in the past decade, my time in Oxford with Carrick would put him in the top 5 people I’ve seen in terms of raw work ethic.

He’s also brilliant. I’ll skip shallow measures of this[1] and just say that I saw Carrick’s intellect firsthand when he pivoted to thinking about pandemic preparedness during COVID. There he would rapidly digest information and connect it to what needed to be done. He’s likely the only person running for congress who has nuanced opinions on pathogen sequencing, platform vaccine tech, and what should go into the strategic national stockpile.

Finally, many people who are as brilliant and driven as Carrick are assholes, and Carrick is emphatically not. He is one of the most warm, caring, and supportive people I know. He cares for animals too, and he and his wife are vegan (and not in an asshole-like way).

I also know that Carrick is fiercely loyal. If he wins, his constituents in Oregon will have a representative that takes this duty seriously and puts their interests above his own.

The importance of a champion in Congress

After 9/11, the U.S. government spent over $1 trillion on counterterrorism to ensure something similar wouldn’t happen again. After losing almost 900,000 lives and $7.6 trillion in economic damage to COVID, what do you think Congress is doing to prevent the next pandemic?

The Biden administration released a fantastic $65 billion plan that aims to prevent future pandemics. Congress has funded practically none of it. Part of the problem is that nobody in congress has made pandemic preparedness a ‘core issue.’ Congressional members don’t oppose the president’s plan, and there are some standout champions, but none of t...