Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cause area: Short-sleeper genes, published by JohnBoyle on August 10, 2022 on The Effective Altruism Forum.

The claim

There is decent evidence that there are "short sleeper genes": individual genetic mutations that cause a human to naturally get only 4-6.5 hours of sleep per night (instead of ~8) while remaining healthy, happy, and productive. There don't seem to be any negative side effects, although there might be positive ones.

So far, 5 short-sleeper mutations across 4 genes have been identified. Probably at least one of these is a good candidate for developing a drug to mimic the effects of the gene.

Such a drug would be enormously valuable. If it worked, presumably it would give people at least one extra waking hour per day, and maybe the "higher resilience and productivity" side effects would be real too. If you care about overall utility, then consider the value of 1 extra waking hour per day, times 365 days per year, and imagine that 100 million people ended up taking the drug; if we say each hour is worth at least $1 (a lowball), then that's $36 billion per year.

Just to illustrate the point, if an investment yielding 3% returns were that profitable, the investment would be $1 trillion. Even if you assume the probability of success were as low as 1%, that'd be worth $10 billion. Such an opportunity is clearly worth looking into.

The evidence

I've skimmed several of the papers on the subject of familial natural short sleepers (FNSS), most of which were written by UCSF professor Ying-Hui Fu and colleagues. (Disclosure: I've emailed Ying-Hui, video chatted with her, and consider her something of a friend.) As I understand it, what they've done is collect people who might be natural short sleepers, validate that they are, look among family members, find candidate short-sleeper genes, and test those genes in mice or similar organisms, and publish papers if an effect is demonstrated in the mice.

The DEC2 mutation reduced sleep in mice by about an hour. The ADRB1 mutation reduced by 55 minutes. NPSR1 by 71 minutes, and the two GRM1 mutations (roughly equivalent to each other) by 25 minutes. Ying-Hui says it's expected that the effects in mice are smaller than in humans.

All the genes in question are known to affect wakefulness. For example, the DEC2 protein suppresses expression of a neuropeptide called orexin, and mutant DEC2 suppresses it less; orexin is known to promote wakefulness (e.g. people who can't produce orexin are narcoleptic). This part of the mechanism seems reasonably clear.

What's less clear is how these genes accelerate the repairs/maintenance the body performs during sleep. (If they did not do any such acceleration, then you'd expect these people to be miserable, unhealthy insomniacs.) But there is at least one experiment that seems to show that they do accelerate some type of repair:

They took mice that were engineered to generate amyloid plaques and tau protein tangles—pathologies that seem associated with Alzheimer's disease (though the causality seems unclear and you should take such research with a grain of salt). Anyway, they gave some of them the DEC2 and others the NPSR1 mutations, and found that both genes significantly reduced those Alzheimer-correlated pathologies.

Side effects?

I'll just directly quote this summary article:

[Short sleepers] report greater flexibility around sleep timing and less subjective deficit after sleep deprivation. They often deny experiencing jetlag (unpublished data). However, it is not just sleep duration that characterizes this group of individuals. There is a high behavioral drive among those with FNSS, and individuals report a need to always be mentally active resulting commonly in high profile, high pressure jobs or holding multiple jobs. FNSS individuals also appear to have high pain thresholds and rela...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The History, Epistemology and Strategy of Technological Restraint, and lessons for AI (short essay), published by MMMaas on August 10, 2022 on The Effective Altruism Forum.

In short: some expect that we could not meaningfully slow or halt the development of A(G)I even if we expect extreme risks. Yet there is a surprisingly diverse historical track record of technological delay and restraint, even for strategically promising technologies that were seen as 'obvious' and near-inevitable in their time. Epistemic hurdles around studying 'undeployed' technologies make it likely that we underestimate the frequency of restraint decisions, or misinterpret their causes. From an outside view, this should lead us to be slightly more optimistic about the viability of restraint for future technologies. This analysis does not show that restraint for AGI is currently desirable; that it would be easy; that it would be a wise strategy (given its consequences); or that it is an optimal or competitive approach relative to other available AI governance strategies.

However, possible conditions for- or pathways towards restraint should be explored in greater detail, as part of a comprehensive toolset that offers strategic clarity regarding all options on the table.

Disclaimers:

Background: this short essay was published on the legal academic forum Verfassungsblog, as part of a symposium debate sequence on 'Longtermism and the Law' (see summary twitter thread here). As such it approaches the topic from the perspective of 'legal longtermism', but many points apply to longtermist (or AI risk/governance) debates generally. This post has been lightly edited.

Epistemic status: this is an initial primer on an in-progress research project, and many of the case studies will need further analysis;

My take: it is not yet my view that restraint is warranted as a strategy--nor that it currently compares well to other work. However, I do expect it should be explored amongst other avenues;

Feedback & future work: I will cover this in greater detail, including a discussion of advantages and risks of pursuing restraint, in an upcoming profile of the 'Containing' approach, as part of my 'Strategic Perspectives on Long-term AI Governance' sequence. I welcome feedback.

Abstract: If the development of certain technologies, such as advanced, unaligned A(G)I, would be as dangerous as some have suggested, a longtermist (legal) perspective might advocate a strategy of technological delay—or even restraint—to avoid a default outcome of catastrophe. To many, restraint–a decision to withhold indefinitely from the development, or at least deployment, of the technology–might look implausible. However, history offers a surprising array of cases where strategically promising technologies were delayed, abandoned, or left unbuilt (see in-progress list), even though many at the time perceived their development as inevitable. They range from radiological- and weather weapons to atomic planes, from dozens of voluntarily cancelled state nuclear weapons programs, to a Soviet internet, and many more. It is easy to miss these cases, or to misinterpret their underlying causes, in ways that lead us to be too pessimistic about future prospects for restraint.

That does not mean that restraint for future technologies like advanced AI will be easy, or a wise strategy. Yet investigating when and why restraint might be needed, where it is viable, and how legal interventions could contribute to achieving and maintaining it, should be a key pillar within a long-termist legal research portfolio.

The question of restraint around AI development

In a famous 2000 essay, entitled ‘Why the Future Doesn’t Need Us’, computer scientist Bill Joy grimly reflected on the potential range of new technological threats that could await us in the 21st centu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Cause Area: Differential Neurotechnology Development, published by mwcvitkovic on August 10, 2022 on The Effective Altruism Forum.

Research assistance from AJ Kourabi, Sabrina Singh, and Neiman Mathew.

A shorter, self-contained version of this document was submitted to the Open Philanthropy Cause

Exploration Prizes.

In three sentences

Neurotechnology could have extremely positive or negative impacts on the wellbeing of humanity and other beings in the near- and long-term future.

Almost no efforts have the stated goal of differential neurotechnology development.

There are fundable projects that could help preferentially advance the development of beneficial and risk-reducing neurotechnologies.

Summary

A neurotechnology is any tool that directly, exogenously observes or manipulates the state of biological nervous systems, especially the human brain. Brain-computer interfaces and antidepressant drugs are familiar examples.

Importance

Neurotechnologies could have profound impacts on the near- and long-term future.

In the positive direction, neurotechnologies have the potential to:

Address the growing ~21% share of global disease burden from neurological and neuropsychiatric disorders

Generally reduce suffering and improve subjective wellbeing

Improve individual reasoning and cognition

Inform decision-making on topics like the suffering of non-human minds

Help develop safe AI systems

In the negative direction, neurotechnologies have the potential to:

Lead to addiction or other personal harms

Exacerbate authoritarianism

Irreparably corrupt human values

Increase risks from AI systems

Differential neurotechnology development is the project of preemptively creating the most beneficial or risk-reducing neurotechnologies before any others.

Investing in differential neurotechnology development now could be timely. On current trajectories, neurotechnologies in clinical trials today could have large-scale impacts in 1-5 decades, with 30 years as a mean estimate. And with concerted effort, neurotechnologies currently in clinical and preclinical development could be advanced in 10 to 20 years to the point where they might meaningfully benefit AI safety, in addition to other, potentially less-urgent benefits.

Neglectedness

Of the ~$20B/year that goes toward funding neuroscience overall, ~$4B/year goes toward non-drug, central nervous system neurotechnology development. But almost no efforts have the stated goal of differential neurotechnology development.

Tractability

Fundable opportunities today for a new philanthropist that might help achieve differential neurotechnology development include:

Research and forecasting relevant to differential neurotechnology development, particularly on the cost-effectiveness of specific interventions (<$100k in the near-term)

R&D infrastructure like open-source research software and clinical cohort recruitment (tens of thousands to millions USD)

A patent pool (<$100k total)

Startups or Focused Research Organizations to directly build differentially beneficial neurotechnologies (up to millions of USD per year per project)

Importance

A neurotechnology is any tool that directly, exogenously observes or manipulates the state of biological nervous systems, especially the human brain.[1] Readers may be familiar with neurotechnologies like electrode-based brain-computer interfaces (BCIs), antidepressant drugs, MRI, transcranial magnetic stimulation, or optogenetics.

Neurotechnologies alter the most intrinsic part of ourselves: our minds. So it’s perhaps unsurprising that neurotechnology could have extremely positive or negative impacts on the welfare of humanity and other beings in the near- and long-term future.

Differential neurotechnology development is the project of preemptively creating the most beneficial or risk-reducing neurotechnologies before an...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What’s alive right now?, published by rosehadshar on August 10, 2022 on The Effective Altruism Forum.

In this post, I lay out some different ways of looking at what’s alive right now.

This seems worth knowing about to me, but also hard to grasp because of how huge the numbers are. A lot of what I’m doing in the post is trying to find ways to get my human brain to understand big numbers about how the world is.

I don’t have a considered view on which organisms are sentient or how much, or on how much different organisms matter morally compared to other organisms.

Biomass

I like this graphic a lot:

It’s from Our World in Data article on Biodiversity and Wildlife [I recommend following the link so you can zoom in on the graphic], which pulls from a 2018 paper by Bar-On, Phillips and Milo.

Some of the headlines that OWID pull out:

Life on earth is dominated by plants – they make up 82% of global biomass.

The animal kingdom makes up just 0.4% of global biomass.

Humans account for just 0.01% of biomass. However, our livestock outweighs wild mammals and birds ten-fold.

The reason to look at life on earth in terms of biomass rather than individual organisms is an intuition that this makes it more comparable: otherwise the numbers get dominated by loads and loads of tiny things, and the bigger things (which most people care more about) barely show up.

Numbers

I’m still kind of interested in the absolute numbers though. I think this is partly because my brain is better at conceptualising ‘individual organisms’ than ‘weight in tonnes of carbon’.

Bar-On, Phillips and Milo do give estimates for the absolute numbers (which they call ‘abundance’), to the nearest order of magnitude:

TaxonBillion tonnes of carbonAbundancePlantsTrees 45010^13BacteriaTerrestrial deep subsurface60 10^30Marine deep subsurface7 10^29Soil7 10^29Marine1.3 10^29Total 7010^30Fungi 1210^27ArchaeaTerrestrial deep subsurface4 10^29Marine deep subsurface3 10^29Soil0.5 10^28Marine0.3 10^28Total 710^29Protists 410^27AnimalsChordatesFish0.7 10^15Livestock0.1 10^10Humans0.06 10^10Wild mammals0.007 -Wild birds0.002 10^11ArthropodsTerrestrial0.2 10^18Marine1 10^20Annelids0.2 10^18Molluscs0.2 10^18Cnidarians0.1 10^16Nematodes0.02 10^21Total 210^21Viruses 0.210^31

(See also this post by Brian Tomasik for another set of estimates and much number crunching.)

I want to zoom in on the animals part of this table. First, for people who like me don’t intuit how big the difference between 10^10 and 10^18 is, here is the animals part of the same table with zeros:

ChordatesFishLivestockHumansWild mammals-Wild birdsArthropodsTerrestrialMarineAnnelidsMolluscsCnidariansNematodes

1,000,000,000,000,000

10,000,000,000

10,000,000,000

100,000,000,000

100,000,000,000,000,000

10,000,000,000,000,000,000

100,000,000,000,000,000

100,000,000,000,000,000

10,000,000,000,000,000

100,000,000,000,000,000,000

So for every one human, there’s something like 10 billion nematodes.

The estimates Bar-On, Phillips and Milo give are orders of magnitude (which makes sense given how big most of the numbers are). For some of the smaller numbers, we can get more precise estimates. Here’s a table I made with some more numbers I was interested and able to find:

Order of magnitude from Bar-On, Phillips and Milo EstimateYearFish1 quadrillion (a million billions)Farmed fish126.5 billion2015Wild fish999,873.5 billion (999.9 trillion)-Livestock10 billionPoultry25.7 billion2018Cows1.5 billion2018Sheep1.2 billion2014Goats1 billion2014Pigs1 billion2018Humans10 billion-8 billion2022Wild mammals--550 billion-

My brain can track these numbers better than the raw orders of magnitude. But even smaller numbers would be even easier for me to make sense of.

One way of getting smaller numbers is to think about how many organisms of different types there are for every individual hum...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Proposal: Consider not using distance-direction-dimension words in abstract discussions, published by moridinamael on August 9, 2022 on LessWrong.

Epistemic Status: Less Wrong Classic™, a bold and weird proposal to possibly improve your rationality a little bit, maybe.

Overview

It is common to use words and phrases implying distance, directionality, and dimension when discussing ideas and concepts that do not inherently possess these properties. This usually doesn't help and often actively harms our ability to communicate and reason correctly. I am running an experiment to see the effect it has on my thinking and speaking habits.

Rationale

We implicitly translate a thought from its natural form as a clean, clear, distinct concept into a spatial/continuum metaphor all the time, casually and without regard. A mathematician would not try to pull a mathematical object with a thousand dimensions, half of which are binary or non-Euclidean dimensions, into a two-dimensional projection without carefully thinking about the implications of doing so, but we do the equivalent of this all the time, casually and without regard for subtle consequences.

When you mentally impose dimension/distance/direction onto an idea, you risk bringing it into the part of your mind where you are allowed to freely leverage spatial metaphors when thinking about the idea. You will likely have no awareness of all the ways in which you are abusing these spatial/geometric metaphors, because humans are not really smart enough to keep track of all the implications of our casually asserted metaphors. I think that even genius-level humans can be extremely blind to the damage that's done when they implicitly translate a thought from its natural form as a clean, clear, distinct concept into a spatial/continuum metaphor.

I think there's a reason we like to compress things to one or two dimensions and use spatial metaphors. The Curse of Dimensionality applies to humans as well as machines. The way we deal with it is by squinting until all but a couple of the dimensions go away and then reasoning as if only a couple of dimensions exist. Perhaps this is not very generous to humans, but I don't think it's too far off the mark. We come with pre-built circuitry for reasoning in three or fewer dimensions, so it's not surprising that we want to compress things.

It makes sense to use "continuum" language iff what you're talking about has natural and clear continuum properties. "Is this paint color 'more red' than that paint color?" Otherwise you are, at best, muddying up your thoughts and communication for no gain, and often actively damaging things.

Examples

Non-political examples include the following:

Referring to two ideas as being "distant" or "far apart," or "close" or "nearby," or some analogous phrase. Perhaps you feel like sparrows are closer to finches, eagles are distant from pigeons, but not as distant and penguins are from ostriches. In other words, you're trying to use the language of spatial relationships to describe your mental model of what a similarity cluster looks like. The real problem, though, is that birdiness is not really something that lends itself to intuitions of a spatial continuum. Certain individual dimensions of birdiness, such as "can it fly?" are almost perfectly binary, and thus not really dimensions at all in the spatial sense. Your spatial intuitions will lead you astray if you actually put any weight on your metaphor - so, of what use is the metaphor?

Referring to two positions, claims, proposals, or policies as being "orthogonal." What is being gestured at here is the idea that there is no "intersection" (another geometry word!) between the ideas; that there is no relative relevance between the ideas; that the implications of Idea 1 have no bearing on Idea 2. This is usually not t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Team Shard Status Report, published by David Udell on August 9, 2022 on LessWrong.

Team Shard is a nebulous alignment research collective, on paper siloed under John Wentworth's SERI MATS program, but in reality extending its many tendrils far across the Berkeley alignment community. "Shard theory" -- a name spoken of in hushed, mildly confused tones at many an EA hangout. This is their story (this month).

Epistemic status: A very quick summary of Team Shard's current research, written up today. Careful summaries and actual results are forthcoming, so skip this unless you're specifically interested in a quick overview of what we're currently working on.

Introduction

This past month, Team Shard began its research into the relationship between the reinforcement schedules and learned values of RL agents. Our core MATS team is composed of yours truly, Michael Einhorn, and Quintin Pope. The greater Team Shard, however, is legion -- its true extent only dimly suggested by the author names on its LessWrong writeups.

Our current path to impact is to (1) distill and expound shard theory and preregister its experimental predictions, (2) run RL experiments testing shard theory's predictions about learned values, and (3) climb the interpretability tech tree, starting with finetuned-on-values-text large language models, to unlock more informative experiments. In the 95th percentile, best-case possible world, we learn a bunch about how to reliably induce chosen values in extant RL agents by modulating the agent's reinforcement schedule and are able to probe the structure of those induced values within the models with interpretability tools.

Distillations

If you don't understand shard theory's basic claims and/or its relevance to alignment, stay tuned! A major distillation is forthcoming.

Natural Shard Theory Experiments in Minecraft

Uniquely, Team Shard already has a completed (natural) experiment under its belt! However, this experiment has a couple of nasty confounds, and even without those it would only have yielded a single bit of evidence for or against shard theory. But to summarize: OpenAI's MineRL agent is able to, in the best case, craft a diamond pickaxe in Minecraft in 4 minutes (!). Usually, the instrumental steps the MineRL agent must pursue to craft the diamond pickaxe . lie on the most efficient path to crafting the diamond pickaxe. So we can't disentangle from the model's ordinary gameplay data whether the model terminally values the journey or the destination: is reinforcement the model's optimization target, or are its numerous in-distribution proxies among its terminal goals?

One Karolis Ramanauskas, thankfully, already did the hard work of finding out for us!

When you give the MineRL agent a full stack of diamonds at the get-go . it starts punching trees and crafting the basic Minecraft tools, rather than immediately crafting as many diamond pickaxes as possible. Theories that predict the journey rather than the destination rejoice!

Now, there's a confound here, because the model was trained via reward shaping -- it was rewarded some lesser amount for all the instrumental steps along the way to the diamond pickaxe. Also, the model is quite stupid, despite its best-case properties. Whatever's true about its terminal values, it may just be spazzing around and messing up. Given the significant difficulty of even running (yet alone further finetuning) the OpenAI MineRL model, along with the model's stupidity confound, we opted to conduct our RL experiments in a more tractable (but still appreciably complex) environment.

Learned Values in CoinRun

So we now have an RL agent playing CoinRun well!

We're going to take it off-distribution and see whether it terminally values (1) just the coins, (2) a small handful of in-distribution proxies for getting coins, or (3) all ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Military Service as an Option to Build Career Capital, published by Molly on August 9, 2022 on The Effective Altruism Forum.

Background

You might be interested in this post if you’re just kind of curious about the military or if you’re actually considering it as a career option.

I was commissioned as an officer in the U.S. Army on July 4th, 2011. There aren’t a lot of EAs with a military background, so I thought it’d be useful to offer some thoughts and perspective on what EAs and might get out of military service. This post is U.S.-centric.

My service in the military has been as a lawyer, so my experience has been atypical in a lot of ways; then again, there is no one typical military experience. My perspective is also Army-centric, and I have less insight into the Navy or Air Force (or Marines or Space Force or Coast Guard). But some things about military service are pretty universal - I’ll try to focus more on those.

The post is organized thematically by aspects of military service that could be valuable to EAs.

Internalizing a Foreign Ethic

When I first joined the Army, everything about being a soldier was absurdly foreign to me. I basically had to play-act to get through my day. Whenever I used the words “sir” or “ma’am” I felt like I was secretly mocking people. Lifting my hand to my hat to salute felt so unnatural, I actually started humming a patriotic little horn ditty under my breath to try to convince my brain that saluting was the proper thing to do.

But if you do anything long enough it begins to feel natural.

This happened to me with a million little habits of military life: blousing my boots, doing push-ups (even lawyers do push-ups), standing at attention when a general enters the room, putting my hair in a bun, saying “roger” as an acknowledgement. After a while, my memories morphed in adaptation. I found myself recalling a pre-military conversation with my beloved political theory professor, an aged Gandhi scholar, and in my memory I called him “sir.” In reality he would’ve physically recoiled had I ever done such a thing.

These microdoses of cultural conditioning were related to and symbolic of a greater underlying shift that occurred in me over the years. I internalized a culture and an ethic very different from my native one. I began to see the beauty in things that had once been an anathema to me: traditional gender roles, hierarchy, nationalism.

I didn’t lose my original values; I expanded on them. I became more humble about them, and related better to people who didn’t hold them.

I think EAs are a uniquely skilled group at steelmanning others’ points of view. But noticing when something is in need of steelmanning is its own skill, different from the skill of actually doing the steelmanning. Having some emotional connection to an ethic very different from prevailing EA ethics, or the prevailing ethics of cultures that EAs tend to come from, makes this kind of noticing easy in contexts where those two ethics are in conflict.

But my suspicion is that an emotional connection to multiple ethical frameworks allows for general practice in perspective-taking (though not everyone will be inclined to practice perspective-taking). As a consequence, someone who has two (or more) different ethical frameworks that they feel a deep connection to may be more likely to notice when an entirely different ethical framework is being strawmanned or should be better steelmanned.

On the flip side, it’s possible that being acculturated to a different ethical framework could also help EAs be better calibrated about which criticisms of EA should be taken seriously. As a group, EAs tend to be so aware of the failure mode of not taking feedback well, that we forget about the failure mode of incorporating feedback that wasn’t actually in furtherance of our goals, or was ju...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Risks from atomically precise manufacturing - Problem profile, published by Benjamin Hilton on August 9, 2022 on The Effective Altruism Forum.

This problem profile was written for 80,000 Hours. It's based largely on Thoughts on nanotechnology strategy research as an EA cause area and OpenPhil’s cause report into atomically precise manufacturing, and is written for an audience broadly unfamiliar with EA.

Summary

Both the risks and benefits of advances in atomically precise manufacturing seem like they might be significant, and there is currently little effort to shape the trajectory of this technology. However, there is also relatively little investment going into developing atomically precise manufacturing, which reduces the urgency of the issue.

Our overall view

Potentially top depending on fit

Working on this problem could be among the best ways of improving the long-term future, but there are fewer high-impact opportunities to work on this issue than on our top priority problems.

Why could risks from atomically precise manufacturing be a pressing problem?

Atomically precise manufacturing is a form of particularly advanced nanotechnology. With atomically precise manufacturing we could build products out of individual atoms and molecules, allowing us to perfectly create a very wide range of products with very few flaws. Effectively, this would be like having perfect 3D printers that can produce anything.

Atomically precise manufacturing might be feasible, and there’s incentive to develop it

Molecular machines – tiny mechanical machines on the scale of just a few individual molecules – exist in nature, which means they’re definitely possible to create. In fact, we’ve actually produced a wide variety of simple artificial molecular machines. With time, we may be able to design and produce machines as small and complex as biological organisms.

This could be useful for a number of reasons:

Cheap energy — we may be able to produce even more efficient batteries and solar cells. The use of semiconductors in these devices means that they’re already one of the things we make to the highest degree of precision – it’s possible that even more precision could improve these devices further.

Carbon capture and storage — we may be able to produce cheap, high-performance nanoscale devices to remove CO2 from the atmosphere.

Medicine — we may be able to produce devices on the scale of human cells for targeting medical issues.

Cheap manufacturing — we may be able to take apart most things (even e.g. trash) and use the individual atoms to produce anything else, meaning that everything we manufacture could theoretically be made in one location using only extremely cheap inputs.

While we’re confident it’s possible to create the technology required for simple atomically precise manufacturing, it’s not clear that we could create technology advanced enough to do everything in the above list. But if we can do any of the above, it seems like there will be significant incentive to develop and produce this technology.

There are risks associated with atomically precise manufacturing

There appear to be substantial (and perhaps even existential) risks associated with developing atomically precise manufacturing:

Widespread access to atomically precise manufacturing could lead to widespread ability to unilaterally produce things as destructive as nuclear weapons or catastrophic pandemics.

States with this technology may have an incentive to produce new kinds of weapons with atomically precise manufacturing (even if most people don’t have access to this technology).

Manufacturing that can disassemble and produce products from extremely cheap inputs could lead to a much faster design and prototyping cycle for weapons.

We may be able to produce more powerful computers, which could increase risks from tra...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are "Bad People" Really Unwelcome in EA?, published by 𝕮𝖎𝖓𝖊𝖗𝖆 on August 9, 2022 on The Effective Altruism Forum.

Epistemic Status

Written in a hurry while frustrated. I kind of wanted to capture my feelings in the moment and not sanitise it when I'm of clearer mind.

Context

This is mostly a reply to these comments:

Exhibit A

1) One way to see the problem is that in the past we used frugality as a hard-to-fake signal of altruism, but that signal no longer works.

Exhibit B

Agree.

Fully agree we need new hard-to-fake signals. Ben's list of suggested signals is good. Other things I would add are vegan and cooperates with other orgs / other worldviews. But I think we can do more as well as increase the signals. Other suggestions of things to do are:

Testing for altruism in hiring (and promotion) processes. EA orgs could put greater weight on various ways to test or look for evidence of altruism and kindness in their hiring processes. There could also be more advice and guidance for newer orgs on the best ways to look for and judge this when hiring. Decisions to promote staff should seek feedback from peers and direct reports.

Zero tolerance to funding bad people. Sometimes an org might be tempted to fund or hire someone they know / have reason to expect it is a bad person or primarily seeking power or prestige not impact. Maybe this person has relevant skills and can do a lot of good. Maybe on a naïve utilitarian calculus it looks good to hire them as we can pay them for impact. I think there is a case to be heavily risk adverse here and avoid hiring or funding such people.

A Little Personal Background

I've been involved in the rationalist community since 2017 and joined EA via social osmosis (I rarely post on the forum and am mostly active on social media [currently Twitter]). I was especially interested in AI risk and x-risk mitigation more generally, and still engage mostly with the existential security parts of EA.

Currently, my main objective in life is to help create a much brighter future for humanity (that is, I am most motivated by the prospect of creating a radically better world as opposed to securing our current one from catastrophe). I believe strongly that one is possible (nothing in the fundamental laws prohibit it), and effective altruism seems like the movement for me to realise this goal.

I am currently training (learning maths, will start a CS Masters this autumn and hopefully a PhD afterwards) to pursue a career as an alignment researcher.

I'm a bit worried that people like me are not welcome in EA.

Motivations

Since my mid to early teens, I've always wanted to have a profound impact on the world. It was how I came to grasp with mortality. I felt like people like Newton, Einstein, etc. were immortalised by their contributions to humanity. Generations after their deaths, young children learn about their contributions in science class.

I wanted that. To make a difference. To leave a legacy behind that would immortalise me. I had plans for the world (these changed as I grew up, but I never permanently let go of my desire to have an impact).

Nowadays, it's mostly not a mortality thing (I aspire to [greatly] extended life), but the core idea of "having an impact" persists. Even if we cure aging, I wouldn't be satisfied with my life if it were insignificant — if I weren't even a footnote in the story of human civilisation — I want to be the kind of person who moves the world.

Argument

Purity Tests Aren't Effective

I want honour and glory, status, and prestige. I am not a particularly kind, generous, selfless, or altruistic person. I'm not vegan, and I'd only stop eating meat when it becomes convenient to do so. I want to be affluent and would enjoy (significant) material comfort. Nonetheless, I feel that I am very deeply committed to making the world a much ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bahamian Adventures: An Epic Tale of Entrepreneurship, AI Strategy Research and Potatoes, published by Jaime Sevilla on August 9, 2022 on The Effective Altruism Forum.

In previous episodes of my life:

Jaime suspended his PhD. He started working with Open Philanthropy on modelling the development of transformative Artificial Intelligence. His independent research group picked up some steam. But whatever happened to the errand researcher afterwards?

In short: lots. After leaving my PhD, what followed was one of the most hectic and exciting stages of my career so far.

Let’s start with the highlights:

Together with the rest of my research group we have launched our own research organisation: Epoch. We are working on anticipating the future of Artificial Intelligence. Back when I was interning at the Future of Humanity Institute, I resolved that one of my career goals would be to coordinate a research group. This has now happened five years earlier than I expected, which makes me slightly nervous about biting off more than I can chew. Still, I think it made sense to take this step. Our work so far has been well received, I got encouraging feedback on my leadership style and our founding team is awesome.

I’ve been participating in the FTX Bahamas EA Fellowship program. This has been an amazing opportunity to mingle and cowork with some of the smartest people I’ve ever met. The paradisiac background hasn’t been too shabby either.

My research hasn’t stopped:

I am still working on macroeconomic modelling of AI Takeoffs with Open Philanthropy. Working with my manager, Tom Davidson, has been amazingly stimulating and instructive. On the downside, I feel like I severely underestimated how long the project would take. And in hindsight, I think I should have committed to work on it full time in order to do it justice. We have advanced quite a lot though, and with the support of Epoch the project is going more smoothly now.

The Epoch team wrote Compute Trends Across Three Eras of Machine Learning. We studied how state-of-the-art Machine Learning models are becoming more compute-intensive over time.This paper has been very successful, and it’s been featured in the WCCI 2022 conference, Our World in Data and The Economist.

I published Principled extremizing of aggregated forecasts, in which I propose a new forecast aggregation method based on a paper by Eric Neyman. The method performs quite well on Metaculus public data, and people at Metaculus have been experimenting with some variations.

Pablo Villalobos and I wrote a review of The Potato's Contribution to Population and Urbanization: Evidence from a Historical Experiment for the Forethought Foundation. Turns out potatoes really are key for the development of the world outside the Americas.

Jonathan Lindblum and I wrote A Bayesian Model of Records. We explored the problem of forecasting how we should expect the fastest times in several athletic events to evolve over time. The framework is very general, and I am hopeful that it will help us solve other problems like forecasting records in speedrunning.

Ege Erdil and I wrote A time-invariant version of Laplace’s rule. We showed an inconsistency in the application of Laplace’s rule of succession, and recommended forecasters to adopt a new version. This is a continuation of my work on heuristics for forecasting.

I have participated in a number of Epoch papers and blogposts, including Machine Learning Model Sizes and the Parameter Gap, What’s the backward-forward FLOP ratio for Neural Networks?, Projecting compute trends in Machine Learning and Estimating Training Compute of Deep Learning Models.

Alongside these successes, there were also some failures:

I got offered to write another article about potatoes. I wrote a few drafts and sent them, but I never got around to editing them prop...