Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All AGI safety questions welcome (especially basic ones) [monthly thread], published by plex on July 16, 2022 on LessWrong.

tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb!

Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics.

As requested in the previous thread, we'll be putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask.

It's okay to ask uninformed questions, and not worry about having done a careful search before asking.

Stampy's Interactive AGI Safety FAQ

Additionally, this will serve as a soft-launch of the project Rob Miles' volunteer team has been working on: Stampy - which will be (once we've got considerably more content) a single point of access into AGI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that! You can help by adding other people's questions and answers to Stampy or getting involved in other ways!

We're not at the "send this to all your friends" stage yet, we're just ready to onboard a bunch of editors who will help us get to that stage :)

We welcome feedback and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase. You are encouraged to add other people's answers from this thread to Stampy if you think they're good, and collaboratively improve the content that's already on our wiki.

We've got a lot more to write before he's ready for prime time, but we think Stampy can become an excellent resource for everyone from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets.

Guidelines for Questioners:

No previous knowledge of AGI safety is required. If you want to watch a few of the Rob Miles videos, read either the WaitButWhy posts, or the The Most Important Century summary from OpenPhil's co-CEO first that's great, but it's not a prerequisite to ask a question.

Similarly, you do not need to try to find the answer yourself before asking a question (but if you want to test Stampy's in-browser tensorflow semantic search that might get you an answer quicker!).

Also feel free to ask questions that you're pretty sure you know the answer to, but where you'd like to hear how others would answer the question.

One question per comment if possible (though if you have a set of closely related questions that you want to ask all together that's ok).

If you have your own response to your own question, put that response as a reply to your original question rather than including it in the question itself.

Remember, if something is confusing to you, then it's probably confusing to other people as well. If you ask a question and someone gives a good response, then you are likely doing lots of other people a favor!

Guidelines for Answerers:

Linking to the relevant canonical answer on Stampy is a great way to help people with minimal effort! Improving that answer means that everyone going forward will have a better experience!

This is a safe space for people to ask stupid questions, so be kind!

If this post works as intended then it will produce many answers for Stampy's FAQ. It may be worth keeping this in mind as you write your answer. For example, in some cases it might be worth giving a slightly longer / more expansive / more detailed...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Confused about "making people happy" vs. "making happy people", published by tobiasleenaert on July 16, 2022 on The Effective Altruism Forum.

Hi,(first post, hope I'm doing everything more or less right).You’re probably familiar with the phrase (I don’t know who framed it this way) that “we care about making people happy, but we’re indifferent to making happy people.” I nicely summarizes the idea that while it is important to provide currently living people with as much wellbeing as possible (because they are here), creating more humans doesn’t really matter morally, even if they would be very happy, because the unborn can’t care about being born (I hope I'm doing an okay job at paraphrasing).I share this view (I'm pretty indifferent about making happy people - except if more people has an impact on people already existing). In fact, I can’t intuitively understand why someone could have the opposite opinion. But clearly I must be missing something, because it seems in the EA community many or most people do care about creating as many (happy) people as possible.

I have wrestled with this topic for a long time, and watching a new Kurtzgesagt video on longtermism made me want to write this post. In that (wonderfully made) video, the makers clearly are of the opnion that making happy people is a good thing. The video contains things like

“If we screw up the present so many people may not come to exist. Quadrillions of unborn humans are at our mercy. The unborn are the largest group of people and the most disenfranchised. Someone who might be born in a thousand or even a million years, deeply depends on us today for their existence.”

This doesn’t make that much sense to me (except in the context when more people means more happiness for everyone, not just additional happiness because there’s more people), and I don't understand how the makers of this video present the “making happy people” option as if it is not up for debate. Unless... it is not up for debate?

My questions, if you want:

1. how do you estimate is the division within the EA community? How many people are indifferent to making happy people, and how many care about making happy people?

2. if you are of the opposite opinion: what am I not seeing if I'm indifferent to making happy people? Is this stance still a respectable opinion? Or is it not at all?

Thank you!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How would a language model become goal-seeking?, published by David Mears on July 16, 2022 on The Effective Altruism Forum.

Or what should I read to understand this?

It seems like some people expect descendants of large language models to pose a risk of becoming superintelligent agents. (By ‘descendants’ I mean adding scale and non-radical architectural changes: GPT-N.)

I accept that there’s no reason in principle that LLM intelligence (performance on tasks) should be capped at the human level.

But I don’t know why to believe that at some point language models would develop agency / goal-seeking behaviour, where they start to try to achieve things in the real world instead of continuing to perform their ‘output predicted text’ behaviour.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: List of Lists of Concrete Biosecurity Project Ideas, published by Tessa on July 16, 2022 on The Effective Altruism Forum.

I was working on my own list of biorisk reduction project ideas, and ended up creating this list of lists, which seemed silly not to share.

I have quoted a few words describing each project, but it’s well worth clicking through to the original lists. None of them describe projects for longer than two pages, and most for a paragraph or so. Several of the EA Forum posts also contain links for getting in touch with teams working on these projects. If you’re interested in that, you might also want to fill in the longtermist census.

In general, I am so tired of people dying in pandemics. It does not necessarily make me optimistic to see all these project ideas― the world should already be much better at this, and there is so much work to be done. However, it does make me feel hopeful: the world can be much better, and there are projects that you could work on to help it become so.

Concrete Biosecurity Projects (some of which could be big)

January 2022 EA Forum Post

Early Detection Center

Early detection of a biothreat increases the amount of time we have to respond (e.g. designing tailored countermeasures, using protective equipment, heading to bunkers, etc)

See also: Nucleic Acid Observatory announcement from Will Bradshaw, Anjali Gopal, Mike McLaren, related post from Jeff Kaufman

Super PPE

it seems plausible that with good materials science and product design we could come up with next-generation PPE that is simultaneously highly effective in extreme cases, easy to use, reliable over long periods of time, and cheap/abundant.

Medical countermeasures

“radically improved medical countermeasures against GCBR-class threats, either by 1) producing targeted countermeasures against particularly concerning threats (or broad-spectrum countermeasures against a class of threats), or by 2) creating rapid response platforms that are reliable even against deliberate adversaries

See also: Alvea Vax announcement from Kyle Fish

BWC Strengthening

there is more scope for creative ways of strengthening the treaty (e.g. whistleblowing prizes), or creating new bilateral agreements and avoiding bureaucratic gridlock

Sterilization technology

Sterilization techniques that rely on physical principles . have the advantage of being broadly applicable, difficult to engineer around, and having little dual-use downside potential

See also: post on UV-C air purifiers

Refuges

specially designed refuges that safeguard against catastrophic pandemics

See also: Fønix: Bioweapons shelter project launch, Help Us Make Civilizational Refuges Happen from Linch Zhang and Ajay Karpur

Future Fund Project Ideas

As of July 2022

(I have not repeated ideas that pull directly from the above post, though the FTX write-ups are not word-for-word identical)

Crisis-aware regulation for global catastrophic biological risks

public health measures that would have a poor risk-benefit balance in normal times may be warranted or even critical in a crisis in which the benefits to eradicating or delaying a public health threat are unusually large” (link)

See also: original Project Ideas Competition submission by Mackenzie Arnold, with contributions by Kyle Fish

Rapid development and approval of emergency vaccination and therapeutics

imagine a team of high-powered vaccine developers, large-scale manufacturing capability on standby, and ready-to-go infrastructure for rapid human challenge trials in a regulatorily compliant manner” (link)

New ideas for mitigating biotechnology misuse

July 2022 EA Forum Post

Record-keeping for strong attribution

a record-based system for strong attribution of biological agents may be a powerful mechanism to deter biological weapons development and use”

Responsible access to genetic s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: QNR Prospects, published by PeterMcCluskey on July 16, 2022 on LessWrong.

Approximately a book review: Eric Drexler's QNR paper.

[Epistemic status: very much pushing the limits of my understanding. I've likely made several times as many mistakes as in my average blog post. I want to devote more time to understanding these topics, but it's taken me months to produce this much, and if I delayed this in hopes of producing something better, who knows when I'd be ready.]

This nearly-a-book elaborates on his CAIS paper (mainly chapters 37 through

39), describing a path for AI capability research enables the CAIS approach to remain competitive as capabilities exceed human levels.

AI research has been split between symbolic and connectionist camps for as long as I can remember. Drexler says it's time to combine those approaches to produce systems which are more powerful than either approach can be by itself.

He suggests a general framework for how to usefully combine neural networks and symbolic AI. It's built around structures that combine natural language words with neural representations of what those words mean.

Drexler wrote this mainly for AI researchers. I will attempt to explain it to a slightly broader audience.

Components

What are the key features that make this more powerful than GPT-3 alone, or natural language alone?

QNR extends natural language by incorporating features of deep learning, and mathematical structures used in symbolic AI.

words are associated with neural representations. Those representations are learned via a process that focuses on learning a single concept at a time with at least GPT-3-level ability to understand the concept. BERT exemplifies how to do this.

words can be related to other word via graphs (such as syntax trees).

words, or word-like concepts, can be created via compositions of simpler concepts.

corresponding to phrases, sentences, books, and entities that we have not yet conceived.

QNR feels much closer than well-known AI's to how I store concepts within my mind, as opposed to the stripped down version that I'm using here to help you reconstruct some of that in your mind.

Drexler contrasts QNR to foundation models, but I don't find "foundation models" to be a clear enough concept to be of much value.

Importance?

I've noticed several people updating this year toward earlier AGI timelines, based on an increasing number of results demonstrating what look to me like marginally more general-purpose intelligence. [I wrote this before Gato was announced, and have not yet updated on that.]

I've also updated toward somewhat faster timelines, but I'm mainly reacting to Drexler's vision of how to encode knowledge in a more general-purpose, scalable form.

I expect that simple scaling up of GPT-3 will not generate human-level generality with a plausible amount of compute, possibly just because it relearns the basics from scratch each time a new version is tried. With

QNR, new services would build on knowledge that is represented in much more sophisticated forms than raw text.

Effects on AI Risks

The QNR approach focuses on enhancing knowledge corpora, not fancier algorithms. It enables the AI industry to create more value, possibly at an accelerating rate, without making software any more agent-like. So it could in principle eliminate the need for risky approaches to AI.

However, that's not very reassuring by itself, as the QNR approach is likely to accelerate learning abilities of agenty AIs if those are built. Much depends on whether there are researchers who want to pursue more agenty approaches.

I can imagine QNR enabling more equality among leading AI's if a QNR corpus is widely available.

A QNR-like approach will alter approaches to interpretability.

The widely publicized deep learning results such as GPT-3 create enormous inscrutable fl...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Report on Social Returns to Productivity Growth, published by Tom Davidson on July 15, 2022 on The Effective Altruism Forum.

Historically, economic growth has had huge social benefits, lifting billions out of poverty and improving health outcomes around the world. This leads some to argue that accelerating economic growth, or at least productivity growth, should be a major philanthropic and social priority going forward.

I’ve written a report in which I evaluate this view in order to inform Open Philanthropy’s Global Health and Wellbeing (GHW) grantmaking. Specifically, I use a relatively simple model to estimate the social returns to directly funding research and development (R&D). I focus on R&D spending because it seems like a particularly promising way to accelerate productivity growth, but I think broadly similar conclusions would apply to other innovative activities.

My estimate, which draws heavily on the methodology of Jones and Summers (2020), asks two primary questions:

How much would a little bit of extra R&D today increase people’s incomes into the future, holding fixed the amount of R&D conducted at later times?

How much welfare is produced by this increase in income?

In brief, I find that:

The social returns to marginal R&D are high, but typically not as high as the returns in other areas we’re interested in. Measured in our units of impact (where “1x” is giving cash to someone earning $50k/year) I estimate that the cost-effectiveness of funding R&D is 45x. This is ~4% as impactful as the (roughly 1,000x) GHW bar for funding.

Put another way, I estimate that $20 billion to “average” R&D has the same welfare benefit as increasing the incomes of 180 million people by 10% each for one year.

That said, the best R&D projects might have much higher returns. So could projects aimed at increasing the amount of R&D (for example, improving science policy).

This estimate is very rough, and I could readily imagine it being off by a factor of 2-3 in either direction, even before accounting for the limitations below.

Returns to R&D were plausibly much higher in the past. This is because R&D was much more neglected, and because of feedback loops where R&D increased the amount of R&D occurring at later times.

My estimate has many important limitations. For example, it omits potential downsides to R&D (e.g. increasing global catastrophic risks), and it focuses on a specific scenario in which historical rates of return to R&D continue to apply even as population growth stagnates.

Alternative scenarios might change the bottom line. For instance, R&D today might speed up the development of some future technology that drastically accelerates R&D progress. This would significantly increase the returns to R&D, but in my view would also strengthen the case for Open Phil to focus on reducing risks from that technology rather than accelerating its development.

Overall, the model implies that the best R&D-related projects might be above our GHW bar, but it also leaves us relatively skeptical of arguments that accelerating innovation should be the primary social priority going forward.

In the full report, I also discuss:

How alternative scenarios might affect social returns to R&D.

What these returns might have looked like in the year 1800.

How my estimates compare to those of economics papers that use statistical techniques to estimate returns to R&D growth.

The ways in which my current views differ from those of certain thinkers in the Progress Studies movement.

If environmental constraints require that we reduce our use of various natural resources, productivity growth can allow us to maintain our standards of living while using fewer of these scarce inputs.

For example: in Stubborn Attachments, Tyler Cowen argues that the best way to improve the long-run future is to maximize ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Safety Implications of LeCun's path to machine intelligence, published by Ivan Vendrov on July 15, 2022 on LessWrong.

Yann LeCun recently posted A Path Towards Autonomous Machine Intelligence, a high-level description of the architecture he considers most promising to advance AI capabilities.

This post summarizes the architecture and describes some implications for AI safety work if we accept the hypothesis that the first transformative AI will have this architecture.

Why is this a hypothesis worth considering? My reasons for elevating this hypothesis, in order of importance, are

LeCun has a track record of being ahead of mainstream academic research, from working on CNNs in the 90s to advocating for semi-supervised learning back in 2014-2016 when supervised learning was ascendant.

LeCun runs Meta AI (formerly FAIR) which has enormous resources and influence to advance his research agenda, making it more likely that his proposed architecture will be built at scale. In general I think this is an underrated factor; AI research exhibits a great deal of path dependence, and most plausible paths to AI are not taken primarily because nobody is willing to take a big risk on them.

The architecture is dramatically different from the architectures commonly assumed (implicitly) in much AI alignment work, such as model-free deep RL and "GPT-3 but scaled up 10000x". This makes it a good robustness check for plans that are overly architecture-specific.

Architecture Overview

The Overall Agent

At a high level, the proposed architecture is a set of specialized cognitive modules, nearly all of which are trained end-to-end with gradient descent. With the exception of the Actor and the Intrinsic Cost (see below) they are all deep neural networks.

What is this agent doing, exactly? It's meant to be a general architecture for any autonomous AI agent, but LeCun repeatedly emphasizes video inputs and uses self-driving cars as a recurrent example, so the central use case is embodied agents taking actions in the physical world. Other talks I've seen by LeCun suggest he thinks understanding video is essential for intelligence, both by analogy to humans and by a heuristic argument about the sheer amount of data it contains.

The World Model

More than half the body of the paper is about designing and training the world model, the predictive model of the environment that the AI uses to plan its actions. LeCun explicitly says that "designing architectures and training paradigms for the world model constitute the main obstacles towards real progress in AI over the next decades."

Why are world models so important? Because the main limitation of current AI systems, according to LeCun, is their sample inefficiency - they need millions of expensive, dangerous real-world interactions to learn tasks that humans can learn with only a few examples. The main way to progress capabilities is to reduce the number of interactions a system needs before it learns how to act, and the most promising way is to learn predictive world models on observational data. (The GPT-3 paper Large Language Models are Few Shot Learners is a great example of this - a good enough predictive model of language enables much more sample-efficient task acquisition than supervised learning).

What will these world models look like? According to LeCun, they will be

Predictive but not generative: They will predict high-level features of the future environment but not be able to re-generate the whole environment. This is especially obvious for high-dimensional data like video, where predicting the detailed evolution of every pixel is vastly overkill if you're doing planning. But it could also apply to language agents like chatbots, for whom it may be more important to predict the overall sentiment of a user's reply than the exact sequence of ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My local EA group has an unfriendly and impersonal vibe (via r/EffectiveAltruism), published by Tyner on July 15, 2022 on The Effective Altruism Forum.

I am not the original poster; cross-posting because I thought it could be useful feedback.

"Hello, this is a bit of a rant about my personal experience with my local EA group.

I've been showing up to my local EA group for about a year because the core principles resonate with me (effectiveness, the intrinsic value of life, etc.), but I haven't been able to build trust with these folks.

Don't get me wrong, there are friendly members of course. However, I've found the "core" members to be quite unfriendly. They all know each other, and I've found it very difficult to feel part of their group.

I find that a lot of what I say is met with dismissiveness and elitist attitudes. It is really exhausting when every conversation quickly becomes an impersonal philosophical debate about morality.

Sometimes when I talk to these folks I start doubting myself, asking questions like "Why are they so hostile towards me? Is there something wrong with me? Am I dumb? Am I unfriendly? Am I a bad person?" However, there is plenty of evidence from other aspects of my life demonstrating that I'm a reasonable person.

I'm afraid that showing up to local EA meetups is ruining my perceptions towards effectiveness and altruism. It just isn't an enjoyable vibe. I took a friend with me to the meetup once and following the meetup he quite earnestly said "I felt incredibly unwelcomed."

I went to EAG and that was great. I met some amazing people who I've kept in touch with and it showed me that the wider EA community isn't as hostile as the local group in my city. Does anyone else feel this unfriendly vibe with their local group(s)? Are some local groups doing more damage to altruism than good?"

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Existential Biorisk vs. GCBR, published by Will Aldred on July 15, 2022 on The Effective Altruism Forum.

I'm confused by why existential biorisk seems to not exist as a term.

The context here is that I've now talked to around a dozen people working in (or seriously considering working in) the field/cause area of global catastrophic biological risk (GCBR). Some of these folks are longtermist and/or x-risk oriented, and seem to care exclusively about the tail end of GCBRs (i.e., the part of the GCBR distribution that might lie in existential territory). Others aren't so focused on tail/existential risks. To me, lumping these two together under the single banner of GCBR leads to a degree of confusion and talking past one another. Anecdotally, I've witnessed a discussion on GCBR research directions where half an hour went by before those involved realised the source of their disagreement was x-risk focus, or lack thereof.

(I've heard the argument, in response to my "existential biorisk" terminology proposal, that all GCBR work is useful for reducing x-risk, and that this is why the distinction I point to is not made. While I do agree that some or even all non-x-risk focused GCBR work could turn out useful for reducing x-risk, it'd be a suspicious convergence if the current portfolio of GCBR work is optimal for reducing bio-related x-risk.)

Which brings me back to my question: why is "existential biorisk," or a similar term, not in use?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What if we don't need a "Hard Left Turn" to reach AGI?, published by Eigengender on July 15, 2022 on The Effective Altruism Forum.

Epistemic Status: My day job involves building Large Language Models (LLMs), but I only read AI safety literature as a casual hobby, and may have missed some important points. All predictions I make about the future of AI naturally have a very high level of uncertainty.

Over the last few years, I've noticed a contradiction in the beliefs of AGI prognosticators in the rationalist-adjacent community. The amazing progress in AI over the last few years has caused many to dramatically update their timelines for the emergence of AGI. While there have been advances in other areas of AI, most of the major accomplishments come from Large Language Models (LLMs) such as GPT-3, and I don't believe that the non-LLM progress would elicit anywhere near the same level of excitement without LLMs. That is, LLM progress has caused many to update their timelines for AGI. However, it has not caused those same people to update their predictions for how an AGI might act and what conceptual frameworks we should use to understand its behavior.

Language Models can do more than model language: 2022 has been a good year for AI research, but I believe that, in retrospect, Minerva will be considered the most important advance of the year. Minerva is a LLM trained to take math word problems and generate a step-by-step solution. In a particularly impressive result, Minerva got ~50% accuracy on the MATH dataset, much better than the previous 10% state-of-the-art result. This may not sound impressive until you realize how tough the MATH dataset is. This isn't a dataset of 'plug in the numbers' word problems from an introductory algebra class. This dataset is taken from high-school math competitions, designed to challenge the best mathematics students. Estimates of human performance are spotty, but the original MATH dataset paper gave the MATH dataset to a smart Computer Science undergrad student and got a performance around ~40%. Personally, I have a degree in statistics and I'd probably struggle to beat Minerva at these problems.

It's conservative to say that Minerva is around 90th percentile of human performance in solving math problems.

Language models are trained to the predict the next token in a large corpus of unstructured text. But the results on the MATH dataset can't be explained by simple pattern matching. To easily adapt to performing complex and difficult math problems, Minerva has That's not to say that Minerva is an AGI - it clearly isn't. But something important is going on here.

Maybe scaling really is all we need: There's no reason to expect performance to plateau at this level. My very conservative estimate is that, within the next two years, a model will be able to surpass the performance of very smart people who haven't specifically trained on math problems (ie. 80-90% accuracy on MATH). What comes next?

I find it helpful to think of AI progress in terms of order-of-magnitude (OOM) improvements in effective model capacity, over, say, the next 25 years. Scaling can come from three places: more spending, better hardware, and better algorithms.

The manhattan project cost ~20 billion in todays dollars, and current large models cost ~20 million. If AGI gets increasingly more plausible, it's possible to imagine the size of current models increasing 2-5 OOMs from increased spending alone.

GPU performance/$ doubles every 2.5 years, so that means 2-4 OOMs from more efficient hardware (where the upper end of this range is driven by increasingly specialized hardware).

Progress from improved algorithms is harder to estimate, but we can make some guesses. Chain-of-thought prompting alone, released this year, improves performance on downstream math reasoning by 1-2...