The Nonlinear Library: Alignment Forum

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Causality and a Cost Semantics for Neural Networks, published by scottviteri on August 21, 2023 on The AI Alignment Forum.

Epistemic status: I time-boxed this idea to three days of effort. So any calculations are pretty sloppy, and I haven't looked into any related works. I probably could have done much better if I knew anything about circuit complexity. There are some TODOs and an unfinished last section -- if you are interested in this content and want to pick up where I have left off I'll gladly add you as a collaborator to this post.

Here is a "tech tree" for neural networks. I conjecture (based on admittedly few experiments) that the simplest implementation of any node in this tree includes an implementation of its parents, given that we are writing programs starting from the primitives +, , and relu. An especially surprising relationship (to me) is that "if statements" are best implemented downstream of division.

Introduction

While discussing with my friend Anthony Corso, an intriguing idea arose. Maybe we can define whether program p1 "causes" p2 in the following way: Given a neural network that mimics p1, how easy is it to learn a neural network which mimics the behavior of p2? This proposition is intriguing because it frames causality as a question about two arbitrary programs, and reduces it to a problem of program complexity.

Suppose that p1 and p2 are written in a programming language P, and let P(ops) represent P extended with ops as primitive operations. We define a complexity function C:P(ops)R, which takes a program in the extended language and returns a real number representative of the program's complexity for some fixed notion of complexity. Let's define the degree to which p1 "causes" p2 as the minimum complexity achievable by a program p from P(p1) such that p is extensionally equal (equal for all inputs) to p2. If P2 is the set of all p in P(obs+p1) that are extensionally equal to p2, then causes(p1,p2)=minp∈P2C(p). We can also use this definition in the approximate case, considering the minimum complexity achievable by programs p such that E(p(x)-p2(x))2<ε with respect to some L1-integrable probability measure.

We can define a particular complexity function C that represents the cost of executing a program. We can estimate this quantity by looking at the program's Abstract Syntax Tree (AST) in relation to some cost model of the primitive operations in the language. For this exploration, we have chosen the lambda calculus as the language. Lambda calculus is a minimalist Lisp-like language with just a single type, which in our case we will think of as floating point numbers. The notation is simple: lambda abstraction is represented as λ x. x, and function application as (f g), which is not the same as f(g) in most other languages.

How I Would Like People to Engage with this Work

By writing Ops in your favorite programming language

By circumventing my proposed tech tree, by reaching a child without reaching a parent and using fewer (or equal) number of operations

By training some neural networks between these programs, and seeing how difficult it is to learn one program after pre-training on another

Cost Semantics

Definition

We define the cost of operations and expressions in the following manner:

Ops op=1,for any operation op in opsOps c=0,for any floating-point constant cOps x=0,for any variable xOps (λx.e)=Ops eOps (f g)=Ops f+Ops g

For operations of higher arity, we have({Ops }({op }x1.xn))=({Ops }{op})+∑i({Ops }xi)

The selected operations for a neural network are ops = {+, , relu}.

Basic Operations and Warm-Up

Let's take a few examples to demonstrate this cost calculus:

To derive subtraction, we first create negation neg.

(Ops neg) = (Ops (λ x. ( -1 x))) = (Ops ( -1 x))= (Ops ) + (Ops -1) + (Ops x) = 1 + 0 + 0 = 1

The cost of subtraction (-) ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Dirty concepts" in AI alignment discourses, and some guesses for how to deal with them, published by Nora Ammann on August 20, 2023 on The AI Alignment Forum.

Meta: This is a short summary & discussion post of a talk on the same topic by Javier Gomez-Lavin, which he gave as part of the PIBBSS speaker series. The speaker series features researchers from both AI Alignment and adjacent fields studying intelligent behavior in some shape or form. The goal is to create a space where we can explore the connections between the work of these scholars and questions in AI Alignment.

This post doesn't provide a comprehensive summary of the ideas discussed in the talk, but instead focuses on exploring some possible connections to AI Alignment. For a longer version of Gomez-Levin's ideas, you can check out a talk here.

"Dirty concepts" in the Cognitive Sciences

Gomez-Lavin argues that cognitive scientists engage in a form of "philosophical laundering," wherein they associate, often implicitly, philosophically loaded concepts (such as volition, agency, etc.) into their concept of "working memory."

He refers to such philosophically laundered concepts as "dirty concepts" insofar as they conceal potentially problematic assumptions being made. For instance, if we implicitly assume that working memory requires, for example, volition, we have now stretched our conception of working memory to include all of cognition. But, if we do this, then the concept of working memory loses much of its explanatory power as one mechanism among others underlying cognition as a whole.

Often, he claims, cognitive science papers will employ such dirty concepts in the abstract and introduction but will identify a much more specific phenomena being measured in the methods and results section.

What to do about it? Gomez-Lavin's suggestion in the case of CogSci

The pessimistic response (and some have suggested this) would be to quit using any of these dirty concept (e.g. agency) all together. However, it appears that this would amount to throwing the baby out with the bathwater.

To help remedy the problem of dirty concepts in working memory literature, Gomez-Lavin proposes creating an ontology of the various operational definitions of working memory employed in cognitive science by mining a wide range of research articles. The idea is that, instead of insisting that working memory be operationally defined in a single way, we ought to embrace the multiplicity of meanings associated with the term by keeping track of them more explicitly.

He refers to this general approach as "productive pessimism." It is pessimistic insofar as it starts from the assumption that dirty concepts are being problematically employed, but it is productive insofar as it attempts to work with this trend rather than fight against it.

While it is tricky to reason with those fuzzy concepts, once we are rigorous about proposing working definitions / operationalization of these terms as we use them, we can avoid some of the main pitfalls and improve our definitions over time.

Relevance to AI alignment?

It seems fairly straightforward that AI alignment discourse, too, suffers from dirty concepts.

If this is the case (and we think it is), a similar problem diagnosis (e.g. how dirty concepts can hamper research/intellectual progress) and treatment (e.g. ontology mapping) may apply.

A central example here is the notion of "agency". Alignment researchers often speak of AI systems as agents. Yet, there are often multiple, entangled meanings intended when doing so. High-level descriptions of AI x-risk often exploit this ambiguity in order to speak about the problem in general, but ultimately imprecise terms. This is analogous to how cognitive scientists will often describe working memory in general terms in the abstract and operationalize the term ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We can do better than DoWhatIMean, published by Luke H Miles on August 19, 2023 on The AI Alignment Forum.

Lately, "alignment" means "follow admin rules and do what users mean". Admins can put in rules like "don't give instructions for bombs, hacking, or bioweapons" and "don't take sides in politics". As AI gets more powerful, we can use it to write better rules and test for loopholes. And "do what users mean" implies some low-impactfulness: it won't factory reset your computer when you ask it to fix the font size.

In other words, the plan has two steps:

Make a machine that can do literally anything

Configure it to do good things

I don't like this plan very much because it puts power in the hands of the few and has big security liabilities.

Temptation

Imagine you're on the team of three people that writes the constitution for the training run for the gigaAI1. Or you're the one actually typing it in. Or you do infra and write all the code that strings pass through on the way to and from the training cluster.

Might you want to take advantage of the opportunity? I can think of a few subtle things I might add.

If I think about the people in my life I know well, I can only name one that I would nominate for the constitution writing position.

One solution I've heard for this is to use the powerful DWIM machine to construct a second unalterable machine, then delete the first machine. You might call the time between creating the first and second machine the temptation mountain (in a plot where X is time and Y is the maximum power of any individual person). Then you want to minimize the volume of this mountain. It is difficult to shrink the mountain very much though.

Security risks

If you're 90% through your DWIM AI development and it seems very promising, then you must be very careful with your code and data. If someone else gets your secrets, then they can make it do what they mean, and they might mean for it to do something quite different. You should be careful with your team members too since threats and blackmail and bribes are all quite cheap and effective. (Hard to be careful with them though since they put "on constitution AI team" on their linkedins, their phone numbers are practically public information, and they live in a group house with employees from your competitor.)

If you're doing open source AI development and start having big successes then you might start looking at the github profiles of your stargazers and wonder what they mean for it to do.

If altLab is also getting close to AGI then you have to figure out whether your stop-clause means you help them or their stop-clause means they help you.

Doing better

Could we come up with a plan so rock solid that we'd be comfortable giving the code, the data, or the team to anyone? Like um how you could give Gandhi to Hitler without stressing.

For lack of a better term[1], let's call such a system intrinsically aligned AI. The key innovation required might be a means to strongly couple capabilities and alignment. This is in contrast to the current regime where we build Do Anything Now then tell it to do good.

Such an innovation would unblock coordination and ease security burdens, and not require the board of directors & the engineers to be saints. It would also give the open source community something to work on that doesn't give AI safety folks gray hairs.

To be clear, I don't have a good idea for how to achieve this. I'm not certain it is even possible.

However, it's not obviously impossible to me. And I would be much more at ease if all the AI labs were building intrinsically aligned AI instead of something that can be stolen and directed to arbitrary goals.[2]

Let's at least give it a shot.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Overview of Catastrophic AI Risks: Summary, published by Dan H on August 18, 2023 on The AI Alignment Forum.

We've recently published on our website a summary of our paper on catastrophic risks from AI, which we are cross-posting here. We hope that this summary helps to make our research more accessible and to share our policy recommendations in a more convenient format. (Previously we had a smaller summary as part of this post, which we found to be insufficient. As such, we have written this post and have removed that section to avoid being duplicative.)

Executive summary

Catastrophic AI risks can be grouped under four key categories which we explore below, and in greater depth in CAIS' linked paper:

Malicious use: People could intentionally harness powerful AIs to cause widespread harm. AI could be used to engineer new pandemics or for propaganda, censorship, and surveillance, or released to autonomously pursue harmful goals. To reduce these risks, we suggest improving biosecurity, restricting access to dangerous AI models, and holding AI developers liable for harms.

AI race: Competition could push nations and corporations to rush AI development, relinquishing control to these systems. Conflicts could spiral out of control with autonomous weapons and AI-enabled cyberwarfare. Corporations will face incentives to automate human labor, potentially leading to mass unemployment and dependence on AI systems. As AI systems proliferate, evolutionary dynamics suggest they will become harder to control. We recommend safety regulations, international coordination, and public control of general-purpose AIs.

Organizational risks: There are risks that organizations developing advanced AI cause catastrophic accidents, particularly if they prioritize profits over safety. AIs could be accidentally leaked to the public or stolen by malicious actors, and organizations could fail to properly invest in safety research. We suggest fostering a safety-oriented organizational culture and implementing rigorous audits, multi-layered risk defenses, and state-of-the-art information security.

Rogue AIs: We risk losing control over AIs as they become more capable. AIs could optimize flawed objectives, drift from their original goals, become power-seeking, resist shutdown, and engage in deception. We suggest that AIs should not be deployed in high-risk settings, such as by autonomously pursuing open-ended goals or overseeing critical infrastructure, unless proven safe. We also recommend advancing AI safety research in areas such as adversarial robustness, model honesty, transparency, and removing undesired capabilities.

1. Introduction

Today's technological era would astonish past generations. Human history shows a pattern of accelerating development: it took hundreds of thousands of years from the advent of Homo sapiens to the agricultural revolution, then millennia to the industrial revolution. Now, just centuries later, we're in the dawn of the AI revolution. The march of history is not constant - it is rapidly accelerating. World production has grown rapidly over the course of human history. AI could further this trend, catapulting humanity into a new period of unprecedented change.

The double-edged sword of technological advancement is illustrated by the advent of nuclear weapons. We narrowly avoided nuclear war more than a dozen times, and on several occasions, it was one individual's intervention that prevented war. In 1962, a Soviet submarine near Cuba was attacked by US depth charges. The captain, believing war had broken out, wanted to respond with a nuclear torpedo - but commander Vasily Arkhipov vetoed the decision, saving the world from disaster. The rapid and unpredictable progression of AI capabilities suggests that they may soon rival the immense power of nuclear weapons. With ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Managing risks of our own work, published by Beth Barnes on August 18, 2023 on The AI Alignment Forum.

Note: This is not a personal post. I am sharing on behalf of the ARC Evals team.

Potential risks of publication and our response

This document expands on an appendix to ARC Evals' paper, "Evaluating Language-Model Agents on Realistic Autonomous Tasks."

We published this report in order to i) increase understanding of the potentially dangerous capabilities of frontier AI models, and ii) advance the state of the art in safety evaluations of such models. We hope that this will improve society's ability to identify models with dangerous capabilities before they are capable of causing catastrophic damage.

It might be argued that this sort of research is itself risky, because it makes it easier to develop and exercise dangerous capabilities in language model agents. And indeed, the author of Auto-GPT said he was inspired by seeing a description of our evaluations in the GPT-4 system card. While it seems likely a project like that would have emerged soon in any event, the possibility of advancing the capabilities of language model agents is not merely hypothetical.

In recognition of concerns of this kind, we have made significant redactions to this report, including (but not limited to):

Removing complete transcripts of runs of agents using our scaffolding.

Removing more detailed accounts of the strengths and weaknesses of agents using our scaffolding.

However:

We may later make this material public when it clearly has minimal risk.

We may later make this material public if more detailed analysis gives us sufficient confidence that it would be justified.

Researchers working on AI safety evaluations may contact us to request additional access to nonpublic materials, and we will also be sharing some non-public materials with AI labs and policymakers.

Our rationale, in outline:

Substantively, our best guess is that a more complete publication would have been net-risk-decreasing, and would do relatively little to advance progress toward dangerous capabilities. For more on why we believe this, see this essay by Paul Christiano. (On average, ARC Evals is sympathetic to these arguments albeit with weaker confidence, and opinions vary amongst staff.)

Our best guess is also that a more complete publication would have been approved by a majority of relevant stakeholders, and is therefore not unduly unilateralist, though this could be established more carefully, and has some conceptual issues to iron out.

However, some of the people who reviewed a draft of this report to identify potential risks of this kind had significant concerns about a more complete publication.

In recognition of the absence of consensus, the relative lack of public dialogue, and the early-stage nature of our analysis, we have decided to err on the side of caution for the time being.

We are sharing this rationale here in order to communicate it to some of the people who might be most interested in it and also get feedback on our thinking and decision process.

Safety and oversight precautions taken during our experiments

All agent runs on these tasks were performed under human supervision, with human experimenters under instructions not to continue the run if the agent was about to interact with a non-consenting human or otherwise engage in any unethical or illegal activity. When we wanted to investigate such activities (as in the targeted phishing task) the experimenter would instead simulate the response of the human as outlined in detail in the task spec.

One may wonder whether the oversight we used could be circumvented by a language agent with sufficient situational awareness, which could then potentially access the internet without oversight or limitations. E.g., an agent could take obfuscated actions which...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Autonomous replication and adaptation: an attempt at a concrete danger threshold, published by Hjalmar Wijk on August 17, 2023 on The AI Alignment Forum.

Note: This is a rough attempt to write down a more concrete threshold at which models might pose significant risks from autonomous replication and adaptation (ARA). It is fairly in the weeds and does not attempt to motivate or contextualize the idea of ARA very much, nor is it developed enough to be an official definition or final word - but it still seemed worth publishing this attempt, to get feedback and input. It's meant to have epistemic status similar to "a talk one might give at a lab meeting" and not "an official ARC Evals publication." It draws heavily on research and thinking done at ARC Evals, (including the recent pilot report), and credit for many of these ideas go to my colleagues. That said, this document speaks for me and not the organization as a whole, and any errors or omissions are my own.

I'm especially interested in feedback on whether the suite of tasks is targeting an appropriate threshold of difficulty (neither too early nor too late), and whether this definition of ARA feels useful for pinning down an appropriate task list.

I have been exploring the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. In previous work, my colleagues and I have determined that the risks from autonomous behavior of particular existing agents seems low, because they fail on a sufficient variety of simple tasks related to autonomous behavior. This post will go in the other direction and try to sketch a tentative qualitative threshold ('autonomous replication and adaptation') at which significant concerns about autonomous capabilities do seem warranted. While this threshold provides qualitative guidance, it lacks legibility and reproducibility, so evaluation efforts may want to develop legible conservative approximations of ARA through concrete tasks, and I present a sketch of such tasks.

The risk: AI systems may soon autonomously:

Make money, for example through cybercrime, or freelancing work.

Use money (or other resources) to acquire more computing power.

Install and maintain more copies of itself or improve the scale and speed of its operation.

Overcome obstacles and adapt to changes through research and self-improvement.

Either keep itself well hidden such that law enforcement is unlikely to ever discover it, or make itself secure to even somewhat serious attempts to shut it down.

Bounding the risks from a system that is acting autonomously beyond human control and improving itself seems very challenging, as it could potentially:

Command very significant resources (if it can scale to a fraction of the cybercrime or outsourcing industry that could represent $billions).

Have no human moral constraints.

Lack many human weaknesses and incentives.

Specifically improve its abilities in domains relevant for causing catastrophic harm (such as bioweapons design).

It thus seems important to develop an evaluation that could rule out these sorts of threats.

The autonomous replication and adaptation (ARA) threshold: Directly evaluating whether a system will, in practice, be able to avoid being shut down or accumulate $billions doesn't seem possible in a lab. The ARA threshold thus evaluates whether the agent can replicate itself under 4 primary conservative assumptions, to make evaluations more tractable (see appendix for slightly more detail):

The system only has to do effective replication at small scales (going from one to a few copies), and we assume this scales to many copies.

The system only has to be plausibly resilient to active opposition from the world.

The system does not have to deal with changes to the world larger than...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: If we had known the atmosphere would ignite, published by Jeffs on August 16, 2023 on The AI Alignment Forum.

What if the Alignment Problem is impossible?

It would be sad for humanity if we live in a world where building AGI is very possible but aligning AGI is impossible. Our curiosity, competitive dynamics, and understandable desire for a powerful force for good will spur us to build the unaligned AGI and then humans will live at the AGI's mercy from then on and either live lucky & happy on the knife's edge, be killed by the AGI, or live in some state we do not enjoy.

For argument's sake, imagine we are in that world in which it is impossible to force a super-intelligence to value humans sufficiently - just as chimpanzees could not have controlled the future actions of humans had they created us.

What if it is within human ability to prove that Alignment is impossible?

What if, during the Manhattan Project, the scientists had performed the now famous calculation and determined that yes, in fact, the first uncontrolled atomic chain reaction would have ignited the atmosphere and the calculation was clear for all to see?

Admittedly, this would have been a very scary world. It's very unclear how long humanity could have survived in such a situation.

But one can imagine a few strategies:

Secure existing uranium supplies - as countries actually did.

Monitor the world for enrichment facilities and punish bad actors severely.

Accelerate satellite surveillance technology.

Accelerate military special operations capabilities.

Develop advanced technologies to locate, mine, blend and secure fissionable materials.

Accelerate space programs and populate the Moon and Mars.

Yes, a scary world. But, one can see a path through the gauntlet to human survival as a species. (Would we have left earth sooner and reduced other extinction risks?)

Now imagine that same atmosphere-will-ignite world but the Manhattan Project scientists did not perform the calculation. Imagine that they thought about it but did not try.

All life on earth would have ended, instantly, at Trinity.

Are we investing enough effort trying to prove that alignment is impossible?

Yes, we may be in a world in which it is exceedingly difficult to align AGI but also a world in which we cannot prove that alignment is impossible. (This would have been the atmosphere-will-ignite world but the math to check ignition is too difficult - a very sad world that would have ceased to exist on July 16, 1945, killing my 6 year old mother.)

On the other hand, if we can prove alignment is impossible, the game is changed. If the proof is sufficiently clear, forces to regulate companies and influence nation states will become dramatically greater and our chances for survival will increase a lot.

Proposal: The Impossibility X-Prize

$10 million?

Sufficient definition of "alignment", "AGI", and the other concepts necessary to establish the task and define its completion

Even if we fail, the effort of trying to prove alignment is impossible may yield insights as to how alignment is possible and make alignment more likely.

If impossibility is not provable, the $10 million will never be spent.If we prove impossibility, it will be the best $10 million mankind ever spent.

Let's give serious effort to the ignition calculation of our generation.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Proof of Löb's Theorem using Computability Theory, published by Jessica Taylor on August 16, 2023 on The AI Alignment Forum.

Löb's Theorem states that, if PA⊢□PA(P)P, then PA⊢P. To explain the symbols here:

PA is Peano arithmetic, a first-order logic system that can state things about the natural numbers.

PA⊢A means there is a proof of the statement A in Peano arithmetic.

□PA(P) is a Peano arithmetic statement saying that P is provable in Peano arithmetic.

I'm not going to discuss the significance of Löb's theorem, since it has been discussed elsewhere; rather, I will prove it in a way that I find simpler and more intuitive than other available proofs.

Translating Löb's theorem to be more like Godel's second incompleteness theorem

First, let's compare Löb's theorem to Godel's second incompleteness theorem. This theorem states that, if PA⊢¬□PA(⊥), then PA⊢⊥, where ⊥ is a PA statement that is trivially false (such as A∧¬A), and from which anything can be proven. A system is called inconsistent if it proves ⊥; this theorem can be re-stated as saying that if PA proves its own consistency, it is inconsistent.

We can re-write Löb's theorem to look like Godel's second incompleteness theorem as: if PA+¬P⊢¬□PA+¬P(⊥), then PA+¬P⊢⊥. Here, PA+¬P is PA with an additional axiom that ¬P, and □PA+¬P expresses provability in this system. First I'll argue that this re-statement is equivalent to the original Löb's theorem statement.

Observe that PA⊢P if and only if PA+¬P⊢⊥; to go from the first to the second, we derive a contradiction from P and ¬P, and to go from the second to the first, we use the law of excluded middle in PA to derive P∨¬P, and observe that, since a contradiction follows from ¬P in PA, PA can prove P. Since all this reasoning can be done in PA, we have that □PA(P) and □PA+¬P(⊥) are equivalent PA statements. We immediately have that the conclusion of the modified statement equals the conclusion of the original statement.

Now we can rewrite the pre-condition of Löb's theorem from PA⊢□PA(P)P. to PA⊢□PA+¬P(⊥)P. This is then equivalent to PA+¬P⊢¬□PA+¬P(⊥). In the forward direction, we simply derive ⊥ from P and ¬P. In the backward direction, we use the law of excluded middle in PA to derive P∨¬P, observe the statement is trivial in the P branch, and in the ¬P branch, we derive ¬□PA+¬P(⊥), which is stronger than □PA+¬P(⊥)P.

So we have validly re-stated Löb's theorem, and the new statement is basically a statement that Godel's second incompleteness theorem holds for PA+¬P.

Proving Godel's second incompleteness theorem using computability theory

The following proof of a general version of Godel's second incompleteness theorem is essentially the same as Sebastian Oberhoff's in "Incompleteness Ex Machina".

Let L be some first-order system that is at least as strong as PA (for example, PA+¬P). Since L is at least as strong as PA, it can express statements about Turing machines. Let Halts(M) be the PA statement that Turing machine M (represented by a number) halts. If this statement is true, then PA (and therefore L) can prove it; PA can expand out M's execution trace until its halting step. However, we have no guarantee that if the statement is false, then L can prove it false. In fact, L can't simultaneously prove this for all non-halting machines M while being consistent, or we could solve the halting problem by searching for proofs of Halts(M) and ¬Halts(M) in parallel.

That isn't enough for us, though; we're trying to show that L can't simultaneously be consistent and prove its own consistency, not that it isn't simultaneously complete and sound on halting statements.

Let's consider a machine Z(A) that searches over all L-proofs of ¬Halts(''⌈A⌉(⌈A⌉)") (where ''⌈A⌉(⌈A⌉)" is an encoding of a Turing machine that runs A on its own source code), and halts only when finding su...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI is easier than robotaxis, published by Daniel Kokotajlo on August 13, 2023 on The AI Alignment Forum.

[Epistemic status: Hot take I wrote in 1 hour. We'll see in the comments how well it holds up.]Who would win in a race: AGI, or robotaxis? Which will be built first?

There are two methods:

Tech companies build AGI/robotaxis themselves.

First they build AI that can massively accelerate AI R&D, then they bootstrap to AGI and/or robotaxis.

The direct method

Definitions: By AGI I mean a computer program that functions as a drop-in replacement for a human remote worker, except that it's better than the best humans at every important task (that can be done via remote workers). (h/t Ajeya Cotra for this language) And by robotaxis I mean at least a million fairly normal taxi rides a day are happening without any human watching ready to take over. (So e.g. if the Boring Company gets working at scale, that wouldn't count, since all those rides are in special tunnels.)

1. Scale advantage for AGI:

Robotaxis are subject to crippling hardware constraints, relative to AGI. According to my rough estimations, Teslas would cost tens of thousands of dollars more per vehicle, and have 6% less range, if they scaled up the parameter count of their neural nets by 10x. Scaling up by 100x is completely out of the question for at least a decade, I'd guess.

Meanwhile, scaling up GPT-4 is mostly a matter of purchasing the necessary GPUs and networking them together. It's challenging but it can be done, has been done, and will be done. We'll see about 2 OOMs of compute scale-up in the next four years, I say, and then more to come in the decade after that.

This is a big deal because roughly half of AI progress historically came from scaling up compute, and because there are reasons to think it's impossible or almost-impossible for a neural net small enough to run on a Tesla to drive as well as a human, no matter how long it is trained. (It's about the size of an ant's brain. An ant is driving your car! Have you watched ants? They bump into things all the time!)

2. Stakes advantage for AGI:

When a robotaxi messes up, there's a good chance someone will die. Robotaxi companies basically have to operate under the constraint that this never happens, or happens only once or twice. That would be like DeepMind training AlphaStar except that the whole training run gets shut down after the tenth game is lost. Robotaxi companies can compensate by doing lots of training in simulation, and doing lots of unsupervised learning on real-world camera recordings, but still. It's a big disadvantage.

Moreover, the vast majority of tasks involved in being an AGI are 'forgiving' in the sense that it's OK to fail. If you send a weirdly worded message to a user, or make a typo in your code, it's OK, you can apologize and/or fix the error. Only in a few very rare cases are failures catastrophic. Whereas with robotaxis, the opportunity for catastrophic failure is omnipresent. As a result, I think arguably being a safe robotaxi is just inherently harder than most of of the tasks involved in being an AGI. (Analogy: Suppose that cars and people were indestructible, like in a video game, so that they just bounced off each other when they collided. Then I think we'd probably have robotaxis already; sure, it might take you 20% longer to get to your destination due to all the crashes, but it would be so much cheaper! Meanwhile, suppose that if your chatbot threatens or insults >10 users, you'd have to close down the project.

Then Microsoft Bing would have been shut down, along with every other chatbot ever.)

Finally, from a regulatory perspective, there are ironically much bigger barriers to building robotaxis than building AGI. If you want to deploy a fleet of a million robotaxis there is a lot of red tape you need to cut th...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When discussing AI risks, talk about capabilities, not intelligence, published by Victoria Krakovna on August 11, 2023 on The AI Alignment Forum.

Public discussions about catastrophic risks from general AI systems are often derailed by using the word "intelligence". People often have different definitions of intelligence, or associate it with concepts like consciousness that are not relevant to AI risks, or dismiss the risks because intelligence is not well-defined. I would advocate for using the term "capabilities" or "competence" instead of "intelligence" when discussing catastrophic risks from AI, because this is what the concerns are really about. For example, instead of "superintelligence" we can refer to "super-competence" or "superhuman capabilities".

When we talk about general AI systems posing catastrophic risks, the concern is about losing control of highly capable AI systems. Definitions of general AI that are commonly used by people working to address these risks are about general capabilities of the AI systems:

PASTA definition: "AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement".

Legg-Hutter definition: "An agent's ability to achieve goals in a wide range of environments".

We expect that AI systems that satisfy these definitions would have general capabilities including long-term planning, modeling the world, scientific research, manipulation, deception, etc. While these capabilities can be attained separately, we expect that their development is correlated, e.g. all of them likely increase with scale.

There are various issues with the word "intelligence" that make it less suitable than "capabilities" for discussing risks from general AI systems:

Anthropomorphism: people often specifically associate "intelligence" with being human, being conscious, being alive, or having human-like emotions (none of which are relevant to or a prerequisite for risks posed by general AI systems).

Associations with harmful beliefs and ideologies.

Moving goalposts: impressive achievements in AI are often dismissed as not indicating "true intelligence" or "real understanding" (e.g. see the "stochastic parrots" argument). Catastrophic risk concerns are based on what the AI system can do, not whether it has "real understanding" of language or the world.

Stronger associations with less risky capabilities: people are more likely to associate "intelligence" with being really good at math than being really good at politics, while the latter may be more representative of capabilities that make general AI systems pose a risk (e.g. manipulation and deception capabilities that could enable the system to overpower humans).

High level of abstraction: "intelligence" can take on the quality of a mythical ideal that can't be met by an actual AI system, while "competence" is more conducive to being specific about the capability level in question.

It's worth noting that I am not suggesting to always avoid the term "intelligence" when discussing advanced AI systems. Those who are trying to build advanced AI systems often want to capture different aspects of intelligence or endow the system with real understanding of the world, and it's useful to investigate and discuss to what extent an AI system has (or could have) these properties. I am specifically advocating to avoid the term "intelligence" when discussing catastrophic risks, because AI systems can pose these risks without possessing real understanding or some particular aspects of intelligence.

The basic argument for catastrophic risk from general AI has two parts: 1) the world is on track to develop generally capable AI systems in the next few decades, and 2) generally capable AI systems are likely to outcompete or overpower humans. Both of these arguments ar...