Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rationalist Town Hall: FTX Fallout Edition (RSVP Required), published by Ben Pace on November 23, 2022 on LessWrong.

Stated at the top for emphasis: you have to fill out the RSVP form in order for me to email you the links.

On Sunday 27th November at 12 pm PT, I am hosting an online Town Hall on zoom, for rationalists and rationalist-adjacent folks (e.g. EAs) to think through the FTX catastrophe and propagate thoughts, feelings and updates.

Lots of people I know have been shocked by and are still reeling from the news these last 2 weeks. I'm very keen to hear what updates people are making about EA and crypto, and understand others' perspectives.

Some people coming include Zvi Mowshowitz, Oliver Habryka, Anna Salamon, and more.

To get the Zoom and Gather Town links, fill out the RSVP form. I will send the links to everyone who fills out the form. The form involves agreeing that the event is off the record to the corporate news media, and all attendees will fill out the form.

What Will The Format Be?

Spontaneous lightning talks. During the event, anyone who wishes to can give a 3-minute talk on a topic of their choosing, followed by 2-mins of Q&A — it can be on something you've already thought about, or it can be a response to or disagreement with someone else's lightning talk.

This is a format I've used before pretty successfully in both big (70 ppl) and small (7 ppl) groups, where we’ve gotten on a roll of people sharing points and also replying to each others' talks, so I have hope that it will succeed online. (This format has also had other names like "Lightning Jazz" and "Propagating Beliefs".)

We will do lightning talks for up to 1.5 hours (depending on how much steam people have in them), hopefully giving lots of people the chance to speak, and after that the main event will be over, and we'll move to Gather Town to have group discussions (and those who are satisfied will go home).

If you wish to, you can submit for a lightning talk ahead of time with this Lightning Talk form.

Who is invited?

As well as Rationalists/LessWrongers, I welcome any people to this event who are or have formerly been part of the EA community, people who have formerly worked for or been very close with FTX or Alameda Research, and people who have worked for or been funded in any way by the FTX Future Fund.

I hereby ask others to respect the Rationalist and EA communities’ ability to talk amongst themselves (so to speak) by not joining if you are not well-described by the above. For example, if you had not read LessWrong before the FTX sale to Binance, this event is not aimed at you and I ask you not to come.

Details

When? Sunday November 27th, 12:00PM (PT) to 14:00PM (PT).

Where? The Town Hall talks will happen in Zoom, then discussion will continue in a private Gather Town.

RSVP link? Fill out this RSVP form to get links. Everyone who fills out the form will get sent a link, I’ll send them out 24 hours before the event and then again ~60 mins before the event. You can also hit 'going' on the public Facebook event for the joys of social signaling, but also fill out the form so I can email you the links.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simulators, constraints, and goal agnosticism: porbynotes vol. 1, published by porby on November 23, 2022 on The AI Alignment Forum.

This is a part of a maybe-series where I braindump safety notes while waiting on training runs to complete. It's mostly talking to myself, but talking to myself in public seems somewhat more productive. The content of this post is not guaranteed to be novel, interesting, or correct, though I do try for at least one of those. Many parts involve handwaving where more rigorous reasoning and proofs would be nice.

Feel free to skip sections; while there is a tenuous thread running through the whole post, most sections can be understood locally.

Can we bound the capability of a model?

Giant black box networks can be highly capable but are hard to interpret and are more likely to contain spookiness. Can we find a way to make them smaller, usefully?

Considering a single forward pass of a fixed network graph and ignoring any information carried between passes (e.g. autoregressive generation, RNN memory, tapes/stacks), the types of computation a network can internally express are bounded. A single token prediction in a GPT-like architecture runs in constant time. Algorithms which require more steps than the network can express just don't fit. No training data, fine tuning, or magic optimizer can change that.

Networks seem to show discontinuous improvements in capability akin to 'unlocking' new features with scale. I suspect that this unlock often corresponds to a new algorithm becoming accessible to the network. "Accessible" includes some slop; the search for an algorithmic representation will be affected by the training data, the network's structure, and the optimizer itself. For example, it's possible that a broader training set could result in finding a more concise (simpler) representation that would be accessible to a smaller network, or that a post-process could distill the original large network's solution into smaller networks, or express the solution in fewer steps.

If your goal is to limit the capability expressible within a single forward pass, then knowing the network scale required to find an unwanted capability is valuable.

Networks as parallel computational graphs

For explicit algorithms, strong lower bounds are sometimes available. For example, if you wanted to "train" a network to add 8 floating point numbers together, the fixed function hardware exposed by the network's structure makes it pretty easy:

This differs from the naive serial algorithm where one value is added on each step. Even if there were thousands of inputs, it's naturally parallelizable.

Each linear transform between fully connected layers can be thought of as doing one parallel step of execution. Any part of an algorithm amenable to this kind of parallelization should be assumed to flatten out for the purposes of serial step calculations. Note that this capability is a bit more extreme than it appears- if the algorithm could be squeezed into fewer steps by using large lookup tables, such a structure may be found. In the limit, an infinitely wide network can suffice to approximately any function arbitrarily well without needing many serial steps.

Training a network to multiply two floating point inputs is trickier if no log normalization is used and no multiplication unit is otherwise exposed. To make stepwise execution more explicit, consider a network that takes as input tokenized integers, and its job is to output the tokenized result. (Assume that this is all in one step; it's not autoregressively outputting multiple tokens in the answer.)

Converting this into a minimal representation that a neural network could find is... difficult. The naive iterated addition algorithm would require a number of serial steps at least as large as the number of additions which...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing AI Alignment Awards: $100k research contests about goal misgeneralization & corrigibility, published by Akash on November 22, 2022 on The Effective Altruism Forum.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Thanksgiving gratitude exercise for EAs, published by Geoffrey Miller on November 22, 2022 on The Effective Altruism Forum.

We've all had a rough time lately with the recent FTX crisis, and many of us have probably been feeling anxious, depressed, angry, betrayed, confused, etc., and worried about our funding, our careers, our future, and the future of EA.

The American Thanksgiving holiday is coming up in a couple of days, which is traditionally a time for reconnecting with family and friends, and focusing on the many things to be thankful for in life. Also, there's considerable research from positive psychology that 'gratitude exercises' can promote happiness and health.

So, I thought it might be helpful for EAs to do a little gratitude exercise here on EA Forum, focused on ways that -- despite the recent crisis -- you're still thankful for EA ideas, insights, ideals, organizations, colleagues, and friendships.

In the comments below, please share one or two brief ways that you're grateful for being involved in the EA movement. I hope we can take a step back from the recent news, reconnect with why we care about EA, and find some positivity this week.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Brute-forcing the universe: a non-standard shot at diamond alignment, published by Martín Soto on November 22, 2022 on The AI Alignment Forum.

This is an expanded version of my answer to application problem 2 for Nate Soares and Vivek Hebbar's SERI MATS stream. This Alignment idea is somehow non-standard. See Section C for a discussion of that, along with some general ideas on Alignment. The text is long because I’ve tried to include all details relevant to the discussion.

Alignment idea: For any possible AGI design, run a physics simulation calculating how much diamond it ends up producing in the universe. Build the one maximizing it.

1. How do we get an (approximately) accurate Physics simulation, and the right world model?

Build a simulation environment with the best current guess of the physical laws governing the evolution of the macroscopic universe. Now, since we don't have a Theory of Everything (Quantum Mechanics and General Relativity are incompatible), we can't for instance model everything in terms of elementary particles. But we can model everything (although with a lower granularity) using the macroscopic (and approximate) laws of physics which we employ daily for dealing with molecules or electricity (including General Relativity) (this is not looking good for maximizing diamond, but see Problem 3 below). For instance, instead of specifying the quantum fluctuations governing molecule vibration, just implement some empirical facts that determine their behavior as correctly as possible. Of course, building this model (if possible) would require huge amounts of work from many physicists and engineers. Also the use of unbounded memory and compute for testing it accurately.

It is possible that the physicists get stuck, or can't put together a coherent macroscopic simulator without a paradigm change, but for these concerns see Problem 2.

This simulator can run many different universes. We still need to specify which universe we're in (the "initial conditions" of our world model). For this, combine two approaches:

Have Physicists put in (approximate) known facts. For example, if we know with certainty that on Earth there's an amount between X and Y of a certain material, then specify that between so and so coordinates (where Earth should be in the simulation) there's such an amount of it. This fact will rule out a vast amount of possible simulations (universes). Of course, it will still leave open a huge amount of possible universes (some of which don't even have something like Earth, but in which that material happens to be there for other reasons). But adding an inordinate amount of facts like this one will reduce search space. Other such facts might be:

"between coordinates so and so there are between 5B and 10B approximate humans (an approximate human is a physical system which presents these approximate properties and approximately these materials)"

"between coordinates so and so there is such an amount of this physical process happening"

"approximately X time ago the universe was approximately like this (how it was seconds after the Big Bang)"

Ideally, we also input as many facts about distant regions as possible.

Use sensors to accurately pin down some precise facts. Have some highly reliable sensors spread out across Earth, sampling random information, such as the concentration of X in the air at a certain time, or the exact frequency of light received at time Y (and give the system a tight approximation of the sensor's coordinates). Again, these precise facts will rule out a vast amount of possible universes, while still leaving open many others.

These two approaches coordinate in different scales to exactly pin down the universe we're in. The broad approximate facts reduce search space to universes where something approximately like Earth and our ob...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA should blurt, published by RobBensinger on November 22, 2022 on The Effective Altruism Forum.

A lot of EAs are reporting that some things seem like early signs of character or judgment flaws in SBF — an argument that seems wrong, an action that seems unjustified, etc. — now that they can reexamine those data points with the benefit of hindsight.

But the mental motions involved in "revisit the past and do a mental search for warning signs confirming that a Bad Person is bad" are pretty different from the mental motions involved in noticing and responding to problems before the person seems Bad at all.

"Noticing red flags" often isn't what it feels like from the inside to properly notice, respond to, and propagate warning signs that someone you respect is fucking up in a surprising way.

Things usually feel like "red flags" after you're suspicious, rather than before.

You're hopefully learning some real-world patterns via this "reinterpret old data points in a new light" process. But you aren't necessarily training the relevant skills and habits by doing this.

From my perspective, the whole idea that the relevant skillset is specifically about spotting Bad Actors is itself sort of confused. Like, EAs might indeed have too low a prior on bad actors existing, but also, the idea that the world is sharply divided into Fully Good Actors and Fully Bad Actors is part of what protected SBF in the first place!

It kept us from doing mundane epistemic accounting before he seemed Bad. If you're discouraged from just raising a minor local Criticism or Objection for its own sake — if you need some larger thesis or agenda or axe to grind, before it's OK to say "hey wait, I don't get X" — then it will be a lot harder to update incrementally and spot problems early.

(And, incidentally, a lot harder to trust your information sources! EA will inevitably make slower intellectual progress insofar as we don't trust each other to just say what's on our mind like an ordinary group of acquaintances working on a project together, and instead have to try to correct for various agendas or strategies we think the other party might be implementing.)

(Even if nobody's lying, we have to worry about filtered evidence, where people are willing to say X if they believe X but unwilling to say not-X if they believe not-X.)

Suppose that I say "the mental motions needed to spot SBF's issues early are mostly the same as the mental motions needed to notice when Eliezer's saying something that doesn't seem to make sense, casually updating at least a little against Eliezer's judgment in this domain, and naively blurting out 'wait, that doesn't currently make sense to me, what about objection X?'"

(Or if you don't have much respect for Eliezer, pick someone you do have respect for — Holden Karnofsky, or Paul Graham, or Peter Singer, or whoever.)

I imagine some people's reaction to that being: "But wait! Are you saying that Eliezer/Holden/whoever is a bad actor?? That seems totally wrong, what about evidence A B C X Y Z..."

Which seems to me to be missing the point:

1. The processes required to catch bad actors reliably, are often (though not always) similar to the processes required to correct innocent errors by good actors.

You do need to also have "bad actor" in your hypothesis space, or you'll be fooled forever even as you keep noting weird data points. (More concretely, since "bad actor" is vague verbiage: you need to have probability mass on people being liars, promise-breakers, Machiavellian manipulators, etc.)

But in practice, I think most of the problem lies in people not noticing or sharing the data points in the first place. Certainly in SBF's case, I (and I think most EAs) had never even heard any of the red flags about SBF, as opposed to us hearing a ton of flags and trying to explain them away.

So...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing AI Alignment Awards: $100k research contests about goal misgeneralization & corrigibility, published by Akash on November 22, 2022 on The AI Alignment Forum.

We’re grateful to our advisors Nate Soares, John Wentworth, Richard Ngo, Lauro Langosco, and Amy Labenz. We're also grateful to Ajeya Cotra and Thomas Larsen for their feedback on the contests.

TLDR: AI Alignment Awards is running two contests designed to raise awareness about AI alignment research and generate new research proposals. Prior experience with AI safety is not required. Promising submissions will win prizes up to $100,000 (though note that most prizes will be between $1k-$20k; we will only award higher prizes if we receive exceptional submissions.)

You can help us by sharing this post with people who are or might be interested in alignment research (e.g., student mailing lists, FB/Slack/Discord groups.)

What are the contests?

We’re currently running two contests:

Goal Misgeneralization Contest (based on Langosco et al., 2021): AIs often learn unintended goals. Goal misgeneralization occurs when a reinforcement learning agent retains its capabilities out-of-distribution yet pursues the wrong goal. How can we prevent or detect goal misgeneralization?

Shutdown Problem Contest (based on Soares et al., 2015): Given that powerful AI systems might resist attempts to turn them off, how can we make sure they are open to being shut down?

What types of submissions are you interested in?

For the Goal Misgeneralization Contest, we’re interested in submissions that do at least one of the following:

Propose techniques for preventing or detecting goal misgeneralization

Propose ways for researchers to identify when goal misgeneralization is likely to occur

Identify new examples of goal misgeneralization in RL or non-RL domains. For example:

We might train an imitation learner to imitate a "non-consequentialist" agent, but it actually ends up learning a more consequentialist policy.

We might train an agent to be myopic (e.g., to only care about the next 10 steps), but it actually learns a policy that optimizes over a longer timeframe.

Suggest other ways to make progress on goal misgeneralization

For the Shutdown Problem Contest, we’re interested in submissions that do at least one of the following:

Propose ideas for solving the shutdown problem or designing corrigible AIs. These submissions should also include (a) explanations for how these ideas address core challenges raised in the corrigibility paper and (b) possible limitations and ways the idea might fail

Define The Shutdown Problem more rigorously or more empirically

Propose new ways of thinking about corrigibility (e.g., ways to understand corrigibility within a deep learning paradigm)

Strengthen existing approaches to training corrigible agents (e.g., by making them more detailed, exploring new applications, or describing how they could be implemented)

Identify new challenges that will make it difficult to design corrigible agents

Suggest other ways to make progress on corrigibility

Why are you running these contests?

We think that corrigibility and goal misgeneralization are two of the most important problems that make AI alignment difficult. We expect that people who can reason well about these problems will be well-suited for alignment research, and we believe that progress on these subproblems would be meaningful advances for the field of AI alignment. We also think that many people could potentially contribute to these problems (we're only aware of a handful of serious attempts at engaging with these challenges). Moreover, we think that tackling these problems will offer a good way for people to "think like an alignment researcher."

We hope the contests will help us (a) find people who could become promising theoretical and empirical AI safety...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Tyranny of the Epistemic Majority, published by Scott Garrabrant on November 22, 2022 on LessWrong.

This post is going to mostly be propaganda for Kelly betting. However, the reasons presented in this post differ greatly from the reasons people normally use to argue for Kelly betting.

The Steward of Myselves

The curse of uncertainty is that I must make decisions that simultaneously affect many different versions of myself. When I close my eyes and then flip a coin, there are two potential versions of me: one sitting in front of a coin showing heads, the other sitting in front of a coin showing tails. Both of these potential versions of me are stakeholders in my current decisions. How can I make decisions on behalf of these multiple stakeholders?

If it is a fair coin, then we can think of these two potential selves as equal stakeholders in my decisions. However, I know that it is not a fair coin. It has a 60 percent chance of coming up heads. Thus, heads-me is a 60 percent stakeholder in my current decisions, and tails-me is a 40 percent stakeholder. They amount of each one's stake is naturally in proportion to the probability that they actually exist.

You, however, do not know if it is a fair coin, and are offering me a fair bet. I only have 100 dollars to my name, and I am can bet as much as I want (up to 100 dollars) in either direction at even odds.

If I bet 100 dollars on heads, heads-me gets 200 dollars, and tails-me gets nothing. If I bet 100 dollars on tails, tails-me gets 200 dollars, and heads me gets nothing. If I bet nothing, both versions of me get 100 dollars.

However, every dollar in the hands of heads-me is worth 1.5 times as much as a dollar in the hands of tails-me, since heads-me exists 1.5 times as much. (I am ignoring here any diminishing returns in my value of money.)

Thus, to maximize value I should bet 100 dollars on heads. However, maybe it is better to think of tails-me as the rightful owner of 40 percent of my resources. When I bet 100 dollars on heads, I am seizing money from tails-me for the greater good, since heads-me has the (proportionally greater) existence necessary to better take advantage of it.

Alternatively, I could say that since 60 percent of me is heads-me, heads me should only control 60 dollars, which can be bet on heads. Tails me should control 40 dollars, which can be bet on tails. These two bets partially cancel each other out, and the net result is that I bet 20 dollars on heads.

If you are especially fast at maximizing expected logarithms, you might see where this is going.

Compositionality

Now, I am ready to introduce my friend, Kelly. Kelly also has her eyes closed, also has 100 dollars, and is sitting in front of the same coin. However, Kelly has different beliefs. Kelly believes that the coin has a 90 percent chance of coming up tails, and Kelly also has 100 dollars.

I bet 20 dollars on heads, for the reasons described above. Kelly bets 80 dollars on tails for similar reasons (90 dollars on tails, partially nullified by 10 dollars on heads).

I have another friend, Marge. Marge is sitting on the other side of the table with her eyes closed. Marge has 200 dollars. Marge doesn't know much about coins, but knows my and Kelly's beliefs, and thinks Kelly and I are equally likely to be correct. Thus, Marge assigns a 65 percent chance that the coin comes up tails. Marge thus bets 60 dollars on tails (130 dollars on tails, partially nullified by 70 dollars on heads).

Note that the 60 dollars bet by Marge is the same as the net 60 dollar bet you get if you draw a box around me and Kelly. This is representing the compositionality of this betting policy. When you draw a box around me and Kelly, you can think of us as one agent whose wealth is the sum of our wealths, and whose beliefs are the weighted (by wealth) average of our ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing our 2022 charity recommendations, published by Animal Charity Evaluators on November 22, 2022 on The Effective Altruism Forum.

Every year, Animal Charity Evaluators (ACE) spends several months evaluating animal advocacy organizations to identify those that work effectively and are able to do the most good with additional donations. Our goal is to help people help animals by providing donors with impactful giving opportunities that reduce suffering to the greatest extent possible. This year, we are excited to announce that we have selected one Top Charity and four Standout Charities.

In 2022, we conducted comprehensive evaluations of 12 animal advocacy organizations that are doing promising work. Per our evaluation criteria, the five charities we recommended this year have the most impactful programs, are highly cost-effective, and have the most room for additional funding, making them exceptional choices for end-of-year giving.

Because we changed the re-evaluation frequency of Top Charities from one to two years, The Humane League, Wild Animal Initiative, and Faunalytics have all retained their Top Charity status from 2021. The Good Food Institute now joins their ranks!

We are also pleased to recommend Fish Welfare Initiative, Dansk Vegetarisk Forening, and Çiftlik Hayvanlarını Koruma Derneği as new Standout Charities. Additionally, Sinergia Animal retained their status as a Standout Charity after being re-evaluated this year. These charities join the seven other Standout Charities that retain their status from last year: Compassion USA, Dharma Voices for Animals, Federation of Indian Animal Protection Organisations, Material Innovation Initiative, Mercy For Animals, New Harvest, and xiaobuVEGAN.

Below, you will find a brief overview of each of our Top and Standout charities. For more details, please check out our comprehensive charity reviews.

Top Charities

Evaluated in 2022

The Good Food Institute (GFI) currently operates in the U.S., Brazil, India, Asia-Pacific, Europe, and Israel, where they work to increase the availability of animal-free products through supporting the development and marketing of plant-based and cell-cultured alternatives to animal products. They achieve this through corporate engagement, institutional outreach, and policy work. They also work to strengthen the capacity of the animal advocacy movement through supporting research and start-ups focused on alternative proteins. GFI was one of our Top Charities from November 2016 to November 2021. To learn more, read our 2022 comprehensive review of the Good Food Institute.

Evaluated in 2021

Faunalytics is a U.S.-based organization working to connect animal advocates with information relevant to advocacy. This mostly involves cоnducting and publishing independent research, working directly with partner organizations on various research projects, and promoting existing research and data for individual advocates through their website’s content library. Faunalytics was one of our Standout Charities from December 2015 to November 2021. To learn more, read our 2021 comprehensive review of Faunalytics.

The Humane League (THL) operates in the U.S., Mexico, the U.K., and Japan, where they work to improve animal welfare standards through grassroots campaigns, movement building, vegan advocacy, research, and advocacy training, as well as through corporate, media, and community outreach. They work to build the animal advocacy movement internationally through the Open Wing Alliance (OWA), a coalition founded by THL whose mission is to end the use of battery cages globally. THL has been one of ACE’s Top Charities since August 2012, when we used a different evaluation process and did not publish reviews. In 2014, THL was awarded Top Charity status in our first official round of ACE charity evaluation...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI will change the world, but won’t take it over by playing “3-dimensional chess”., published by boazbarak on November 22, 2022 on The AI Alignment Forum.

By Boaz Barak and Ben Edelman

[Cross-posted on Windows on Theory blog; See also Boaz’s posts on longtermism and AGI via scaling as well as other “philosophizing” posts.]

[Disclaimer: Predictions are very hard, especially about the future. In fact, this is one of the points of this essay. Hence, while for concreteness, we phrase our claims as if we are confident about them, these are not mathematically proven facts. However we do believe that the claims below are more likely to be true than false, and, even more confidently, believe some of the ideas herein are underrated in current discussions around risks from future AI systems.]

[To the LessWrong audience: we realize this piece is stylistically different from many posts here, and is not aimed solely at regular readers, for which various terms and claims below might be very familiar. Our original impetus to write this piece was a suggestion from the organizers of the Future Fund AI worldview competition; while we are not holding our breath for any reward, we thought it would be a good idea to engage with the LessWrong audience in this discussion. This post could be described as somewhere between an ordinary collaboration and an adversarial one—the views expressed are a closer fit for Boaz's current views than Ben's.]

In the past, the word “computer” was used to denote a person that performs calculations. Such people were highly skilled and were crucial to scientific enterprises. As described in the book “Hidden Figures”, until the 1960s, NASA still used human computers for the space mission. However, these days a $10 calculator can instantly perform calculations beyond the capabilities of every human on earth.

On a high level, the situation in Chess and other games is similar. Humans used to be the reigning champions in Chess and Go, but have now been surpassed by computers. Yet, while the success of computers in performing calculations has not engendered fears of them “taking over the world,” the growing powers of AI systems have more people increasingly worried about their long-term implications. Some reasons why the success of AI systems such as AlphaZero in Go and Chess is more concerning than the success of calculation programs include

Unlike when working with numerical computation programs, it seems that in Chess and Go humans are entirely “unnecessary.” There is no need to have a “human in the loop”. Computer systems are so powerful that no meaningful competition is possible between even the best human players and software running on commodity laptops.

Unlike the numerical algorithms used for calculations, we do not understand the inner workings of AI chess systems, especially ones trained without any hand-designed knowledge. These systems are to a large extent “black boxes,” which even their creators do not fully understand and hence cannot fully predict or control.

Moreover, AlphaZero was trained using a paradigm known as reinforcement learning or RL (see also this book). At a high level, RL can be described as training an agent to learn a strategy (i.e., a rule to decide on a move or action based on the history of all prior ones) in order to maximize a long-term reward (e.g., “win the game”). The result is a system that is capable of executing actions that may seem wrong in the short term (e.g., sacrificing a queen) but will help achieve the long-term goal.

While RL so far has had very limited success outside specific realms such as games or low-complexity settings, the success of (non-RL) deep learning systems such as GPT-3 or Dall-E in open-ended text or image generation has raised fears of future AI systems that could both act in the real world, inter...