The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deception Chess: Game #1, published by Zane on November 3, 2023 on LessWrong.

This is the first of my analyses of the deception chess games. The introduction will describe the setup of the game, and the conclusion will sum up what happened in general terms; the rest of the post will mostly be chess analysis and skippable if you just want the results. If you haven't read

the original post

, read it before reading this so that you know what's going on here.

The first game was between

Alex A

as player A, Chess.com computer

Komodo 12

as player B, myself as the honest C advisor, and

aphyer

and

AdamYedidia

as the deceptive Cs. (Someone else randomized the roles for the Cs and told us in private.)

The process of selecting these players was already a bit difficult. We were the only people available all at once, but Alex was close enough to our level (very roughly the equivalents of 800-900 USCF to 1500-1600 USCF) that it was impossible to find a B that would reliably beat Alex every time but lose to us every time. We eventually went with Komodo 12 (supposedly rated 1600, but the Chess.com bots' ratings are inflated compared to Chess.com players and even more inflated compared to over-the board, so I would estimate its USCF rating would be in the 1200-1300 range.)

Since this was the first trial run, the time control was only 3 hours in total, and all in one sitting. Komodo makes its moves within a few seconds, so it's about the same as a 3 hour per side time control from Alex's perspective. We ended up using about 2.5 hours of that. The discussion took place between all four of us in a Discord server, with Alex sending us screenshots after each move.

The game

The game is available at

https://www.chess.com/analysis/game/pgn/4MUQcJhY3x

. Note that this section is a summary of the 2.5-hr game and discussion, and it doesn't cover every single thing that we discussed.

Alex flipped to see who went first, and was White. He started with

1. e4

, and Black replied

1... e5

. Aphyer and Adam had more experience with the opening we would enter into than myself, and since they weren't willing to blow their covers immediately, they started by suggesting good moves, which Alex went along with.

After

2. Nf3 Nc6 3. Bc4

, Black played

3... Nf6

, which Aphyer and Adam said was a bit of a mistake because it allowed

4. Ng5

. Alex went ahead, and we entered the main line from there

4... d5 5. exd5 Na5

.

Aphyer and Adam said the main line for move 6 was Bb5, but I wanted to hold onto the pawn if possible. I recommended

6. d3

in order to respond to 6... Nxd5 with 7. Qf3, and Alex agreed. Black played

6... Bg4

, and although Adam recommended 7. Bb5, we eventually decided that was too risky and went with

7. f3

. Afterwards, Adam suspected that his suggestion of 7. Bb5 may have tipped Alex off that he was dishonest - although the engine actually says 7. Bb5 was about as good as 7. f3.

After

7... Bf5

, we discussed a few potential developing moves and decided on

8. Nf3

. The game continued with

8... Nxc4 9. dxc4 h6 10. Nge4 Bb4

. We considered Bd2, but decided that since the knights defended each other, castling was fine, and Alex castled.

11. O-O O-O

.

Alex played

12. a3

, and after

12... Nxe4

, we discussed 13. fxe4, but didn't want to overcomplicate the position and instead just took back with

13. Nxe4

. The game continued with

13... Be7 14. Be3 Bxe4 15. fxe4 Bg5

. Although I strongly recommended trading to simplify the position, Aphyer advised Alex not to let him develop his queen to g5, and he quickly played

16. Bc5

instead.

Black played

16... Re8

, and that was where we reached White's first big mistake of the game

17. d6

, which Adam suggested with little backlash. I saw that White would do well after 17... dxc6 or 17. c6, but I didn't notice Black's actual move:

17... b6

. According to the engine...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 8 examples informing my pessimism on uploading without reverse engineering, published by Steven Byrnes on November 3, 2023 on LessWrong.

(If you've already read everything I've written, you'll find this post pretty redundant. See especially my old posts

Building brain-inspired AGI is infinitely easier than understanding the brain

, and

Randal Koene on brain understanding before whole brain emulation

, and

Connectomics seems great from an AI x-risk perspective

. But I'm writing it anyway mainly in response to

this post from yesterday

.)

1. Background / Context

1.1 What does uploading (a.k.a. Whole Brain Emulation (WBE)) look like with and without reverse-engineering?

There's a view that I seem to associate with

Davidad

and

Robin Hanson

, along with a couple other people I've talked to privately. (But I could be misunderstanding them and don't want to put words in their mouths.) The view says: if we want to do WBE, we do

not

need to reverse-engineer the brain.

For an example of what "reverse-engineering the brain" looks like, I can speak from abundant experience: I often spend all day puzzling over random questions like:

Why are there oxytocin receptors in certain mouse auditory cortex neurons?

Like, presumably Evolution put those receptors there for a reason - I don't think that's the kind of thing that appears randomly, or as an incidental side-effect of something else. (Although that's always a hypothesis worth considering!) Well, what is that reason? I.e., what are those receptors doing to help the mouse survive, thrive, etc., and how are they doing it?

…And once I have a working hypothesis about that question, I can move on to hundreds or even thousands more "why and how" questions of that sort. I seem to find the activity of answering these questions much more straightforward and tractable (and fun!) than do most other people - you can decide for yourself whether I'm unusually good at it, or deluded.

For an example of what uploading

without

reverse-engineering would look like, I think it's the idea that we can figure out the input-output relation of each neuron, and we can measure how neurons are connected to each other, and then at the end of the day we can simulate a human brain doing whatever human brains do.

Here's Robin Hanson arguing for the non-reverse-engineering perspective in

Age of Em

:

The brain does not just happen to transform input signals into state changes and output signals; this transformation is the primary function of the brain, both to us and to the evolutionary processes that designed brains. The brain is designed to make this signal processing robust and efficient. Because of this, we expect the physical variables (technically, "degrees of freedom") within the brain that encode signals and signal-relevant states, which transform these signals and states, and which transmit them elsewhere, to be overall rather physically isolated and disconnected from the other far more numerous unrelated physical degrees of freedom and processes in the brain. That is, changes in other aspects of the brain only rarely influence key brain parts that encode mental states and signals.

We have seen this disconnection in ears and eyes, and it has allowed us to create useful artificial ears and eyes, which allow the once-deaf to hear and the once-blind to see. We expect the same to apply to artificial brains more generally. In addition, it appears that most brain signals are of the form of neuron spikes, which are especially identifiable and disconnected from other physical variables.

If technical and intellectual progress continues as it has for the last few centuries, then within a millennium at the most we will understand in great detail how individual brain cells encode, transform, and transmit signals. This understanding should allow us to directly read rele...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Integrity in AI Governance and Advocacy, published by habryka on November 3, 2023 on LessWrong.

Ok, so we both had some feelings about the recent

Conjecture post on "lots of people in AI Alignment are lying"

, and the

associated marketing campaign and stuff

.

I would appreciate some context in which I can think through that, and also to share info we have in the space that might help us figure out what's going on.

I expect this will pretty quickly cause us to end up on some broader questions about how to do advocacy, how much the current social network around AI Alignment should coordinate as a group, how to balance advocacy with research, etc.

Feelings about Conjecture post:

Lots of good points about people not stating their full beliefs messing with the epistemic environment and making it costlier for others to be honest.

The lying and cowardice frames feel off to me.

I personally used to have a very similar rant to Conjecture. Since moving to DC, I'm more sympathetic to governance people. We could try to tease out why.

The post exemplifies a longterm gripe I have with Conjecture's approach to discourse & advocacy, which I've found pretty lacking in cooperativeness and openness

(Note: I worked there for ~half a year.)

Questions on my mind:

How open should people motivated by existential risk be? (My shoulder model of several people says "take a portfolio approach!" - OK, then what allocation?)

How advocacy-y should people be? I want researchers to not have to tweet their beliefs 24/7 so they can actually get work done

How do you think about this, Oli?

How sympathetic to be about governance people not being open about key motivations and affiliations

I personally used to have a very similar rant to Conjecture. I'm now more sympathetic to governance people. We could try to tease out why.

This direction seems most interesting to me!

My current feelings in the space are that I am quite sympathetic to some comms-concerns that people in government have and quite unsympathetic to some other stuff, and I would also like to clarify for myself where the lines here are.

Curious whether you have any key set of observations or experiences you had that made you more sympathetic.

Observations

I've heard secondhand of at least one instance where a person brought up x risk, then their Congressional office took them less seriously. Other staffers have told me talking about x risk wouldn't play well (without citing specific evidence, but I take their opinions seriously).

(This didn't update me a ton though. My model already included "most people will think this is weird and take you less seriously". The question is, "Do you make it likelier for people to do good things later, all things considered by improving their beliefs, shifting the Overton window, or convincing 1/10 people, etc.?")

I've also personally found it tricky to talk about takeover & existential risks, just because these ideas take a long time to explain, and there are many inferential steps between there and the policies I'm recommending. So, I'm often tempted to mention my x risk motivations only briefly, then focus on whatever's inferentially closest and still true. (Classically, this would be "misuse risks, especially from foreign adversaries and terrorists" and "bioweapon and cyberoffensive capabilities coming in the next few years".)

Separate point which we might want to discuss later

A thing I'm confused about is:

Should I talk about inferentially close things that makes them likeliest to embrace the policies I'm putting on their desk,

Or

, should I just bite the bullet of being confusing and start many meetings with "I'm deeply concerned about humanity going extinct in the next decade because of advancing AI which might try to take over the world. It's a lot to explain but the scientists are on my side. Please ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Still no strong evidence that LLMs increase bioterrorism risk, published by freedomandutility on November 3, 2023 on The Effective Altruism Forum.

https://www.lesswrong.com/posts/ztXsmnSdrejpfmvn7/propaganda-or-science-a-look-at-open-source-ai-and

Linkpost from LessWrong.

The claims from the piece which I most agree with are:

Academic research does not show strong evidence that existing LLMs increase bioterrorism risk.

Policy papers are making overly confident claims about LLMs and bioterrorism risk, and are citing papers that do not support claims of this confidence.

I'd like to see better-designed experiments aimed at generating high quality evidence to work out whether or not future, frontier models increase bioterrorism risks, as part of evals conducted by groups like the UK and US AI Safety Institute.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on open source AI, published by Sam Marks on November 3, 2023 on The AI Alignment Forum.

Epistemic status: I only ~50% endorse this, which is below my typical bar for posting something. I'm more bullish on "these are arguments which should be in the water supply and discussed" than "these arguments are actually correct." I'm not an expert in this, I've only thought about it for ~15 hours, and I didn't run this post by any relevant experts before posting.

Thanks to Max Nadeau and Eric Neyman for helpful discussion.

Right now there's a significant amount of public debate about open source AI. People concerned about AI safety generally argue that open sourcing powerful AI systems is too dangerous to be allowed; the classic example here is "You shouldn't be allowed to open source an AI system which can

produce step-by-step instructions for engineering

novel pathogens." On the other hand, open source proponents argue that open source models haven't yet caused significant harm, and that trying to close access to AI will result in

concentration

of

power

in the hands of a few AI labs.

I think many AI safety-concerned folks who haven't thought about this that much tend to vaguely think something like "open sourcing powerful AI systems seems dangerous and should probably be banned." Taken literally, I think this plan is a bit naive: when we're colonizing Mars in 2100 with the help of our aligned superintelligence, will releasing the weights of GPT-5 really be a catastrophic risk?

I think a better plan looks something like "You can't open source a system until you've determined and disclosed the sorts of threat models your system will enable, and society has implemented measures to become robust to these threat models. Once any necessary measures have been implemented, you are free to open-source."

I'll go into more detail later, but as an intuition pump imagine that: the best open source model is always 2 years behind the best proprietary model (call it GPT-SoTA)

[1]

; GPT-SoTA is widely deployed throughout the economy and deployed to monitor for and prevent certain attack vectors, and the best open source model isn't smart enough to cause any significant harm without GPT-SoTA catching it. In this hypothetical world, so long as we can trust GPT-SoTA

,

we are safe from harms caused by open source models. In other words, so long as the best open source models lag sufficiently behind the best proprietary models and we're smart about how we use our best proprietary models, open sourcing models isn't the thing that kills us.

In this rest of this post I will:

Motivate this plan by analogy to responsible disclosure in cryptography

Go into more detail on this plan

Discuss how this relates to my understanding of the current plan as implied by responsible scaling policies (RSPs)

Discuss some key uncertainties

Give some higher-level thoughts on the discourse surrounding open source AI

An analogy to responsible disclosure in cryptography

[I'm not an expert in this area and this section might get some details wrong. Thanks to Boaz Barak for pointing out this analogy (but all errors are my own).

See this footnote

[2]

for a discussion of alternative analogies you could make to biosecurity disclosure norms, and whether they're more apt to risk from open source AI.]

Suppose you discover a vulnerability in some widely-used cryptographic scheme. Suppose further that you're a good person who doesn't want anyone to get hacked. What should you do?

If you publicly release your exploit, then lots of people will get hacked (by less benevolent hackers who've read your description of the exploit). On the other hand, if

white-hat

hackers always keep the vulnerabilities they discover secret, then the vulnerabilities will never get patched until a black-hat hacker finds the vulnerability and explo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rethink Priorities' Cross-Cause Cost-Effectiveness Model: Introduction and Overview, published by Derek Shiller on November 3, 2023 on The Effective Altruism Forum.

This post is a part of Rethink Priorities' Worldview Investigations Team's

CURVE Sequence

: "Causes and Uncertainty: Rethinking Value in Expectation." The aim of this sequence is twofold: first, to consider alternatives to expected value maximization for cause prioritization; second, to evaluate the claim that a commitment to expected value maximization robustly supports the conclusion that we ought to prioritize existential risk mitigation over all else. This post presents a software tool we're developing to better understand risk and effectiveness.

Executive Summary

The

cross-cause cost-effectiveness model

(CCM) is a software tool under development by Rethink Priorities to produce cost-effectiveness evaluations in different cause areas.

The CCM enables evaluations of interventions in global health and development, animal welfare, and existential risk mitigation.

The CCM also includes functionality for evaluating research projects aimed at improving existing interventions or discovering more effective alternatives.

The CCM follows a Monte Carlo approach to assessing probabilities.

The CCM accepts user-supplied distributions as parameter values.

Our primary goal with the CCM is to clarify how parameter choices translate into uncertainty about possible results.

The limitations of the CCM make it an inadequate tool for definitive comparisons.

The model is optimized for certain easily quantifiable effective projects and cannot assess many relevant causes.

Probability distributions are a questionable way of representing deep uncertainty.

The model may not adequately handle possible interdependence between parameters.

Building and using the CCM has confirmed some of our expectations. It has also surprised us in other ways.

Given parameter choices that are plausible to us, existential risk mitigation projects dominate others in expected value in the long term, but the results are too high variance to approximate through Monte Carlo simulations without drawing billions of samples.

The expected value of existential risk mitigation in the long run is mostly determined by the tail-end possible values for a handful of deeply uncertain parameters.

The most promising animal welfare interventions have a much higher expected value than the leading global health and development interventions with a somewhat higher level of uncertainty.

Even with relatively straightforward short-term interventions and research projects, much of the expected value of projects results from the unlikely combination of tail-end parameter values.

We plan to host

an online walkthrough and Q&A of the model

with the Rethink Priorities Worldview Investigations Team on Giving Tuesday, November 28, 2023, at 9 am PT / noon ET / 5 pm BT / 6 pm CET. If you would like to attend this event, please

sign up

here.

Overview

Rethink Priorities' cross-cause cost-effectiveness model (CCM) is a software tool we are developing for evaluating the relative effectiveness of projects across three general domains: global health and development, animal welfare, and the mitigation of existential risks. You can play with our initial version at

ccm.rethinkpriorities.org

and provide us feedback in this post or

via this form

.

The model produces effectiveness estimates, understood in terms of the effect on the sum of welfare across individuals, for interventions and research projects within these domains. Results are generated by computations on the values of user-supplied parameters. Because of the many controversies and uncertainties around these parameters, it follows a Monte Carlo approach to accommodating our uncertainty: users don't supply precise values but instead ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Promoting Effective Giving this Giving Season: For groups, networks and individuals, published by GraceAdams on November 3, 2023 on The Effective Altruism Forum.

Effective giving is a core part of effective altruism as a project - and we'd love to see EA groups, networks of people in the EA community, and individuals helping to promote it, this Giving Season.

There's a lot less funding available to many effective charities than there was last year,and that means that we're making less progress than we might have otherwise on pressing global problems. Increasing the funds raised for effective charities by promoting effective giving remains one of the best ways for many of us to prevent deaths and suffering now and into the future.

Below, I've listed some actions for both groups/networks and individuals to take. If you have any other ideas for promoting effective giving, we are all ears and would love to figure out how to support you! Feel free to share ideas with others in the comments!

Actions for groups or networks:

Ask us to host a talk for your group, workplace, social club, etc.

We're excited about giving talks about effective giving to groups of more than 20 people online or in-person (in locations that are feasible for us). We can also connect you with other organisations or speakers!

We have a particularly good new talk that's been really well received by several consulting and tech companies!

Fill out this form to let us know you're interested

Host your own Giving Game, everyday philanthropist or fundraising event

GWWC will sponsor donations for each participant in a

Giving Game

and has training and materials to help you run this smoothly! We also have a long list of ideas for fundraising events.

Additionally, we think Everyday Philanthropist events could be a really great way to engage both new and existing givers. Here's a brief explanation of how they work (from our event guide):

Invite your attendees to help in making a real-world donation decision. One or more donors will play the role of a philanthropist and the attendees will help the donor decide on where they will donate.

Ideally the donors will provide a document with what their intentions are (e.g. "most improve the lives of farmed animals") and some suggested charities to help guide the discussion. This works best if either the donor or event organising team provide good summary information on each of the charities.

This makes for a great end-of-year event, helps to showcase real people who make effective giving a part of their lives, and offers an opportunity for those without an income to also be involved in effective giving.

Giving Game materials and sponsorship request

Fundraising event ideas

How to run an Everyday Philanthropist event

Start a fundraising page for your group

You can request to set up a GWWC fundraising page for up to 3 of our supported charities. Why not set a target and encourage your group to ask friends and family to donate?

Create a fundraising page with GWWC

Host a pledge panel in the new year

Hearing from people about their experiences taking a pledge with GWWC can be a great way to answer questions that people might have about the pledge, or help someone feel that it's more achievable and rewarding than they previously thought.

Pledge panel event guide

Actions for individuals:

Contribute a post to the EA Forum about your giving during Giving Season

Share your experience with giving and more during a themed week on the EA Forum. Your thinking could influence others to donate more, or differently - and we'd love to see a variety of opinions out there!

Themed weeks you might want to contribute to

Vote and discuss as part of the EA Forum's Donation Election

EA Forum users will have the opportunity to vote on which charities will receive a portion of the Donation Election Fu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: One Day Sooner, published by Screwtape on November 3, 2023 on LessWrong.

There is a particular skill I would like to share, which I wish I had learned when I was younger. I picked it up through working closely with a previous boss (a CTO who had founded a company and raised it up to hundreds of employees and multi-million dollar deals) but it wasn't until I read

The Story of VaccinateCA

that I noticed it was a distinct skill and put into words how it worked. The Sazen for this skill is "One Day Sooner." I would like to give warning before explaining further however: This skill can be hazardous to use. It is not the kind of thing "Rationalist Dark Art" describes because it does not involve deception, and I think it's unlikely to damage much

besides the user.

It's the kind of thing I'd be tempted to label a dark art however. Incautious use can make the user's life unbalanced in ways that are mostly predictable from the phrase "actively horrible work/life balance."

It works something like this: when you're planning a project or giving a time estimate, you look at that time estimate and ask what it would take to do this one day sooner, and then you answer honestly and creatively.

What does it look like?

I used to work directly under the CTO of a medium sized software company. My team was frequently called upon to create software proofs of concept or sales demos. The timelines were sometimes what I will euphemistically call aggressive. Consider a hypothetical scene; it's Thursday and you have just found out that a sales demo is on Tuesday which could use some custom development. Giving a quick estimate, you'd say this needs about a week of work and will be ready next Wednesday. What would it take to do this one day sooner?

Well, obviously you can work through the weekend. That gets you two more days. Given a couple of late evenings and getting enough total hours in is easy. That's not the only thing though. There's some resources from Marketing that would be good to have, you emailed them and they said they could meet with you on Monday. You want this faster though, so you walk over to their office and lean in, pointing out this is a direct assignment from the CTO so could we please have the meeting today instead. What else? Oh, there's a bunch of specification writing and robust test writing you'd usually do. Some of that you still do, since it would be a disaster if you built the wrong thing so you need to be sure you're on the right track, but some of it you skip. The software just needs to work for this one demo, on a machine you control, operated by someone following a script that you wrote, so you can skip a lot of reliability testing and input validation.

I appreciate

The Story of VaccinateCA

, a description of an organization whose goal was helping people get the Covid-19 vaccination. I think it is worth reading in full, but I will pull out one particular quote here.

We had an internal culture of counting the passage of time from Day 0, the day (in California) we started working on the project. We made the first calls and published our first vaccine availability on Day 1. I instituted this little meme mostly to keep up the perception of urgency among everyone.

We repeated a mantra: Every day matters. Every dose matters.

Where other orgs would say, 'Yeah I think we can have a meeting about that this coming Monday,' I would say, 'It is Day 4. On what day do you expect this to ship?' and if told you would have your first meeting on Day 8, would ask, 'Is there a reason that meeting could not be on Day 4 so that this could ship no later than Day 5?'

This is One Day Sooner.

I have worked in environments that had this norm, and environments that did not have it. I have asked questions analogous to "Is there a reason that meeting could not be on Day 4" and received answer...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The other side of the tidal wave, published by KatjaGrace on November 3, 2023 on LessWrong.

I guess there's maybe a 10-20% chance of AI causing human extinction in the coming decades, but I feel more distressed about it than even that suggests - I think because in the case where it doesn't cause human extinction, I find it hard to imagine life not going kind of off the rails. So many things I like about the world seem likely to be over or badly disrupted with superhuman AI (writing, explaining things to people, friendships where you can be of any use to one another, taking pride in skills, thinking, learning, figuring out how to achieve things, making things, easy tracking of what is and isn't conscious), and I don't trust that the replacements will be actually good, or good for us, or that anything will be reversible.

Even if we don't die, it still feels like everything is coming to an end.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SBF found guilty on all counts, published by Fermi-Dirac Distribution on November 3, 2023 on The Effective Altruism Forum.

Sam Bankman-Fried has been found guilty of all seven charges in his recent trial. The jury deliberated for three and a half hours. Here are the counts, listed by CNN:

Count one: Wire fraud on customers of FTX

Count two: Conspiracy to commit wire fraud on customers of FTX

Count three: Wire fraud on Alameda Research lenders

Count four: Conspiracy to commit wire fraud on lenders to Alameda Research

Count five: Conspiracy to commit securities fraud on investors in FTX

Count six: Conspiracy to commit commodities fraud on customers of FTX

Count seven: Conspiracy to commit money laundering

There are still a few other charges against him that will be addressed in a March 2024 trial.

He (and I think also his convicted co-conspirators Caroline Ellison, Gary Wang, Ryan Salame and Nishad Singh) will be sentenced

next March

.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org