Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Letter to the Bulletin of Atomic Scientists, published by John G. Halstead on November 23, 2022 on The Effective Altruism Forum.

Tldr: This is a letter I wrote to the Climate Contributing Editor of the Bulletin Atomic Scientists, Dawn Stover, about Emile Torres' latest piece criticising EA. In short:

In advance of the publication of the article, Ms Stover reached out to us to check on what Torres calls their most "disturbing" claim viz. that Will MacAskill lied about getting advice from five climate experts.

We showed them that this was false.

The Bulletin published the claim anyway, and then tweeted it.

In my opinion, this is outrageous, so I have asked them to issue a correction and an apology.

Dear Ms Stover,

I have long admired the work of the Bulletin of the Atomic Scientists. However, I am extremely disappointed by your publication of the latest piece by Emile Torres.

I knew long ago that Torres would publish a piece critical of What We Owe the Future, and on me following my report on climate change. However, I am surprised that the Bulletin has chosen to publish this particular piece in its current form. There are many things wrong with the piece, but the most important is that it accuses Will MacAskill and his research assistants of research misconduct. Specifically, Torres contends that five of the climate experts we listed in the acknowledgements for the book were not actually consulted.

Ms Stover: you contacted us about this claim in advance of the article’s publication, and we informed you that it was not true. Overall, we consulted around 106 experts in the research process for What We Owe The Future. Torres suggests that five experts were never consulted at all, but this is not true — as Will stated in his earlier email to you, four of those five experts were consulted. I am happy to provide evidence for this. The article would have readers think that we made up the citations out of thin air. One of them was contacted but didn’t have time to give feedback, and was incorrectly credited in the acknowledgements, which we will change in future editions: this was an honest mistake. The Bulletin also went on to tweet the false claim that multiple people hadn’t been consulted at all.

The acknowledgements are also clear that we are not claiming that those listed checked and agreed with every claim in the book. Immediately after the acknowledgements of subject-matter experts, Will writes: “These advisers don’t necessarily agree with the claims I make in the book, and all errors in the book are my responsibility alone.”

To accuse someone of research misconduct is a very serious allegation. After you check it and find out that it is false, it is extremely poor form to let the claim go out anyway and then to tweet it. The Bulletin should issue a correction to the article, and to the false claim they put out in a tweet.

I also have concerns about the nature of Torres’ background work for article — they seemingly sent every person that was acknowledged for the book a misleading email, telling them that we lied in the acknowledgements, and making some reviewers quite uncomfortable.

To reiterate, I am very disappointed by the journalistic standards demonstrated in this article. I will be publishing something separately about Torres’ (as usual) misrepresented substantive claims, but the most serious allegation of research misconduct needs to be retracted and we need an apology.

(Also, a more minor point: it's not true that I am Head of Applied Research at Founders Pledge. I left that role in 2019.)

John

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Human-level Diplomacy was my fire alarm, published by Lao Mein on November 23, 2022 on LessWrong.

Epistemic status: AI is my hobby, not my job.

Meta just released their new Diplomacy AI, Cicero. Sure, it was several different modules kludged together in a somewhat clever way, with no fundamentally new ideas or massive scaling, but it was damn impressive.

What surprises me most about Cicero is its ability to go all the way from data input to natural language output with relatively high reliability. The model makes a battle plan, uses natural language to interface with humans, then uses the information gained from said interface to alter its plan. And it does this consistently enough to outmatch the average human Diplomacy player. Maybe it doesn't represent anything truly new in terms of architecture or scaling. But that's exactly why I'm so impressed.

It doesn't have the "human spark". But the lesson that we've been learning over the past few decades is that very impressive tasks, from human level Chess to Go to art, can be accomplished without said spark. I'm starting to suspect that human-level AGI is entirely possible with something like Cicero. If systems like Cicero can remain coherent over longer time scales, we could be seeing the automation of many complex tasks, including the holy grail that is AI research. Writing scientific papers? I'm already starting to use automated hypothesis discovery. Let's break down how the median academic paper actually gets written.

Somewhere in the universe, an academic feels publication pressure. They start looking at their connections, resources, and potential funding.

They start bouncing ideas and try to figure out how to write the highest impact paper they can with the lowest use of resources. Analysis of other peoples' data is very tempting at this point. AI can probably be used to search through public data for underutilized but useful datasets.

Funding applications are written. This part can be automated.

They start pulling threads, using their specialty knowledge to come up with the best novel hypotheses they can, and picks the one with the lowest p-value. No multiplicity testing, of course. Automated hypothesis discovery can help greatly here. A language model can probably also be used to estimate future citations for each hypothesis-paper.

Hypothesis + data -> set of core ideas that is then expanded into an abstract. I think current transformer models can already do this.

The paper is actually written. Fine-tuned models have published short papers already. A rough template, multiple generations, and a bit of human help can probably crank out a full-length paper without suspicion.

The paper sent to an impact-factor 3 journal, gets bounced a few times between their editor and the academic and is finally accepted after the recommended citations get added. Editing a paper according to a series of recommendations sounds like something that specialized transformer models are doing already.

Publication! Remember that the median paper has 0 citations.

Recommendations for follow-up experiments can also be obtained with AI analysis.

I fully anticipate an explosion of low-impact papers in the next few years, especially from China. With even worse English than normal. But how long until this starts improving? A bad paper is one with a boring/flawed hypothesis and bad results (p=0.050, after messing with outliers). The same data set and methods, but a more insightful hypothesis, can generate something a good (actually contributes to human knowledge, acts as a stepping stone for future researchers, etc.) paper. I have crawled down a rabbit hole of an interesting idea, realized it didn't pan out halfway through, and then had to keep working with it since I still had to finish that paper and it was too late to start over. In scientific...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing AI safety Mentors and Mentees, published by mariushobbhahn on November 23, 2022 on The Effective Altruism Forum.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conjecture Second Hiring Round, published by Connor Leahy on November 23, 2022 on The AI Alignment Forum.

Conjecture is hiring! We have open roles for all teams, both technical and non-technical. We have written a bit more about the teams at Conjecture here and you can view all open positions here.

Conjecture is an AI Safety startup that aims to solve the alignment problem.

At Conjecture, we focus on short timeline scenarios. We critically evaluate if the research we conduct actually cuts at alignment, and constantly seek to improve ourselves. We think the amount of work that needs to be done is immense, so we are working to build an organization capable of significantly scaling up and diversifying research directions.

We develop our own large models and research infrastructure. In order to maintain our ability to fund and conduct research at scale, we are also working on building AI driven products that we will soon be offering to a wider audience.

We are based in London, and are excited to be part of the growing alignment scene here. We’re in person by default, sponsor visas for all roles, and support employees’ relocation to London financially and logistically.

Nice views from the Conjecture office :)

For technical teams, the roles we’re most interested in filling are:

ML Engineering Lead

Security Lead

Research Engineer (Engineering Focus)

Research Engineer (Research Focus)

Product Engineer

We’re looking for candidates who can hit the ground running. Much of our research agenda at Conjecture involves working with the latest advancements in machine learning and working with large distributed computing clusters. While we expect driven people can skill up quickly in these areas, we will prioritize candidates who have backgrounds in high-performance computing, particularly those with a track record of owning projects end-to-end.

For non-technical teams, the roles we’re most interested in filling are:

Finance Lead

Strategy Analyst

Candidates don’t need to have prior experience with AI safety. More important is drive, professionalism, and a history of making real things happen in the world. While we have a vision for what these roles will look like, we are open-minded and are eager to hear from driven candidates who have concrete suggestions for actions/plans they’d like to execute alongside Conjecture.

For all roles, we’re looking for self-driven and flexible learners. As a startup, people’s responsibilities change quickly, and having the dexterity to quickly pivot to a new project and learn new skills/technologies is essential.

Our culture has a unique flavor. On our website we say some spicy things about hacker/pirate scrappiness, academic empiricism, and wild ambition. But there’s also a lot of memes, rock climbing, late-night karaoke, and insane philosophizing. Come join us!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conjecture: a retrospective after 8 months of work, published by Connor Leahy on November 23, 2022 on The AI Alignment Forum.

This post is a brief retrospective on the last 8 months at Conjecture that summarizes what we have done, our assessment of how useful this has been, and the updates we are making.

Intro

Conjecture formed in March 2022 with 3 founders and 5 early employees. We spent our first months growing the team, building infrastructure, exploring different research agendas, running Refine, publishing our internal infohazard policy, establishing an operational foundation for the business, and raising investments.

It’s been intense! For many of us at Conjecture, the last eight months have been the hardest we’ve worked in our lives. Working on such an immensely difficult problem as alignment alongside a team of brilliant and driven colleagues is, to say the least, galvanizing.

In some ways, this makes it difficult to step back and critically reflect on our work. It is easy to mistakenly measure progress by effort, and the last thing you want to hear after maxing out effort is that it wasn’t good enough.

However, reality does not grade on a curve. We need to advance significantly faster than traditional science in order to solve alignment on short timelines.

By this standard, the sober reflection is that most of our efforts to date have not made meaningful progress on the alignment problem. Our research has not revealed new methods that make neural networks more interpretable or resolve inner or outer alignment problems, and our coordination efforts have not slowed the pace at which AI capabilities are advancing compared to safety. When measured against p(Doom), our efforts haven’t cut it.

That’s not to say this work has been useless. We have learned a lot about where we went wrong, and made a number of changes that put us in a better position to make progress than we were in March. Measuring ourselves against a high standard enables us to constantly improve and be realistic about the difficulty of the problem ahead of us.

The reason we are writing this reflection is to calibrate ourselves. We do not want to be seen as cutting alignment if we are not. What matters is that we ground ourselves in reality and make public as many of our efforts (and mistakes!) as possible in order to gather feedback and update quickly.

What we have done and how useful we think it is

Infrastructure

We have built our own infrastructure to deploy large language models and do bespoke interpretability research. Our small engineering team has developed an impressive tech stack that is comparable (and in some areas exceeds) those built by many large industry research labs. While this has set us up to conduct research and develop tools/products more efficiently, it is only instrumental to alignment and not progress in-and-of-itself.

Interpretability

Our interpretability team explored a new direction in mechanistic interpretability in an effort to better understand polysemanticity in neural networks. The resulting paper identifies polytopes, rather than neurons, as a potentially fundamental unit of neural networks, and found that polysemanticity is reduced at the polytope level.

While the work brings a new perspective on neural network representations, a significant issue is that there are no clear implications of how to use this framework to better interpret neural networks. When measuring progress in interpretability, the clearest signal comes from new affordances–concrete things we can do differently now that we’ve made a research breakthrough. While there’s a chance that polytopes research may bring future affordances closer, the current, practical utility of polytopes is negligible. We also overinvested in iterating on feedback and polishing this project, and think we could have shipp...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing AI safety Mentors and Mentees, published by Marius Hobbhahn on November 23, 2022 on The AI Alignment Forum.

Executive summary

AI safety Mentors and Mentees is a program that matches mentors with mentees to accelerate their AI safety journey. Our aim is to get more people into full-time AI safety roles faster. We expect nearly all mentees to do this next to their current occupation, the program is not designed to be full-time.

The program is designed to be very flexible and lightweight, e.g. as long as the mentor and mentee agree on the terms, they can do pretty much whatever makes the most sense for them. The organizational overhead for each person is less than 10 minutes for the entirety of the program.

We’re trying to fill some of the niches that other programs can’t provide. We don’t expect the program to ever scale up to huge numbers. Whenever possible, we recommend doing cohort-based programs such as AGI safety fundamentals, SERI MATS, AI safety scholars, the AI safety camp, and the like.

We are looking for more mentors and mentees. You can fill out our interest forms here (mentors, mentees). Feel free to forward these forms to people you think might be a good fit.

The requirement for being a mentor is to have a decent overview of what’s happening in AI safety and be able to point people toward relevant articles, organizations and individual researchers. All of our mentors so far are either already working full-time in AI safety or very likely to be in the near future. We think early-stage professionals in AI safety (Ph.D. students, industry researchers/engineers, or independent contributors) are ideal candidates for mentors. Note, being able to consistently spare 30 minutes per week is sufficient to mentor one mentee, so you're time commitment doesn't have to be very big.

The requirement for being a mentee is to intend to work full-time in AI safety in the near future (e.g. ~2 years). There are no strict formal requirements but we strongly prefer candidates who could directly join the workforce or get independent funding to work on AI safety soon after finishing the M&M program. We expect to point many applicants to the AGI safety fundamentals program or other programs if there is no specific reason for 1-on-1 mentorship.

We have funds for people (both mentors and mentees) who really need them--so if you want to participate but don’t have the funds, we will be able to support you. By default, we suggest that the program is unpaid. In some cases, getting paid creates a better structure and more accountability--we are willing to take this into account as a reason to fund you but it should not be your primary reason.

For the last 5 months, we ran a pilot version of the program with 5 mentors and 10-15 mentees (some took breaks or stopped early). We found the following

On average, the mentors and mentees met 30-60 minutes every 1-4 weeks.

Most of our current mentees are either Master's or Ph.D. students in CS, ML or physics with a strong technical background who want to fast-track their understanding of AI safety.

Most of our current mentors are either already working full-time on AI safety or will likely be in the near future (e.g. currently on an independent research grant or finishing their Ph.D.).

Whenever mentees are unable to consistently commit at least 5 hours per week, the program is probably not worth it for them and their mentor.

In the pilot, the default was that most people got funding but we think that led to some mentees overstating (not with bad intentions, just planning fallacy) the time they were able to actually commit. Thus, the program will be on a voluntary basis with support funds when necessary (but with a low bar for what “necessary” means).

Brief overview

Motivation

There are many great programs to get people deeper into AI safe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Feeling Old: Leaving your 20s in the 2020s, published by squidious on November 22, 2022 on LessWrong.

A younger friend recently asked why I (and many others my age) make so many comments about being old. After some thought, I realized a big part of this was related to experiencing specific meaningful life stages during the pandemic.

In 2019, I could (and often did) stay up late at parties, survive on very little sleep, and run on my own willpower alone. I didn't usually need to think too hard about any of this in terms of trade offs, and thus had the ability to be way more spontaneous about it all. If a friend invited me out, I could say yes that night, with no planning ahead, and have a good time.

Then lockdown hit, and everything shut down. During this time, I got divorced, turned 30, and made a bunch of other huge life changes. In normal times, I probably would have started to see my stamina decline. I would have noticed at first that my body would hurt more the day after a night out. Then I'd start to notice myself losing energy earlier and earlier. But things didn't play out this way.

Instead, we were isolated, slowly watching culture change, with Gen Z gaining more control over social and pop culture as we learned to live online as they always had. Harry Potter wasn't cool anymore, side parts were cringe, and it became clear how out of touch we were.

When things opened up years later, many of us found ourselves suddenly unable to keep up with our past selves. It feels like going out until 3 one night, and the next night, needing to leave at midnight. Suddenly we were no longer growing, but aging.

So when I call myself old, I'm not saying that my best years are behind me, or that I feel I'm nearing the end of my life. I think what I actually mean (and should be saying) is that I feel emotional whiplash from moving out of young adulthood into regular adulthood. It feels like I was robbed of the transition between the two. I wish we could have peacefully handed the baton off to the next generation, rather than waking up one day and finding it gone.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Where are you donating this year, and why? (Open thread), published by Lizka on November 23, 2022 on The Effective Altruism Forum.

This post is intended as an open thread for anyone to share where you donated or plan to donate in 2022, and why.

I encourage you to share regardless of how small or large a donation you’re making! And you shouldn’t feel obliged to share the amount that you’re donating.

You can share as much or as little detail as you want (anything from 1 sentence simply describing where you’re giving, to multiple pages explaining your decision process and key considerations).

And if you have thoughts or feedback on someone else’s donation plans, I’d encourage you to share that in a reply to their “answer”, unless the person indicated they don’t want that. (But remember to be respectful and kind while doing this! See also supportive scepticism.)

Why commenting on this post might be useful:

You might get useful feedback on your donation plan

Readers might form better donation plans by learning about donation options you're considering, seeing your reasoning, etc.

Commenting or reading might help you/other people become or stay inspired to give (and to give effectively)

Related:

Effective Giving Day is coming up — November 28 — next week!

Talk about donations earlier and more

Previous posts of this kind:

Where are you donating in 2021, and why?

Where are you donating in 2020 and why?

Where are you donating this year and why – in 2019? Open thread for discussion.

As a final note: we’re enabling emoji reactions for this thread.

Adapted almost entirely from Where are you donating in 2020 and why?, with permission.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Insurance against FTX clawbacks, published by Grant Demaree on November 23, 2022 on The Effective Altruism Forum.

I've been reviewing funding requests as part of Nonlinear's effort to help FTXF grantees. Many applications sound like this:

FTX Future Fund already paid me, so I have plenty of money. I can't spend any of it, since I'm worried about clawbacks. This leaves me practically destitute.

Imagine you could buy insurance against these clawbacks:

Amy has $20,000 in her bank account, but it's all from an FTX grant

Bill owns an insurance company. He believes the chance of a successful clawback is much less than 25%, and he charges Amy $5,000 for the insurance

Now Amy is comfortable spending her remaining $15,000. If a clawback happens, Bill pays her $20,000, and she's unharmed

Seems like a win-win for Amy and Bill. Is there a way to make it happen?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Human-level Full-Press Diplomacy (some bare facts)., published by strawberry calm on November 22, 2022 on LessWrong.

Key links?

Here's the paper:

Here's the blog post:/

Here's the website:/

What is Diplomacy?

Diplomacy is a negotiation-based board game. There are 7 players corresponding to the 7 major European powers during WW1. In each round, the players write down their moves, which are then executed simultaneously. The moves might be attacking/defending territory, supporting/opposing another player's action, etc.

Unlike Risk, there are no dice or other random mechanism.

In "Full Press Diplomacy", players can also hold private conversations between rounds. The dialogue is used to establish trust and coordinate actions with other players. Players can make agreements, but all these agreements are non-binding.

In "No Press Diplomacy", players can't communicate.

Because the moves are simultaneous, game-play requires recursive theory of mind. The players must reason about the other players, and reason about what others are reasoning about them, and so on.

Diplomacy is notorious for ending friendships, probably because it erodes trust. Anecdotally, I've played Diplomacy twice in my life. Each game lasted at least 6 hours. The first game was played with slightly LessWrong-ish friends and wasn't friendship-destroying. The second was played with un-LessWrong-ish friends and it did cause outside-the-game distrust.

Diplomacy was the favourite game of John F. Kennedy and Henry Kissinger.

How well did CICERO perform?

On Nov 22 (today!) Meta AI presented an AI which achieved human-level performance at Full Press Diplomacy.

CICERO achieved more than double the average score of the other players and ranked in the top 10% of players.

The evaluation consisted of anonymous blitz Diplomacy games against humans on webDiplomacy.net

40 games

82 human players

5,277 messages

72 hours of gameplay

Here is commentary by a Diplomacy expert playing against six CICEROs:

"CICERO is so effective at using natural language to negotiate with people in Diplomacy that they often favoured working with CICERO over other human participants." (blog)

They are open-sourcing the code and models, and "interested researchers can submit a proposal to the CICERO RFP to gain access to the data." (?!!)

How does CICERO behave?

In each game, the other players were convinced this was a human player.

"What impresses me most about CICERO is its ability to communicate with empathy and build rapport while also tying that back to its strategic objectives." — Andrew Goff (3x Diplomacy World Champion)

"CICERO is ruthless. It's resilient. And it's patient. [...] CICERO's dialogue is direct, but it has some empathy. It's surprisingly human." — Andrew Goff

Apparently, CICERO is more honest than most human players. Andrew Goff also suggests that CICERO learned to become more honest over time, which is the same learning curve for high-level human players.

CICERO (in green) de-escalates with another player by reassuring them it will not attack them.

CICERO (in blue) suggests mutually beneficial moves a human missed.

CICERO can both coordinate and negotiate.

Sometimes CICERO generates inconsistent dialogue and makes mistakes. For example, CICERO (below) contradicts its first message asking Italy to move to Venice.

How does CICERO work?

CICERO has two components: a strategy engine and a dialogue engine.

The strategy engine constructs a plan which works for itself and the other players and then the dialogue engine persuades the other players of the plan.

"To build a controllable dialogue model, we started with a 2.7 billion parameter BART-like language model pre-trained on text from the internet and fine-tuned on over 40,000 human games on webDiplomacy.net." (blog)

They automatically labelled the messages in this training set with th...