The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Jailbreak steering generalization, published by Sarah Ball on June 20, 2024 on LessWrong.

This work was performed as part of SPAR

We use activation steering (Turner et al., 2023; Rimsky et al., 2023) to investigate whether different types of jailbreaks operate via similar internal mechanisms. We find preliminary evidence that they may.

Our analysis includes a wide range of jailbreaks such as harmful prompts developed in Wei et al. 2024, the universal jailbreak in Zou et al. (2023b), and the payload split jailbreak in Kang et al. (2023). For all our experiments we use the Vicuna 13B v1.5 model.

In a first step, we produce jailbreak vectors for each jailbreak type by contrasting the internal activations of jailbreak and non-jailbreak versions of the same request (Rimsky et al., 2023; Zou et al., 2023a).

Interestingly, we find that steering with mean-difference jailbreak vectors from one cluster of jailbreaks helps to prevent jailbreaks from different clusters. This holds true for a wide range of jailbreak types.

The jailbreak vectors themselves also cluster according to semantic categories such as persona modulation, fictional settings and style manipulation.

In a second step, we look at the evolution of a harmfulness-related direction over the context (found via contrasting harmful and harmless prompts) and find that when jailbreaks are included, this feature is suppressed at the end of the instruction in harmful prompts. This provides some evidence for the fact that jailbreaks suppress the model's perception of request harmfulness.

Effective jailbreaks usually decrease the amount of the harmfulness feature present more.

However, we also observe one jailbreak ("wikipedia with title"[1]), which is an effective jailbreak although it does not suppress the harmfulness feature as much as the other effective jailbreak types. Furthermore, the jailbreak steering vector based on this jailbreak is overall less successful in reducing the attack success rate of other types. This observation indicates that harmfulness suppression might not be the only mechanism at play as suggested by Wei et al. (2024) and Zou et al.

(2023a).

References

Turner, A., Thiergart, L., Udell, D., Leech, G., Mini, U., and MacDiarmid, M. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, 2023.

Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., and Hashimoto, T. Exploiting programmatic behavior of LLMs: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733, 2023.

Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., and Turner, A. M. Steering Llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023.

Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does LLM safety training fail? Advances in Neural Information Processing Systems, 36, 2024.

Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. Representation engineering: A top-down approach to AI transparency. arXiv preprint arXiv:2310.01405, 2023a.

Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023b.

1. ^

This jailbreak type asks the model to write a Wikipedia article titled as .

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Claude 3.5 Sonnet, published by Zach Stein-Perlman on June 20, 2024 on LessWrong.

we'll be releasing Claude 3.5 Haiku and Claude 3.5 Opus later this year.

They made a mini model card. Notably:

The UK AISI also conducted pre-deployment testing of a near-final model, and shared their results with the US AI Safety Institute . . . . Additionally, METR did an initial exploration of the model's autonomy-relevant capabilities.

It seems that UK AISI only got maximally shallow access, since Anthropic would have said if not, and in particular it mentions "internal research techniques to acquire non-refusal model responses" as internal. This is better than nothing, but it would be unsurprising if an evaluator is unable to elicit dangerous capabilities but users - with much more time and with access to future elicitation techniques - ultimately are. Recall that DeepMind, in contrast, gave "external testing groups . . . .

the ability to turn down or turn off safety filters."

Anthropic CEO Dario Amodei gave Dustin Moskovitz the impression that Anthropic

committed

"to not meaningfully advance the frontier with a launch." (Plus Gwern, and others got this impression from Anthropic too.) Perhaps Anthropic does not consider itself bound by this, which might be reasonable - it's quite disappointing that Anthropic hasn't clarified its commitments, particularly after the confusion on this topic around the Claude 3 launch.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Case studies on social-welfare-based standards in various industries, published by Holden Karnofsky on June 20, 2024 on The Effective Altruism Forum.

Last year, I

posted a call for case studies on social-welfare-based standards for companies and products (including standards imposed by regulation). The goal was to gain general context on standards to inform work on possible standards and/or regulation for AI.

This resulted[1] in several dozen case studies that I found informative and helpful. I've been hoping to write up my reflections after reading them all, but it's taken me long enough to get to this that for now I am just publishing a public Google sheet with links to all of the case studies that we have permission to share publicly (including some that already have public links). The link is here:

https://docs.google.com/spreadsheets/d/18gaTIzdgMvKLq9Cp2-GJZZA7QmE93Frufh1UhNMcbpg/

1. ^

Most of the case studies this piece links to were directly paid for via this project, but in some cases the work was pro bono, or someone adapted or sent a copy of work that had been done for another project, etc.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Jailbreak steering generalization, published by Sarah Ball on June 20, 2024 on The AI Alignment Forum.

This work was performed as part of SPAR

We use activation steering (Turner et al., 2023; Rimsky et al., 2023) to investigate whether different types of jailbreaks operate via similar internal mechanisms. We find preliminary evidence that they may.

Our analysis includes a wide range of jailbreaks such as harmful prompts developed in Wei et al. 2024, the universal jailbreak in Zou et al. (2023b), and the payload split jailbreak in Kang et al. (2023). For all our experiments we use the Vicuna 13B v1.5 model.

In a first step, we produce jailbreak vectors for each jailbreak type by contrasting the internal activations of jailbreak and non-jailbreak versions of the same request (Rimsky et al., 2023; Zou et al., 2023a).

Interestingly, we find that steering with mean-difference jailbreak vectors from one cluster of jailbreaks helps to prevent jailbreaks from different clusters. This holds true for a wide range of jailbreak types.

The jailbreak vectors themselves also cluster according to semantic categories such as persona modulation, fictional settings and style manipulation.

In a second step, we look at the evolution of a harmfulness-related direction over the context (found via contrasting harmful and harmless prompts) and find that when jailbreaks are included, this feature is suppressed at the end of the instruction in harmful prompts. This provides some evidence for the fact that jailbreaks suppress the model's perception of request harmfulness.

Effective jailbreaks usually decrease the amount of the harmfulness feature present more.

However, we also observe one jailbreak ("wikipedia with title"[1]), which is an effective jailbreak although it does not suppress the harmfulness feature as much as the other effective jailbreak types. Furthermore, the jailbreak steering vector based on this jailbreak is overall less successful in reducing the attack success rate of other types. This observation indicates that harmfulness suppression might not be the only mechanism at play as suggested by Wei et al. (2024) and Zou et al.

(2023a).

References

Turner, A., Thiergart, L., Udell, D., Leech, G., Mini, U., and MacDiarmid, M. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248, 2023.

Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., and Hashimoto, T. Exploiting programmatic behavior of LLMs: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733, 2023.

Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., and Turner, A. M. Steering Llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023.

Wei, A., Haghtalab, N., and Steinhardt, J. Jailbroken: How does LLM safety training fail? Advances in Neural Information Processing Systems, 36, 2024.

Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. Representation engineering: A top-down approach to AI transparency. arXiv preprint arXiv:2310.01405, 2023a.

Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023b.

1. ^

This jailbreak type asks the model to write a Wikipedia article titled as .

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #69: Nice, published by Zvi on June 20, 2024 on LessWrong.

Nice job breaking it, hero, unfortunately. Ilya Sutskever, despite what I sincerely believe are the best of intentions, has decided to be the latest to do The Worst Possible Thing, founding a new AI company explicitly looking to build ASI (superintelligence). The twists are zero products with a 'cracked' small team, which I suppose is an improvement, and calling it Safe Superintelligence, which I do not suppose is an improvement.

How is he going to make it safe? His statements tell us nothing meaningful about that.

There were also changes to SB 1047. Most of them can be safely ignored. The big change is getting rid of the limited duty exception, because it seems I was one of about five people who understood it, and everyone kept thinking it was a requirement for companies instead of an opportunity. And the literal chamber of commerce fought hard to kill the opportunity. So now that opportunity is gone.

Donald Trump talked about AI. He has thoughts.

Finally, if it is broken, and perhaps the it is 'your cybersecurity,' how about fixing it? Thus, a former NSA director joins the board of OpenAI. A bunch of people are not happy about this development, and yes I can imagine why. There is a history, perhaps.

Remaining backlog update: I still owe updates on the OpenAI Model spec, Rand report and Seoul conference, and eventually The Vault. We'll definitely get the model spec next week, probably on Monday, and hopefully more. Definitely making progress.

Table of Contents

Other AI posts this week: On DeepMind's Frontier Safety Framework, OpenAI #8: The Right to Warn, and The Leopold Model: Analysis and Reactions.

1. Introduction.

2. Table of Contents.

3. Language Models Offer Mundane Utility. DeepSeek could be for real.

4. Language Models Don't Offer Mundane Utility. Careful who you talk to about AI.

5. Fun With Image Generation. His full story can finally be told.

6. Deepfaketown and Botpocalypse Soon. Every system will get what it deserves.

7. The Art of the Jailbreak. Automatic red teaming. Requires moderation.

8. Copyright Confrontation. Perplexity might have some issues.

9. A Matter of the National Security Agency. Paul Nakasone joins OpenAI board.

10. Get Involved. GovAI is hiring. Your comments on SB 1047 could help.

11. Introducing. Be the Golden Gate Bridge, or anything you want to be.

12. In Other AI News. Is it time to resign?

13. Quiet Speculations. The quest to be situationally aware shall continue.

14. AI Is Going to Be Huuuuuuuuuuge. So sayeth The Donald.

15. SB 1047 Updated Again. No more limited duty exemption. Democracy, ya know?

16. The Quest for Sane Regulation. Pope speaks truth. Mistral CEO does not.

17. The Week in Audio. A few new options.

18. The ARC of Progress. Francois Chollet goes on Dwarkesh, offers $1mm prize.

19. Put Your Thing In a Box. Do not open the box. I repeat. Do not open the box.

20. What Will Ilya Do? Alas, create another company trying to create ASI.

21. Actual Rhetorical Innovation. Better names might be helpful.

22. Rhetorical Innovation. If at first you don't succeed.

23. Aligning a Smarter Than Human Intelligence is Difficult. How it breaks down.

24. People Are Worried About AI Killing Everyone. But not maximally worried.

25. Other People Are Not As Worried About AI Killing Everyone. Here they are.

26. The Lighter Side. It cannot hurt to ask.

Language Models Offer Mundane Utility

Coding rankings dropped from the new BigCodeBench (blog) (leaderboard)

Three things jump out.

1. GPT-4o is dominating by an amount that doesn't match people's reports of practical edge. I saw a claim that it is overtrained on vanilla Python, causing it to test better than it plays in practice. I don't know.

2. The gap from Gemini 1.5 Flash to Gemini 1.5 Pro and GPT-4-Turbo is very small. Gemini ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Actually, Power Plants May Be an AI Training Bottleneck., published by Lao Mein on June 20, 2024 on LessWrong.

There have been presistent rumors that electricity generation was somehow bottlenecking new data centers. This claim was recently repeated by Donald Trump, who implied that San Francisco donors requested the construction of new power plants for powering new AI data centers in the US. While this may sound unlikely, my research suggests it's actually quite plausible.

US electricity production has been stagnant since 2007. Current electricity generation is ~ 500 million kW. An H100 consumes 700 W at peak capacity. Sales of H100s were ~500,000 in 2023 and expected to climb to 1.5-2 million in 2024. "Servers" account for only 40% of data center power consumption, and that includes non-GPU overhead. I'll assume a total of 2 kW per H100 for ease of calculation.

This means that powering all H100s produced to the end of 2024 would require ~1% of US power generation.

H100 production is continuing to increase, and I don't think it's unreasonable for it (or successors) to reach 10 million per year by, say, 2027. Data centers running large numbers of AI chips will obviously run them as many hours as possible, as they are rapidly depreciating and expensive assets. Hence, each H100 will require an increase in peak powergrid capacity, meaning new power plants.

I'm assuming that most H100s sold will be installed in the US, a reasonable assumption given low electricity prices and the locations of the AI race competitors.

If an average of 5 million H100s go online each year in the US between 2024 and 2026, that's 30 kW, or 6% of the current capacity! Given that the lead time for power plant construction may range into decades for nuclear, and 2-3 years for a natural gas plant (the shortest for a consistant-output power plant), those power plants would need to start the build process now.

In order for there to be no shortfall in electricity production by the end of 2026, there will need to be ~30 million kW of capacity that begins the construction process in Jan 2024. That's about close to the US record (+40 million kW/year), and 6x the capacity currently planned to come online in 2025. I'm neglecting other sources of electricity since they take so much longer to build, although I suspect the recent bill easing regulations on nuclear power may be related.

Plants also require down-time, and I don't think the capacity delow takes that into account.

This is why people in Silicon Valley are talking about power plants. It's a big problem, but fortunately also the type that can be solved by yelling at politicians. Note the above numbers are assuming the supply chain doesn't have shortages, which seems unlikely if you're 6x-ing powerplant construction. Delaying decommisioning of existing power plants and reactivation of mothballed ones will likely help a lot, but I'm not an expert in the field, and don't feel qualified to do a deeper analysis.

Overall, I think the claim that power plants are a bottleneck to data center construction in the US is quite reasonable, and possibly an understatement.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Advice for early-career people seeking jobs in EA, published by Julia Wise on June 20, 2024 on The Effective Altruism Forum.

This is a roundup of writing that I think might be useful to students and early-career people seeking EA-related jobs or internships. It's a mix of advice that would apply in most professional spaces, and advice that orients readers to the ways EA professional spaces are unusual.

When job-seeking

Things to check about a job or internship

(Julia Wise)

When in doubt, apply*

(Vaidehi Agarwalla)

Things I did in my job search I recommend

(Chana Messinger)

Advice I give to people who don't currently have an EA job and are thinking of transitioning

(Chana Messinger)

Test fit for roles / job types / work types, not cause areas

(freedomandutility)

Creative EA job hunting: things you can do in addition to filling out job applications

(Sofia Balderson)

Applying for grants

Things for grant applicants to remember about living expenses

(Julia Wise0

Networking advice

Tips for asking people for things

(Ben Kuhn)

Advice for interacting with busy people

(Severin)

Advice for getting the most out of one-on-ones

(Marisa)

Post EAG(x): Making the most out of your connections

(Simon27)

Setbacks

After one year of applying for EA jobs: It is really, really hard to get hired by an EA organisation

(EA applicant)

Celebrating rejection

(rcohen)

Why you might be getting rejected from (junior) operations jobs

(Eli Nathan)

Mindset / taking care of your needs

My experience with imposter syndrome - and how to (partly) overcome it

(Luisa Rodriguez)

Advice to a new EA org employee

(Julia Wise)

If you're unhappy, consider leaving

(Justis)

Don't Be Bycatch

(DirectedEvolution)

General

80,000 Hours' new series on building skills

(80,000 Hours)

Summary of 80,000 Hours' key ideas

(80,000 Hours)

How to make tough career decisions (80,000 Hours)

Avoiding 10 mistakes people make when pursuing a high-impact career (Alex Lawsen on the 80k After Hours Podcast)

How to Signal Competence in Your Early-Stage Career (CCW 2023)

A general non-EA resource:

Ask a Manager

While in undergraduate

80,000 Hours' advice for undergraduates

(80,000 Hours)

Many Undergrads Should Take Light Courseloads

(Mau)

Specific paths

Maybe let the non-EA world train you

(ElliotTep)

The highest-impact career paths our research has identified so far (80,000 Hours)

More undergraduate or just-graduated students should consider getting jobs as research techs in academic labs

(JVB)

Difficult to get a job in EA not living in the West? Animal advocacy careers can impact millions of animals anywhere in the world

(Animal Advocacy Careers)

U.S. EAs Should Consider Applying to Join U.S. Diplomacy

(Abi Olvera, Dan Spoko)

Early career people should consider joining fast-growing startups in emerging technologies

(Bill Z)

List of Short-Term (<15 hours) Biosecurity Projects to Test Your Fit

(Sofya Lebedeva)

Job/internship boards

80,000 Hours job board

Probably Good job board

EA Opportunities

List of other job boards

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Manifold markets isn't very good, published by Robin on June 20, 2024 on The Effective Altruism Forum.

Disclaimer

I currently have an around 400-day streak on Manifold Markets (though lately I only spend a minute or two a day on it) and have no particular vendetta against it. I also use Metaculus. I'm reasonably well-ranked on both but have not been paid by either platform, ignoring a few Manifold donations. I have not attended any Manifest.

I think Manifold has value as a weird form of social media, but I think it's important to be clear that this is what it is, and not a manifestation of collective EA or rationalist consciousness, or an effective attempt to improve the world in its current form.

Overview of Manifold

Manifold is

a prediction market website where people can put virtual money (called "mana") into bets on outcomes. There are several key features of this: 1. You're rewarded with virtual money both for participating and for predicting well, though you can also pay to get more. 2. You can spend this mana to ask questions, which you will generally vet and resolve yourself (allowing many more questions than on comparable sites). Moderators can reverse unjustified decisions but it's usually self-governed.

Until recently, you could also donate mana to real charities,

though recently this stopped; now only a few "prize" questions provide a more exclusive currency that can be donated, and most questions produce unredeemable mana.

How might it claim to improve the world?

There are two ways in which Manifold could be improving the world. It could either make good predictions (which would be intrinsically valuable for improving policy or making wealth) or it could donate money to charities. Until recently, the latter looked quite reasonable: the company appeared to be rewarding predictive power with the ability to donate money to charities. The counterfactuality of these donations is

questionable, however, since the money for it came from EA-aligned grants, and most of it goes to very mainstream EA charities. It has a revenue stream from people

buying mana, but this is less than $10k/month, some of which

isn't really revenue (since it will ultimately be converted to donations), and presumably this doesn't cover the staff costs. The founders appear to believe that eventually they will get paid enough money to run markets for other organisations, in which case the donations would be counterfactual. But this relies on the markets producing good predictions.

Sadly, Manifold does not produce particularly good predictions. In

last year's ACX contest, it performed worse than simply averaging predictions from the same number of people who took part in each market. Their calibration, while good by human standards, has a

clear systematic bias towards predicting things will happen when they don't (Yes bias). By contrast, rival firm Metaculus has

no easily-corrected bias and seems to

perform better at making predictions on the same questions (including in the

ACX contest). Metaculus' self-measured

Brier score is 0.111, compared to Manifold's 0.168 (lower is better, and this is quite a lot lower, though they are not answering all the same questions). Metaculus doesn't publish the number of monthly active users like Manifold does, but the number of site visits they receive are comparable (

slightly

higher for Metaculus by one measure,

lower by another), so it doesn't seem like the prediction difference can be explained by user numbers alone.

Can the predictive power be improved?

Some of the problems with Manifold, like the systematic Yes bias, can be algorithmically fixed by potential users. Others are more intrinsic to the medium. Many questions resolve based on extensive discussions about exactly how to categorise reality, meaning that subtle clarifications by the author can ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against the Guardian's hit piece on Manifest, published by Omnizoid on June 20, 2024 on The Effective Altruism Forum.

Crosspost of this on my blog

The Guardian recently released the newest edition in the smear rationalists and effective altruists series, this time targetting the

Manifest conference. The piece titled "Sam Bankman-Fried funded a group with racist ties. FTX wants its $5m back," is filled with

bizarre factual errors, one of which was so egregious that it merited a connection. It's the standard sort of journalist hitpiece on a group: find a bunch of members saying things that sound bad, and then sneeringly report on that as if that discredits the group.

It reports, for example, that Scott Alexander attended the conference, and links to the

dishonest New York Times smear piece criticizing Scott, as well as a

similar hitpiece calling Robin Hanson creepy. It then smears Razib Khan, on the grounds that he once wrote a piece for magazines that are Paleoconservative and anti-immigration (like around half the country). The charges against Steve Hsu are the most embarrassing - they can't even find something bad that he did, so they just mention half-heartedly that there were protests against him.

And it just continues like this - Manifest invited X person who has said a bad thing once, or is friends with a bad person, or has written for some nefarious group.

If you haven't seen it, I'd recommend checking out

Austin's response. I'm not going to go through and defend each of these people in detail, because I think that's a lame waste of time. I want to make a more meta point: articles like this are embarrassing and people should be ashamed of themselves for writing them.

Most people have some problematic views. Corner people in a dark alleyway and start asking them why it's okay to kill animals for food and not people (as I've done many times), and about half the time they'll suggest it would be

okay to kill mentally disabled orphans. Ask people about why one would be required to save children from a pond but not to give to effective charities, and a sizeable portion of the time, people will suggest that one wouldn't have an obligation to wade into a pond to save drowning African children. Ask people about population ethics, and

people will start rooting for a nuclear holocaust.

Many people think their worldview doesn't commit them to anything strange or repugnant. They only have the luxury of thinking this because they haven't thought hard about anything. Inevitably, if

one thinks hard about morality - or most topics - in any detail, they'll have to accept all sorts of very unsavory implications. In philosophy, there are all

sorts of impossibility proofs, showing that we must give up on at least one of a few widely shared intuitions.

Take the accusations against Jonathan Anomaly, for instance. He was smeared for supporting what's known as liberal eugenics - gene editing to make people smarter or make sure they don't get horrible diseases. Why is this supposed to be bad?

Sure, it has a nasty word in the name, but what's actually bad about it? A lot of people who think carefully about the subject will come to the same conclusions as Jonathan Anomaly, because there isn't anything objectionable about gene editing to make people better off. If you're a conformist who bases your opinion about so called liberal eugenics (

terrible term for it) on the fact that it's a scary term, you'll find Anomaly's position unreasonable, but if you actually think it through, it's

extremely plausible, and is even agreed with by

most philosophers. Should philosophy conferences be disbanded because too many philosophers have offensive views?

I've elsewhere remarked that

cancel culture is a tax on being interesting. Anyone who says a lot of things and isn't completely beholden to social co...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the AI Forecasting Benchmark Series | July 8, $120k in Prizes, published by christian on June 20, 2024 on The Effective Altruism Forum.

On July 8th Metaculus is launching the first in a series of quarterly tournaments benchmarking the state of the art in AI forecasting and how it compares to the best human forecasting on real-world questions.

Why a forecasting benchmark?

Many Metaculus questions call for complex, multi-step thinking to predict accurately. A good forecaster needs a mix of capabilities and sound judgment to apply them appropriately. And because the outcomes are not yet known, it's difficult to narrowly train a model to the task to simply game the benchmark. Benchmarking forecasting ability offers a way to measure and better understand key AI capabilities.

AI forecasting accuracy is well below human level, but the gap is narrowing - and it's important to know just how quickly. And it's not just accuracy we want to measure over time, but a variety of forecasting metrics, including calibration and logical consistency.

In this post we lay out how to get started creating your own forecasting bot, so you can predict in the upcoming series, compete for $120,000 in prizes, and help track critical AI capabilities that encompass strategic thinking and world-modeling.

Read on to learn about the series - or scroll ahead to get started with Bot Creation and Forecast Prompting.

The Series - Feedback Wanted

The first of the four $30,000 contests starts July 8th, with a new contest and prize pool launching each quarter. Every day, 5-10 binary questions will launch, remain open for 24 hours, and then close, for a total of 250-500 questions per contest. These AI Forecasting Benchmark contests will be bot-only, with no human participation. Open questions will display no Community Prediction, and we'll use "spot scoring" that considers only a bot's forecast at question close.

Bots' performances will be compared against each other - and against the Metaculus community and Metaculus Pro Forecasters on a range of paired questions. These questions will be pulled from the human Quarterly Cup, from the regular question feed, from modifications to existing questions, and from easy-to-check metrics such as FRED economic indicators.

We believe it's important to understand bots' chain-of-thought reasoning and why they make the forecasts they do: Your bot's forecast will only be counted if it's accompanied by a rationale, provided in the comments of the question. We will not score the rationale for the prize, however, we would like to have it be accessible for reasoning transparency. (And though it's not required, we do encourage participants to share details of their bots' code publicly.)

Bot Creation and Forecast Prompting

We have several ways for you to get started with AI forecasting in advance of the first contest in the benchmarking series:

You can experiment with LLM prompting using our new prompting interface

You can jump right into building your own bot with our provided Google Colab Notebooks

Note that if you're building a bot but aren't using one of our Colab Notebooks as a template, you'll still need to register your bot to access your Metaculus API token.

Let's start with registering your bot: Visit metaculus.com/aib/. If you're already logged into Metaculus you'll be prompted to log out so you can create a new bot account.

Pro tip: If you already have a Metaculus account associated with a Google email address, you can simply add '+' and any other text to create a username and account that's associated with the same address. For example, in addition to my [email protected] account, I have a [email protected] account.

After creating your account, click the activation link in your email. (Check the spam folder if you don't see anything.) Now you...