Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: UK government to host first global summit on AI Safety, published by DavidNash on June 8, 2023 on The Effective Altruism Forum.

An exciting development in this space, it may be worth people thinking about what to organise around this summit. I've copied the press release in full below.

Summit will bring together key countries, leading tech companies and researchers to agree safety measures to evaluate and monitor the most significant risks from AI

PM and President Biden will take a coordinated approach to the opportunities and challenges of emerging tech when they meet at the White House today

Global companies are expanding their AI work in the UK, as PM confirms new university scholarships to further UK-US tech leadership

As the world grapples with the challenges and opportunities presented by the rapid advancement of Artificial Intelligence, the UK will host the first major global summit on AI safety, the Prime Minister has announced today (Wednesday 7 June).

Breakthroughs from AI continue to improve our lives – from enabling paralysed people to walk to discovering superbug-killing antibiotics. But the development of AI is extraordinarily fast moving and this pace of change requires agile leadership. That is why the UK is taking action, because we have a global duty to ensure this technology is developed and adopted safely and responsibly.

Last week dozens of leading experts warned about the potential for AI to endanger humanity in similar ways to pandemics or nuclear weapons.

In Washington DC today, the Prime Minister will stress the importance of likeminded allies and companies working to develop an international framework to ensure the safe and reliable development and use of AI.

The summit, which will be hosted in the UK this autumn, will consider the risks of AI, including frontier systems, and discuss how they can be mitigated through internationally coordinated action. It will also provide a platform for countries to work together on further developing a shared approach to mitigate these risks.

In recent weeks the Prime Minister has discussed this issue with a number of businesspeople and world leaders. This includes all members of the G7 who were united in their ambition to take a shared approach to this issue at the Hiroshima Summit last month.

In May the PM also met the CEOs of the three most advanced frontier AI labs, OpenAI, DeepMind and Anthropic in Downing Street and the Secretary of State for Science, Innovation and Technology also hosted a roundtable with senior AI leaders. The work at the AI safety summit will build on recent discussions at the G7, OECD and Global Partnership on AI.

In July the Foreign Secretary will also convene the first ever briefing of the UN Security Council on the opportunities and risks of Artificial Intelligence for international peace and security.

The UK is well-placed to convene discussions on the future of AI. The UK is a world-leader in AI – ranking third behind the US and China. Our AI sector already contributes £3.7 billion to the UK economy and employs 50,000 people across the country.

Our departure from the EU also allows us to act more quickly and agilely in response to this rapidly changing market. The UK was one of the first leading nations to set out a blueprint for the safe and responsible development of AI, which will be adaptive to the speed of advances in this technology. And the UK has launched an expert taskforce to help build and adopt the next generation of safe AI, backed by £100 million of funding, alongside a commitment to spend £900 million developing compute capacity, including an exascale supercomputer in the UK.

The Prime Minister said:

AI has an incredible potential to transform our lives for the better. But we need to make sure it is developed and used in a way that is safe and secure.

Time and...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #15: The Principle of Charity, published by Zvi on June 8, 2023 on LessWrong.

The sky is not blue. Not today, not in New York City. At least it’s now mostly white, yesterday it was orange. Even indoors, everyone is coughing and our heads don’t feel right. I can’t think fully straight. Life comes at you fast.

Thus, I’m going with what I have, and then mostly taking time off until this clears up. Hopefully that won’t be more than a few more days.

The Principle of Charity comes into play this week because of two posts, by people I greatly respect as thinkers and trust to want good things for the world, making arguments that are remarkably terrible. I wrote detailed responses to the arguments within, then realized that was completely missing the point, and deleted them. Instead, next week I plan to explain my model of what is going on there – I wish they’d stop doing what they are doing, and think they would be wise to stop doing it, but to the extent I am right about what is causing these outputs, I truly sympathize.

For a day we were all talking about a Vice story that sounded too good (or rather, too perfect) to be true, and then it turned out that it was indeed totally made up. Time to take stock of our epistemic procedures and do better next time.

Table of Contents

Introduction

Table of Contents

Language Models Offer Mundane Utility. Quite a lot, actually.

Language Models Don’t Offer Mundane Utility. Not with that attitude.

Deepfaketown and Botpocalypse Soon. Talk to your. parents?

Fun with Image Generation. Investors, man.

Vigilance. It must be eternal, until it won’t be enough.

Introducing. Falcon the open source model, Lightspeed Grants and more.

In Other AI News. Senator asks Meta the question we all ask: Llama?

They Took Our Jobs. First they came for the copywriters.

Out of the Box Thinking. A hard to detect attack on audio activation devices.

I Was Promised Driverless Cars. You’ll get them. Eventually.

If It Sounds Too Good to be True. Guess what?

Quiet Speculations. Algorithmic improvements take center stage for a bit.

A Very Good Sentence. Including a claim it was a very bad sentence.

The Quest for Sane Regulation. Devil is in the details. UK PM steps up.

The Week in Podcasts and Other Audio and Video. Don’t look.

What Exactly is Alignment? It’s a tricky word.

Rhetorical Innovation. Several good ideas.

Have You Tried Not Doing Things That Might Kill Everyone? It’s a thought.

People Who Are Worried People Aren’t Reasoning From World Models.

People Are Worried About AI Killing Everyone. But let’s keep our cool, shall we?

Other People Are Not Worried About AI Killing Everyone. Because of reasons.

The Wit and Wisdom of Sam Altman. The little things are yours forever.

The Lighter Side. Sorry about all that.

Language Models Offer Mundane Utility

Claim that you can get GPT-4 or GPT-3.5 into some sort of ‘logic mode’ where it can play chess ‘better than the old stockfish 8.’

Shako reports crazy amounts of GPT-4 mundane utility , similar to Google before it.

I remember using google circa 2000. Basically as soon as I used it, I began to use it for everything. It was this indexed tool that opened up entire worlds to me. I could learn new things, discover communities, answer questions.

But even as the years went by, other than some nerds or particularly smart people, people barely used it? They didn’t integrate it into their workflow. It was baffling to me.

I remember a single instance where I asked a dr a question, and he googled it in front of me and searched some sources, and being impressed that he was thoughtful enough to use google to improve his problem solving on-the-fly.

In some sense my entire career was built on google. I’m self taught in most things I know, and while I have learned from a lot of sources, the google index tied them all together and let me ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Seeking important GH or IDEV working papers to evaluate, published by ryancbriggs on June 8, 2023 on The Effective Altruism Forum.

I am a Global Health and Development editor at The Unjournal. In brief, our mission is to organize and fund public, journal-independent evaluation and rating of working papers. Right now we are focusing on quantitative work informing global priorities, and my area focuses on global health and international development.

I am writing because I want people to send me working papers (pre-publication) that are decision-relevant to organizations working in global health and international development. Anyone can submit paper suggestions to [email protected]. If you prefer, authors can submit work through our quick submission form HERE, and others can suggest work using this form.

I think two especially useful cases are:

Authors sending their own papers so that they can get high quality feedback before submission. This feedback will be public, but we are open to discussing delaying or embargoing feedback on a case-by-case basis.

Grant-making staff in orgs like GiveWell or OpenPhil sending in papers that are relevant to their work so that they can read expert reviews before they use the results to inform decisions.

The rest of this post explains in more detail: what The Unjournal is, our review process, why we’re doing this, what we’re looking for in a paper, and who can submit.

The Unjournal

The Unjournal (evaluations are here) is the brainchild of David Reinstein. Its mission is to offer clear, public pre-publication expert review of working papers. There are two motivations behind this project. The first is that researchers can do better work if they are able to more easily get high quality feedback on their work. The second is that the entire research ecosystem can benefit from these reviews being public. We pay expert reviewers for their time and we give awards for especially high quality research that we evaluate.

The initial focus of The Unjournal is on quantitative work that informs global priorities, especially in economics, policy, and social science. I’m an editor for global health and international development.

The process

While our process runs similarly to an academic journal, we do not publish articles. Instead, we link to working papers and publish reviews (which we call 'evaluations') and author responses.

The process starts with editors screening lists of working papers created from submissions or based on our knowledge of the research area. Once a working paper is selected, we contact the authors to ask if they would like to be involved in the process. Author involvement allows authors to converse with reviews and/or respond to reviews. It is not mandatory, but we prefer if authors are involved.

We then aim to find two reviewers per paper. Reviewers are experts in the research area and are paid for their reviews. Reviewers have the choice to remain anonymous or not. They read the work and fill out standardized review forms.

We then publish the reviews of the work on our webpage (see here) alongside brief comments from the editor and responses from the authors if they so wish.

Why do this?

If the journal system already exists, then why have something that looks kind of like a journal and offers sort of similar evaluations?

We think The Unjournal adds value to the existing journal system in a number of important ways. For example:

Unlike most academic journals, we have no bias against null results. This means that we’re just as likely to review important work that “fails” to find an effect as important work that rejects the null.

We’re transparent. We publish reviews online so that anyone can see not only the working paper but also what expert reviewers thought of it. Most social science journals do not do this.

We’re experiment...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The current alignment plan, and how we might improve it | EAG Bay Area 23, published by Buck on June 8, 2023 on The Effective Altruism Forum.

I gave a talk at EAG SF where I tried to describe my current favorite plan for how to align transformatively powerful AI, and how I thought that this plan informs current research prioritization. I think this talk does a reasonable job of representing my current opinions, and it was a great exercise for me to write it.

I have one major regret about this talk, which is that my attitude towards the risks associated with following the proposed plan now seems way too blasé to me. I think that if labs deploy transformative AI in the next ten years after following the kind of plan I describe, there's something like 10-20% chance that this leads to AI takeover.

Obviously, this is a massive amount of risk. It seems totally unacceptable to me for labs to unilaterally impose this kind of risk on the world, from both the longtermist perspective and also from any common-sense perspective. The potential benefits of AGI development are massive, but not massive enough that it’s worth accepting massive risk to make the advent of AGI happen a decade sooner.

I wish I'd prefixed the talk by saying something like: "This is not the plan that humanity deserves. In almost all cases where AI developers find themselves in the kind of scenario I described here, I think they should put a lot of their effort into looking for alternative options to deploying their dangerous models with just the kinds of safety interventions I was describing here. From my perspective, it only makes sense to follow this kind of plan if the developers are already in exceptionally dire circumstances which were already a massive failure of societal coordination."

So if I think that this plan is objectively unacceptable, why did I describe it?

Substantially, it's just that I had made the unforced error of losing sight of how objectively terrible a 10-20% chance of AI takeover is.

This is partially because my estimate of x-risk from AI used to be way higher, and so 10-20% intuitively feels pretty chill and low to me, even though obviously it's still objectively awful. I spend a lot of time arguing with people whose P(AI takeover) is way higher than mine, and so I kind of naturally fall into the role of trying to argue that a plan is more likely to succeed than you might have thought.

I realized my error here mostly by talking to Ryan Greenblatt and Beth Barnes. In particular, Beth is focused on developing safety standards that AI labs follow when developing and deploying their AI systems; plans of the type that I described in this talk are probably not able to be robust enough that they would meet the safety standards that Beth would like to have in place.

I think of my job as an alignment researcher as trying to do technical research such that AI developers are able to make the best of whatever empirical situation they find themselves in; from this perspective, it’s someone else’s job to reduce the probability that we end up in a situation where the alignment techniques are under a lot of strain. And so the fact that this situation is objectively unacceptable is often not really on my mind.

Since this talk, I’ve also felt increasingly hopeful that it will be possible to intervene such that labs are much more careful than I was imagining them being in this talk, and I think it’s plausible that people who are concerned about AI takeover should be aiming for coordination such that we’d have an overall 1% chance of AI takeover rather than just shooting for getting to ~10%. The main levers here are:

Go slower at the end. A lot of the risk of developing really powerful AI comes from deploying it in high-leverage situations before you’ve had enough time to really understand its propert...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Exercise to Build Intuitions on AGI Risk, published by Lauro Langosco on June 7, 2023 on LessWrong.

Epistemic status: confident that the underlying idea is useful; less confident about the details, though they're straightforward enough that I expect they're mostly in the right direction.

TLDR: This post describes a pre-mortem-like exercise that I find useful for thinking about AGI risk. It is the only way I know of to train big-picture intuitions about what solution attempts are more or less promising and what the hard parts of the problem are. The (simple) idea is to iterate between constructing safety proposals ('builder step') and looking for critical flaws in a proposal ('breaker step').

Introduction

The way that scientists-in-training usually develop research taste is to smash their heads against reality until they have good intuitions about things like which methods tend to work, how to interpret experimental results, or when to trust their proof of a theorem. This important feedback loop is mostly absent in AGI safety research, since we study a technology that does not exist yet (AGI). As a result, it is hard to develop a good understanding of which avenues of research are most promising and what the hard bits of the problem even are.

The best way I know of to approximate that feedback loop is an iterative exercise with two steps: 1) propose a solution to AGI safety, and 2) look for flaws in the proposal. The idea is simple, but most people don’t do it explicitly or don’t do it often enough.

Multiple rounds of this exercise tend to bring up details about one’s assumptions and predictions that would otherwise stay implicit or unnoticed. Writing down specific flaws of a specific proposal helps ground more general concepts like instrumental convergence or claims like ‘corrigibility is unnatural’. And after some time, the patterns in the flaws (the ‘hard bits’) become visible on their own.

I ran an earlier version of this exercise as a workshop (an important component is to discuss your ideas with others, so a workshop format is convenient). Here are the slides.

The exercise

The exercise consists of two phases: a builder phase in which you write down a best guess / proposal for how we might avoid existential risk from AGI, and a breaker phase in which you dig into the details until you understand how the proposal fails.

Importantly, in the context of this exercise the only thing that counts is your own inside view, that is your own understanding of the technical or political feasibility of the proposal. You might have thoughts like “There’s smart people who have thought about this much longer than I have, and they think X; why should I disagree?”. Put that aside for now; the point is to develop your own views, and that works best when you don’t think too much about other people’s views except to inform your own thoughts.

Builder phase

Write down the proposal: a plausible story for how we might avoid human extinction or disempowerment due to AGI. It doesn’t need to be very detailed yet; that comes in the breaker phase.

The proposal will look very different depending on the assumptions you’re starting with.

If you’re relatively optimistic about AGI risk, the proposal might look very much like things continuing to go on the current trajectory. Write down how you expect the future to go in very broad terms: if we build AGI, how do we know it’ll behave like we want it to? If we don’t build AGI, how come?

If you’re more pessimistic: what’s the most plausible way in which we could move things from their current trajectory? Is it plausible AI labs can coordinate to not build AGI? Or is there a technical solution that seems promising?

The builder phase is complete when you have a proposal (can be as short as a paragraph) that seems to you like it stands a decent chance ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LEAst-squares Concept Erasure (LEACE), published by tricky labyrinth on June 7, 2023 on LessWrong.

"Ever wanted to mindwipe an LLM?

Our method, LEAst-squares Concept Erasure (LEACE), provably erases all linearly-encoded information about a concept from neural net activations. It does so surgically, inflicting minimal damage to other concepts.

LEACE has a closed-form solution that fits on a T-shirt. This makes it orders of magnitude faster than popular concept erasure methods like INLP and R-LACE, which require gradient-based optimization. And the solution can be efficiently updated to accommodate new data."

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Strategy Fortnight (June 12-24), published by Ben West on June 8, 2023 on The Effective Altruism Forum.

Tl;dr: I’m kicking off a push for public discussions about EA strategy that will be happening June 12-24. You’ll see new posts under this tag, and you can find details about people who’ve committed to participating and more below.

Motivation and what this is(n’t)

I feel (and, from conversations in person and seeing discussions on the Forum, think that I am not alone in feeling) like there’s been a dearth of public discussion about EA strategy recently, particularly from people in leadership positions at EA organizations.

To help address this, I’m setting up an “EA strategy fortnight” — two weeks where we’ll put in extra energy to make those discussions happen. A set of folks have already volunteered to post thoughts about major strategic EA questions, like how centralized EA should be or current priorities for GH&W EA.

This event and these posts are generally intended to start discussion, rather than give the final word on any given subject. I expect that people participating in this event will also often disagree with each other, and participation in this shouldn’t imply an endorsement of anything or anyone in particular.

I see this mostly as an experiment into whether having a simple “event” can cause people to publish more stuff. Please don't interpret any of these posts as something like an official consensus statement.

Some people have already agreed to participate

I reached out to people through a combination of a) thinking of people who had shared private strategy documents with me before that still had not been published b) contacting leaders of EA organizations, and c) soliciting suggestions from others. About half of the people I contacted agreed to participate. I think you should view this as a convenience sample, heavily skewed towards the people who find writing Forum posts to be low cost. Also note that I contacted some of these people specifically because I disagree with them; no endorsement of these ideas is implied.

People who’ve already agreed to post stuff during this fortnight [in random order]:

Habryka - How EAs and Rationalists turn crazy

MaxDalton - In Praise of Praise

MichaelA - Interim updates on the RP AI Governance & Strategy team

William_MacAskill - Decision-making in EA

Michelle_Hutchinson - TBD

Ardenlk - On reallocating resources from EA per se to specific fields

Ozzie Gooen - Centralize Organizations, Decentralize Power

Julia_Wise - EA reform project updates

Shakeel Hashim - EA Communications Updates

Jakub Stencel - EA’s success no one cares about

lincolnq - Why Altruists Can't Have Nice Things

Ben_West and 2ndRichter - FTX’s impacts on EA brand and engagement with CEA projects

jeffsebo and Sofia_Fogel - EA and the nature and value of digital minds

Anonymous – Diseconomies of scale in community building

Luke Freeman and Sjir Hoeijmakers - Role of effective giving within E

kuhanj - Reflections on AI Safety vs. EA groups at universities

Joey - The community wide advantages of having a transparent scope

JamesSnowden - Current priorities for Open Philanthropy's Effective Altruism, Global Health and Wellbeing program

Nicole_Ross - Crisis bootcamp: lessons learned and implications for EA

Rob Gledhill - AIS vs EA groups for city and national groups

Vaidehi Agarwalla - The influence of core actors on the trajectory and shape of the EA movement

Renan Araujo - Thoughts about AI safety field-building in LMICs

ChanaMessinger - Reducing the social miasma of trust

particlemania - Being Part of Systems

jwpieters - Thoughts on EA community building

MichaelPlant - The Hub and Spoke Model of Effective Altruism

Quadratic Reciprocity - Best guesses for how public discourse and interest in AI existential risk over the past few months should update EA's...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What will GPT-2030 look like?, published by jsteinhardt on June 7, 2023 on LessWrong.

GPT-4 surprised many people with its abilities at coding, creative brainstorming, letter-writing, and other skills. Surprises in machine learning are not restricted to GPT-4: I was previously surprised by Minerva’s mathematical abilities, as were many competitive forecasters.

How can we be less surprised by developments in machine learning? Our brains often implicitly make a zeroth-order forecast: looking at the current state of the art, and adding on improvements that “feel reasonable”. But what “seems reasonable” is prone to cognitive bias, and will underestimate progress in a fast-moving field like ML. A more effective approach is first-order forecasting: quantifying the historical rate of progress and extrapolating it forward, while also considering reasons for possible slowdowns or speedups.

In this post, I’ll use this approach to forecast the properties of large pretrained ML systems in 2030. I’ll refer throughout to “GPT2030”, a hypothetical system that has the capabilities, computational resources, and inference speed that we’d project for large language models in 2030 (but which was likely trained on other modalities as well, such as images). To forecast GPT2030’s properties, I consulted a variety of sources, including empirical scaling laws, projections of future compute and data availability, velocity of improvement on specific benchmarks, empirical inference speed of current systems, and possible future improvements in parallelism.

GPT2030’s capabilities turn out to be surprising (to me at least). In particular, GPT2030 will enjoy a number of significant advantages over current systems, as well as (in at least some important respects) current human workers:

GPT2030 will likely be superhuman at various specific tasks, including coding, hacking, and math, and potentially protein engineering (Section 1).

GPT2030 can “work” and “think” quickly: I estimate it will be 5x as fast as humans as measured by words processed per minute [range: 0.5x-20x], and that this could be increased to 125x by paying 5x more per FLOP (Section 2).

GPT2030 can be copied arbitrarily and run in parallel. The organization that trains GPT2030 would have enough compute to run many parallel copies: I estimate enough to perform 1.8 million years of work when adjusted to human working speeds [range: 0.4M-10M years] (Section 3). Given the 5x speed-up in the previous point, this work could be done in 2.4 months.

GPT2030's copies can share knowledge due to having identical model weights, allowing for rapid parallel learning: I estimate 2,500 human-equivalent years of learning in 1 day (Section 4).

GPT2030 will be trained on additional modalities beyond text and images, possibly including counterintuitive modalities such as molecular structures, network traffic, low-level machine code, astronomical images, and brain scans. It may therefore possess a strong intuitive grasp of domains where we have limited experience, including forming concepts that we do not have (Section 5).

These capabilities would, at minimum, accelerate many areas of research while also creating serious vectors for misuse (Section 6). Regarding misuse, GPT2030's programming abilities, parallelization, and speed would make it a potent cyberoffensive threat. Additionally, its rapid parallel learning could be turned towards human behavior and thus used to manipulate and misinform with the benefit of thousands of "years" of practice.

On acceleration, a main bottleneck will be autonomy. In a domain like mathematics research where work can be checked automatically, I’d predict that GPT2030 will outcompete most professional mathematicians. In machine learning, I’d predict that GPT2030 will independently execute experiments and generates plots and...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What will GPT-2030 look like?, published by Jacob Steinhardt on June 7, 2023 on The AI Alignment Forum.

GPT-4 surprised many people with its abilities at coding, creative brainstorming, letter-writing, and other skills. Surprises in machine learning are not restricted to GPT-4: I was previously surprised by Minerva’s mathematical abilities, as were many competitive forecasters.

How can we be less surprised by developments in machine learning? Our brains often implicitly make a zeroth-order forecast: looking at the current state of the art, and adding on improvements that “feel reasonable”. But what “seems reasonable” is prone to cognitive bias, and will underestimate progress in a fast-moving field like ML. A more effective approach is first-order forecasting: quantifying the historical rate of progress and extrapolating it forward, while also considering reasons for possible slowdowns or speedups.

In this post, I’ll use this approach to forecast the properties of large pretrained ML systems in 2030. I’ll refer throughout to “GPT2030”, a hypothetical system that has the capabilities, computational resources, and inference speed that we’d project for large language models in 2030 (but which was likely trained on other modalities as well, such as images). To forecast GPT2030’s properties, I consulted a variety of sources, including empirical scaling laws, projections of future compute and data availability, velocity of improvement on specific benchmarks, empirical inference speed of current systems, and possible future improvements in parallelism.

GPT2030’s capabilities turn out to be surprising (to me at least). In particular, GPT2030 will enjoy a number of significant advantages over current systems, as well as (in at least some important respects) current human workers:

GPT2030 will likely be superhuman at various specific tasks, including coding, hacking, and math, and potentially protein engineering (Section 1).

GPT2030 can “work” and “think” quickly: I estimate it will be 5x as fast as humans as measured by words processed per minute [range: 0.5x-20x], and that this could be increased to 125x by paying 5x more per FLOP (Section 2).

GPT2030 can be copied arbitrarily and run in parallel. The organization that trains GPT2030 would have enough compute to run many parallel copies: I estimate enough to perform 1.8 million years of work when adjusted to human working speeds [range: 0.4M-10M years] (Section 3). Given the 5x speed-up in the previous point, this work could be done in 2.4 months.

GPT2030's copies can share knowledge due to having identical model weights, allowing for rapid parallel learning: I estimate 2,500 human-equivalent years of learning in 1 day (Section 4).

GPT2030 will be trained on additional modalities beyond text and images, possibly including counterintuitive modalities such as molecular structures, network traffic, low-level machine code, astronomical images, and brain scans. It may therefore possess a strong intuitive grasp of domains where we have limited experience, including forming concepts that we do not have (Section 5).

These capabilities would, at minimum, accelerate many areas of research while also creating serious vectors for misuse (Section 6). Regarding misuse, GPT2030's programming abilities, parallelization, and speed would make it a potent cyberoffensive threat. Additionally, its rapid parallel learning could be turned towards human behavior and thus used to manipulate and misinform with the benefit of thousands of "years" of practice.

On acceleration, a main bottleneck will be autonomy. In a domain like mathematics research where work can be checked automatically, I’d predict that GPT2030 will outcompete most professional mathematicians. In machine learning, I’d predict that GPT2030 will independently execute experiments and g...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Would it make sense for EA funding to be not so much focused on top talent?, published by Franziska Fischer on June 7, 2023 on The Effective Altruism Forum.

Across loads of EA project, career development services and organisations in general there's a strong sentiment towards focusing on 'top talent'. For example in AI safety there are a few very well funded but extremely competitive programmes for graduates who want to do research in the field. Naturally their output is then limited to relatively small groups of people. An opposing trend seems to have gained traction in AI capability research as e.g. the "We have no moat" paper argued, where a load of output comes from the sheer mass of people working on the problem with a breadth-first approach. A corresponding opposite strategy for EA funds and career development services could be to spread the limited ressouces they have over a larger amount of people.

This concentration of funds on the development on a small group of top talent rather than distributing it over a wider group of people seems to me is a general sentiment quite prominent in the US economy and much less so in EU-countries like Germany, Scandinavia, the netherlands etc. I could imagine that EA origins in US/UK are a major reason for this structural focus.

Has anyone pointers to research on effectiveness comparisons between focusing on top talent vs a broader set of people, ideally in the context of EA? Or any personal thoughts/anecdotes to share on this?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org