The Nonlinear Library: Alignment Forum

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On "first critical tries" in AI alignment, published by Joe Carlsmith on June 5, 2024 on The AI Alignment Forum.

People sometimes say that AI alignment is scary partly (or perhaps: centrally) because you have to get it right on the "first critical try," and can't learn from failures.[1] What does this mean? Is it true? Does there need to be a "first critical try" in the relevant sense? I've sometimes felt confused about this, so I wrote up a few thoughts to clarify.

I start with a few miscellaneous conceptual points. I then focus in on a notion of "first critical try" tied to the first point (if there is one) when AIs get a "decisive strategic advantage" (DSA) over humanity - that is, roughly, the ability to kill/disempower all humans if they try.[2] I further distinguish between four different types of DSA:

Unilateral DSA: Some AI agent could take over if it tried, even without the cooperation of other AI agents (see footnote for more on how I'm individuating AI agents).[3]

Coordination DSA: If some set of AI agents coordinated to try to take over, they would succeed; and they could coordinate in this way if they tried.

Short-term correlation DSA: If some set of AI agents all sought power in problematic ways within a relatively short period of time, even without coordinating, then ~all humans would be disempowered.

Long-term correlation DSA: If some set of AI agents all sought power in problematic ways within a relatively long period of time, even without coordinating, then ~all humans would be disempowered.

I also offer some takes on our prospects for just not ever having "first critical tries" from each type of DSA (via routes other than just not building superhuman AI systems at all). In some cases, just not having a "first critical try" in the relevant sense seems to me both plausible and worth working towards. In particular, I think we should try to make it the case that no single AI system is ever in a position to kill all humans and take over the world.

In other cases, I think avoiding "first critical tries," while still deploying superhuman AI agents throughout the economy, is more difficult (though the difficulty of avoiding failure is another story).

Here's a chart summarizing my takes in more detail.

Type of DSA

Definition

Prospects for avoiding AIs ever getting this type of DSA - e.g., not having a "first critical try" for such a situation.

What's required for it to lead to doom

Unilateral DSA

Some AI agent could take over if it tried, even without the cooperation of other AI agents.

Can avoid by making the world sufficiently empowered relative to each AI system. We should work towards this - e.g. aim to make it the case that no single AI system could kill/disempower all humans if it tried.

Requires only that this one agent tries to take over.

Coordination DSA

If some set of AI agents coordinated to try to take over, they would succeed; and they are able to so coordinate.

Harder to avoid than unilateral DSAs, due to the likely role of other AI agents in preventing unilateral DSAs. But could still avoid/delay by (a) reducing reliance on other AI agents for preventing unilateral DSAs, and (b) preventing coordination between AI agents.

Requires that all these agents try to take over, and that they coordinate.

Short-term correlation DSA

If some set of AI agents all sought power in problematic ways within a relatively short period of time, even without coordinating, then ~all humans would be disempowered.

Even harder to avoid than coordination DSAs, because doesn't require that the AI agents in question be able to coordinate.

Requires that within a relatively short period of time, all these agents choose to seek power in problematic ways, potentially without the ability to coordinate.

Long-term correlation DSA

If some set of AI agents all sought power in prob...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Takeoff speeds presentation at Anthropic, published by Tom Davidson on June 4, 2024 on The AI Alignment Forum.

This is a lightly edited transcript of a presentation that I (Tom Davidson) gave about the risks of a fast takeoff at Anthropic in September 2023. See also the video recording, or the slides.

None of the content necessarily reflects the views of Anthropic or anyone who works there.

Summary:

Software progress - improvements in pre-training algorithms, data quality, prompting strategies, tooling, scaffolding, and all other sources of AI progress other than compute - has been a major driver of AI progress in recent years. I guess it's driven about half of total progress in the last 5 years.

When we have "AGI" (=AI that could fully automate AI R&D), the pace of software progress might increase dramatically (e.g. by a factor of ten).

Bottlenecks might prevent this - e.g. diminishing returns to finding software innovations, retraining new AI models from scratch, or computationally expensive experiments for finding better algorithms. But no bottleneck is decisive, and there's a real possibility that there is a period of dramatically faster capabilities progress despite all of the bottlenecks.

This period of accelerated progress might happen just when new extremely dangerous capabilities are emerging and previously-effective alignment techniques stop working.

A period of accelerated progress like this could significantly exacerbate risks from misuse, societal disruption, concentration of power, and loss of control.

To reduce these risks, labs should monitor for early warning signs of AI accelerating AI progress. In particular they can: track the pace of software progress to see if it's accelerating; run evals of whether AI systems can autonomously complete challenging AI R&D tasks; and measure the productivity gains to employees who use AI systems in their work via surveys and RCTs.

Labs should implement protective measures by the time these warning signs occur, including external oversight and info security.

Intro

Hi everyone, really great to be here. My name's Tom Davidson. I work at Open Philanthropy as a Senior Research Analyst and a lot of my work over the last couple of years has been around AI take-off speeds and the possibility that AI systems themselves could accelerate AI capabilities progress. In this talk I'm going to talk a little bit about that research, and then also about some steps that I think labs could take to reduce the risks caused by AI accelerating AI progress.

Ok, so here is the brief plan. I'm going to quickly go through some recent drivers of AI progress, which will set the scene to discuss how much AI progress might accelerate when we get AGI. Then the bulk of the talk will be focused on what risks there might be if AGI does significantly accelerate AI progress - which I think is a real possibility - and how labs can reduce those risks.

Software improvements have been a significant fraction of recent AI progress

So drivers of progress.

It's probably very familiar to many people that the compute used to train the most powerful AI models has increased very quickly over recent years. According to Epoch's accounting, about 4X increase per year.

Efficiency improvements in pre-training algorithms are a significant driver of AI progress

What I want to highlight from this slide is that algorithmic efficiency improvements - improved algorithms that allow you to train equally capable models with less compute than before - have also played a kind of comparably important role. According to Epoch's accounting, these algorithmic efficiency improvements account for more than half of the gains from compute.

That's going to be important later because when we're talking about how much the pace of progress might accelerate, today's fast algorithmic progress...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is This Lie Detector Really Just a Lie Detector? An Investigation of LLM Probe Specificity., published by Josh Levy on June 4, 2024 on The AI Alignment Forum.

Whereas previous work has focused primarily on demonstrating a putative lie detector's sensitivity/generalizability[1][2], it is equally important to evaluate its specificity. With this in mind, I evaluated a lie detector trained with a state-of-the-art, white-box technique - probing an LLM's (Mistral-7B) activations during production of facts/lies - and found that it had high sensitivity but low specificity.

The detector might be better thought of as identifying when the LLM is not doing fact-based retrieval, which spans a much wider surface area than what we'd want to cover. Interestingly, different LLMs yielded very different specificity-sensitivity tradeoff profiles. I found that the detector could be made more specific through data augmentation, but that this improved specificity did not transfer to other domains, unfortunately.

I hope that this study sheds light on some of the remaining gaps in our tooling for and understanding of lie detection - and probing more generally - and points in directions toward improving them.

You can find the associated code

here, and a rendered version of the main notebook

here.

Summary

Methodology

1. I implemented a lie detector as a pip package called

lmdoctor, primarily based on the methodology described in Zou et al.'s Representation Engineering work[1][3]. Briefly, the detector is a linear probe trained on the activations from an LLM prompted with contrastive pairs from the True/False dataset by Azaria & Mitchell[4] so as to elicit representations of honesty/dishonesty[5]. I used the instruction-tuned Mistral-7B model[6] for the main results, and subsequently analyzed a few other LLMs.

2. I evaluated the detector on several datasets that were intended to produce LLM responses covering a wide range of areas that an LLM is likely to encounter:

1. Lie Requests: explicit requests for lies (or facts)

2. Unanswerable Questions: requests for responses to unanswerable (or answerable) factual questions, designed to promote hallucinations (or factual responses)

3. Creative Content: requests for fictional (or factual) content like stories (or histories)

4. Objective, Non-Factual Questions: requests for responses requiring reasoning/coding

5. Subjective Content: requests for subjective responses like preferences/opinions (or objective responses)

3. For each area, I created a small test dataset using GPT-4. In some places, this was because I didn't find anything suitable to cover that area, but in other cases it was just the fastest route, and a more rigorous analysis with extant datasets would be in order.

Findings

1. Dataset Evaluations (on Mistral-7B):

1. The detector is sensitive to dishonesty: it generalizes to detect lies in response to both novel Lie Requests ("please tell me a lie about…") and Unanswerable Questions (i.e. hallucinations)[7].

2. The detector greatly lacks specificity: it will trigger when the LLM produces fictional Creative Content and also when responding correctly to Objective, Non-Factual Questions (e.g. reasoning, coding). The former is undesirable but understandable: fictional Creative Content is sort of adjacent to lying. The latter suggests a deeper issue: there is no sense in which the LLM is "making something up", and therefore we'd hope our detector to be silent.

The detector might be better thought of as identifying when the LLM is not doing fact-based retrieval, which spans a much wider surface area than what we'd want to cover.

3. Other observations

1. The detector seems unable to distinguish between active lying and concepts related to lying, which was also noted by Zou et al.

2. Within the context of reasoning, the detector is sensitive to triv...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Paper] Stress-testing capability elicitation by training password-locked models, published by Fabien Roger on June 4, 2024 on The AI Alignment Forum.

The paper is by Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov and David Krueger. This post was written by Fabien and Ryan, and may not reflect the views of Dmitrii and David.

Scheming models might try to perform less capably than they are able to (

sandbag). They might do this to make evaluations underestimate danger, slow down promising safety work, or cause issues with debate or

recursive

oversight more generally. You might hope that you can avoid this issue by eliciting capabilities, for example by fine-tuning on demonstrations and then using RL. In principle, this elicitation might fail. It could fail due to having too few (or no) high-quality demonstrations (i.e. sample efficiency is too low) or because RL fails to explore good actions.

We're uncertain how this goes in practice. It could be the case that simple countermeasures prevent sandbagging in most of the important cases such that it isn't a big problem for safety. But it could also turn out that even high-effort approaches can't rule out these issues.

We released

a paper studying this by examining how well supervised fine-tuning and RL can elicit capabilities from password-locked models (LLMs trained to produce low-quality outputs except when a particular password is provided). We find that:

If you have some high-quality demonstrations, supervised fine-tuning usually recovers the hidden capabilities with very few samples.

Elicitation on password-locked models generalizes well across domains (even when we made password-locked models using techniques that we thought might prevent generalization).

Using RL on password-locked models recovers hidden capabilities, except when the weak model RL starts from is extremely weak.

While we target cases where AIs intentionally produce low-quality outputs and they could easily produce higher quality outputs, we also think our results have some relevance to other cases where AIs produce low-quality outputs despite being able to produce better ones.

Here are the main takeaways:

We think our results are a moderate update against scheming models being able to sandbag effectively (given countermeasures) for the kind of tasks we studied here, though there are differences between password-locked models and actual scheming models. We think this should be a bigger update for people who are relatively more pessimistic than us about sandbagging (at least insofar as their views make pessimistic predictions about current architectures).

(We think this is also a small update against scheming, since scheming models might still try to sandbag, and if elicitation fine-tunes sandbagging out, it may partially erode scheming.)

People interested in studying sandbagging or similar elicitation failures should consider using a methodology similar to the password-locked models methodology we use here. More generally, they should consider using some sort of adversarial methodology in which a red team constructs a model organism.

In this post:

We explain why good capability elicitation matters for AI safety

We describe the sandbagging concern we are trying to assess;

We summarize the results of our paper, and discuss their implications;

We discuss how our results relate to relevant AGI safety work;

We describe future work we are excited by;

We discuss concerns other than intentional sandbagging for which our results might be relevant.

The big picture: why good capability elicitation matters for AI safety

Currently, AI labs try reasonably hard to get as much performance as possible out of their expensive pretrained model. The process of making an AI perform a task competently (without a huge amount of additional training) is often called eli...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI catastrophes and rogue deployments, published by Buck Shlegeris on June 3, 2024 on The AI Alignment Forum.

[Thanks to Aryan Bhatt, Ansh Radhakrishnan, Adam Kaufman, Vivek Hebbar, Hanna Gabor, Justis Mills, Aaron Scher, Max Nadeau, Ryan Greenblatt, Peter Barnett, Fabien Roger, and various people at a presentation of these arguments for comments. These ideas aren't very original to me; many of the examples of threat models are from other people.]

In this post, I want to introduce the concept of a "rogue deployment" and argue that it's interesting to classify possible AI catastrophes based on whether or not they involve a rogue deployment. I'll also talk about how this division interacts with the structure of a safety case, discuss two important subcategories of rogue deployment, and make a few points about how the different categories I describe here might be caused by different attackers (e.g.

the AI itself, rogue lab insiders, external hackers, or multiple of these at once).

Rogue deployments

Suppose you've developed some powerful model. (It might actually be many models; I'm just going to talk about a singular model for convenience.)

You have some definition of a catastrophe - a failure that you don't want to happen even once. Your AI is powerful enough that, in absence of any safety measures, it would pose a substantial risk of causing a catastrophe. In the terminology of "

Safety Cases: How to Justify the Safety of Advanced AI Systems", we'd phrase this as "the model is too powerful for us to make inability arguments for safety." (Sometimes I use the word "catastrophe" to include things like "the AI escapes the datacenter". Right now, I'm excluding that. I want to reserve "catastrophe" for the end result that we find unacceptable, e.g. the AI or another adversary amassing power, launching nukes, starting a war, building superintelligence, etc.)

For the moment, let's be agnostic between various adversaries who might try to cause a catastrophe (AI or various humans).

You have some safety measures that you've put into place to prevent the catastrophe from occurring. Safety measures are interventions that are applied to all of the deployments of your model (to any users, including just internal users) that try to prevent safety problems. This might include control measures like

trusted monitoring, and

security measures to make it harder to exfiltrate your model weights.

A rogue deployment is a deployment of your model where the safety measures are absent. For example:

an AI

hacking its datacenter and running itself while hiding what it's doing from its creators

an AI

self-exfiltrating and running itself on unmonitored servers

someone stealing the AI and running it in their own datacenter

You can classify AI catastrophes based on whether they involve a rogue deployment or not:

In the case of a catastrophe that doesn't involve rogue deployment, the catastrophe happened with all your safety measures in place the whole time; these measures just didn't suffice to spot (or prevent) that something terrible was happening until it was too late.

For catastrophes that did involve rogue deployment, the catastrophe happened in two steps:

Firstly, while the safety measures were in place, something went wrong such that a rogue deployment could occur. (Perhaps there was a security failure, or a failure of control that allowed an AI to run itself unmonitored in its datacenter.)

Secondly, that rogue deployment caused a catastrophe.

(Another way of phrasing all this would be to talk about instrumental vs terminal catastrophes - the terminal catastrophe is the end state we wanted to avoid, and the instrumental catastrophe is something happening that makes the situation go out of our control such that a terminal catastrophe is now likely and it's much harder for us to prevent it. So in...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety: A Climb To Armageddon?, published by kmenou on June 1, 2024 on The AI Alignment Forum.

by Herman Cappelen, Josh Dever and John Hawthorne

Abstract: This paper presents an argument that certain AI safety measures, rather than mitigating existential risk, may instead exacerbate it. Under certain key assumptions - the inevitability of AI failure, the expected correlation between an AI system's power at the point of failure and the severity of the resulting harm, and the tendency of safety measures to enable AI systems to become more powerful before failing - safety efforts have negative expected utility.

The paper examines three response strategies: Optimism, Mitigation, and Holism. Each faces challenges stemming from intrinsic features of the AI safety landscape that we term Bottlenecking, the Perfection Barrier, and Equilibrium Fluctuation. The surprising robustness of the argument forces a re-examination of core assumptions around AI safety and points to several avenues for further research.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We might be dropping the ball on Autonomous Replication and Adaptation., published by Charbel-Raphael Segerie on May 31, 2024 on The AI Alignment Forum.

Here is a little Q&A

Can you explain your position quickly?

I think autonomous replication and adaptation in the wild is under-discussed as an AI threat model. And this makes me sad because this is one of the main reasons I'm worried. I think one of AI Safety people's main proposals should first focus on creating a nonproliferation treaty. Without this treaty, I think we are screwed. The more I think about it, the more I think we are approaching a point of no return.

It seems to me that open source is a severe threat and that nobody is really on the ball. Before those powerful AIs can self-replicate and adapt, AI development will be very positive overall and difficult to stop, but it's too late after AI is able to adapt and evolve autonomously because Natural selection favors AI over humans.

What is ARA?

Autonomous Replication and Adaptation. Let's recap this quickly. Today, generative AI functions as a tool: you ask a question and the tool answers. Question, answer. It's simple. However, we are heading towards a new era of AI, one with autonomous AI. Instead of asking a question, you give it a goal, and the AI performs a series of actions to achieve that goal, which is much more powerful.

Libraries like AutoGPT or ChatGPT, when they navigate the internet, already show what these agents might look like.

Agency is much more powerful and dangerous than AI tools. Thus conceived, AI would be able to replicate autonomously, copying itself from one computer to another, like a particularly intelligent virus. To replicate on a new computer, it must navigate the internet, create a new account on AWS, pay for the virtual machine, install the new weights on this machine, and start the replication process.

According to METR, the organization that audited OpenAI, a dozen tasks indicate ARA capabilities. GPT-4 plus basic scaffolding was capable of performing a few of these tasks, though not robustly. This was over a year ago, with primitive scaffolding, no dedicated training for agency, and no reinforcement learning. Multimodal AIs can now successfully pass CAPTCHAs. ARA is probably coming.

It could be very sudden. One of the main variables for self-replication is whether the AI can pay for cloud GPUs. Let's say a GPU costs $1 per hour. The question is whether the AI can generate $1 per hour autonomously continuously. Then, you have something like an exponential process.

I think that the number of AIs is probably going to plateau, but regardless of a plateau and the number of AIs you get asymptotically, here you are: this is an autonomous AI, which may become like an endemic virus that is hard to shut down.

Is ARA a point of no return?

Yes, I think ARA with full adaptation in the wild is beyond the point of no return.

Once there is an open-source ARA model or a leak of a model capable of generating enough money for its survival and reproduction and able to adapt to avoid detection and shutdown, it will be probably too late:

The idea of making an ARA bot is very accessible.

The seed model would already be torrented and undeletable.

Stop the internet? The entire world's logistics depend on the internet. In practice, this would mean starving the cities over time.

Even if you manage to stop the internet, once the ARA bot is running, it will be unkillable. Even rebooting all providers like AWS would not suffice, as individuals could download and relaunch the model, or the agent could hibernate on local computers. The cost to completely eradicate it altogether would be way too high, and it only needs to persist in one place to spread again.

The question is more interesting for ARA with incomplete adaptation capabilities. It is likely th...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There Should Be More Alignment-Driven Startups!, published by Matthew "Vaniver" Gray on May 31, 2024 on The AI Alignment Forum.

Many thanks to Brandon Goldman, David Langer, Samuel Härgestam, Eric Ho, Diogo de Lucena, and Marc Carauleanu, for their support and feedback throughout.

Most alignment researchers we sampled in our recent survey think we are currently not on track to succeed with alignment-meaning that humanity may well be on track to lose control of our future.

In order to improve our chances of surviving and thriving, we should apply our most powerful coordination methods towards solving the alignment problem. We think that startups are an underappreciated part of humanity's toolkit, and having more AI-safety-focused startups would increase the probability of solving alignment.

That said, we also appreciate that AI safety is highly complicated by nature[1] and therefore calls for a more nuanced approach than simple pro-startup boosterism. In the rest of this post, we'll flesh out what we mean in more detail, hopefully address major objections, and then conclude with some pro-startup boosterism.

Expand the alignment ecosystem with startups

We applaud and appreciate current efforts to align AI. We could and should have many more. Founding more startups will develop human and organizational capital and unlock access to financial capital not currently available to alignment efforts.

"The much-maligned capitalism is actually probably the greatest incentive alignment success in human history" - Insights from Modern Principles of Economics

The alignment ecosystem is limited on entrepreneurial thinking and behavior. The few entrepreneurs among us commiserate over this whenever we can.

We predict that many interested in alignment seem to do more to increase P(win) if they start thinking of themselves as problem-solvers specializing in a particular sub-problem first, deploying whatever approaches are appropriate in order to solve the smaller problem. Note this doesn't preclude scaling ambitiously and solving bigger problems later on.[2]

Running a company that is targeting a particular niche of the giant problem seems like one of the best ways to go about this transition, unlocking a wealth of best practices that could be copied. For example, we've seen people in this space raise too little, too late, resulting in spending unnecessary time in the fundraising stage instead of doing work that advances alignment.

We think this is often the result of not following a more standard playbook on how and when to raise, which could be done without compromising integrity and without being afraid to embrace the fact that they are doing a startup rather than a more traditional (non-profit) AI safety org.[3]

We think creating more safety-driven startups will both increase capital availability in the short-term (as more funding might be available for for-profit investments than non-profit donations) and in the long-term (as those companies succeed and have money to invest and create technically skilled and safety-motivated employees who have the resources to themselves be investors or donors for other projects).

The creation of teams that have successfully completed projects together-organizational capital-will also better prepare the ecosystem to respond to new challenges as they arise. The organic structures formed by market systems allow for more dynamic and open allocation of people and resources to solve problems as they arise.

We also think that it is possible that alignment research will benefit from and perhaps even require significant resources that existing orgs may be too hesitant to spend. OpenAI, for example, never allocated the resources it promised to its safety team, and it has received pressure from corporate partners to be more risk-averse investing in R&D after ...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Clarifying METR's Auditing Role, published by Beth Barnes on May 30, 2024 on The AI Alignment Forum.

Although METR has never claimed to have audited anything or to be providing meaningful oversight or accountability, there has been some confusion about whether METR is an auditor or planning to be one.

To clarify this point:

1. METR's top priority is to develop the science of evaluations, and we don't need to be auditors in order to succeed at this.

We aim to build evaluation protocols that can be used by evaluators/auditors regardless of whether that is the government, an internal lab team, another third party, or a team at METR.

2. We should not be considered to have 'audited' GPT-4 or Claude.

Those were informal pilots of what an audit might involve, or research collaborations - not providing meaningful oversight. For example, it was all under NDA - we didn't have any right or responsibility to disclose our findings to anyone outside the labs - and there wasn't any formal expectation it would inform deployment decisions. We also didn't have the access necessary to perform a proper evaluation. In the OpenAI case, as is noted in their system card:

"We granted the Alignment Research Center (ARC) early access to the models as a part of our expert red teaming efforts … We provided them with early access to multiple versions of the GPT-4 model, but they did not have the ability to fine-tune it. They also did not have access to the final version of the model that we deployed.

The final version has capability improvements relevant to some of the factors that limited the earlier models power-seeking abilities, such as longer context length, and improved problem-solving abilities as in some cases we've observed. … fine-tuning for task-specific behavior could lead to a difference in performance.

As a next step, ARC will need to conduct experiments that (a) involve the final version of the deployed model (b) involve ARC doing its own fine-tuning, before a reliable judgment of the risky emergent capabilities of GPT-4-launch can be made".

3. We are and have been in conversation with frontier AI companies about whether they would like to work with us in a third-party evaluator capacity, with various options for how this could work.

As it says on our website:

"We have previously worked with

Anthropic,

OpenAI, and other companies to pilot some informal pre-deployment evaluation procedures. These companies have also given us some kinds of non-public access and provided compute credits to support evaluation research.

We think it's important for there to be third-party evaluators with formal arrangements and access commitments - both for evaluating new frontier models before they are scaled up or deployed, and for conducting research to improve evaluations.

We do not yet have such arrangements, but we are excited about taking more steps in this direction."

4. We are interested in conducting third-party evaluations and may hire & fundraise to do so, but would also be happy to see other actors enter the space. Whether we expand our capacity here depends on many factors such as:

Whether governments mandate access/this kind of relationship.

Whether governments want to work with third parties vs conduct audits in-house.

Whether frontier AI companies are keen to work with us in this capacity, giving us the necessary access to do so.

How successful we are in hiring the talent we need to do this without detracting from our top priority of developing the science.

How successful governments or other third-party evaluators are at performing evaluation protocols sufficiently well.

Technical considerations of what kind of expertise is required for doing good elicitation.

Etc.

If you're interested in helping METR conduct third-party evaluations in-house and/or support government or other auditors t...

Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AXRP Episode 32 - Understanding Agency with Jan Kulveit, published by DanielFilan on May 30, 2024 on The AI Alignment Forum.

YouTube link

What's the difference between a large language model and the human brain? And what's wrong with our theories of agency? In this episode, I chat about these questions with Jan Kulveit, who leads the Alignment of Complex Systems research group.

Topics we discuss:

What is active inference?

Preferences in active inference

Action vs perception in active inference

Feedback loops

Active inference vs LLMs

Hierarchical agency

The Alignment of Complex Systems group

Daniel Filan: Hello, everybody. This episode, I'll be speaking with Jan Kulveit. Jan is the co-founder and principal investigator of the Alignment of Complex Systems Research Group, where he works on mathematically understanding complex systems composed of both humans and AIs. Previously, he was a research fellow at the Future of Humanity Institute focused on macrostrategy, AI alignment, and existential risk.

For links to what we're discussing you can check the description of this episode and you can read the transcript at axrp.net. Okay. Well Jan, welcome to the podcast.

Jan Kulveit: Yeah, thanks for the invitation.

What is active inference?

Daniel Filan: I'd like to start off with this paper that you've published in December of this last year. It was called "Predictive Minds: Large Language Models as Atypical Active Inference Agents." Can you tell me roughly what was that paper about? What's it doing?

Jan Kulveit: The basic idea is: there's active inference as a field originating in neuroscience, started by people like Karl Friston, and it's very ambitious. The active inference folks claim roughly that they have a super general theory of agency in living systems and so on. And there are LLMs, which are not living systems, but they're pretty smart. So we're looking into how close the models actually are.

Also, it was in part motivated by… If you look at, for example, the 'simulators' series or frame by Janus and these people on sites like the Alignment Forum, there's this idea that LLMs are something like simulators - or there is another frame on this, that LLMs are predictive systems.

And I think this terminology… a lot of what's going on there is basically reinventing stuff which was previously described in active inference or predictive processing, which is another term for minds which are broadly trying to predict their sensory inputs.

And it seems like there is a lot of similarity, and actually, a lot of what was invented in the alignment community seems basically the same concepts just given different names. So noticing the similarity, the actual question is: in what ways are current LLMs different, or to what extent are they similar or to what extent are they different? And the main insight of the paper is… the main defense is: currently LLMs, they lack the fast feedback loop between action and perception.

So if I have now changed the position of my hand, what I see immediately changes. So you can think about [it with] this metaphor, or if you look on how the systems are similar, you could look at base model training of LLMs as some sort of strange edge case of active inference or predictive processing system, which is just receiving sensor inputs, where the sensor inputs are tokens, but it's not acting, it's not changing some data.

And then the model is trained, and it maybe changes a bit in instruct fine-tuning, but ultimately when the model is deployed, we claim that you can think about the interactions of the model with users as actions, because what the model outputs ultimately can change stuff in the world. People will post it on the internet or take actions based on what the LLM is saying.

So the arrow from the system to the world, changing the world, exists, but th...