Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding Gradient Hacking, published by peterbarnett on December 10, 2021 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

Additional thanks to Michele Campolo, Oliver Zhang, Adam Shimi, and Leo Gao for their thoughts and feedback on this post. This work is additionally supported by CEEALAR.

TL;DR: Gradient hacking describes how a deceptively misaligned mesa-optimizer could influence its own training process via its outputs and thinking process. This is mainly considered in the context of training a neural network with gradient descent. A deceptive mesa-optimizer would have strong incentives to gradient hack; to protect its mesa-objective from being overwritten by the training process, to prevent humans from discovering the deception, and to build its own misaligned capabilities. I think gradient hacking is only plausible if the model’s ability to gradient hack is strongly coupled to the model’s general capability to perform well on tasks, such that attempting to remove the gradient hacking ability causes a large drop in the model’s performance. If there is not this strong coupling I expect the gradient hacker to not be stable, and to be erased by gradient descent.

When attempting to prevent gradient hacking, we do not want to be in the position where we are attempting to look for a model performing gradient hacking; if a model is able to gradient hack then it may be able to competently hide itself from us. Rather we should attempt to design our training procedures such that gradient hacking never happens.

What is gradient hacking?

Gradient hacking is a potential phenomenon where a machine learning model is able to influence its training process via its outputs and push its parameters in directions it desires. When a model is being trained by a local-optimization procedure such as gradient descent, it can deliberately choose certain outputs or ways of thinking to ensure the gradients are what it wants, and hence the parameter updates are in line with its own objectives. Here it is choosing its outputs and thought process to ‘hack’ the gradients and influence its training. This is most worrying if we have a deceptively aligned model, which is attempting to pretend to be aligned while actually having its own goals which are not aligned with ours. Such a gradient hacking model may choose its outputs in such a way that optimizes its mesa-objective rather than doing what we actually want. So far gradient hacking is still theoretical and seems to require a relatively powerful mesa-optimizer.

But if a powerful mesa-optimizer does develop in a prosaic AI model, then it seems likely that it would attempt to gradient hack.

A gradient hacker may choose to protect parts of itself from various aspects of the training process which would cause it to be erased, or it may seek to deliberately modify its weights to better achieve its mesa-objective. For example, if we were using transparency tools to deliberately look for deceptive alignment in our model, then a gradient hacker could change its weights to trick us or better conceal itself. For this reason, if gradient hacking is plausible then it is essential that our training process never produces one in the first place. We cannot be in the position of ‘putting out fires’ here; once a model has started to gradient hack it may already be too late because our transparency tools and other attempts to discover misalignment may be rendered useless.

Gradient hacking has a seemingly recursive nature; the model has to reason about its training process to choose how to affect the gradients of its parameters, but this very reasoning process influences the gradients. The model is made up...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding Gradient Hacking, published by peterbarnett on December 10, 2021 on The AI Alignment Forum.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

Additional thanks to Michele Campolo, Oliver Zhang, Adam Shimi, and Leo Gao for their thoughts and feedback on this post. This work is additionally supported by CEEALAR.

TL;DR: Gradient hacking describes how a deceptively misaligned mesa-optimizer could influence its own training process via its outputs and thinking process. This is mainly considered in the context of training a neural network with gradient descent. A deceptive mesa-optimizer would have strong incentives to gradient hack; to protect its mesa-objective from being overwritten by the training process, to prevent humans from discovering the deception, and to build its own misaligned capabilities. I think gradient hacking is only plausible if the model’s ability to gradient hack is strongly coupled to the model’s general capability to perform well on tasks, such that attempting to remove the gradient hacking ability causes a large drop in the model’s performance. If there is not this strong coupling I expect the gradient hacker to not be stable, and to be erased by gradient descent.

When attempting to prevent gradient hacking, we do not want to be in the position where we are attempting to look for a model performing gradient hacking; if a model is able to gradient hack then it may be able to competently hide itself from us. Rather we should attempt to design our training procedures such that gradient hacking never happens.

What is gradient hacking?

Gradient hacking is a potential phenomenon where a machine learning model is able to influence its training process via its outputs and push its parameters in directions it desires. When a model is being trained by a local-optimization procedure such as gradient descent, it can deliberately choose certain outputs or ways of thinking to ensure the gradients are what it wants, and hence the parameter updates are in line with its own objectives. Here it is choosing its outputs and thought process to ‘hack’ the gradients and influence its training. This is most worrying if we have a deceptively aligned model, which is attempting to pretend to be aligned while actually having its own goals which are not aligned with ours. Such a gradient hacking model may choose its outputs in such a way that optimizes its mesa-objective rather than doing what we actually want. So far gradient hacking is still theoretical and seems to require a relatively powerful mesa-optimizer.

But if a powerful mesa-optimizer does develop in a prosaic AI model, then it seems likely that it would attempt to gradient hack.

A gradient hacker may choose to protect parts of itself from various aspects of the training process which would cause it to be erased, or it may seek to deliberately modify its weights to better achieve its mesa-objective. For example, if we were using transparency tools to deliberately look for deceptive alignment in our model, then a gradient hacker could change its weights to trick us or better conceal itself. For this reason, if gradient hacking is plausible then it is essential that our training process never produces one in the first place. We cannot be in the position of ‘putting out fires’ here; once a model has started to gradient hack it may already be too late because our transparency tools and other attempts to discover misalignment may be rendered useless.

Gradient hacking has a seemingly recursive nature; the model has to reason about its training process to choose how to affect the gradients of its parameters, but this very reasoning process influences the gradients. The model is made up...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 12/9: Counting Down the Days, published by Zvi on December 9, 2021 on LessWrong.

Omicron is coming. We are not preparing for it. That is the news that matters.

In addition to that, Delta is a growing problem as we face a winter wave. Future problems don’t protect us from current problems, no matter how inevitable and predictable the current problems might be. Here’s where we round those things up.

Note that due to recent events, and the new data on what people actually benefit from and want to share, I think it increasingly makes less sense to focus on weekly posts, and to focus more on addressing particular issues when they appear, so I’ll be transitioning towards being more inclined to split things up.

Omicron Posts: #1, #2, #3, #4, #5 (today)

Last Week’s Weekly Post.

Executive Summary

Omicron is coming.

Boosters won’t be updated in time.

Paxlovid remains illegal.

On Paxlovid, update is that final Pfizer data is expected in a few days, a perfect timeline gets the whole thing done by end of month, the 24% at Polymarket for that is probably somewhat low. Nothing’s going wrong, except for the total lack of urgency.

There’s various fights over masks and vaccine mandates, but game that matters is Omicron.

In the meantime, on with the show. Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 630k cases (+20%) and 8,000 deaths (+26%)

Results: 736k cases (+39%) and 8,355 deaths (+32%).

Prediction for next week: 750k cases (+2%) and 8,800 deaths (+5%).

I was being somewhat cowardly making my prediction. On reflection, I should have been more confident in what was obviously happening.

This week I’m predicting smaller increases because I expect that some of this week’s big increases were due to data dumps, and because I expect some amount of additional caution due to fear of Omicron, whether or not that yet makes physical sense.

Note that Omicron cases aren’t going to be noticeable this coming week.

Deaths

Cases

You’ll want to smooth out numbers over the last few weeks. Either way, it’s clear that the Delta situation continues to get worse in addition to the threat from Omicron. It won’t become a crisis before Omicron becomes primary.

Vaccinations

Fear of Omicron is working, at least somewhat, and there have been reports of difficulty getting appointments. Getting an appointment for my son’s second shot took a week, whereas my own booster was in my arm an hour after I went online to book an appointment.

Some potentially good persuasion here.

Vaccine Mandates

Boosters work, and are important to protect yourself from both Delta and Omicron, and there isn’t time to wait for an Omicron booster. I got boosted and I suggest you do the same.

That doesn’t mean that I think vilifying those without boosters, or making them mandatory, is a good idea. On the contrary: This is going to make a lot of people very angry, and is widely regarded as a bad move. It’s a tough crowd out there.

Yet it’s going to happen anyway. Here’s a sample opening shot across the bow.

I realize that it’s a universal rule that the answer to any headline in the form of a question will ever and always be “NO!” but every rule has its exceptions. Yes, of course it’s safe to hang out with the ‘unboosted.’ It’s also safe to hang out with the unvaccinated, that’s the whole point (ok, fine, most of the the point) of getting boosted.

The other front is to continue applying the squeeze to the unvaccinated. In Europe this is taking the form of outright mandates. Germany is locking down the unvaccinated.

Here, we prefer to utilize our uniquely perverse system of charging people for medical care, where ‘pay out of pocket’ means ‘we treat you without telling you the price or letting you say no, then we charge you ten times what it actually costs and then take your house.’

The principle behind this m...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 12/9: Counting Down the Days, published by Zvi on December 9, 2021 on LessWrong.

Omicron is coming. We are not preparing for it. That is the news that matters.

In addition to that, Delta is a growing problem as we face a winter wave. Future problems don’t protect us from current problems, no matter how inevitable and predictable the current problems might be. Here’s where we round those things up.

Note that due to recent events, and the new data on what people actually benefit from and want to share, I think it increasingly makes less sense to focus on weekly posts, and to focus more on addressing particular issues when they appear, so I’ll be transitioning towards being more inclined to split things up.

Omicron Posts: #1, #2, #3, #4, #5 (today)

Last Week’s Weekly Post.

Executive Summary

Omicron is coming.

Boosters won’t be updated in time.

Paxlovid remains illegal.

On Paxlovid, update is that final Pfizer data is expected in a few days, a perfect timeline gets the whole thing done by end of month, the 24% at Polymarket for that is probably somewhat low. Nothing’s going wrong, except for the total lack of urgency.

There’s various fights over masks and vaccine mandates, but game that matters is Omicron.

In the meantime, on with the show. Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 630k cases (+20%) and 8,000 deaths (+26%)

Results: 736k cases (+39%) and 8,355 deaths (+32%).

Prediction for next week: 750k cases (+2%) and 8,800 deaths (+5%).

I was being somewhat cowardly making my prediction. On reflection, I should have been more confident in what was obviously happening.

This week I’m predicting smaller increases because I expect that some of this week’s big increases were due to data dumps, and because I expect some amount of additional caution due to fear of Omicron, whether or not that yet makes physical sense.

Note that Omicron cases aren’t going to be noticeable this coming week.

Deaths

Cases

You’ll want to smooth out numbers over the last few weeks. Either way, it’s clear that the Delta situation continues to get worse in addition to the threat from Omicron. It won’t become a crisis before Omicron becomes primary.

Vaccinations

Fear of Omicron is working, at least somewhat, and there have been reports of difficulty getting appointments. Getting an appointment for my son’s second shot took a week, whereas my own booster was in my arm an hour after I went online to book an appointment.

Some potentially good persuasion here.

Vaccine Mandates

Boosters work, and are important to protect yourself from both Delta and Omicron, and there isn’t time to wait for an Omicron booster. I got boosted and I suggest you do the same.

That doesn’t mean that I think vilifying those without boosters, or making them mandatory, is a good idea. On the contrary: This is going to make a lot of people very angry, and is widely regarded as a bad move. It’s a tough crowd out there.

Yet it’s going to happen anyway. Here’s a sample opening shot across the bow.

I realize that it’s a universal rule that the answer to any headline in the form of a question will ever and always be “NO!” but every rule has its exceptions. Yes, of course it’s safe to hang out with the ‘unboosted.’ It’s also safe to hang out with the unvaccinated, that’s the whole point (ok, fine, most of the the point) of getting boosted.

The other front is to continue applying the squeeze to the unvaccinated. In Europe this is taking the form of outright mandates. Germany is locking down the unvaccinated.

Here, we prefer to utilize our uniquely perverse system of charging people for medical care, where ‘pay out of pocket’ means ‘we treat you without telling you the price or letting you say no, then we charge you ten times what it actually costs and then take your house.’

The principle behind this m...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Monitoring Wild Animal Welfare via Vocalizations, published by HannahMc on December 10, 2021 on The Effective Altruism Forum.

Summary / Overview

This article describes how various types and features of vocalizations could act as welfare metrics for wild animals and how a remote acoustic sensing network could be used to collect this type of data non-invasively.

Animal vocalizations vary with emotional state (due to physiological changes)

Certain types of vocalizations are only produced in certain situations (e.g. alarm calls), which could give us information about the lives of wild animals

Vocalizations can be recorded from the wild remotely, using microphones placed in the field

Machine learning techniques could be applied to recognize the species and analyze welfare-indicating features in the vocalizations, or recognize what type of call it is

This may help indicate the welfare status of whole groups of populations of wild animals

Studying affective vocalizations could give insight into animal welfare in many different contexts, not just remote sensing

This article is designed to (i) describe how animal vocalizations can be used as welfare indicators, (ii) describe how vocalization and acoustic data can be both collected and analyzed, and (iii) evaluate the benefits and limitations of remote acoustic sensing for understanding and building a field of wild animal welfare (WAW).

People who are generally interested in animal welfare as a cause area should find this article worthwhile, as well as people who might be interested in pursuing this idea as a research project. The sections on costs and equipment are included for someone who might want to start remote acoustic monitoring for WAW today, but I anticipate the whole project would have more payoff in the medium term. This is both because it will take time to learn about the welfare indicators in vocalizations of many more species and because field-building and gaining the traction in interest and funding will also take time.

If you are not someone looking to execute this idea, but are interested in the viability or potential of vocalizations as animal welfare-indicators (especially in the context of WAW), I recommend reading the ‘Introduction to Affective Vocalizations’ and then the ‘Scope, Neglectedness, Tractability’ sections.

Introduction to Affective Vocalizations

What is an affective state?

An affective state is a multifaceted response to internal or external stimuli that arises to facilitate survival – it can be thought of as an emotional or mental state. They evolved via natural selection to, broadly, lead animals toward survival-optimizing conditions (positive affective states) and away from survival-threatening conditions (negative affective states). Affective states can be seen as a component of welfare. There are many descriptive and operational definitions of welfare, here I broadly comply with Wild Animal Initiative’s definition, which describes welfare as only the mental experiences of an animal. That said, I do think the idea proposed here (to monitor wild animal vocalizations at a large scale) would still contribute to building a picture of welfare under other’s definitions, which often encompass physical health, mental wellbeing and an animal having the opportunity to express biologically normal behaviors.

Welfare is generally thought about on the level of individual animals (while it is convenient to speak of the “aggregate welfare” of a population, welfare is experienced only by individuals, not groups).

Why, biologically, are affective states expressed in vocalizations?

In the wild, animals vocalize to convey information about themselves, others and the environment. Vocalizations serve many functions. Animals may communicate strategically with allies of the same species, or warn competitors of th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Monitoring Wild Animal Welfare via Vocalizations, published by HannahMc on December 10, 2021 on The Effective Altruism Forum.

Summary / Overview

This article describes how various types and features of vocalizations could act as welfare metrics for wild animals and how a remote acoustic sensing network could be used to collect this type of data non-invasively.

Animal vocalizations vary with emotional state (due to physiological changes)

Certain types of vocalizations are only produced in certain situations (e.g. alarm calls), which could give us information about the lives of wild animals

Vocalizations can be recorded from the wild remotely, using microphones placed in the field

Machine learning techniques could be applied to recognize the species and analyze welfare-indicating features in the vocalizations, or recognize what type of call it is

This may help indicate the welfare status of whole groups of populations of wild animals

Studying affective vocalizations could give insight into animal welfare in many different contexts, not just remote sensing

This article is designed to (i) describe how animal vocalizations can be used as welfare indicators, (ii) describe how vocalization and acoustic data can be both collected and analyzed, and (iii) evaluate the benefits and limitations of remote acoustic sensing for understanding and building a field of wild animal welfare (WAW).

People who are generally interested in animal welfare as a cause area should find this article worthwhile, as well as people who might be interested in pursuing this idea as a research project. The sections on costs and equipment are included for someone who might want to start remote acoustic monitoring for WAW today, but I anticipate the whole project would have more payoff in the medium term. This is both because it will take time to learn about the welfare indicators in vocalizations of many more species and because field-building and gaining the traction in interest and funding will also take time.

If you are not someone looking to execute this idea, but are interested in the viability or potential of vocalizations as animal welfare-indicators (especially in the context of WAW), I recommend reading the ‘Introduction to Affective Vocalizations’ and then the ‘Scope, Neglectedness, Tractability’ sections.

Introduction to Affective Vocalizations

What is an affective state?

An affective state is a multifaceted response to internal or external stimuli that arises to facilitate survival – it can be thought of as an emotional or mental state. They evolved via natural selection to, broadly, lead animals toward survival-optimizing conditions (positive affective states) and away from survival-threatening conditions (negative affective states). Affective states can be seen as a component of welfare. There are many descriptive and operational definitions of welfare, here I broadly comply with Wild Animal Initiative’s definition, which describes welfare as only the mental experiences of an animal. That said, I do think the idea proposed here (to monitor wild animal vocalizations at a large scale) would still contribute to building a picture of welfare under other’s definitions, which often encompass physical health, mental wellbeing and an animal having the opportunity to express biologically normal behaviors.

Welfare is generally thought about on the level of individual animals (while it is convenient to speak of the “aggregate welfare” of a population, welfare is experienced only by individuals, not groups).

Why, biologically, are affective states expressed in vocalizations?

In the wild, animals vocalize to convey information about themselves, others and the environment. Vocalizations serve many functions. Animals may communicate strategically with allies of the same species, or warn competitors of th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Promise and Peril of Finite Sets, published by davidad on December 10, 2021 on The AI Alignment Forum.

Has this ever happened to you?

But how can we decide whether this {species/moral theory/consensus node/quantum world} is actually 2 different things, or just 1 thing? Or does it even count at all?

The underlying cause of such dilemmas is that somebody, somewhere in the setup of your ontology and the framing of your problem statement, explicitly or implicitly, made the following move:

For simplicity of analysis, we assume that X is a finite set.

The Promise

This is an incredibly popular simplifying assumption, easily competitive with "we assume this distribution is Gaussian" and "we neglect higher-order terms". It's plausibly the single most popular mathematical modeling assumption in the world, because one needs so little background knowledge to make use of it.

It's also extremely powerful. Here's just some of the mathematical structure you get for free if you accept the assumption:

Computable representation. Saying that X is a finite set can be formalized as ∃n:N.X≅{i:N|i

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Promise and Peril of Finite Sets, published by davidad on December 10, 2021 on The AI Alignment Forum.

Has this ever happened to you?

But how can we decide whether this {species/moral theory/consensus node/quantum world} is actually 2 different things, or just 1 thing? Or does it even count at all?

The underlying cause of such dilemmas is that somebody, somewhere in the setup of your ontology and the framing of your problem statement, explicitly or implicitly, made the following move:

For simplicity of analysis, we assume that X is a finite set.

The Promise

This is an incredibly popular simplifying assumption, easily competitive with "we assume this distribution is Gaussian" and "we neglect higher-order terms". It's plausibly the single most popular mathematical modeling assumption in the world, because one needs so little background knowledge to make use of it.

It's also extremely powerful. Here's just some of the mathematical structure you get for free if you accept the assumption:

Computable representation. Saying that X is a finite set can be formalized as ∃n:N.X≅{i:N|i

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EU AI Act now has a section on general purpose AI systems, published by MathiasKB on December 9, 2021 on The Effective Altruism Forum.

The EU AI Act is currently undergoing the ordinary legislative procedure, in which the European Parliament and Council can propose changes to the act.A brief summary of the act is that systems defined as high-risk will be required to undergo conformity assessments, which among other things requires the system to be monitored and have a working off-switch (longer summary and analysis here).The Council's amendments have recently been circulated. Most importantly for longtermists, they include a new section for general purpose AI systems. For the first time ever regulating general AI is on the table, and for an important government as well!The article reads:

Article 52a - General purpose AI systems

The placing on the market, putting into service or use of general purpose AI systems shall not, by themselves only, make those systems subject to the provisions of this Regulation.

Any person who places on the market or puts into service under its own name or trademark or uses a general purpose AI system made available on the market or put into service for an intended purpose that makes it subject to the provisions of this Regulation shall be considered the provider of the AI system subject to the provisions of this Regulation.

Paragraph 2 shall apply, mutatis mutandis, to any person who integrates a general purpose AI system made available on the market, with or without modifying it, into an AI system whose intended purpose makes it subject to the provisions of this Regulation.

The provisions of this Article shall apply irrespective of whether the general purpose AI system is open source software or not.

Or in plain English, General Purpose AI Systems will not be considered high-risk, unless they are explicitly intended to be used for a high-risk purpose.What are your reactions to this development?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EU AI Act now has a section on general purpose AI systems, published by MathiasKB on December 9, 2021 on The Effective Altruism Forum.

The EU AI Act is currently undergoing the ordinary legislative procedure, in which the European Parliament and Council can propose changes to the act.A brief summary of the act is that systems defined as high-risk will be required to undergo conformity assessments, which among other things requires the system to be monitored and have a working off-switch (longer summary and analysis here).The Council's amendments have recently been circulated. Most importantly for longtermists, they include a new section for general purpose AI systems. For the first time ever regulating general AI is on the table, and for an important government as well!The article reads:

Article 52a - General purpose AI systems

The placing on the market, putting into service or use of general purpose AI systems shall not, by themselves only, make those systems subject to the provisions of this Regulation.

Any person who places on the market or puts into service under its own name or trademark or uses a general purpose AI system made available on the market or put into service for an intended purpose that makes it subject to the provisions of this Regulation shall be considered the provider of the AI system subject to the provisions of this Regulation.

Paragraph 2 shall apply, mutatis mutandis, to any person who integrates a general purpose AI system made available on the market, with or without modifying it, into an AI system whose intended purpose makes it subject to the provisions of this Regulation.

The provisions of this Article shall apply irrespective of whether the general purpose AI system is open source software or not.

Or in plain English, General Purpose AI Systems will not be considered high-risk, unless they are explicitly intended to be used for a high-risk purpose.What are your reactions to this development?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.