Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Working towards AI alignment is better, published by Johannes C. Mayer on December 9, 2022 on The AI Alignment Forum.

There are people not motivated to solve AI alignment, who do work related to AI alignment. E.g. people work on adversarial robustness, understanding how to do science mechanically, or on advancing other paradigms that are more interpretable than modern ML. These people might be interested in the science, or work on it for some other personal reason. They probably will do a worse job, compared to, if they would try to advance AI alignment, even when they work on something that is useful for AI alignment.

This basic idea was mentioned by Buck in a talk.

The following is a list of reasons why somebody who tries to solve alignment directly, would be better at solving alignment (though this sentence alone makes it sound obvious):

They are more likely to switch directions once they realize that they could be doing something better with their time, to make progress on AI alignment.

E.g. somebody who is interested in type theory and then learns that they can help AI alignment might be excited to help, but when there is lots of evidence that they should do something that does not involve type theory, they will keep sticking to doing things with type theory until the end, because their interest in type theory outweighs their desire to advance AI alignment.

The path that they take to solve the problem might look very different.

It is less likely that they take unpromissing but interesting sidetracks.

Simplifications that they make to the problem decrease the value of a solution less, in expectation.

In general, if there are multiple ways to solve the problem, the solution we end up with will likely be more relevant for alignment.

They can employ the full power of their consequentialist reasoning and be agentic about what to do, without starting to goodhart.

E.g. if you just let them do whatever, they are likely to discover things that are useful that you did not think of before. If somebodies main objective is not to solve AI alignment, it is likely that they will follow whatever looks best to their real motivation, as long as they can find some plausible explanation for why this is useful for AI alignment so that they have an excuse (in the case where they are payed to work on this to advance AI alignment).

There are probably many more points I have not thought of. How much you want to solve alignment compared to other things is a spectrum. What you care about might naturally drift. When you work on something for a long time, you get attached to your work. That's something to keep in mind.

It's interesting to think about the difference, between trying to solve alignment and just doing related work. It can help to notice when we fall into this trap ourselves. Also, getting clear on this might help in doing the good things (e.g. the things in the list above) even more.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: You can still fetch the coffee today if you're dead tomorrow, published by davidad (David A. Dalrymple) on December 9, 2022 on The AI Alignment Forum.

"You can't fetch the coffee if you're dead."—Stuart Russell, on the instrumental convergence of shutdown-avoidance

Note: This is presumably not novel, but I think it ought to be better-known. The technical tl;dr is that we can define time-inhomogeneous reward, and this provides a way of "composing" different reward functions; while this is not a way to build a shutdown button, it is a way to build a shutdown timer, which seems like a useful technique in our safety toolbox.

"Utility functions" need not be time-homogeneous

It's common in AI theory (and AI alignment theory) to assume that utility functions are time-homogeneous over an infinite time horizon, with exponential discounting. If we denote the concatenation of two world histories/trajectories by ⊳, the time-consistency property in this setting can be written as

This is property is satisfied, for example, by the utility-function constructions in the standard Wikipedia definitions of MDP and POMDP, which are essentially

Under such assumptions, Alex Turner's power-seeking theorems show that optimal agents for random reward functions R will systematically tend to disprefer shutting down (formalized as "transitioning into a state with no transitions out").

Exponential discounting is natural because if an agent's preferences are representable using a time-discount factor that depends only on relative time differences and not absolute time, then any non-exponential discounting form is exploitable (cf. Why Time Discounting Should Be Exponential).

However, if an agent has access to a clock, and if rewards are bounded by an integrable nonnegative function of time, the agent may be time-inhomogeneous in nearly arbitrary ways without actually exhibiting time inconsistency:

Any utility function with the above form still obeys an analogous version of our original time-consistency property that is modified to index over initial time t0:

Note that time-homogenous utility functions are a special case in which U(t,h)=γtU(0,h).

Time-bounded utility functions can be sequentially composed

We define a time-bounded utility function as a dependent tuple

i.e., a family of utility functions indexed by times within a given fixed range. The intended semantics of a time-bounded utility function in (τ,R) form is:

Given two time-bounded utility functions (in the same environment), they can be concatenated into a new time-bounded utility function:

You can check that ⊳ is a monoid, with the neutral element given by (0,∅).

How to build a shutdown timer

Let R1 be the reward function for a time-bounded task and τ1 be the time limit for the task, after which we want this agent to shut down. Assume that R1 also has bounded output, with per-stage reward always between R1––– and ˆR1. We define

We can then define τ2 to be 1 or indeed any positive integer. If an agent does not reach a shutdown state before τ1 is up, then it will realize a cost in R2 that outweighs all other rewards it could receive during the episode by a factor of C (a constant greater than 1). Therefore, optimal agents for (τ1,R1)⊳(τ2,R2) must shut down within time τ1 with probability ≥1−1/C (if the shutdown state is reachable in that time by any agent).

Proof

Suppose that the optimal policy π∗ results in a shutdown probability p<1−1/C, but there exists a policy π′ which shuts down deterministically (with probability 1). Then

which contradicts the optimality of π∗.

Comparison with the shutdown switch problem

Several years ago, MIRI's Agent Foundations group worked on how to make a reflectively stable agent with a shutdown switch, and (reportedly) gave up after failing to find a solution where the agent neither tries to manipulate the ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Smallpox eradication, published by Lizka on December 9, 2022 on The Effective Altruism Forum.

Today (December 9) is Smallpox Eradication Day. 43 years ago, smallpox was confirmed to have been eradicated after killing hundreds of millions of people. This was a major achievement in global health.

So I'm link-posting Our World in Data’s data explorer on smallpox (and here’s the section on how decline & eradication was achieved).

This post shares a summary of the history of the eradication of smallpox and selected excerpts from the data explorer.

A summary of the history of smallpox eradication

Smallpox was extremely deadly, probably killing 300 million people in the 20th century alone. The last known cases occurred in 1977, and smallpox is now the only human disease that has been completely eradicated.

So how was this accomplished?

Before we had a smallpox vaccine, we had the practice of variolation — deliberately exposing people to material from smallpox scabs or pus, in order to protect them against the disease (variolation traces back to 16th century China). While variolation made cases of smallpox much less severe, variolation infected the patient and could spread the disease to others, and the severity of the infection could not be easily controlled. So variolation did not lead to the elimination of smallpox from the population.

In the late 18th century, Edward Jenner demonstrated that exposure to cowpox — a much less severe disease that turns out to be related — protected people against smallpox. This, in turn, led to the invention of a vaccine against smallpox (the first vaccine ever).

In the 19th and 20th centuries, further improvements were made to the smallpox vaccine, and many states were running programs to vaccinate significant portions of the population. By 1959, the World Health Organization (WHO) launched a global program to eradicate smallpox . This involved a coordinated effort to immunize large numbers of people, isolate infected individuals, and monitor the spread of the disease. The program used a technique known as ring vaccination, which involved vaccinating people who had been in contact with infected individuals, in order to create a protective "ring" around the infected person and prevent further spread of the disease.

Excerpts from the Our World in Data entry

Introduction

Smallpox is the only human disease that has been successfully eradicated.

Smallpox, an infectious disease caused by the variola virus, was a major cause of mortality in the past, with historic records of outbreaks across the world. Its historic death tolls were so large that it is often likened to the Black Plague.

The eradication of smallpox is therefore a major success story for global health for several reasons: it was a disease that was endemic (and caused high mortality rates) across all continents; but was also crucial to advances in the field of immunology. The smallpox vaccine was the first successful vaccine to be developed.

How many died of smallpox?

In his review paper ‘The eradication of smallpox – An overview of the past, present, and future’ Donald Henderson reports that during the 20th century alone “an estimated 300 million people died of the disease.”

In his book Anderson suggests that in the last hundred years of its existence smallpox killed “at least half a billion people.” 500 million deaths over a century means 5 million annual deaths on average.

Eradication across the world

The last variola major infection was recorded in Bangladesh in October 1975, and the last variola minor infection occurred two years later in Merka, Somalia, on October 26th, 1977. During the following two years, WHO teams searched the African continent for further smallpox cases among those rash-like symptoms (which is a symptom of numerous other diseases). They found no further cas...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Main paths to impact in EU AI Policy, published by JOMG Monnet on December 8, 2022 on The Effective Altruism Forum.

Epistemic status

This is a tentative overview of the current main paths to impact in EU AI Policy. There is significant uncertainty regarding the relative impact of the different paths below

The paths mentioned below were cross-checked with four experts working in the field, but the list is probably not exhaustive

Additionally, this post may also be of interest to those interested in working in EU AI Policy. In particular, consider the arguments against impact here

This article doesn’t compare the potential value of US AI policy careers with those in the EU for people with EU-citizenship. A comparison between the two options is beyond the scope of this post

Summary

People seeking to have a positive impact on the direction of AI policy in the EU may consider the following paths to impact:

Working on (enforcement of) the AI Act, related AI technical standards and adjacent regulation

Working on the current AI Act draft (possibility to have impact immediately)

Working on technical standards and auditing services of the AI Act (possibility to have impact immediately)

Making sure the EU AI Act is enforced effectively (possibility to have impact now and >1/2 years from now)

Working on a revision of the AI Act (possibility to have impact >5 years from now), or on (potential) new AI-related regulation (e.g. possibility to have impact now through the AI Liability Directive)

Working on export controls and using the EU’s soft power (possibility to have immediate + longer term impact)

Using career capital gained from a career in EU AI Policy to work on different policy topics or in the private sector

While the majority of impact for some paths above is expected to be realised in the medium/long term, building up career capital is probably a prerequisite for making impact in these paths later on. It is advisable to create your own personal “theory of change” for working in the field. The list of paths to impact below is not exhaustive and individuals should do their own research into different paths and speak to multiple experts.

Paths to impact

Working on (enforcement of) the AI Act, related AI technical standards and adjacent regulation

Since the EU generally lacks cutting edge developers of AI systems, the largest expected impact from the EU AI Act is expected to follow from a Brussels effect of sorts. For the AI act to have a positive effect on the likelihood of safe, advanced AI systems being developed within organisations outside of the EU, the following assumptions need to be true:

Advanced AI systems are (also) being developed within private AI labs that (plan to) export products to EU citizens.

The AI-developing activities within private AI labs need to be influenced by the EU AI Act through a Brussels Effect of some sort (de facto or de jure). This depends on whether advanced AI is developed within a product development process in which AI-companies take EU AI Act regulation into account.

Requirements on these companies either have a (1) slowing effect on advanced AI development, buying more time for technical safety research or better regulation within the countries where AI is developed (2) direct effect, e.g. through risk management requirements that increase the odds of safe AI development

Working on the final draft of the AI Act

Current status of AI Act

The AI Act is a proposed European law on artificial intelligence (AI) – the first law on AI by a major regulator anywhere. On April 21, 2021 the Commission published a proposal to regulate artificial intelligence in the European Union. The proposal of the EU AI Act will become law once both the Council (representing the 27 EU Member States) and the European Parliament agree on a common version of the te...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Notes on OpenAI’s alignment plan, published by Alex Flint on December 8, 2022 on LessWrong.

This work was supported by the Monastic Academy for the Preservation of Life on Earth.

OpenAI has published a plan for producing aligned AI systems. Very roughly, their plan is to build language models capable of producing human-level alignment research. The output of these models might be natural language essays about alignment, or code that directly implements machine learning systems. In either case, human researchers would spend their time reviewing machine-generated alignment research. Their research agenda goes as follows.

Step 1: Train language models to solve useful problems using reinforcement learning. Rather than having a human review each attempt the model makes, which would be very time-consuming, train a second model that watches a few human reviews and learns to predict what reward the human would have given. Then train the main reinforcement learning model using this second model to provide a lot of feedback. You still need a human to review this second model every so often to make sure it has learned the right thing, but overall it requires less human effort than having a human review every attempt that the reinforcement learning model makes.

Step 2: When tasks become so complex that humans struggle to assess the outputs of the model, use some combination of the following:

Train models that critique their own outputs or the outputs of other models, possibly requiring the initial model to expand its reasoning around certain points in its original output. ("debate")

Train auxiliary models that help humans work out what reward to give to the primary model by solving more tractable sub-problems. In assessing the validity of a philosophy essay, you might train an auxiliary model to identify which paragraphs contain the key reasoning steps, and then use this model as a tool when assessing the philosophy essays. It might be that the sub-problems are themselves very difficult to provide training data for, in which case you can build models to solve sub-sub-problems that help humans to assess the output of the models targeting the sub-problems, and so on recursively. ("Recursive reward modeling")

Train a model that is general-purpose enough that it can itself do useful work to assist a human generating training data for the model’s own training. Each time you train a new version of the model, give that model to humans to use as a tool in generating more training data, and then retrain the model on that improved training data. In training an internet-search assistant, for example, you might have humans use the previous version of the assistant to answer sub-questions that are relevant to assessing a top-level answer or answering a top-level question. If this produces better training data than in the previous iteration then you may also be able to train a better internet-search assistant using this improved training data. You can repeat this process for as long as the thing keeps improving. ("Iterated amplification")

Step 3: use the above to build a language model capable of producing human-level alignment research. Presumably we would put in prompts like "what are the governing dynamics of agency in intelligent systems?" or "how can we avoid mesa-optimization in supervised learning?" and the thing would produce essays containing insights that clarify the alignment problem for us.

Discussion

There is a bit of a leap from "fine-tune a language model" to "produce human-level alignment research". Let’s say we’re training a language model to write insightful essays about alignment. We begin with some language model that has been pre-trained using, say, text from the internet. We give it a prompt such as "what would it mean to measure degrees of agency in arbitrary computer p...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety Seems Hard to Measure, published by HoldenKarnofsky on December 8, 2022 on LessWrong.

In previous pieces, I argued that there's a real and large risk of AI systems' developing dangerous goals of their own and defeating all of humanity - at least in the absence of specific efforts to prevent this from happening.

A young, growing field of AI safety research tries to reduce this risk, by finding ways to ensure that AI systems behave as intended (rather than forming ambitious aims of their own and deceiving and manipulating humans as needed to accomplish them).

Maybe we'll succeed in reducing the risk, and maybe we won't. Unfortunately, I think it could be hard to know either way. This piece is about four fairly distinct-seeming reasons that this could be the case - and that AI safety could be an unusually difficult sort of science.

This piece is aimed at a broad audience, because I think it's important for the challenges here to be broadly understood. I expect powerful, dangerous AI systems to have a lot of benefits (commercial, military, etc.), and to potentially appear safer than they are - so I think it will be hard to be as cautious about AI as we should be. I think our odds look better if many people understand, at a high level, some of the challenges in knowing whether AI systems are as safe as they appear.

First, I'll recap the basic challenge of AI safety research, and outline what I wish AI safety research could be like. I wish it had this basic form: "Apply a test to the AI system. If the test goes badly, try another AI development method and test that. If the test goes well, we're probably in good shape." I think car safety research mostly looks like this; I think AI capabilities research mostly looks like this.

Then, I’ll give four reasons that apparent success in AI safety can be misleading.

“Great news - I’ve tested this AI and it looks safe.” Why might we still have a problem? Problem Key question Explanation The Lance Armstrong problem Did we get the AI to be actually safe or good at hiding its dangerous actions? The King Lear problem The lab mice problem Today's "subhuman" AIs are safe.What about future AIs with more human-like abilities? The first contact problem

When dealing with an intelligent agent, it’s hard to tell the difference between “behaving well” and “appearing to behave well.”

When professional cycling was cracking down on performance-enhancing drugs, Lance Armstrong was very successful and seemed to be unusually “clean.” It later came out that he had been using drugs with an unusually sophisticated operation for concealing them.

The AI is (actually) well-behaved when humans are in control. Will this transfer to when AIs are in control?

It's hard to know how someone will behave when they have power over you, based only on observing how they behave when they don't.

AIs might behave as intended as long as humans are in control - but at some future point, AI systems might be capable and widespread enough to have opportunities to take control of the world entirely. It's hard to know whether they'll take these opportunities, and we can't exactly run a clean test of the situation.

Like King Lear trying to decide how much power to give each of his daughters before abdicating the throne.

Today's AI systems aren't advanced enough to exhibit the basic behaviors we want to study, such as deceiving and manipulating humans.

Like trying to study medicine in humans by experimenting only on lab mice.

Imagine that tomorrow's "human-like" AIs are safe. How will things go when AIs have capabilities far beyond humans'?

AI systems might (collectively) become vastly more capable than humans, and it's ... just really hard to have any idea what that's going to be like. As far as we know, there has never before been anything in the galaxy that's vastly m...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Monitoring & Evaluation Specialists – a new career path profile from Probably Good, published by Probably Good on December 8, 2022 on The Effective Altruism Forum.

Probably Good is excited to share a new path profile for careers in monitoring and evaluation. Below, we’ve included a few excerpts from the full profile.

Monitoring and evaluation (M&E) specialists collect, track, and analyze data to assess the value and impact of different programs and interventions, as well as translate these assessments into actionable insights and strategies to increase the impact of an organization.

M&E specialist careers might be a promising option for some people. If you’re an exceptional fit, especially if you’re based in a low- or middle-income country where there’s lots of scope for implementing global health and development interventions, then it may be worth considering these careers.

However, the impact you’ll be able to have will be determined in large part by the organization you enter – making it particularly important to seek out the best organizations and avoid those that only superficially care about evaluating their impact. Additionally, if you’re a good fit for some of the top roles in this path, it’s likely you’ll also be a good fit for other highly impactful roles, so we’d recommend you consider other paths, too.

How promising is this path?

Monitoring and evaluation is important for any organization aiming to have an impact. Without collecting evidence and data, it’s easy to seem like an intervention or program is having an impact, even when it’s not. Here are a few ways in which M&E might be able to generate impact:

Discover effective interventions that do a lot of good. For example, rigorous evaluation by J-PAL affiliates and Evidence Action found that placing chlorine-treated water dispensers in rural African villages reduced under-5 child mortality by as much as 63%. Evidence Action has now pledged to double the size of its water-treatment program, reaching 9 million people.

Make improvements to known effective interventions. Improving the efficacy of an already-impactful intervention by even a little bit can generate a large impact, especially if the intervention is rolled out on a large scale. Consider this study run by malaria charity TAMTAM, which found that charging even a nominal price for malaria bednets decreased demand by up to 60%, leading a number of large organizations to offer them for free instead.

Identify ineffective or harmful interventions, so that an organization can change course. A great example of this is animal advocacy organization the Humane League, which determined that their current strategy of performing controversial public stunts was ineffective, and pivoted its strategy towards corporate campaigns. In doing so, they convinced Unilever to stop killing male chicks, saving millions of baby chicks from gruesome deaths.

Advantages

Clear links to effectiveness - Because M&E is explicitly concerned with measuring the impact of interventions, there’s often a clear “theory of change” for how your work might translate into positive impact.

Leverage - If you’re working in a large organization, or working on an intervention with a large pool of potential funders and implementers, your work can influence where large amounts of money is spent, or how large amounts of other resources are distributed.

Flexible skill set - the skills and qualifications you’ll need for a career in M&E are robustly useful across a range of careers. As such, it’s likely that M&E work will provide you with flexible career capital for pursuing other paths.

Disadvantages

Narrow range of cause areas - Within our top recommended cause areas, there are far more M&E roles within global health and development than the others. This means M&E may be a promising career path if y...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SFF is doubling speculation (rapid) grant budgets; FTX grantees should consider applying, published by JueYan on December 8, 2022 on The Effective Altruism Forum.

The Survival and Flourishing Fund (SFF) funds many longtermist, x-risk, and meta projects, and has distributed $18mm YTD. While SFF’s focus areas are similar to those of the FTX Future Fund, SFF has received few applications since the latest round closed in August.

This is a reminder that projects can apply to be considered for expedited speculation grants at any time. Speculation grants can be approved in days and paid out as quickly as within a month. Past speculation grants have ranged from $10,000 to $400,000, and applicants for speculation grants will automatically be considered for the next main SFF round. In response to the recent extraordinary need, Jaan Tallinn, the main funder of SFF, is doubling speculation budgets. Grantees impacted by recent events should apply.

SFF funds charities and projects hosted by organizations with charity status. You can get a better idea of SFF’s scope from its website and its recent grants. I encourage relevant grantees to consider applying to SFF, in addition to the current array of efforts led by Open Phil, Mercatus, and Nonlinear.

For general information about the Survival and Flourishing Fund, see:/

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Presenting: 2022 Incubated Charities (Charity Entrepreneurship), published by KarolinaSarek on December 8, 2022 on The Effective Altruism Forum.

We are proud to announce that 5 new charitable organizations have been launched from our June-August 2022 Incubation Program. Nine high-potential individuals graduated from our two-month intensive training program. The Incubation Program has been designed to teach participants everything they need to know to launch and run an extremely effective, high-impact charity. From analyzing the cost-effectiveness of an intervention, all the way to crafting a proposal for funding, the program participants are equipped with the knowledge and confidence they need to see their chosen intervention become a reality. Eight have gone on to start new effective nonprofits focused on policy, mental health, family planning and EA meta cause areas, and one participant was hired by Charity Entrepreneurship as a Research Analyst. They will be joining our 2023 cohort.

Thanks to our generous CE Seed Network of funders, we have helped to secure $732,000 in funding for the organizations, and will further support them with mentorship, operational support, free co-working space in London, and access to a constantly growing entrepreneurial community of funders, advisors, interns and other charity founders.

The 2022 incubated charities are:

Center for Effective Aid Policy- identifying and promoting high-impact development policies and interventions.

Centre for Exploratory Altruism Research (CEARCH)- conducting cause prioritization research and outreach.

Maternal Health Initiative- producing transformative benefits to women’s health, agency, and income through increased access to family planning.

Kaya Guides- reducing depression and anxiety among youth in low-and middle-income countries.

Vida Plena- building strong mental health in Latin America.

CENTER FOR EFFECTIVE AID POLICY

Co-founders: Jacob Wood, Mathias BondeWebsite: aidpolicy.orgEmail address: [email protected] CE incubation grant: $170,000

Description of the intervention:

The Center for Effective Aid Policy will work on identifying and advocating for effective solutions in aid policy. This may include:

Increasing international development aid

Increasing budget allocation to specific effective development programs

Introducing new effective development interventions into aid budgets

Revising processes which result in improved development aid effectiveness

Background of the intervention:

$179 billion was spent on development aid in 2021 - that is roughly 240x the amount of money that GiveWell has moved since 2009. While well-intentioned, there is a broad consensus among experts, think tanks, and implementing partners alike that aid effectiveness can be vastly improved.

The Center for Effective Aid Policy believes tractable interventions exist in the development aid space that will result in improved aid spending and better outcomes for its recipients. You can read more in their recent EA Forum post.

Near-term plans:

In 2022-2023, The Center for Effective Aid Policy will identify policy windows and formulate impactful and practical-to-implement policies, which they will advocate to governments and NGOs. They conservatively estimate their chances of advocacy success at $5.62 per DALY - more than an order of magnitude higher than multiple GiveWell-recommended charities.

CENTRE FOR EXPLORATORY ALTRUISM RESEARCH (CEARCH)

Founder: Joel TanWebsite: exploratory-altruism.orgContact: incubation grant: $100,000

Description of the intervention:

CEARCH conducts cause prioritization research and outreach - identifying the most important problems in the world and directing resources towards solving them, so as to maximize global welfare.

Background of the intervention:

There are many potential cause areas (e.g.,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Machine Learning Consent, published by jefftk on December 8, 2022 on LessWrong.

For years, researchers have trained machine learning systems on whatever data they could find. People mostly haven't cared about this or paid attention, I think because the systems hadn't been very good. Recently, however, some very impressive systems have come out, including ones that

answer questions,

complete code, and

generate

images

from prompts.

Because these are so capable a lot more people are paying attention now, and there are big questions around whether it's ok that these systems were trained this way. Code that I uploaded to GitHub and the writing that I've put into this blog went into training these models: I didn't give permission for this kind of use, and no one asked me if it was ok. Doesn't this violate my copyrights?

The machine learning community has generally assumed that training models on some input and using it to generate new output is legal, as long as the output is sufficiently different from the input. This relies on the doctrine of "fair use", which does not require any sort of permission from the original author as long as it is sufficiently "transformative". For example, if I took a book and replaced every instance of the main characters name with my own I doubt any court would consider that sufficiently transformative, and so my book would be considered a "derivative work" of the original book. On the other hand, if I took the words in the book and painstakingly reordered them to tell a completely unrelated story, there's a sense in which my book was "derived" from the original one but I think it would pretty clearly be transformative enough that I wouldn't need any permission from the copyright holder.

These models can be used to create things that are clearly derivative works of their input. For example, people very quickly realized that Copilot would complete the code for Greg Walsh's fast inverse square root implementation verbatim, and if you ask any of the image generators for the Mona Lisa or Starry Night you'll get something close enough to the original that it's clearly a knock-off. This is a major issue with current AI systems, but it's also a relatively solvable one. It's already possible to slowly check that the output doesn't excessively resemble any input, and I think it's likely they'll soon figure out how to do that efficiently. On the other hand, all of the examples of this I've seen (and I just did some

looking) have been people trying to elicit plagiarism.

The normal use case is much more interesting, and more controversial. While the transformative fair use justification I described above is widely assumed within the machine learning community as far as I can tell it hasn't been tested in court. There is currently a large class action lawsuit over Copilot, and it's possible this kind of usage will turn out not qualify. Speculating, I think it's pretty unlikely that the suit will succeed, but I've created a prediction market on it to gather information:

Aside from the legal question, however, there is also a moral or social question: is it ok to train a model on someone's work without their permission? What if this means that they and others in their profession are no longer able to earn a living?

On the second question, you could imagine someone creating a model where they used only data that was either in the public domain or which they'd purchased appropriate licenses for. While that's great for the particular people who agree and get paid, a much larger number would still be out of work without compensation. I do think there's potentially quite a bad situation, where as these systems get better more and more people are unable to add much over an automated system, and we get massive technological unemployment. Now, historically worries...