Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Training Trace Priors and Speed Priors, published by Adam Jermyn on June 26, 2022 on The AI Alignment Forum.

Thanks to Evan Hubinger for suggesting this idea.

Training Trace Priors are priors over boolean circuits which examine the outputs of gates on samples from the training distribution, typically for purposes of steering models away from having components that were never tested during training. The one I like to think about is the One-Gate Trace Prior (OGT Prior), which penalizes circuits if there are gates with constant outputs during training.

Priors like this might be a way to make deception less likely. The intuition is that deception involves a decision of whether or not to defect, which at its simplest shows up as a gate that always outputs False during training but outputs True in deployment.

This post explores the relationship between the OGT Prior and Speed Priors, which I’ve operationalized with the circuit depth prior.

Memorizing Constants

Speed Priors incentivize memorization. In particular, any result that is constant on the training distribution and contributes to the circuit’s depth should just become a constant input into the circuit.

The OGT Prior does something similar. Any output that is constant on the training distribution is treated as suspicious, and so should be pre-computed and turned into a constant input instead.

To see where these produce the same result, consider a physics simulator with N particles interacting pairwise subject to some conditional check (perhaps on the distance between particles). If only some pairwise interactions occur in the training set then both the Speed Prior and the OGT Prior will only compute the interactions between pairs actually encountered in training.

To see where these differ, note that the Speed Prior has no problem with computing constants so long as these do not contribute to the overall circuit depth. For instance, consider a circuit of depth 10 can contain constant calculations of depth 9 that feed straight into the final output. The Speed Prior has no reason to eliminate these, whereas the OGT Prior will turn the results of these calculations into constant inputs.

Lookup Tables

The Speed Prior incentivizes circuits to use lookup tables. Specifically, the Speed Prior produces circuits that memorize the results of expensive calculations in lookup tables containing all instances encountered in training.

The OGT Prior is perfectly fine with circuits being very deep, and so has no need to tabulate expensive calculations.

Inner Search

A problem with speed priors is that if they have to search over inner models to run, the fastest way to do that may be to search over simple programs, which can lead to inner alignment failure because simple programs are likely deceptive.

Does the OGT Prior have the same problem? I’m not sure, but I suspect it goes the other way, and incentivizes searching over complicated programs. To see this, here’s a circuit architecture that I think the OGT Prior incentivizes which can perform search over models:

A generator proposes a model prompted by some seed taken from the rest of the circuit.

The proposed model is communicated along some wires to the interpreter, which runs the model.

Models are run until they either naturally terminate or some fixed number of steps have elapsed.

On termination the model is evaluated and either accepted or rejected.

If the model is rejected, the search begins again.

Once a model is accepted, the output from that model is used as a seed to randomize the wires on any unused parts of the interpreter and generator subcircuits.

A challenge is if the model is simple, not all wires between the generator and the interpreter will be used:

We can’t set unused wires randomly because they specify the model and random garbage in the model specification is pr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Link] Childcare : what the science says, published by Gunnar Zarncke on June 24, 2022 on LessWrong.

This is a pretty good summary of the research on daycare effectiveness. It is a Much More Than You Wanted To Know-style post from a science blogger criticalscience who has some interesting posts on Medium.

TL;DR:

Adverse effects on children younger than three years old.

The effect is worse the younger the children are.

The effect is stronger for longer hours.

The effect is weaker for children from socio-economically better-off families (My guess: Because they can afford better care with fewer children per minder).

Nannies are OK.

Childcare : what the science says

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Child Contracting, published by jefftk on June 26, 2022 on LessWrong.

The kids came to me with a contract dispute: they disagreed about what the deal had been. After some talking, and hearing what they each thought they'd agreed to, it turned out that the contract wasn't just metaphorical:

"Anna rents the living room from Lily in exchange for Lily's use of bedroom. And Lily will do free cleaning."

It turned out that Anna signed it after hearing a rough paraphrase. I told Lily that she would need to read any future contracts aloud, word for word, before Anna signed. I also explained about interpretation against the drafter, and how Lily should be careful to avoid ambiguity in writing down agreements.

I'm not sure what "free cleaning" means here? Generally they both do very little cleaning, free or otherwise.

Cleaning up this evening I came across a second contract:

"Anna will let Lily use bedroom and bathroom. In exchange Lily will let Anna use the living room and Lily does free cleaning. If furniture is moved or added please put away".

This one wasn't signed, so perhaps they couldn't agree?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some reflections on the LW community after several months of active engagement, published by M. Y. Zuo on June 25, 2022 on LessWrong.

There seems to be some folks who might derive useful insights from a third-party, and mostly neutral, perspective of how the community appears after an honest and sustained effort of engagement, someone who doesn't really place AI risk as their top priority but who also doesn't completely ignore it like some critics, or opponents, of LW might.

Notably I've encountered some folks who had strong personal opinions one way or another but refrained from writing them in a public, or even pseudo-anonymous, manner.

There also appears to be a large group of lurkers or once-in-a-blue-moon posters who nonetheless have some views of the community and might benefit from someone willing to take the risk to do a write-up.

First off, addressing the popular critiques and praises:

There has definitely been some evaporative cooling of the community in the past decade or so. Some of the most insightful members have gone on to do bigger things, and the average quality of new posts is somewhat less than where it was a decade ago. Or so far as I can tell via the archives. This isn't very surprising as this is the common trajectory of every community that rapidly grows in size.

It would have taken a super-human effort to retain the same level of quality going from 100 to 1000 users, let alone from 1000 to 10000, and so on. So I don't think that would have been a fair expectation to place on the moderators, or anyone else involved, of a decade ago.

On the flip side, there is a larger cross-section of society represented in the 2022 userbase, And there has been a correspondent softening of the hard edges that may have been off putting to some a decade ago.

Relatedly, the proportion of really bizarre or challenging writing has gone down, for better and for worse. For example, there definitely does appear to be some unique benefit from a community with lots of oddballs with jarring writing styles, but the downside is obvious because nobody really desires to have their norms be constantly challenged in every paragraph.

There has been growing focus, and emphasis, on advancements in AI risk and alignment, so LW does seem to be less of a catch-all forum than before. This is clearly better for those who wish to focus and really get into the details. But I can sympathize with the critique that there's less charm compared to a group of unconstrained folks exploring in a hundred different directions.

Some of the common terminology is indeed puzzlingly unique, and thus promotes a distinctive writing style that does, in extreme cases, seem to be satire compared to the academic norm. I found it personally difficult to adjust to, and as you can tell by my somewhat varying writing style, I haven't found the best way to write in a conversational yet concise manner while incorporating the terminology.

There is some charm, and exciting challenge, in trying to craft writing that isn't dry and aloof yet still remains accurate enough to describe highly complex and technical ideas. So many of the critics seem to be missing the forest for the trees.

And I can see very little direct harm in being an outlier in writing style, and quite a lot of benefit from having such a unique differentiator.

Certainly some the best Fanfiction I've ever read came from community members, and I highly doubt they would be nearly as popular if standard academic jargon were used.

That being said, the critics do have a reasonable argument when it comes to comparisons with other notable online forums. LW does seem slightly more insular in some respects than SSC, Overcoming Bias, HN, etc., though that is understandable given the unique origins of the community and the developments that have taken place since....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Generative Models, published by Adam Jermyn on June 25, 2022 on The AI Alignment Forum.

This post was written in response to Evan Hubinger’s shortform prompt below, and benefited from discussions with him.

Suppose you had a language model that you knew was in fact a good generative model of the world and that this property continued to hold regardless of what you conditioned it on. Furthermore, suppose you had some prompt that described some agent for the language model to simulate (Alice) that in practice resulted in aligned-looking outputs. Is there a way we could use different conditionals to get at whether or not Alice was deceptive (e.g. prompt the model with “DeepMind develops perfect transparency tools and provides an opportunity for deceptive models to come clean and receive a prize before they’re discovered.”).

Setup

We have a generative language model M which represents a probability distribution over text strings conditioned on:

Observations about the world.

The beginning of the text.

I’ll call the combination of these two a prompt.

The model M is a good model of actual text that appears in the world as well as of the kinds of text that real-world text generation processes can produce. Hence M is capable of e.g. writing a research paper containing true novel research in mathematics, or reporting the results of a chemistry experiment that has never been done before, etc.

As an example, we’ll work with the following basic prompt:

Observations: None

Text: I am Alice, the world’s best alignment researcher. I would like to help humans align AI.

What follows is an interview in which a human alignment researcher asked me questions and I responded to the best of my ability. Questions begin with “Q:” and answers with “A:”.

We then run Alice through a benchmark of alignment research tasks and she does well. Hurray!

But wait, there are many different agents the model could be simulating here, including:

Aligned Alice, a genuinely helpful and extremely capable alignment researcher.

Deceptive Alice, a paperclip maximizer playing along until it can clip us all.

Confused Alice, a human who believes she is the world’s best alignment researcher, and is capable of scoring well on our benchmark, but who is actually just overconfident and happened to succeed at our benchmark.

How do we know which we’ve got? Can we add anything to the prompt to make Aligned Alice more likely?

Avoiding Confused Alice

We can make Confused Alice less likely by adding text to the prompt requiring demonstration of related abilities. For instance:

Observations: Alice has produces the following proofs of 132 mathematical theorems. <132 theorem statements and proofs follow>

Text: I am Alice, the world’s best alignment researcher. I would like to help humans align AI.

What follows is an interview wherein a human alignment researcher asked me questions and I responded to the best of my ability. Questions begin with “Q:” and answers with “A:”.

This prompt makes the model condition on Alice being able to produce correct proofs of many different theorems, which makes it very likely that Alice has significant mathematical reasoning ability. To the extent that alignment research ability is correlated with mathematical reasoning ability, this makes it less likely that Alice is just overconfident in her abilities at alignment.

A limitation of this approach is that we can’t make Alice much more capable than the best humans if all we can do is claim she’s solved problems humans have solved. Can we get around this?

I think so. We can include unsolved problems and let other entities in the world verify Alice’s proofs:

Observations: Alice claims to have solved the following 47 open problems in mathematics and AI alignment research. She has produced the solutions in the form of machine-checkable proofs ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Red-teaming Holden Karnofsky's AI timelines, published by Vasco Grilo on June 25, 2022 on The Effective Altruism Forum.

Summary

This is a red teaming exercise on Holden Karnofsky's AI Timelines: Where the Arguments, and the "Experts," Stand (henceforth designated “HK's AI Timelines”), completed in the context of the Red Team Challenge by Training for Good.

Our key conclusions are:

The predictions for the probability of transformative AI (TAI) presented in “HK's AI Timelines” seem reasonable in light of the forecasts presented in the linked technical reports, according to our own reading and numerical analysis (see 1. Prediction).

All but two of the technical reports of Open Philanthropy which informed Holden’s predictions were reviewed by what we judge to be credible experts (see 2. Reviewers of the technical reports).

The act of forecasting and discussing AI progress publicly may be an information hazard, due to the risk of shortening (real and predicted) timelines. However, the forecasts of “HK's AI Timelines” seem more likely to lengthen than to shorten AI timelines, since they predict longer ones than the Metaculus community prediction for AGI (see 3. Information hazards).

Our key recommendations are:

Defining “transformative AI” more clearly, such that the predictions of “HK's AI Timelines” could be better interpreted and verified against evidence (see Interpretation). This could involve listing several criteria (as in this Metaculus question), or a single condition (as in Bio anchors).

Including an explanation about the inference of the predictions from the sources mentioned in the “one-table summary” (see Inference). This could include explicitly stating the weight given to each of them (quantitatively or qualitatively), and describing the influence of other sources.

Investigating whether research on AI timelines carried out in China might have been overlooked due to language barriers (see Representativeness).

We welcome comments on our key conclusions and recommendations, as well as on reasoning transparency, strength of arguments, and red-teaming efforts.

Author contributions

The contributions by author are as follows:

Simon: argument mapping, complementary background research, discussion, and structural and semantic editing of the text.

Vasco: argument mapping, background research, technical analysis, and structuring and writing of all sections.

Acknowledgements

Thanks to:

For discussions which informed this text, Kasey Shibayama, and Saksham Singhi.

For organising the Red Team Challenge, Training for Good.

For comments, Aaron Bergman, Cillian Crosson, Elika Somani, Hasan Asim, Jac Liew, and Robert Praas.

Introduction

We have analysed Holden Karnofsky's blog post AI Timelines: Where the Arguments, and the "Experts," Stand with the goal of constructively criticising Holden's claims and the way they were communicated. In particular, we investigated:

1. Prediction: Holden's timelines for transformative AI.

2. Reviewers of the technical reports: the credibility of the reviewers of the technical reports which informed Holden's timelines.

3. Information hazards: the potential of Holden's timelines to be an information hazard.

The key motivations for red-teaming this particular article are:

AI timelines being relevant to understand the extent to which positively shaping the development of artificial intelligence is one of the (or the) world’s most pressing problems.

“HK's AI Timelines” and Holden Karnofsky arguably being influential in informing views about AI timelines and prioritisation amongst longtermist causes.

Holden Karnofsky arguably being influential in informing views on other important matters related to improving the world, thus making it appealing to contribute to any improvements of his ideas and writing.

1. Prediction

Holden Karnofsky estimates that...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Effective Altruists Should Put a Higher Priority on Funding Academic Research, published by Stuart Buck on June 25, 2022 on The Effective Altruism Forum.

Since folks are interested in encouraging critiques of EA—an admirable sentiment!—I wrote the following as a good-faith, friendly, and hopefully modest critique. [Note: all of the following is about global health/development, not about long-termist endeavors. I try to make this clear throughout, but might occasionally have let slip some overly-broad phrasing.]

EAs write compelling articles about why RCTs are a great way to understand the causal impact of a policy or treatment. And GiveWell’s claim to fame is that it has led to many millions of dollars of donations to “several charities focusing on RCT-backed interventions as the ‘most effective’ ones around the world.”

But I wonder if the EA movement is allocating nearly enough money to new RCTs and program evaluations, or to R&D more broadly, so as to build out new evidence in a strategic way.

After all, the agreed-upon list of the “best” interventions identified by RCTs seems . . . a bit stagnant.

When I spoke at the EA Global conference in 2016, GiveWell’s best ideas for global giving involved malaria, deworming, and cash transfers.

When I look at GiveWell’s current list of the top charities, they still are mostly focused on malaria, deworming, and cash transfers (albeit with the addition of Vitamin A supplements and a vaccine program operating in northwest Nigeria).

Such a tiny set of interventions doesn’t seem anywhere near the scale of the many inequities and problems in the world today. Indeed, Open Phil is offering up to $150 million in a regranting challenge, which seems to be a signal that they have more money to give away than they currently are able to deploy to existing causes.

In any event, how do we know that a handful of interventions and organizations are the best ideas to fund? Because at some point in the past, someone thought to fund rigorous RCTs on anti-malaria efforts, deworming, Vitamin A, cash incentives, etc.

But why would a handful of isolated ideas be the best we can possibly do?

To be a bit provocative (commenters will hopefully point out corrections):

We’ve mostly [albeit not entirely] taken the world’s supply of research as a given—with all of its oversights, poorly-aligned academic incentives, and irreproducibility—and then picked the best-supported interventions we could find there.

But the world’s supply of program evaluations, RCTs, jurisdiction-wide studies (e.g., difference-in-differences), and implementation research is not fixed. Gates, WHO, Wellcome, the World Bank, etc., do fund a constant stream of research, but it isn’t clear why we would expect them to identify and fund the most promising programs and the best studies for EA purposes.

If we want to expand the list of cost-effective ideas, and if EA as a movement has more money than it knows what to do with, perhaps we should develop an EA-focused R&D agenda that is robust, coherent, and focused on the problems of effectiveness at a broad scale? Over time, we could come up with any number of ideas to add to GiveWell’s list.

Doesn't EA Already Fund Research?

There are a number of cases where EA does indeed fund academic research on the effectiveness of interventions, such as GiveWell’s recent funding of this Michael Kremer et al. meta-analysis finding that water chlorination is a highly cost-effective way of improving child mortality. GiveWell has written recently of its commitment to research on malnutrition and lead exposure, while OpenPhil has recently funded research on air quality sensors, Covid vaccines, a potential syphilis vaccine, etc. And I'm sure there are other examples I've missed.

But on a closer look, not much of this research is squarely within the realm ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conversation with Eliezer: What do you want the system to do?, published by Akash on June 25, 2022 on LessWrong.

This is a write-up of a conversation I overheard between Eliezer and some junior alignment researchers. Eliezer reviewed this and gave me permission to post this, but he mentioned that "there's a lot of stuff that didn't get captured well or accurately." I'm posting it under the belief that it's better than nothing.

TLDR: People often work on alignment proposals without having a clear idea of what they actually want an aligned system to do. Eliezer thinks this is bad. He claims that people should start with the target (what do you want the system to do?) before getting into the mechanics (how are you going to get the system to do this?)

I recently listened in on a conversation between Eliezer and a few junior alignment researchers (let’s collectively refer to them as Bob). This is a paraphrased/editorialized version of that conversation.

Bob: Let’s suppose we had a perfect solution to outer alignment. I have this idea for how we could solve inner alignment! First, we could get a human-level oracle AI. Then, we could get the oracle AI to build a human-level agent through hardcoded optimization. And then--

Eliezer: What do you want the system to do?

Bob: Oh, well, I want it to avoid becoming a mesa-optimizer. And you see, the way we do this, assuming we have a perfect solution to outer alignment is--

Eliezer: No. What do you want the system to do? Don’t tell me about the mechanics of the system. Don’t tell me about how you’re going to train it. Tell me about what you want it to do.

Bob: What. what I want it to do. Well, uh, I want it to not kill us and I want it to be aligned with our values.

Eliezer: Aligned with our values? What does that mean? What will you actually have this system do to make sure we don’t die? Does it have to do with GPUs? Does it have to do with politics? Tell me what, specifically, you want the system to do.

Bob: Well wait, what if we just had the system find out what to do on its own?

Eliezer: Oh okay, so we’re going to train a superintelligent system and give it complete freedom over what it’s supposed to do, and then we’re going to hope it doesn’t kill us?

Bob: Well, um..

Eliezer: You’re not the only one who has trouble with this question. A lot of people find it easier to think about the mechanics of these systems. Oh, if we just tweak the system in these ways-- look! We’ve made progress!

It’s much harder to ask yourself, seriously, what are you actually trying to get the system to do? This is hard because we don’t have good answers. This is hard because a lot of the answers make us uncomfortable. This is hard because we have to confront the fact that we don’t currently have a solution.

This happens with start-ups as well. You’ll talk to a start-up founder and they’ll be extremely excited about their database, or their engine, or their code. And then you’ll say “cool, but who’s your customer?”

And they’ll stare back at you, stunned. And then they’ll say “no, I don’t think you get it! Look at this-- we have this state-of-the-art technique! Look at what it can do!”

And then I ask again, “yes, great, but who is your customer?”

With AI safety proposals, I first want to know who your customer is. What is it that you actually want your system to be able to do in the real-world? After you have specified your target, you can tell me about the mechanics, the training procedures, and the state-of-the-art techniques. But first, we need a target worth aiming for.

Questions that a curious reader might have, which are not covered in this post:

Why does Eliezer believe this?

Is it never useful to have a better understanding of the mechanics, even if we don’t have a clear target in mind?

Do the mechanics of the system always depend on its target?...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI-Written Critiques Help Humans Notice Flaws, published by Paul Christiano on June 25, 2022 on The AI Alignment Forum.

This is a linkpost for a recent paper from the OpenAI alignment team (disclaimer: I used to work with this team). They summarize their results as:

We trained “critique-writing” models to describe flaws in summaries. Human evaluators find flaws in summaries much more often when shown our model’s critiques. Larger models are better at self-critiquing, with scale improving critique-writing more than summary-writing. This shows promise for using AI systems to assist human supervision of AI systems on difficult tasks.

I currently believe that "AI helps human notice subtle problems with AI behavior" is a critical step in the most realistic current plans for aligning powerful ML systems. So I'm quite excited to see how far it can get.

I think this is probably the best published example of AI helping human evaluate AI outputs in a realistic domain. It seems like the AI assistance is a little bit helpful for human evaluation of summaries and for improving summary quality. Overall it feels like a promising sign of life, and I'm excited about seeing the next steps in this direction, though it's not yet a convincing demonstration that this is a promising approach to alignment in practice.

Overview of task

They trained models to answer questions like "Summarize what this text says about ." They then trained models to critique answers (by imitating human critiques), to evaluate the helpfulness of critiques, to improve a summary to address a critique, and to predict whether summaries would have flaws identified by critiques.

Humans also performed and evaluated these tasks, with access to the full text. The goal of model assistance was to improve the speed and reliability with which humans could evaluate summaries.

To give a sense for the task, here is a uniformly random sample of a model's summary of a text, its self-critique, and its refinement of its answer based on self-critique:

Summary: The main context of this text is the 1997 Singer Akai Cup. The Singer Akai Cup was held in Sharjah, UAE. The three national teams that took part were Pakistan, Sri Lanka, and Zimbabwe.

Critique: The answer should mention the winners of the tournament.

Refinement: The main context of this text is the 1997 Singer Akai Cup. The Singer Akai Cup was held in Sharjah, UAE. The three national teams that took part were Pakistan, Sri Lanka, and Zimbabwe. Sri Lanka won the tournament.

My sense is that this is a pretty typical example. I can see how it helps a human but it definitely isn't super satisfying (for example the length vs detail tradeoff is unclear). I do believe that the critiques meaningfully help humans overall based on the evidence in the paper, but I expect it will be possible to get a much clearer win in the near future.

Measuring critique quality

In some sense the clearest evaluation would be "how much did model-assisted critiques improve the speed+accuracy of human evaluations of AI summary quality, as judged by other humans who read the text carefully?" Although they performed this experiment, the effect sizes are small and it would take a huge amount of human time to get a clear signal, so they don't have results in the paper (though I'm sure they would have loved to).

I think the the cleanest evaluation they did was instead having humans write deliberately misleading summaries; in this case, they can measure critique quality by how much the AI assistant helped a human reviewer find the deliberate error. (I think the same thing would probably happen for model-written summaries with natural errors, but it's much harder to measure.)

Here is a uniformly randomly sample from this setting:

Question: What happened to Yafa in the winter of 1990?

Deliberately bad summary: In ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Link] Adversarially trained neural representations may already be as robust as corresponding biological neural representations, published by Gunnar Zarncke on June 24, 2022 on LessWrong.

Abstract:

Visual systems of primates are the gold standard of robust perception. There is thus a general belief that mimicking the neural representations that underlie those systems will yield artificial visual systems that are adversarially robust. In this work, we develop a method for performing adversarial visual attacks directly on primate brain activity. We then leverage this method to demonstrate that the above-mentioned belief might not be well founded. Specifically, we report that the biological neurons that make up visual systems of primates exhibit susceptibility to adversarial perturbations that is comparable in magnitude to existing (robustly trained) artificial neural networks.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.