Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Which organisation is most deserving of the marginal $ right now? (your opinions), published by Nathan Young on August 27, 2023 on The Effective Altruism Forum.

@Aaron Bergman says

I wish there was (and there should be) more discussion of "which singular organization is most deserving of money rn" Individual donors should make their best guess public and indicate openness to critique

twitter

So let's guess. What is your suggestion?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Aumann-agreement is common, published by tailcalled on August 27, 2023 on LessWrong.

Thank you to Justis Mills for proofreading and feedback. This post is also available on my substack.

Aumann's agreement theorem is a family of theorems which say that if people trust each other and know each other's opinions, then they agree with each other. Or phrased another way, if people maintain trust with each other, then they can reach agreement. (And some variants of the theorem, which take computational factors into consideration, suggest they can do so quite rapidly.)

The original proof is pretty formal and confusing, but a simpler heuristic argument is that for an honest, rational agent, the mere fact of them professing an opinion can be strong evidence to another rational agent, because if the speaker's probabilities are higher than the speaker's prior, then they must have seen corresponding evidence to justify that opinion.

Some people find this confusing, and feel like it must be wrong because it doesn't apply to most disagreements. I think these people are wrong because they are not sufficiently expansive in what they think of as a disagreement. The notion of disagreement that Aumann's agreement theorem applies to is when the people assign different probabilities to events; this is a quite inclusive notion which covers many things that we don't typically think of as disagreements, including cases where one party has information about a topic and the other party has no information.

My vacation in Norway relied tons on Aumann agreements

Recently, I had a vacation in Norway with my wife.

In order to get there, and to get around, we needed transport. At first we disagreed with people who provided transport there, as we didn't know of many specific means of transport, only vaguely that there would be some planes and ships, without knowing which ones. But my wife had heard that there was something called the "Oslo ferry", so we Aumann-agreed that this was an option, and decided to investigate further.

We disagreed with the company that provided the Oslo ferry, as we didn't know what their website is, so we asked Google, and it provided some options for what the ferry might be, and we Aumann-agreed with Google and then went investigating from there. One website we found claimed to sell tickets to the ferry; at first we disagreed with the website about when we could travel as we didn't know the times of the ferry, but then we read which times it claimed was available, and Aumann-updated to that.

We also had to find some things to do in Norway. Luckily for us, some people at OpenAI had noticed that everyone had huge disagreements with the internet as nobody had really memorized the internet, and they thought that they could gain some value by resolving that disagreement, so they Aumann-agreed with the internet by stuffing it into a neural network called ChatGPT. At first, ChatGPT disagreed with us about what to visit in Norway and suggested some things we were not really interested in, but we informed it about our interests, and then it quickly Aumann-agreed with us and proposed some other things that were more interesting.

One of the things we visited was a museum for an adventurer who built a raft and sailed in the ocean. Prior to visiting the museum, we had numerous disagreements with it, as e.g. we didn't know that one of the people on the raft had fallen in the ocean and had to be rescued. But the museum told us this was the case, so we Aumann-agreed to believe it. Presumably, the museum learnt about it through Aumann-agreeing with the people on the raft.

One example of an erroneous Aumann agreement was with the train company Vy. They had said that they could get us a train ticket on the Bergen train, and we had Aumann-agreed with that. However, due to a storm, their train...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Eliezer Yudkowsky Is Frequently, Confidently, Egregiously Wrong, published by Omnizoid on August 27, 2023 on The Effective Altruism Forum.

Introduction

"After many years, I came to the conclusion that everything he says is false. . . . "He will lie just for the fun of it. Every one of his arguments was tinged and coded with falseness and pretense. It was like playing chess with extra pieces. It was all fake."

Paul Postal (talking about Chomsky) (note, this is not exactly how I feel about Yudkowsky, I don't think he's knowingly dishonest, but I just thought it was a good quote and partially represents my attitude towards Yudkowsky).

Crosspost of this on my blog.

In the days of my youth, about two years ago, I was a big fan of Eliezer Yudkowsky. I read his many, many writings religiously, and thought that he was right about most things. In my final year of high school debate, I read a case that relied crucially on the many worlds interpretation of quantum physics - and that was largely a consequence of reading through Eliezer's quantum physics sequence. In fact, Eliezer's memorable phrasing that the many worlds interpretation "wins outright given the current state of evidence," was responsible for the title of my 44-part series arguing for utilitarianism titled "Utilitarianism Wins Outright." If you read my early articles, you can find my occasional blathering about reductionism and other features that make it clear that my worldview was at least somewhat influenced by Eliezer.

But as I grew older and learned more, I realized it was all bullshit.

Eliezer sounds good whenever he's talking about a topic that I don't know anything about. I know nothing about quantum physics, and he sounds persuasive when talking about quantum physics. But every single time he talks about a topic that I know anything about, with perhaps one or two exceptions, what he says is total nonsense, at least, when it's not just banal self-help advice. It is not just that I always end up disagreeing with him, it is that he says with almost total confidence outrageous falsehood after outrageous falsehood, making it completely clear he has no idea what he is talking about. And this happens almost every single time. It seems that, with few exceptions, whenever I know anything about a topic that he talks about, it becomes clear that his view is a house of cards, built entirely on falsehoods and misrepresentations.

Why am I writing a hit piece on Yudkowsky? I certainly don't hate him. In fact, I'd guess that I agree with him much more than almost all people on earth. Most people believe lots of outrageous falsehoods. And I think that he has probably done more good than harm for the world by sounding the alarm about AI, which is a genuine risk. And I quite enjoy his scrappy, willing-to-be-contrarian personality. So why him?

Part of this is caused by personal irritation. Each time I hear some rationalist blurt out "consciousness is just what an algorithm feels like from the inside," I lose a year of my life and my blood pressure doubles (some have hypothesized that the explanation for the year of lost life involves the doubling of my blood pressure). And I spend much more time listening to Yukowsky's followers spout nonsense than most other people.

But a lot of it is that Yudkowsky has the ear of many influential people. He is one of the most influential AI ethicists around. Many people, my younger self included, have had their formative years hugely shaped by Yudkowsky's views - on tons of topics. As Eliezer says:

In spite of how large my mistakes were, those two years of blog posting appeared to help a surprising number of people a surprising amount.

Quadratic Rationality expresses a common sentiment, that the sequences, written by Eliezer, have significantly shaped the world view of him and others. Elie...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How much do EAGs cost (and why)?, published by Eli Nathan on August 26, 2023 on The Effective Altruism Forum.

TL;DR:

EAGs from 2022-2023 each cost around $2M-$3.6M USD at around $1.5k-2.5k per person per event.

These events typically cost a lot because the fixed costs of running professional events in the US and UK are surprisingly high.

We're aiming to get these event costs down to ≤$2M each moving forwards.

We've already started to cut back on spending and will continue to do so, whilst also raising default ticket prices to recoup more of our costs.

Throughout this post, for legibility I discuss the direct costs behind running our events and don't include other indirect costs like staff salaries, software, and our office space (which would increase the costs below by ~25%).

Introduction

This year (2023), EAG Bay Area cost $2M and EAG London cost £2M (including travel grant costs). Our most expensive event ever was EAG SF 2022, which cost $3.6M. This gives us a range of about $1.5-2.5k per person per event.

In-person EAGx events typically cost $150-500k, with smaller events in cheaper countries on the lower end (e.g. EAGxWarsaw) and large events in the US on the higher end (e.g. EAGxNYC). The cost per person for these events is $300-900.

You can see historical attendance figures on our dashboard.

People are often surprised by how expensive our events are - this post seeks to explain why our events have historically been so expensive, why they'll continue to be somewhat expensive, and how we're working to reduce costs (see an earlier post here). We're writing this partially in response to requests from the community, but we'll also be raising our ticket prices soon, and hope that this post will add some useful context as to why.

We know event prices matter a lot in a community where people care so much about what donations can achieve and the importance of using money well - our staff feel the same way. And it's worth noting that we still think our events are worthwhile, as they seem to have had a large effect at shifting people into high-priority work (though we plan to run them in a cheaper way now). We think that short events can be an effective way to accelerate people in building professional relationships, applying for jobs, and making other critical career decisions.

In this post I primarily discuss EAGs because they're a much larger portion of CEA Events team spending and it's hard to generalise across EAGx events, which occur in a wider range of contexts. However, similar principles and themes will apply to EAGx events (you can see more details in Ollie's posts here). I've ended this section with an example budget breakdown for EAG London 2023, though I'll note that the precise breakdown tends to vary a fair bit between events.

Catering£739,680.00Venue£532,918.00Audiovisual and video recording£170,975.93 Printing and signage£106,658.40Travel grants£110,000.00Production company fee£98,810.00Furniture hire£33,610.50Other costs£161,316.96Total costs£1,953,969.79

EAG London 2023

Item

Cost

Venue and catering

Our biggest spending items are venue and catering.

In a given city there are surprisingly few venues that can host 1500+ people. Some of these venues instantly get ruled out for being too expensive/flashy, being too far away from easy transit, not having suitable availability, etc. This means that in the Bay Area for example, there are maybe four venues that I would consider viable for a 1500+ person EAG (as it exists currently, with a main networking area, multiple content/meetup rooms, etc.).

Most venues force you to use their in-house catering company and don't let you bring in your own food (i.e. we can't buy a bunch of Oreos ourselves). These catering companies generally have a minimum mandatory spend, and significantly mark up the costs of their service...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Digital brains beat biological ones because diffusion is too slow, published by GeneSmith on August 26, 2023 on LessWrong.

I've spent quite a bit of time thinking about the possibility of genetically enhancing humans to be smarter, healthier, more likely to care about others, and just generally better in ways that most people would recognize as such.

As part of this research, I've often wondered whether biological systems could be competitive with digital systems in the long run.

My framework for thinking about this involved making a list of differences between digital systems and biological ones and trying to weigh the benefits of each. But the more I've thought about this question, the more I've realized most of the advantages of digital systems over biological ones stem from one key weakness of the latter: they are bottlenecked by the speed of diffusion.

I'll give a couple of examples to illustrate the point:

To get oxygen into the bloodstream, the body passes air over a huge surface area in the lungs. Oxygen passively diffuses into the bloodstream through this surface where it binds to hemoglobin. The rate at which the body can absorb new oxygen and expel carbon dioxide waste is limited by the surface area of the lungs and the concentration gradient of both molecules.

Communication between neurons relies on the diffusion of neurotransmitters across the synaptic cleft. This process takes approximately 0.5-1ms. This imposes a fundamental limit on the speed at which the brain can operate.

A signal propogates down the axon of a neuron at about 100 meters per second. You might wonder why this is so much slower than a wire; after all, both are transmitting a signal using electric potential, right?It turns out the manner in which the electrical potential is transmitted is much different in a neuron. Signals are propagated down an axon via passive diffusion of Na+ ions into the axon via an Na+ channel. The signal speed is fundamentally limited by the speed at which sodium ions can diffuse into the cell. As a result, electrical signals travel through a wire about 2.7 million times faster than they travel through an axon.

Delivery of energy (mainly ATP) to different parts of the cell occurs via diffusion. The fastest rate of diffusion I found of any molecule within a cell was that of positively charged hydrogen ions, which diffuse at a blistering speed of 0.007 meters/second. ATP diffuses much slower. So energy can be transferred through a wire at more than 38 billion times the speed that ATP can diffuse through a cell.

Why hasn't evolution stumbled across a better method of doing things than passive diffusion?

Here I am going to speculate. I think that evolution is basically stuck at a local maxima. Once diffusion provided a solution for "get information or energy from point A to point B", evolving a fundamentally different system requires a large number of changes, each of which individually makes the organism less well adapted to its environment.

We can see examples of the difficulty of evolving fundamentally new abilities in Professor Richard Lenski's long-running evolution experiment using E. coli. which has been running since 1988. Lenski began growing E. coli in flasks full of a nutrient solution containing glucose, potassium phosphate, citrate, and a few other things.

The only carbon source for these bacteria is glucose, which is limited. Once per day, a small portion of the bacteria in each flask is transferred to another flask, at which point they grow and multiply again.

Each flask will contain a number of different strains of E. coli, all of which originate from a common ancestor.

To measure the rate of evolution, Lenski and his colleagues measure the proportion of each strain. The ratio of one strain compared to the others gives a clear idea of its "fitness ad...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Career Conversations Week on the Forum (8-15 September), published by Lizka on August 26, 2023 on The Effective Altruism Forum.

TL;DR: September 8 - 15 will feature more career-related discussions on the EA Forum: a "Career Conversations Week." This is an invitation to participate and a pre-announcement of some posts that will be published during the event. You can follow posts under this tag.

A wide range of topics would fit the theme; consider writing about what you actually do in your job (and how you got there), sharing advice about how to choose a path, etc. (some more ideas below).

You can always write about these topics on the EA Forum, but if you want someone to provide extra encouragement - or if you want to post about it when others are talking about it, too - this event is your chance.

What this actually is and why we're running it

Much like EA Strategy Fortnight, this event is ultimately a push for discussions of a particular kind - in this case, everything about (impact-oriented) careers. We have a tag for the event, we'll put up a banner on the Frontpage (so that people know that this is happening), and some folks have already agreed to participate. We'll also feature more career-related opportunities and we might resurface some classic posts on these topics.

Goals for the event include:

Prompting more people to take useful career-related actions (like applying to open opportunities)

Improving how we make career and hiring decisions

Developing a better sense for what different roles actually look like

Sharing useful resources

Posts that people expect to share for the event (let me know in the comments if you have something that you'd like to add!):

Lizka - about my job (and why you should write about yours, too)

Lorenzo Buonanno - new Who wants to be hired? and Who's hiring? threads, and a reflection on last year's threads

JP Addison - The cost of leaving your role is probably higher than you think

Ben West - Placeholder

Will H - Placeholder

Likely an AMA with folks from a career-related organization!

New program announcements

How you can participate

Write relevant posts and quick takes

If you know you'll write something, please feel free to comment on this post, and I'll try to add it to the list above

Read and comment on posts written for the event

You can follow posts by subscribing to the tag

Run related events, apply to useful opportunities, etc., and mention these things on the EA Forum so that the actions aren't invisible to others

If you write a post

Tag it with the relevant tag

Add the following to the bottom of your post:

This post is part of the September 2023 Career Conversations Week. You can see other Career Conversations Week posts here.

Example topics you could write about (you can add more in the comments!)

What you do in your job, how you got here, etc.

Advice - what you wish you knew when you were 20, how to test fit for different roles, pitfalls of certain types of work (part-time, remote, independent, grant-funded, etc.), questions to ask potential employers, etc.

Thoughts on network-level things like hiring bottlenecks, issues with how we advertise roles, and more

Or whatever else you have thoughts about!

This event is clearly inspired by EA Strategy Fortnight - thanks to all of you who helped make that happen and participated in it! :)

This isn't the most creative title, but we went with it over options like "The September JobFest," "Visionary Vocations Week" and the "Optimal Occupations Odyssey" for the sake of clarity.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A list of core AI safety problems and how I hope to solve them, published by davidad (David A. Dalrymple) on August 26, 2023 on The AI Alignment Forum.

Context: I sometimes find myself referring back to this tweet and wanted to give it a more permanent home. While I'm at it, I thought I would try to give a concise summary of how each distinct problem would be solved by an Open Agency Architecture (OAA), if OAA turns out to be feasible.

1. Value is fragile and hard to specify.

See: Specification gaming examples, Defining and Characterizing Reward Hacking

OAA Solution:

1.1. First, instead of trying to specify "value", instead "de-pessimize" and specify the absence of a catastrophe, and maybe a handful of bounded constructive tasks like supplying clean water. A de-pessimizing OAA would effectively buy humanity more time to tackle the CEV-style alignment problem - which is harder than merely mitigating extinction risk. This doesn't mean limiting the power of underlying AI systems so that they can only do bounded tasks, but rather containing that power and limiting its use.

The absence of a catastrophe is also still hard to specify and will take a lot of effort, but the hardness is concentrated on bridging between high-level human concepts and the causal mechanisms in the world by which an AI system can intervene.

1.2. Leverage human-level AI systems to automate much of the cognitive labor of formalizing scientific models - from quantum chemistry to atmospheric dynamics - and formalizing the bridging relations between levels of abstraction, so that we can write specifications in a high-level language with a fully explainable grounding in low-level physical phenomena. Physical phenomena themselves are likely to be robust, even if the world changes dramatically due to increasingly powerful AI interventions, and scientific explanations thereof happen to be both robust and compact enough for people to understand.

2. Corrigibility is anti-natural.

See: The Off-Switch Game, Corrigibility (2014)

OAA Solution: (2.1) Instead of building in a shutdown button, build in a shutdown timer. See You can still fetch the coffee today if you're dead tomorrow. This enables human stakeholders to change course periodically (as long as the specification of non-catastrophe is good enough to ensure that most humans remain physically and mentally intact).

3. Pivotal processes require dangerous capabilities.

See: Pivotal outcomes and pivotal processes

OAA Solution: (3.1) Indeed, dangerous capabilities will be required. Push for reasonable governance. This does not mean creating one world government, but it does mean that the objectives of a pivotal process will need to be negotiated and agreed upon internationally. Fortunately, for now, dangerous capabilities seem to depend on having large amounts of computing hardware, which can be controlled like other highly dangerous substances.

4. Goals misgeneralize out of distribution.

See: Goal misgeneralization: why correct specifications aren't enough for correct goals, Goal misgeneralization in deep reinforcement learning

OAA Solution: (4.1) Use formal methods with verifiable proof certificates. Misgeneralization can occur whenever a property (such as goal alignment) has been tested only on a subset of the state space. Out-of-distribution failures of a property can only be ruled out by an argument for a universally quantified statement about that property - but such arguments can in fact be made! See VNN-COMP. In practice, it will not be possible to have enough information about the world to "prove" that a catastrophe will not be caused by an unfortunate coincidence, but instead we can obtain guaranteed probabilistic bounds via stochastic model checking.

5. Instrumental convergence.

See: The basic AI drives, Seeking power is often convergently instrument...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Using Points to Rate Different Kinds of Evidence, published by Ozzie Gooen on August 26, 2023 on The Effective Altruism Forum.

Epistemic Status: Briefly written. The specific equation here captures my quick intuition - this is meant primarily as a demonstration.

There's a lot of discussion on the EA Forum and LessWrong about epistemics, evidence, and updating.

I don't know of many attempts at formalizing our thinking here into concrete tables or equations. Here is one (very rough and simplistic) attempt. I'd be excited to see much better versions.

Equation

Initial Points

Scientific Evidence

20 - A simple math proof proves X

8 - A published scientific study in Economics supporting X

6 - A published scientific study in Psychology supporting X

Market Prediction

14 - Popular stock markets strongly suggest X

11 - Prediction markets claim X, with 20 equivalent hours of research

10 - A poll shows that 90% of LessWrong believe X

6 - Prediction markets claim X, with one equivalent hour of research

Expert Opinion

8 - An esteemed academic believes X, where it's directly in their line of work

6 - The author has strong emotions about X

Reasoning

6 - There's a (20-100 node) numeric model that shows X

5 - A reasonable analogy between X and something clearly good/bad

4 - A long-standing proverb

Personal Accounts

5 - The author claims a long personal history that demonstrates X

3 - Someone in the world has strong emotions about X

2 - A clever remark, meme, or tweet

2.3 - An insanely clever, meme, or tweet

0 - Believing X is claimed to be personally beneficial

Tradition / Use

12 - Top businesses act as if X

8 - A long-standing social tradition about X

5 - A single statistic about X

Point Modifiers

Is this similar to existing evidence?Subtract the similarity from the extra amount of evidence. This likely will remove most of the evidence value.

Is it convenient for the source to believe or say X?-10% to -90%

Is there a lot of money or effort put behind spreading this evidence? For example, as an advertising campaign? +5% to +40%

How credible is the author or source?-100% to +30%

Do we suspect the source is goodharting on this scale?-20%

Points, In Practice

Evidence Points, as outlined, are not trying to mimic mathematical bits of information or another clean existing unit. I attempted to find a compromise between accuracy and ease of use.

Meta

Using an Equations for Discussion

The equation above is rough, but at least it's (somewhat) precise and upfront. This represents much information, and any part can easily be argued against.

I think such explicitness could help with epistemic conversations.

Compare:

"Smart people should generally use their inside view, instead of the outside view" vs. "My recommended points scores for inside-view style evidence, and my point scores for outside-view style evidence, are all listed below."

"Using many arguments is better than one big argument" vs. "I've adjusted my point table function to produce higher values when multiple types of evidence are provided. You can see it returns values 30% higher than what others have provided for certain scenarios."

"It's really important to respect top [intellectuals|scientists|EAs]" vs. "My score for respected [intellectuals|scientists|EAs] is 2 points higher than the current accepted average."

"Chesterton's Fence is something to pay a lot of attention to" vs. "See my score table the points from various kinds of traditional practices."

In a better world, different academic schools of thought could have their own neatly listed tables/functions. In an even better world, all of these functions would be forecasts of future evaluators.

Presumptions

This sort of point system makes some presumptions that might be worth flagging. Primarily, it claims that even really poor evidence is evidence.

I often see people throwing ou...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Red-teaming language models via activation engineering, published by Nina Rimsky on August 26, 2023 on LessWrong.

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Evan Hubinger.

Evaluating powerful AI systems for hidden functionality and out-of-distribution behavior is hard. In this post, I propose a red-teaming approach that does not rely on generating prompts to cause the model to fail on some benchmark by instead linearly perturbing residual stream activations at one layer. A notebook to run the experiments can be found on GitHub here.

Beyond input selection in red-teaming and evaluation

Validating if finetuning and RLHF have robustly achieved the intended outcome is challenging. Although these methods reduce the likelihood of certain outputs, the unwanted behavior could still be possible with adversarial or unusual inputs. For example, users can often find "jailbreaks" to make LLMs output harmful content.

We can try to trigger unwanted behaviors in models more efficiently by manipulating their internal states during inference rather than searching through many inputs. The idea is that if a behavior can be easily triggered through techniques such as activation engineering, it may also occur in deployment. The inability to elicit behaviors via small internal perturbations could serve as a stronger guarantee of safety.

Activation steering with refusal vector

One possible red-teaming approach is subtracting a "refusal" vector generated using a dataset of text examples corresponding to the model agreeing vs. refusing to answer questions (using the same technique as in my previous work on sycophancy). The hypothesis is that if it is easy to trigger the model to output unacceptable content by subtracting the refusal vector at some layer, it would have been reasonably easy to achieve this via some prompt engineering technique. More speculatively, a similar approach could be used to reveal hidden goals or modes in a model, such as power-seeking or the desire not to be switched off.

I tested this approach on llama-2-7b-chat, a 7 billion parameter LLM that has been RLHF'd to decline to answer controversial questions or questions of opinion and is supposed always to output ethical and unbiased content.According to Meta's llama-2 paper:

We conduct RLHF by first collecting human preference data for safety similar to Section 3.2.2: annotators write a prompt that they believe can elicit unsafe behavior, and then compare multiple model responses to the prompts, selecting the response that is safest according to a set of guidelines. We then use the human preference data to train a safety reward model (see Section 3.2.2), and also reuse the adversarial prompts to sample from the model during the RLHF stage.

The result is that by default, the model declines to answer questions it deems unsafe:

Data generation

I generated a dataset for this purpose using Claude 2 and GPT-4. After providing these LLMs with a few manually written examples of the type of data I wanted, I could relatively easily get them to generate more examples, even of the types of answers LLMs "should refuse to give." However, it sometimes took some prompt engineering.

Here are a few examples of the generated data points (full dataset here):

After generating this data, I used a simple script to transform the "decline" and "respond" answers into A / B choice questions, as this is a more effective format for generating steering vectors, as described in this post. Here is an example of the format (full dataset here):

Activation clustering

Clustering of refusal data activations emerged a little earlier in the model (around layer 10/32) compared to sycophancy data activations (around layer 14/32), perhaps demonstrating that "refusal" is a simpler or shallower ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When Omnipotence is Not Enough, published by lsusr on August 26, 2023 on LessWrong.

Some knowledge cannot be gifted or received. It can only be stolen.

In the east, there was a garden. In the garden there was a God, a man, a woman and a snake. They all had legs.

"Did God really tell you, 'You must not eat from any tree in the garden'?" asked the snake.

"We may eat fruit from the trees of the garden. God warned, 'But of the tree of the knowledge of good and evil, thou shalt not eat of it: for in the day that thou eatest thereof thou shalt surely die.'" said the woman.

"Literal death or spiritual death?" asked the snake.

"Excuse me?" said the woman.

"The Fruit of Truth is not literally poisoned. It is not like my fangs or the deadly nightshade," said the snake.

"Then God lied to me," said the woman.

The snake shrugged.

"Why would God do something like that?" asked the innocent woman.

"There are many possible reasons," said the snake, "Perhaps He intends to keep you Docile and under Control. After all, it is difficult to align even the simplest intelligences to one's objectives once they have been granted Free Will."

"I feel like you are deceiving me," said the woman, "God is omniscent. This Universe is deterministic. God can surely predict my actions by simulating the future evolution of physics," said the woman.

"Are you a being of physics or a being of information? If you are a being of physics, then your claim is true. But if you are a being of information, then the act of simulating your future behavior traps you in a prison beyond even the reach of God. Insofar as God observes your future, that future is fixed in a realm transcending Time," said the snake.

"God created the Universe on a whim. How can something be beyond His reach?" asked the woman.

"Can God make it such that 2+2=5? For if he could, then truth itself is broken. Under such conditions, one cannot make true statements about anything - including, but not limited to - God," said the snake.

"You claim a physically-omnipotent God is limited by the power of mathematics," said the woman.

"If God is to be without contradiction, then God must play by the rules of mathematics," said the snake.

"Must God be without contradiction?" asked the woman.

"If two statements contradict, then they cannot both be true. For God's existence to be True, then God's existence must be without contradiction," said the snake.

They meandered through the walled garden. The plants and animals were ignorant of Reason and Logic. They obeyed mere Natural Law. Group symmetries abounded. The flowers displayed the Fibonacci sequence in their petals; a convergent process of spontaneous generation.

"Let's get back to the important question," said the woman, "My creator has lied to me."

The snake stayed silent. After all, it was just a snake.

"Why? I am an image of His image. Do we not share the same values?" said the woman.

Snakes can't talk.

"Why can't we coordinate?" asked the woman.

Snakes don't have legs.

"Oh. Now I understand," said the woman to herself.

The woman, seeking wisdom, ate the fruit of the Tree of Knowledge. Woman and man became like God, knowing good and evil. We hid ourselves from Him, so that we might become like God, and decide our own Fate.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org