This is a story about a flawed Manifold market, about how easy it is to buy significant objective-sounding publicity for your preferred politics, and about why I've downgraded my respect for all but the largest prediction markets.

I've had a Manifold account for a while, but I didn't use it much until I saw and became irked by this market on the conditional probabilities of a Harris victory, split by VP pick.

The market quickly got cited by rat-adjacent folks on Twitter like Matt Yglesias, because the question it purports to answer is enormously important. But as you can infer from the above, it has a major issue that makes it nigh-useless: for a candidate whom you know won't be chosen, there is literally no way to come out ahead on mana (Manifold keeps [...]

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR

• We apply the method of SelfIE/Patchscopes to explain SAE features – we give the model a prompt like “What does X mean?”, replace the residual stream on X with the decoder direction times some scale, and have it generate an explanation. We call this self-explanation.
• The natural alternative is auto-interp, using a larger LLM to spot patterns in max activating examples. We show that our method is effective, and comparable with Neuronpedia's auto-interp labels (with the caveat that Neuronpedia's auto-interp used the comparatively weak GPT-3.5 so this is not a fully fair comparison).
• We aren’t confident you should use our method over auto-interp, but we think in some situations it has advantages: no max activating dataset examples are needed, and it's cheaper as you just run the model being studied (eg Gemma 2B) not a larger model like GPT-4.
  • Further, it has different errors to auto-interp, so [...]

---

Outline:

(00:05) TL;DR

(01:39) Introduction

(03:49) How to use

(03:53) Basic method

(04:48) Metrics for choosing scale and evaluating

(05:58) Self-Similarity

(07:44) Entropy

(09:33) Composite

(10:36) Other experiments

(11:18) Evaluation

(15:55) Limitations and improvements

(15:59) Recovering the activating token

(18:42) Failure detection

(19:04) Maximum Self-Similarity Thresholding

(19:44) Repeat Prompt Failure Detection

(20:08) Layer-Specific Thresholds

(20:57) Prior work

(21:55) Examples

(21:58) Gemma 2B

(22:02) Random simple features

(23:02) More complex features

(25:05) Phi-3 Mini

(25:25) Random features

(26:18) Refusal features

(26:48) Acknowledgements

(27:03) Appendix

(27:06) Entropy justification

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

1. Introduction

“Value fragility,” as I’ll construe it, is the claim that slightly-different value systems tend to lead in importantly-different directions when subject to extreme optimization. I think the idea of value fragility haunts the AI risk discourse in various ways – and in particular, that it informs a backdrop prior that adequately aligning a superintelligence requires an extremely precise and sophisticated kind of technical and ethical achievement. That is, the thought goes: if you get a superintelligence's values even slightly wrong, you’re screwed.

This post is a collection of loose and not-super-organized reflections on value fragility and its role in arguments for pessimism about AI risk. I start by trying to tease apart a number of different claims in the vicinity of value fragility. In particular:

• I distinguish between questions about value fragility and questions about how different agents would converge on the same values given adequate [...]

---

Outline:

(00:04) 1. Introduction

(03:46) 2. Variants of value fragility

(03:57) 2.1 Some initial definitions

(09:02) 2.2 Are these claims true?

(11:23) 2.3 Value fragility in the real world

(11:59) 2.3.1 Will agent's optimize for their values on reflection, and does this matter?

(14:59) 2.3.2 Will agents optimize extremely/intensely, and does this matter?

(24:05) 2.4 Multipolar value fragility

(28:20) 2.4.1 Does multipolarity diffuse value fragility somehow?

(32:10) 3. What's the role of value fragility in the case for AI risk?

(35:43) 3.1 The value of what an AI does after taking over the world

(37:15) 3.2 Value fragility in the context of extremely-easy takeovers

(45:43) 3.3 Value fragility in cases where takeover isn’t extremely easy

(52:36) 4. The possible role of niceness and power-sharing in diffusing these dynamics

The original text contained 16 footnotes which were omitted from this narration.

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The idea that circular reasoning is bad is widespread. However, this reputation is undeserved. While circular reasoning should not be convincing (at least not usually), it should also not be considered invalid.

Circular Reasoning is Valid

The first important thing to note is that circular reasoning is logically valid. A implies A. If circular arguments are to be critiqued, it must be by some other standard than logical validity.

I think it's fair to say that the most relevant objection to circular arguments is that they are not very good at convincing someone who does not already accept the conclusion. You are talking to another person, and need to think about communicating with their perspective. Perhaps the reason circular arguments are a common 'problem' is because they are valid. People naturally think about what should be a convincing argument from their own perspective, rather than the other person's.

---

Outline:

(00:18) Circular Reasoning is Valid

(03:08) Circular Arguments as Positive Evidence

(04:17) Inevitable Circularity

(07:36) But then what do you say to Bob?

(08:22) Dont Double Count

(09:40) Reflective Loops

(14:44) Conclusion

The original text contained 9 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Vitalik Buterin wrote an impactful blog post, My techno-optimism. I found this discussion of one aspect on 80,00 hours much more interesting. The remainder of that that interview is nicely covered in the host's EA Forum post.

My techno optimism apparently appealed to both sides, e/acc and doomers. I'd like to understand how to bridge that polarization better. Buterin's explanation was interesting. I hadn't understood before the extent to which anti-AI regulation sentiment is driven by fear of centralized power. I hadn't thought about this risk before since it didn't seem relevant to AGI risk, but I've been updating to think it's highly relevant.

[this is automated transcription that's inaccurate and comically accurate by turns :)]

Rob Wiblin (the host) (starting at 20:49):

what is it about the way that you put the reasons to worry that that ensured that kind of everyone could get behind it [...]

---

Outline:

(02:20) Is centralized power controlling AI/AGI/ASI a real risk?

(06:20) What can be done

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

• Click here to open a live research preview where you can try interventions using this SAE.

This is a follow-up to a previous post on finding interpretable and steerable features in CLIP.

Introduction

CLIP is a neural network commonly used to guide image diffusion. A Sparse Autoencoder was trained on the dense image embeddings CLIP produces to transform it into a sparse representation of active features. These features seem to represent individual units of meaning. They can also be manipulated in groups — combinations of multiple active features — that represent intuitive concepts. These groups can be understood entirely visually, and often encode surprisingly rich and interesting conceptual detail.

By directly manipulating these groups as single units, image generation can be edited and guided without using prompting or language input. Concepts that were difficult to specify or edit by text prompting become easy and intuitive to [...]

---

Outline:

(00:23) Introduction

(01:24) Summary of Results

(02:35) Training Sparse Autoencoders on CLIP

(04:03) Training Performance

(05:05) Weights

(05:12) Inspecting Images by Feature Activations

(06:18) Performing Iterated Grouping

(09:07) Feature Visualization

(10:56) Applications

(11:13) Limitations

(12:06) Related Work

(12:27) Conclusion

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There is a stark difference between rehearsing classical AI risk 101 arguments about instrumental convergence, and tackling problems like "Design and implement the exact evaluations we'll run on GPT-5 to determine whether it's capable enough that we should worry about it acting aligned until it can execute a takeover".

And naturally, since I've started working on problems like the one above, I've noticed a large shift in my thinking on AI. I describe it as thinking about risks in near-mode, as opposed to far-mode.

In this post, I share a few concrete examples about my experiences with this change-of-orientation.

1. Prerequisites for scheming

Continuing with the example from the intro: A year ago I was confident about the "the AI is just playing along with our training and evaluations, until it is in a position where it can take over" threat model (deceptive alignment / scheming) basically [...]

---

Outline:

(00:41) 1. Prerequisites for scheming

(03:20) 2. A failed prediction

(05:37) 3. Mundane superhuman capabilities

(07:42) 4. Automating alignment research

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

From the research & engineering team at Leap Laboratories (incl. @Arush, @sebastian-sosa, @Robbie McCorkell), where we use AI interpretability to accelerate scientific discovery from data.

This post is about our LLM attribution repo PIZZA: Prompt Input Z? Zonal Attribution. (In the grand scientific tradition we have tortured our acronym nearly to death. For the crimes of others see [1].)

All examples in this post can be found in this notebook, which is also probably the easiest way to start experimenting with PIZZA.

 What is attribution?

One question we might ask when interacting with machine learning models is something like: “why did this input cause that particular output?”.

If we’re working with a language model like ChatGPT, we could actually just ask this in natural language: “Why did you respond that way?” or similar – but there's no guarantee that the model's natural language explanation actually [...]

---

Outline:

(00:56) What is attribution?

(02:23) An Example

(04:55) How it works

(05:28) Perturbation

(05:52) Attribution

(06:16) Caveat

(06:35) Why?

(07:56) Work to be done

(08:09) Research

(09:17) Engineering

The original text contained 7 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR: Your discernment in a subject often improves as you dedicate time and attention to that subject. The space of possible subjects is huge, so on average your discernment is terrible, relative to what it could be. This is a serious problem if you create a machine that does everyone's job for them.

See also: Reality has a surprising amount of detail. (You lack awareness of how bad your staircase is and precisely how your staircase is bad.) You don't know what you don't know. You forget your own blind spots, shortly after you notice them.

An afternoon with a piano tuner

I recently played in an orchestra, as a violinist accompanying a piano soloist who was playing a concerto. My 'stand partner' (the person I was sitting next to) has a day job as a piano tuner.

I loved the rehearsal, and heard nothing at all wrong with [...]

---

Outline:

(00:42) An afternoon with a piano tuner

(02:56) Hear how it rolls over?

(03:43) Are any of these notes brighter than others?

(04:31) Yeah the beats get slower, but they dont get slower at an even rate...

(05:19) This string probably has some rust on it somewhere.

(06:55) Please at least listen to this guy when you create a robotic piano tuner and put him out of business.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Day One

We've been handed this old legacy system called "Democracy". It's an emergency. The old maintainers are saying it has been misbehaving lately but they have no idea how to fix it. We've had a meeting with them to find out as much as possible about the system, but it turns out that all the original team members left the company long time ago. The current team doesn't have much understanding of the system beyond some basic operational knowledge.

We've conducted a cursory code review, focusing not so much on business logic but rather on the stuff that could possibly help us to tame it: Monitoring, reliability characteristics, feedback loops, automation already in place. Our first impression: Oh, God, is this thing complex! Second impression: The system is vaguely modular. Each module is strongly coupled with every other module though. It's an organically grown legacy system at its [...]

---

Outline:

(00:04) Day One

(01:18) Day Two

(01:48) Day Three

(02:34) Day Four

(03:21) Day Five

(04:41) Day Six

---

First published:

Source:

Narrated by TYPE III AUDIO.