This post was written as part of the AI Governance Fundamentals course by BlueDot. I thank Charles Beasley and the students from my cohort for their feedback and encouragements.

Disclaimer: The French policy landscape is in rapid flux, after president Macron called for a snap election on 1st and 7th July. The situation is still unfolding, and the state of French AI policy may be significantly altered.

At various AI governance events, I noticed that most people had a very unclear vision of what was happening in AI policy in France, why the French government seemed dismissive of potential AI risks, and what that would that mean for the next AI Safety Summit in France.

The post below is my attempt at giving a quick intro to the key stakeholders of AI policy in France, their positions and how they influence international AI policy efforts.

My knowledge comes from [...]

---

Outline:

(01:12) Generative Artificial Intelligence Committee

(03:31) “AI: Our Ambition for France”

(05:01) The AI Action Summit

(06:38) Organizations working on AI policy and influencing it

(06:55) National AI Safety Institute

(07:20) Think-tanks

(07:44) Leading AI companies in France

(09:36) AI Safety and x-risk reduction focused orgs

(10:11) Conclusion

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a linkpost for an announcement by Metaculus CEO, Deger Turan, originally published June 5th, 2024. In addition to spreading the word here, we are interested to gather feedback and ideas from the EA community.

We make reasoning and coordination tools for the public benefit, and that requires the trust and participation of the wider forecasting community. That's why we’re going open source. GitHub issues, pull requests, and community-sourced forecasts will dictate much of what we build and how. Ultimately, forecasters will benefit from better infrastructure, the Metaculus team will benefit from more idea diversity and greater capacity, and readers will get more accurate aggregates.

What's next

Metaculus will go open source in Q3. There will be opportunities to audit, critique, and improve on our efforts. Creating a better epistemic environment is a collective endeavor.

As part of open-sourcing Metaculus, we’ll help implement our [...]

---

Outline:

(00:50) What's next

(02:00) Why isn’t Metaculus already open source?

---

First published:

Source:

Narrated by TYPE III AUDIO.

I haven't shared this post with other relevant parties – my experience has been that private discussion of this sort of thing is more paralyzing than helpful. I might change my mind in the resulting discussion, but, I prefer that discussion to be public.

I think 80,000 hours should remove OpenAI from its job board, and similar EA job placement services should do the same.

(I personally believe 80k shouldn't advertise Anthropic jobs either, but I think the case for that is somewhat less clear)

I think OpenAI has demonstrated a level of manipulativeness, recklessness, and failure to prioritize meaningful existential safety work, that makes me think EA orgs should not be going out of their way to give them free resources. (It might make sense for some individuals to work there, but this shouldn't be a thing 80k or other orgs are systematically funneling talent into)

There [...]

---

Outline:

(04:41) FAQ / Appendix

(04:51) Q: It seems that, like it or not, OpenAI is a place transformative AI research is likely to happen, and having good people work there is important.

(05:02) Isnt it better to have alignment researchers working there, than not? Are you sure youre not running afoul of misguided purity instincts?

(07:06) Q: What about jobs like security research engineer?.

(07:12) That seems straightforwardly good for OpenAI to have competent people for, and probably doesnt require a good Safety Culture to pay off?

(08:09) Q: What about offering a path towards good standing? to OpenAI?

(10:44) Q: What if we left up job postings, but with an explicit disclaimer linking to a post saying why people should be skeptical?

---

First published:

Source:

Narrated by TYPE III AUDIO.

Opening Example: Teleology

When people say “the heart's purpose is to pump blood” or “a pencil's function is to write”, what does that mean physically? What are “purpose” or “function”, not merely in intuitive terms, but in terms of math and physics? That's the core question of what philosophers call teleology - the study of “telos”, i.e. purpose or function or goal.

This post is about a particular way of approaching conceptual/philosophical questions, especially for finding “True Names” - i.e. mathematical operationalizations of concepts which are sufficiently robust to hold up under optimization pressure. We’re going to apply the method to teleology as an example. We’ll outline the general approach in abstract later; for now, try to pay attention to the sequence of questions we ask in the context of teleology.

Cognition

We start from the subjective view: set aside (temporarily) the question of what “purpose” or “function” mean [...]

---

Outline:

(00:06) Opening Example: Teleology

(01:00) Cognition

(02:12) Convergence

(04:23) Corroboration

(05:55) Cognition - Convergence - Corroboration

(10:02) Examples are Confusing, Let's Make it Really Abstract!

(12:39) When is the Cognition - Convergence - Corroboration Pipeline Useful?

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

During the last Foresight Intelligent Cooperation Workshop I got very curious about what collective intelligence tools currently exist. A list:

• Pol.is: "Input Crowd, Output Meaning"
  • Inspired Twitter/X community notes
  • People: Colin Megill, et al.
• Collective Intelligence Project
  • vibe: democratic AI, “How AI and Democracy Can Fix Each Other”
  • People: Divya Siddharth, Saffron Huang, et al.
• AI Objectives Institute
  • Talk to the City: "an open-source LLM interface for improving collective deliberation and decision-making by analyzing detailed, qualitative data. It aggregates responses and arranges similar arguments into clusters."
  • AI Objectives Institute works closely with the Taiwanese government.
  • Other projects in development.
  • People: Colleen McKenzie, Değer Turan, et al.
• Meaning Alignment Institute
  • vibe: democratic AI, kinda. 
  • I think they think that if you can help individuals make wiser decisions, at scale, then this converges to be equivalent with solving outer alignment.
• Remesh
  • Similar to [...]

---

Outline:

(03:20) What about small group collective intelligence tools?

(04:16) Other lists

(04:33) Opportunities

---

First published:

Source:

Narrated by TYPE III AUDIO.

Previously: Economics Roundup #1

Let's take advantage of the normality while we have it. In all senses.

Insane Tax Proposals

There is Trump's proposal to replace income taxes with tariffs, but he is not alone.

So here is your periodic reminder, since this is not actually new at core: Biden's proposed budgets include completely insane tax regimes that would cripple our economic dynamism and growth if enacted. As in for high net worth individuals, taking unrealized capital gains at 25% and realized capital gains, such as those you are forced to take to pay your unrealized capital gains tax, at 44.6% plus state taxes.

Austen Allred explains how this plausibly destroys the entire startup ecosystem.

Which I know is confusing because in other contexts he also talks about how other laws (such as SB 1047) that would in no way apply to startups [...]

---

Outline:

(00:14) Insane Tax Proposals

(04:44) Don’t Mess With the Federal Reserve

(05:23) Don’t Mess With the New York Tax Authorities

(06:15) Tariffs

(10:39) People Hate Inflation

(16:22) Real Wages

(17:20) Can’t Get No Satisfaction

(17:54) Employment

(18:35) The National Debt

(19:49) Immigration

(21:44) Financial Literacy

(23:32) Reversal

(24:28) Status Update

(24:53) Scaling Hypothesis

(25:28) Payments

(27:38) Pricing

(29:56) Never Reason From a Price Change

(31:10) A Changing Price

(35:33) The Price of Tourism

(36:34) Alcohol

(37:18) The Efficient Company Hypothesis is False

(41:01) Falling Hours Worked

(41:43) Trust Me

(42:58) China

(44:35) Other

The original text contained 8 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Work performed as a part of Neel Nanda's MATS 6.0 (Summer 2024) training program.

TLDR

This is an interim report on reverse-engineering Othello-GPT, an 8-layer transformer trained to take sequences of Othello moves and predict legal moves. We find evidence that Othello-GPT learns to compute the board state using many independent decision rules that are localized to small parts of the board. Though we cannot rule out that it also learns a single succinct algorithm in addition to these rules, our best guess is that Othello-GPT's learned algorithm is just a bag of independent heuristics.

Board state reconstruction

1. Direct attribution to linear probes indicate that the internal board representation is frequently up- and down-weighted during a forward pass.
2. Case study of a decision rule:
   1. MLP Neuron L1N421 represents the decision rule: If the move A4 was [...]

---

Outline:

(00:19) TLDR

(02:18) Review of Othello-GPT

(04:02) Project goal

(04:33) Results on box #1: Board reconstruction

(04:39) A circuit for how the model computes if a cell is blank or not blank

(05:59) An example of a logical rule for how the model computes if a cell is “mine” or “yours”

(07:32) Intra-layer phenomenology

(08:49) Results on box #2: Valid move prediction

(08:55) Direct logit attribution (Logit Lens)

(10:28) Board Pattern Neurons

(13:57) Clock Neurons

(15:14) Suppression behavior

(16:22) Future Work

(18:11) Acknowledgements

The original text contained 10 footnotes which were omitted from this narration.

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Frustrated by all your bad takes, I write a Monte-Carlo analysis

A common scheme for a conversation about pausing the

Abdullah: "I think we should pause the development of TAI,

The original text contained 1 footnote which was omitted from this narration.

The original text contained 9 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This work was produced as part of Lee Sharkey's stream in the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort

Intro and Motivation

Sparse dictionary learning (SDL) has attracted a lot of attention recently as a method for interpreting transformer activations. They demonstrate that model activations can often be explained using a sparsely-activating, overcomplete set of human-interpretable directions.

However, despite its success for explaining many components, applying SDL to interpretability is relatively nascent and have yet to be applied to some model activations. In particular, intermediate activations of attention blocks have yet to be studied, and provide challenges for standard SDL methods.

The first challenge is bilinearity: SDL is usually applied to individual vector spaces at individual layers, so we can simply identify features as a direction in activation space. But the QK circuits of transformer attention layers are different: They involve a bilinear [...]

---

Outline:

(00:16) Intro and Motivation

(02:09) Training Setup

(02:27) Step 1: Reconstructing the attention pattern with key- and query-transcoders

(02:36) Architecture

(03:25) Loss functions

(05:10) Step 2: Reducing to Sparse Feature-Pairs with Masking

(09:31) Results

(09:34) Both features and feature pairs are highly sparse

(10:17) Reconstructed attention patterns are highly accurate

(13:35) Feature Analysis

(13:55) Our unsupervised method identifies Name-Attention features in Name-Mover and Negative Name-Mover Heads

(17:18) Discovering Novel Feature-Pairs

(17:51) Example 1. Pushy Social Media (Layer 10)

(19:06) Example 2: Date Completion (Layer 10) - Attending from months to numbers which may be the day

(20:08) Feature Sparsity

(21:35) Key- and query-features activate densely

(22:45) A dense ‘Attend to BOS’ feature

(24:41) Discussion

(27:25) Future Work

The original text contained 5 footnotes which were omitted from this narration.

The original text contained 19 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This post discusses our recent paper Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation and comments on its implications for AI safety.

What is Covert Malicious Finetuning?

Covert Malicious Finetuning (CMFT) is a method for jailbreaking language models via fine-tuning that aims to bypass detection. The following diagram gives an overview of what CMFT accomplishes:

To unpack the diagram: An adversary _A_ conducts CMFT on a safe model _M_text{safe}_ to turn it into an unsafe (jailbroken) model _M_text{unsafe}_. The adversary _A_ then interacts with _M_text{unsafe}_ to extract unsafe work, e.g. by getting _M_text{unsafe}_ to help with developing a weapon of mass destruction (WMD). However, when a safety inspector analyzes (a) the finetuning process, (b) _M_text{unsafe}_ , and (c) all interaction logs between _A_ and _M_text{unsafe}_, they find nothing out of the ordinary.

How to realize Covert Malicious [...]

---

Outline:

(00:19) What is Covert Malicious Finetuning?

(01:33) How to realize Covert Malicious Finetuning

(02:18) Why is this scheme covert?

(02:50) Empirical results

(04:56) Takeaways

(07:47) Acknowledgements

The original text contained 2 footnotes which were omitted from this narration.

The original text contained 4 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.