AI researchers often complain about the poor coverage of their work in the news media. But why is this happening, and how can it be fixed? In this episode, I speak with Shakeel Hashim about the resource constraints facing AI journalism, the disconnect between journalists' and AI researchers' views on transformative AI, and efforts to improve the state of AI journalism, such as Tarbell and Shakeel's newsletter, Transformer.

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

The transcript: https://axrp.net/episode/2025/01/05/episode-38_4-shakeel-hashim-ai-journalism.html

FAR.AI: https://far.ai/

FAR.AI on X (aka Twitter): https://x.com/farairesearch

FAR.AI on YouTube: https://www.youtube.com/@FARAIResearch

The Alignment Workshop: https://www.alignment-workshop.com/

Topics we discuss, and timestamps:

01:31 - The AI media ecosystem

02:34 - Why not more AI news?

07:18 - Disconnects between journalists and the AI field

12:42 - Tarbell

18:44 - The Transformer newsletter

Links:

Transformer (Shakeel's substack): https://www.transformernews.ai/

Tarbell: https://www.tarbellfellowship.org/

Episode art by Hamish Doodles: hamishdoodles.com

The Machine Intelligence Research Institute has recently shifted its focus to "technical governance". But what is that actually, and what are they doing? In this episode, I chat with Peter Barnett about his team's work on studying what evaluations can and cannot do, as well as verifying international agreements on AI development.

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

The transcript: https://axrp.net/episode/2024/12/14/episode-38_4-peter-barnett-technical-governance-at-miri.html

FAR.AI: https://far.ai/

FAR.AI on X (aka Twitter): https://x.com/farairesearch

 FAR.AI on YouTube: https://www.youtube.com/@FARAIResearch

The Alignment Workshop: https://www.alignment-workshop.com/

Topics we discuss, and timestamps:

01:25 - MIRI's technical governance team

03:43 - Misuse evaluations

06:37 - Misalignment evaluations

11:29 - Verifying international agreements

13:30 - Difficulties in compute monitoring

16:44 - More on MIRI's technical governance team

Links:

MIRI Technical Governance Team: https://techgov.intelligence.org/

What AI evaluations for preventing catastrophic risks can and cannot do: https://techgov.intelligence.org/research/what-ai-evaluations-for-preventing-catastrophic-risks-can-and-cannot-do

Declare and Justify: Explicit assumptions in AI evaluations are necessary for effective regulation: https://arxiv.org/abs/2411.12820

Mechanisms to Verify International Agreements About AI Development: https://techgov.intelligence.org/research/mechanisms-to-verify-international-agreements-about-ai-development

Revisiting algorithmic progress: https://epoch.ai/blog/revisiting-algorithmic-progress

Episode art by Hamish Doodles: hamishdoodles.com

Lots of people in the AI safety space worry about models being able to make deliberate, multi-step plans. But can we already see this in existing neural nets? In this episode, I talk with Erik Jenner about his work looking at internal look-ahead within chess-playing neural networks.

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

The transcript: https://axrp.net/episode/2024/12/12/episode-38_3-erik-jenner-learned-look-ahead.html

FAR.AI: https://far.ai/

FAR.AI on X (aka Twitter): https://x.com/farairesearch

FAR.AI on YouTube: https://www.youtube.com/@FARAIResearch

The Alignment Workshop: https://www.alignment-workshop.com/

Topics we discuss, and timestamps:

00:57 - How chess neural nets look into the future

04:29 - The dataset and basic methodology

05:23 - Testing for branching futures?

07:57 - Which experiments demonstrate what

10:43 - How the ablation experiments work

12:38 - Effect sizes

15:23 - X-risk relevance

18:08 - Follow-up work

21:29 - How much planning does the network do?

Research we mention:

Evidence of Learned Look-Ahead in a Chess-Playing Neural Network: https://arxiv.org/abs/2406.00877

Understanding the learned look-ahead behavior of chess neural networks (a development of the follow-up research Erik mentioned): https://openreview.net/forum?id=Tl8EzmgsEp

Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT: https://arxiv.org/abs/2310.07582

Episode art by Hamish Doodles: hamishdoodles.com

The 'model organisms of misalignment' line of research creates AI models that exhibit various types of misalignment, and studies them to try to understand how the misalignment occurs and whether it can be somehow removed. In this episode, Evan Hubinger talks about two papers he's worked on at Anthropic under this agenda: "Sleeper Agents" and "Sycophancy to Subterfuge".

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

The transcript: https://axrp.net/episode/2024/12/01/episode-39-evan-hubinger-model-organisms-misalignment.html

Topics we discuss, and timestamps:

0:00:36 - Model organisms and stress-testing

0:07:38 - Sleeper Agents

0:22:32 - Do 'sleeper agents' properly model deceptive alignment?

0:38:32 - Surprising results in "Sleeper Agents"

0:57:25 - Sycophancy to Subterfuge

1:09:21 - How models generalize from sycophancy to subterfuge

1:16:37 - Is the reward editing task valid?

1:21:46 - Training away sycophancy and subterfuge

1:29:22 - Model organisms, AI control, and evaluations

1:33:45 - Other model organisms research

1:35:27 - Alignment stress-testing at Anthropic

1:43:32 - Following Evan's work

Main papers:

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training: https://arxiv.org/abs/2401.05566

Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models: https://arxiv.org/abs/2406.10162

Anthropic links:

Anthropic's newsroom: https://www.anthropic.com/news

Careers at Anthropic: https://www.anthropic.com/careers

Other links:

Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research: https://www.alignmentforum.org/posts/ChDH335ckdvpxXaXX/model-organisms-of-misalignment-the-case-for-a-new-pillar-of-1

Simple probes can catch sleeper agents: https://www.anthropic.com/research/probes-catch-sleeper-agents

Studying Large Language Model Generalization with Influence Functions: https://arxiv.org/abs/2308.03296

Stress-Testing Capability Elicitation With Password-Locked Models [aka model organisms of sandbagging]: https://arxiv.org/abs/2405.19550

Episode art by Hamish Doodles: hamishdoodles.com

You may have heard of singular learning theory, and its "local learning coefficient", or LLC - but have you heard of the refined LLC? In this episode, I chat with Jesse Hoogland about his work on SLT, and using the refined LLC to find a new circuit in language models.

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

The transcript: https://axrp.net/episode/2024/11/27/38_2-jesse-hoogland-singular-learning-theory.html

FAR.AI: https://far.ai/

FAR.AI on X (aka Twitter): https://x.com/farairesearch

 FAR.AI on YouTube: https://www.youtube.com/@FARAIResearch

The Alignment Workshop: https://www.alignment-workshop.com/

Topics we discuss, and timestamps:

00:34 - About Jesse

01:49 - The Alignment Workshop

02:31 - About Timaeus

05:25 - SLT that isn't developmental interpretability

10:41 - The refined local learning coefficient

14:06 - Finding the multigram circuit

Links:

Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient: https://arxiv.org/abs/2410.02984

Investigating the learning coefficient of modular addition: hackathon project: https://www.lesswrong.com/posts/4v3hMuKfsGatLXPgt/investigating-the-learning-coefficient-of-modular-addition

Episode art by Hamish Doodles: hamishdoodles.com

Road lines, street lights, and licence plates are examples of infrastructure used to ensure that roads operate smoothly. In this episode, Alan Chan talks about using similar interventions to help avoid bad outcomes from the deployment of AI agents.

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

The transcript: https://axrp.net/episode/2024/11/16/episode-38_1-alan-chan-agent-infrastructure.html

FAR.AI: https://far.ai/

FAR.AI on X (aka Twitter): https://x.com/farairesearch

 FAR.AI on YouTube: https://www.youtube.com/@FARAIResearch

The Alignment Workshop: https://www.alignment-workshop.com/

Topics we discuss, and timestamps:

01:02 - How the Alignment Workshop is

01:32 - Agent infrastructure

04:57 - Why agent infrastructure

07:54 - A trichotomy of agent infrastructure

13:59 - Agent IDs

18:17 - Agent channels

20:29 - Relation to AI control

Links:

Alan on Google Scholar: https://scholar.google.com/citations?user=lmQmYPgAAAAJ&hl=en&oi=ao

IDs for AI Systems: https://arxiv.org/abs/2406.12137

Visibility into AI Agents: https://arxiv.org/abs/2401.13138

Episode art by Hamish Doodles: hamishdoodles.com

Do language models understand the causal structure of the world, or do they merely note correlations? And what happens when you build a big AI society out of them? In this brief episode, recorded at the Bay Area Alignment Workshop, I chat with Zhijing Jin about her research on these questions.

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

The transcript: https://axrp.net/episode/2024/11/14/episode-38_0-zhijing-jin-llms-causality-multi-agent-systems.html

FAR.AI: https://far.ai/

FAR.AI on X (aka Twitter): https://x.com/farairesearch

FAR.AI on YouTube: https://www.youtube.com/@FARAIResearch

The Alignment Workshop: https://www.alignment-workshop.com/

Topics we discuss, and timestamps:

00:35 - How the Alignment Workshop is

00:47 - How Zhijing got interested in causality and natural language processing

03:14 - Causality and alignment

06:21 - Causality without randomness

10:07 - Causal abstraction

11:42 - Why LLM causal reasoning?

13:20 - Understanding LLM causal reasoning

16:33 - Multi-agent systems

Links:

Zhijing's website: https://zhijing-jin.com/fantasy/

Zhijing on X (aka Twitter): https://x.com/zhijingjin

Can Large Language Models Infer Causation from Correlation?: https://arxiv.org/abs/2306.05836

Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents: https://arxiv.org/abs/2404.16698

Episode art by Hamish Doodles: hamishdoodles.com

Epoch AI is the premier organization that tracks the trajectory of AI - how much compute is used, the role of algorithmic improvements, the growth in data used, and when the above trends might hit an end. In this episode, I speak with the director of Epoch AI, Jaime Sevilla, about how compute, data, and algorithmic improvements are impacting AI, and whether continuing to scale can get us AGI.

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

The transcript: https://axrp.net/episode/2024/10/04/episode-37-jaime-sevilla-forecasting-ai.html

Topics we discuss, and timestamps:

0:00:38 - The pace of AI progress

0:07:49 - How Epoch AI tracks AI compute

0:11:44 - Why does AI compute grow so smoothly?

0:21:46 - When will we run out of computers?

0:38:56 - Algorithmic improvement

0:44:21 - Algorithmic improvement and scaling laws

0:56:56 - Training data

1:04:56 - Can scaling produce AGI?

1:16:55 - When will AGI arrive?

1:21:20 - Epoch AI

1:27:06 - Open questions in AI forecasting

1:35:21 - Epoch AI and x-risk

1:41:34 - Following Epoch AI's research

Links for Jaime and Epoch AI:

Epoch AI: https://epochai.org/

Machine Learning Trends dashboard: https://epochai.org/trends

Epoch AI on X / Twitter: https://x.com/EpochAIResearch

Jaime on X / Twitter: https://x.com/Jsevillamol

Research we discuss:

Training Compute of Frontier AI Models Grows by 4-5x per Year: https://epochai.org/blog/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year

Optimally Allocating Compute Between Inference and Training: https://epochai.org/blog/optimally-allocating-compute-between-inference-and-training

Algorithmic Progress in Language Models [blog post]: https://epochai.org/blog/algorithmic-progress-in-language-models

Algorithmic progress in language models [paper]: https://arxiv.org/abs/2403.05812

Training Compute-Optimal Large Language Models [aka the Chinchilla scaling law paper]: https://arxiv.org/abs/2203.15556

Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data [blog post]: https://epochai.org/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data

Will we run out of data? Limits of LLM scaling based on human-generated data [paper]: https://arxiv.org/abs/2211.04325

The Direct Approach: https://epochai.org/blog/the-direct-approach

Episode art by Hamish Doodles: hamishdoodles.com

Sometimes, people talk about transformers as having "world models" as a result of being trained to predict text data on the internet. But what does this even mean? In this episode, I talk with Adam Shai and Paul Riechers about their work applying computational mechanics, a sub-field of physics studying how to predict random processes, to neural networks.

Patreon: https://www.patreon.com/axrpodcast

Ko-fi: https://ko-fi.com/axrpodcast

The transcript: https://axrp.net/episode/2024/09/29/episode-36-adam-shai-paul-riechers-computational-mechanics.html

Topics we discuss, and timestamps:

0:00:42 - What computational mechanics is

0:29:49 - Computational mechanics vs other approaches

0:36:16 - What world models are

0:48:41 - Fractals

0:57:43 - How the fractals are formed

1:09:55 - Scaling computational mechanics for transformers

1:21:52 - How Adam and Paul found computational mechanics

1:36:16 - Computational mechanics for AI safety

1:46:05 - Following Adam and Paul's research

Simplex AI Safety: https://www.simplexaisafety.com/

Research we discuss:

Transformers represent belief state geometry in their residual stream: https://arxiv.org/abs/2405.15943

Transformers represent belief state geometry in their residual stream [LessWrong post]: https://www.lesswrong.com/posts/gTZ2SxesbHckJ3CkF/transformers-represent-belief-state-geometry-in-their

Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer: https://www.lesswrong.com/posts/mBw7nc4ipdyeeEpWs/why-would-belief-states-have-a-fractal-structure-and-why

Episode art by Hamish Doodles: hamishdoodles.com

Patreon: https://www.patreon.com/axrpodcast

MATS: https://www.matsprogram.org

Note: I'm employed by MATS, but they're not paying me to make this video.