This is a crosspost from our report website for Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts. This report details the work behind our LLM-written paper "The Consistency Confound: Why Stronger Alignment Can Break Black-Box Jailbreak Detection" accepted at Agents4Science 2025, the first scientific conference requiring AI as primary author, where it passed both AI and human review.

TL;DR

• We built 6 AI agents using Gemini 2.5 Pro and Claude Code, mapped to stages of the scientific workflow from idea to hypothesis generation, experiment execution, evaluation and paper writing.
• We tested our agents on 4 research ideas across ML sub-domains such as Multi-Agent RL, World Models, and AI Safety. 3 ideas failed during implementation or evaluation. Only 1 succeeded and was published at Agents4Science 2025.
• We document 6 recurring failure modes: bias toward training data, implementation drift under pressure, memory/context degradation, overexcitement that declares success despite obvious failures, and gaps in domain intelligence and scientific taste.
• We also derive 4 design principles for more robust AI scientist systems, discuss the limitations of training and evaluation data for future autonomous science, and release all prompts, artifacts, and outputs at github.com/Lossfunk/ai-scientist-artefacts-v1.

Problem Definition and System Overview

We [...]

---

Outline:

(00:37) TL;DR

(01:44) Problem Definition and System Overview

(03:55) Our Agents4Science 2025 Submission

(06:25) Observed Failure Modes and Mitigation

(06:40) 1. Bias on Training Data

(07:25) 2. Implementation Drift

(08:04) 3. Memory and Context Issues

(08:55) 4. Overexcitement and Eureka Instinct

(09:50) 5. & 6. Lack of Domain Intelligence and Scientific Taste

(10:44) Design Takeaways for AI Scientist Systems

(10:57) 1. Start Abstract, Ground Later

(11:15) 2. Verify Everything

(11:35) 3. Plan for Failure and Recovery

(11:57) 4. Log Everything

(12:14) Limitations and Discussion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.