This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research. This blogpost is paired with our announcement that Apollo Research is spinning out from fiscal sponsorship into a PBC.

Summary of main claims:

• There is a set of safety tools and research that both meaningfully increases AGI safety and is profitable. Let's call these AGI safety products.
  • By AGI, I mean systems that are capable of automating AI safety research, e.g., competently running research projects that would take an expert human 6 months or longer. I think these arguments are less clear for ASI safety.
• At least in some cases, the incentives for meaningfully increasing AGI safety and creating a profitable business are aligned enough that it makes sense to build mission-driven, for-profit companies focused on AGI safety products.
• If we take AGI and its economic implications seriously, it's likely that billion-dollar AGI safety companies will emerge, and it is essential that these companies genuinely attempt to mitigate frontier risks.
• Automated AI safety research requires scale. For-profits are typically more compatible with that scale than non-profits.
• While non-safety-motivated actors might eventually build safety companies purely for profit, this is arguably too [...]

---

Outline:

(02:07) Definition of AGI safety products

(05:37) Argument 1: Sufficient Incentive Alignment

(06:40) Transfer in time: AGI could be a scaled-up version of current systems

(08:20) Transfer in problem space: Some frontier problems are not too dissimilar from safety problems that have large-scale demand

(09:58) Argument 2: Taking AGI & the economy seriously

(12:13) Argument 3: Automated AI safety work requires scale

(14:59) Argument 4: The market doesn't solve safety on its own

(18:02) Limitations

(21:46) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.