Today we're releasing RoastMyPost, a new experimental application for blog post evaluation using LLMs. Try it Here.

TLDR

• RoastMyPost is a new QURI application that uses LLMs and code to evaluate blog posts and research documents.
• It uses a variety of LLM evaluators. Most are narrow checks: Fact Check, Spell Check, Fallacy Check, Math Check, Link Check, Forecast Check, and others.
• Optimized for EA & Rationalist content with direct import from EA Forum and LessWrong URLs. Other links use standard web fetching.
• Works best for 200 - ~10,000 word documents with factual assertions and simple formatting. It can also do basic reviewing of Squiggle models. Longer documents and documents in LaTeX will experience slowdowns and errors.
• Open source, free for reasonable use[1]. Public examples are here. 
• Experimentation encouraged! We're all figuring out how to best use these tools.
• Overall, we're most interested in using RoastMyPost as an experiment for potential LLM document workflows. The tech is early now, but it's at a good point for experimentation.

How It Works

1. Import a document. Submit markdown text or provide the URL of a publicly accessible post.
2. Select evaluators to run. A few are system-recommended. Others are [...]

---

Outline:

(00:27) TLDR

(01:52) How It Works

(02:30) Screenshots

(03:21) Current AI Agents / Workflows

(03:26) Is it Good?

(04:33) What are Automated Writing Evaluations Good For?

(07:20) Privacy & Data Confidentiality

(07:50) Technical Details

(09:12) Building Custom Evaluators

(10:26) Try it Out

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.