Behind the Craft

📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com

🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so

Shreya and Hamel teach the industry standard course for AI evaluations taken by 4,500+ students from OpenAI, Google, and more. In our interview, I asked them to give live feedback on the AI evals that I built for my skills and demo how anyone can run evals with ChatGPT/Claude and their free evals skill. Don’t miss this up to date primer on evals from the real experts. 🙂

P.S., I asked Shreya and Hamel to give my audience 25% off the next cohort of their AI evals course. It has 4.7 stars with 900 reviews which is almost unheard of on Maven. Sign up via the link here: https://maven.com/parlance-labs/evals?utm_campaign=peteryang&utm_medium=affiliate&utm_source=maven&promoCode=PETERYANG

We talked about:

(00:00) Why AI evals are completely different now with the latest models

(02:30) Demo: Live audit of the evals that I built for my AI skills

(05:14) Top down vs. bottom up evals and why AI sucks at the latter

(08:16) Spinning up AI agents to grade each eval criteria

(14:32) Demo: Using Shreyas AI skill to run evals in Claude

(23:46) How to review output labels to identify failures

(34:37) How to turn failures into reusable evals

(43:19) Where human judgement is still needed

Resources mentioned:

Free eval skills from Shreya: https://github.com/ai-evals-course/evals-skills

Do automated evals work?: https://parlance-labs.com/blog/posts/auto-evals/

Thanks to our sponsors:

Wispr Flow — 4x faster than typing with your voice — https://ref.wisprflow.ai/peteryang

📌 Get the takeaways: https://creatoreconomy.so/p/how-to-build-better-ai-evals-with-claude-code-shreya-hamel

Where to find Shreya and Hamel:

Shreya: https://www.sh-reya.com/

Hamel: https://hamel.dev/

Subscribe to this channel - more interviews coming soon!

📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com

🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so

Riley is a top AI creator and founder of Agent Native. In this episode, he showed me how he uses Codex to run his entire content business from generating engaging thumbnails to writing incredible video hooks. Riley also shared his full AI stack including some lesser known AI tools to edit videos, draw diagrams, and more.

We talked about:

(00:00) How Codex runs Riley’s 1.5M+ follower content business

(03:16) Demo: Using Codex to create animations and B-roll

(07:28) Demo: Codex can reverse-engineer a top YouTube hook

(13:24) The BRENS framework to turn an idea into a strong hook

(17:24) Demo: Riley’s Codex stack for diagrams, thumbnails, and posts

(22:01) How Riley creates so much content every week

(25:41) Demo: Using Codex and Paper for viral YouTube thumbnails

(35:01) How to set up AI agents to find content ideas proactively

Thanks to our sponsors:

Google AI Studio: The fastest way to go from prompt to production https://aistudio.google.com/?utm_source=peteryang&utm_medium=partner&utm_campaign=26Q3-peteryang_ais

Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang

Wispr Flow: 4x faster than typing with your voice https://ref.wisprflow.ai/peteryang

📌 Get the takeaways: https://creatoreconomy.so/p/how-i-run-my-content-business-with-codex-riley-brown

Where to find Riley:

Agent Native, Riley’s new company: https://agentnative.inc/

X: https://x.com/rileybrown

Website: https://rileybrown.xyz/

Subscribe to this channel - more interviews coming soon!

📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com

🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so

Nan and Jacob built Linear Agent, which has transformed how the $1.25B company gets work done. In this episode, they took me from the initial memo to launch, showing how to give an agent tools to find the context it needs, use evals to improve reliability, and more. This episode is a must-watch if you want a concrete, behind-the-scenes example of how to build a production agent end to end.

We talked about:

(00:00) What’s holding AI agents back as models improve?

(00:51) Three things every good agent needs

(03:00) Demo: The internal memo that started Linear Agent

(09:50) The biggest mistake to avoid when building agents

(11:29) Demo: From Slack to code in 6 minutes with the agent

(17:01) When prototyping, “use the biggest model until it works”

(24:02) Give agents tools to find context, not more context

(34:07) The first thing to do before building an agent

Thanks to our sponsors:

Oceans: Hire AI-native EAs http://oceanstalent.com/peter

Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang

Linear: The AI agent platform for modern teams https://linear.app/behind-the-craft

📌 Get the takeaways: https://creatoreconomy.so/p/5-rules-for-building-ai-agents-in-production-linear-nan-jacob

Where to find Nan and Jacob:

Nan on LinkedIn: https://www.linkedin.com/in/thenanyu

Jacob on LinkedIn: https://www.linkedin.com/in/jacob-shumway-314a8366

Linear Agent: https://linear.app/docs/linear-agent

Subscribe to this channel - more interviews coming soon!

📌 Copy my personal AI system to save time and build with AI: https://www.behindthecraft.com/

🤝 Get my free AI newsletter read by 150K+ professionals: https://creatoreconomy.so/

Karan co-founded Nous Research, the company behind Hermes, the #1 personal agent and AI app overall on OpenRouter. I found him incredibly down to earth as we discussed why open-source AI must win, how Hermes gets better the more you use it, and even a live demo of how Hermes can mod your favorite childhood games.

Karan and I talked about:

(00:00) How Hermes differs from Claude Code and Codex

(02:20) Why every AI model now sounds exactly the same

(11:16) How to stop AI from saying “you’re absolutely right”

(15:16) How Hermes creates skills without bloating its context

(16:40) The open vs. closed-source debate on AI safety

(23:40) Using one Kanban board for humans and AI agents

(26:45) Demo: Rebuilding a childhood Sonic world with Hermes

(36:07) How two Discord friends built a self-improving agent

(42:40) How Hermes became its own biggest contributor

Thanks to our sponsors:

Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang

Linear: The AI agent platform for modern teams https://linear.app/behind-the-craft

Wispr Flow: 4x faster than typing with your voice https://ref.wisprflow.ai/peteryang

📌 Get the takeaways: https://creatoreconomy.so/p/hermes-co-founder-on-building-an-ai-agent-that-improves-itself-karan-malhotra

Where to find Karan:

X: https://x.com/karan4d

Website: https://www.karan4d.com/

Hermes Agent: https://github.com/NousResearch/hermes-agent

Subscribe to this channel - more interviews coming soon!

Jason is a DevEx engineer on the Codex team. In our episode, he showed me how he runs his entire workday using Codex, from his chief of staff thread to the way he builds skills that learn how he works. Jason wrote the official OpenAI guidebook on how to get the most out of Codex, so don’t miss this episode.

Jason and I talked about:

(00:00) The only job left after AI can do the work

(02:40) Turn pinned threads into long-running workspaces

(05:50) Build a chief of staff across Slack, email, and Linear

(10:12) Teach Codex to write like you for every audience

(13:33) Mine past sessions for new skills and workflows

(20:01) How Codex revised a launch video while Jason was out

(24:23) Build a drum practice site around how you learn

(25:59) Goal, plan, and worklog for long-running Codex tasks

(33:52) Why taste starts with knowing what feels wrong

Thanks to our sponsor:

Google AI Studio: The fastest way to go from prompt to production https://aistudio.google.com/?utm_source=peteryang&utm_medium=partner&utm_campaign=26Q3-peteryang_ais

Riverside: All-in-one AI studio for podcasts and video

https://creators.riverside.com/PeterYang

📌 Get my personal AI OS with useful AI skills, $270 in AI tool credits, live monthly workshops, and courses on Hermes and Codex agents: https://www.behindthecraft.com/

Where to find Jason:

X: https://x.com/jxnlco

Subscribe to this channel - more interviews coming soon!

Thariq works on the Claude Code team, and I’ve wanted to see how he builds for a long time. In our episode, he showed how to use /goal to keep Claude working, how he plans with Claude to remove unknowns before building, and how he runs a team of agents in Slack. He also shared why his team cut Claude Code’s system prompt by 80% and how he creates polished launch videos with Claude.

Thariq and I talked about:

(00:00) Goals, loops, and workflows to make Claude work longer

(04:48) Live demo: Creating a polished video with one prompt

(08:17) Plan by removing unknowns before you build

(14:32) How Anthropic uses HTML artifacts for planning

(18:35) Running parallel Claude tasks through Slack

(27:34) Using separate creator and verifier sub-agents

(31:08) How Thariq stays focused while Claude works in parallel

(34:07) Why Anthropic cut Claude Code’s system prompt by 80%

(37:01) How to become more technical when AI writes the code

Thanks to our sponsors:

Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang

Linear: The AI agent platform for modern teams https://linear.app/behind-the-craft

Wispr Flow: 4x faster than typing with your voice https://ref.wisprflow.ai/peteryang

📌 Get the takeaways: https://creatoreconomy.so/p/how-i-plan-build-and-run-loops-with-claude-code-thariq-shihipar

📌 Get my personal AI OS with useful AI skills, $270 in AI tool credits, live monthly workshops, and courses on Hermes and Codex agents: https://www.behindthecraft.com/

Where to find Thariq:

LinkedIn: https://x.com/trq212

Website: https://www.thariq.io/

Subscribe to this channel - more interviews coming soon!

Jared is a Builder in Residence at Cognition, the team behind Devin. He showed me why teams often overbuild their first agents, and how simple prompts, better tools, and self-testing work together instead. Then he demoed one manager agent breaking down a task, directing 10 cloud agents, and deciding what a human needs to review.

Jared and I talked about:

(00:00) Why teams overbuild their first AI agent

(02:50) Let the model cook before adding rules

(03:16) Why tools matter more than better prompts

(10:51) Why you should build evals after shipping

(16:06) What cloud agents can do that local agents can’t

(20:30) How to get one agent to manage other agents

(25:39) Demo: One agent launches a team of 10

(28:03) How agents prove their work before you review it

(33:18) The best tasks for agent teams to work on

Thanks to our sponsors:

Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang

Linear: The AI agent platform for modern teams https://linear.app/behind-the-craft

Wispr Flow: 4x faster than typing with your voice https://ref.wisprflow.ai/peteryang

📌 Get the takeaways: https://creatoreconomy.so/p/how-to-build-ai-agents-that-check-their-own-work-jared-cognition-devin

📌 Get my personal AI OS with useful AI skills, $270 in AI tool credits, live monthly workshops, and courses on Hermes and Codex agents: https://www.behindthecraft.com/

Where to find Jared:

X: https://x.com/imjaredz

Devin: https://devin.ai

Subscribe to this channel - more interviews coming soon!

As a product manager for OpenAI Codex, Rohan has a firsthand look at how one of the most AI-native teams in the world operates. He showed me how he uses Codex to trigger work from Slack replies, prototype designs with image generation, replace stale PRDs with live project sites, and more. Watch this episode if you’re tired of all the nonsense around “AI-native PM” and want to see how a real one actually operates.

Rohan and I talked about:

(00:00) Codex can do more than you think

(01:53) Use Codex to pull context from Slack, docs, Linear, and email

(03:28) How Codex flips planning from docs to demos

(13:14) Demo: Do the task once with Codex, then automate it

(14:30) Demo: Trigger Codex automations from Slack replies

(17:26) Demo: Use Image Gen to explore design variants fast

(19:58) Demo: Turn any Codex thread into a reusable skill

(29:55) Demo: Turn project context into a shareable Site

(33:33) Use one Codex thread to manage other Codex threads

(37:18) Aim 10x more ambitious than feels reasonable

Thanks to our sponsors:

Linear: The AI agent platform for modern teams https://linear.app/behind-the-craft

Wispr Flow: 4x faster than typing with your voice https://ref.wisprflow.ai/peteryang

Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang

📌 Get the takeaways: https://creatoreconomy.so/p/openai-pm-reveals-how-he-uses-codex-for-product-work-rohan-varma

📌 Get my personal AI OS with useful AI skills, $270 in AI tool credits, live monthly workshops, and courses on Hermes and Codex agents: https://www.behindthecraft.com/

Where to find Rohan:

LinkedIn: https://www.linkedin.com/in/rohannvarma

X: https://x.com/TheRohanVarma

Codex: https://openai.com/codex/

Subscribe to this channel - more interviews coming soon!

As product lead at Anthropic, Jess has a first-hand look at how Anthropic teams are moving from prompting AI to building long-running agents that work overnight. In our episode, Jess showed me how to build a Claude analytics agent from scratch, then covered how Anthropic PMs use agents internally to understand the codebase, monitor customer feedback, and make better product decisions.

Jess and I talked about:

(00:00) AI agents that work while you sleep

(03:46) Claude Managed Agents help for long-running tasks

(05:22) Demo: Building a Claude analyst agent from scratch

(09:28) Traces: How to see where an agent got stuck

(15:08) Evals: How agents grade their own work

(20:16) 5 ways in which Anthropic PMs use agents internally

(28:06) Processing a 4,000-company waitlist with agents

(32:47) To build your first agent, focus on helping one person

(36:47) What changes when agents can work overnight

Thanks to our sponsors:

Linear: The AI agent platform for modern teams https://linear.app/behind-the-craft

Wispr Flow: 4x faster than typing with your voice https://ref.wisprflow.ai/peteryang

Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang

📌 Get the takeaways: https://creatoreconomy.so/p/inside-anthropics-bet-on-claude-agents-jess-yan

📌 Get my personal AI operating system with all my skills and prompts: https://www.behindthecraft.com/

Where to find Jess:

X: https://x.com/jess__yan

Website: https://platform.claude.com/docs/en/managed-agents/overviewSubscribe to this channel - more interviews coming soon!

Bin and Jake built HyperFrames, HeyGen’s free open-source tool that turns HTML into professional product launch videos. In this episode, they shared how you can one-shot a launch video for any website. Jake also walked through his personal playbook to make amazing videos with AI — from storyboarding to reusing components to polish.

We talked about:

(00:00) From HTML to professional video with AI

(02:37) How to install HyperFrames in Codex or Claude Code

(04:50) Demo: Make a launch video in one shot from Spotify.com

(10:47) Jake’s workflow to level up AI videos

(13:06) frame.md: Turn brand guidelines into video direction

(15:05) storyboard.md: Write the story before building scenes

(17:31) HyperFrames launches: Reuse HeyGen’s real components

(20:17) Get AI to generate static frames for review before rendering

(22:46) Studio: Make visual edits and final polish

Thanks to our sponsors:

Linear: The AI agent platform for modern teams https://linear.app/behind-the-craft

Wispr Flow: 4x faster than typing with your voice https://ref.wisprflow.ai/peter-yang-jun21-pod

Riverside: All-in-one AI studio for podcasts and video https://creators.riverside.com/PeterYang

📌 Get the takeaways: https://creatoreconomy.so/p/full-tutorial-make-professional-launch

📌 Get my personal AI operating system with all my skills and prompts: https://www.behindthecraft.com/

Where to find Bin and Jake:

Bin: https://x.com/liu8in

Jake: https://x.com/JakeFromHeyGen

HyperFrames: https://github.com/heygen-com/hyperframes

Subscribe to this channel - more interviews coming soon!