Lenny’s AI Builders

devs are paying double for the old GPT on the API - GPT-5.6 Sol costs $4 in / $20 out per million tokens, GPT-6.1 Sol $2 / $10 - and swearing it's better than the new one. so i ran both on three of my real jobs (a broken test in Threadify, a video pipeline bug and my landing page copy), two runs each, with a blind judge from a different AI that only saw A and B.

also this week: GPT-6 Pro running agents in parallel, Claude writing 90% of Claude Code, X paying creators to post video, Claude Code answering sooner, and moving your AI memory into Grok Bot.

the practical: run your own model bake-off on your own work. give the prompt below to your agent. my recording cut out before i read it on camera, so here it is in full:

I want to know which AI model is actually better for MY work - not on benchmarks. Run a blind bake-off.

The job: [paste one real task from your week - a bug, a failing test, a page of copy]

Model A: gpt-5.6-sol Model B: gpt-6.1-sol (swap in any two models your CLI can run)

Prices per million tokens (OpenAI pricing page, 17 Oct 2026): gpt-5.6-sol $4 in / $0.40 cached / $20 out; gpt-6.1-sol $2 in / $0.10 cached / $10 out.

1. Make two clean copies of this project in a temp folder. Never touch my real folder.

2. Run the job once in each copy with the same prompt and the same effort setting:

codex exec -m [model] -c model_reasoning_effort=medium --json "[the job]"

Time each run and save its token usage.

3. Run my tests in both copies and note what passed.

4. Hide the model names. Label the two results A and B in a random order. Ask a different AI to judge them

(for example: claude -p) - give it only the job, the two diffs and the test results, and ask for a 0-100 score

and one sentence why for each.

5. Show me one table: tests passed, judge score, minutes, and cost at today's API prices. Then tell me which

model to use for this kind of job - and say plainly if it's too close to call.

Drafts only: don't push, merge or publish anything.

chapters

0:00 devs pay double for the old GPT

0:11 my blind bake-off: 3 real jobs, 2 models

0:34 why devs went back to the 2x model

1:38 this is the LAB

1:54 stop reading benchmarks

2:13 GPT-6 Pro runs agents in parallel

4:21 Claude writes 90% of Claude Code

6:38 sponsor: Threadify

7:53 X pays creators to post video

9:22 more Codex usage, but no number

10:02 Claude Code answers sooner

10:54 move your AI memory into Grok Bot

11:55 run your own model bake-off

12:42 who won: old GPT vs new GPT

13:44 the prompt (full text below)

thanks to today's sponsor, my software Threadify - your lead generation agent:

https://www.threadify.app/home?utm_source=youtube&utm_medium=video&utm_campaign=lab-0024&utm_content=sponsor

want to sponsor a future episode? [email protected]

watch the full episode: https://youtu.be/eIGJmUUqUr8

Meta just gave away Muse Spark Open, which it calls its most capable US open-weight model. the catch: to run it yourself you need two Mac Studios (or rent it on OpenRouter from $1.25 per million tokens in). so i put it on BuilderBench, my own benchmark, against Meta's paid Muse Wonder Full.

the result: Meta's free model scored 27.68. its own paid Muse Wonder Full scored 42.52 and Google's Gemini 4 Argon 45.68 - same tasks only (the desk tasks weren't run), rated by me.

also this week: Opus 5.5 fast mode with no waitlist ($8 in / $40 out per million tokens), Claude Code Review for Team and Enterprise, ChatGPT driving your Chrome (and GPT-Live 1.1), Dan Shipper's "the newsletter is dead" and Marc Lou's MailCheap, and Nick Cammarata on reading an AI's mind.

the practical: Felix Rieseberg's "deeper version, not a bigger one" prompt, tested on my newsletter with a blind judge from a different AI.

Felix's article: https://felixrieseberg.com/how-to-be-more-ambitious-a-tutorial-for-humans-with-models/

chapters

0:00 Meta's free model has a catch

0:22 Muse Spark Open: free weights, two Mac Studios

2:01 this is the LAB

2:17 everything got cheaper, faster or free

2:27 Opus 5.5 fast mode, no waitlist

3:58 Claude Code Review goes live

5:21 sponsor: Threadify

6:43 ChatGPT now drives your Chrome

7:50 the newsletter is dead + MailCheap

10:47 reading an AI's mind

12:14 ask for a deeper version, not a bigger one

13:55 Meta's free model on my bench: the score

14:33 outro

thanks to today's sponsor, my software Threadify - your lead generation agent:

https://www.threadify.app/home?utm_source=youtube&utm_medium=video&utm_campaign=lab-0023&utm_content=sponsor

want to sponsor a future episode? [email protected]

watch the full episode: https://youtu.be/UapnTPlIO04

Claude now runs a whole team of agents - so i pointed four of them at my own landing page. it scored 46/100 on Google's speed test (Lighthouse, mobile). i gave them one prompt - Voxyz's "20 things that slow your site down" - in the new Claude Code desktop app, and let them fix it in one pass.

the result: on a copy of the site with every fix, measured side by side with an untouched copy (which scored 48), 48 → 86 (median of 5 runs each). the phone freeze went from about 4 seconds to 55 ms and the page got 69% lighter (2.3 MB → 0.7 MB). the live site still scores 46 until my teammate Sam ships the fix - that's his call. 86 is the fixed copy, not the live site yet. check the live score yourself: https://pagespeed.web.dev/analysis?url=https%3A%2F%2Fwww.threadify.app%2Fhome

the prompt (credit @Voxyz_ai): https://x.com/Voxyz_ai/status/2110083153930920113

also: the Claude Code desktop app runs lots of agents side by side, Grok Voice 3.0 tops Artificial Analysis's speech-to-speech index (86.9 vs GPT-Live-1's 83.2), Grok Bot opens a connector store and adds plus-aliases, an investor says Grok Bot caught mistakes his accountant missed (his own account - he also invests in Magnetic Tax), Elicit's CEO thinks AI cyberattacks on infrastructure are plausible within a year (his opinion), and Claude Code now compacts idle sessions so coming back costs less.

chapters

0:00 my website scored 46

0:14 Claude Code's desktop app runs your agents

2:02 this is the LAB

2:18 anthropic is so far ahead

2:31 Grok Voice 3.0 tops the voice board

4:55 Grok Bot opens its connector store

6:27 sponsor: Threadify

7:17 Grok Bot checked his accountant

8:25 Elicit's CEO: AI cyberattacks within a year

9:33 Claude Code stops torching idle sessions

10:58 Tibo's day 10: another Codex reset

11:31 one prompt, four agents, a faster site

13:31 outro

thanks to today's sponsor, my software Threadify - your lead generation agent:

https://www.threadify.app/home?utm_source=youtube&utm_medium=video&utm_campaign=lab-0022&utm_content=sponsor

want to sponsor a future episode? [email protected]

watch the full episode: https://www.youtube.com/watch?v=LLxQuQUhAQg

Meta and Google both say their new models caught Opus 5.5. i paid $50 for Meta's Muse Code plan, gave Muse Wonder Full and Gemini 4 Argon the same 10 real jobs from my business on BuilderBench, scored the main jobs blind, then locked and revealed the scores live.

result (BuilderBench v3, 14 Oct 2026, owner-finalized partial personal review): Gemini 4 Argon 45.38/100, Muse Wonder Full 39.91/100. neither beat Opus 5.5's 68.26 - but that score is from pack 1, an older version of the test, so it's a reference line, not a same-test match.

was it fair? same 10 jobs and the same 4-hour window (a storage problem interrupted it and the clock was not reset). not the same effort (Argon's highest setting is high), not the same tools (Muse Code app vs the BuilderBench API harness), and autonomy + proactivity were rated by name. Argon's cost is an API-equivalent estimate (about $8.92, partial coverage); Muse Wonder Full's usage wasn't recorded (it ran on the $50/month plan). neither is a bill.

also: REA takes apart any app, Grok Bot sold a $500 ad (Alex Finn's number), Claude broke git at Anthropic, DigUp searches your Mac by meaning, and a recording device planted in OpenAI's office (charged, not convicted).

chapters

0:00 Meta and Google say they caught Opus

0:15 Gemini 4 Argon + Muse Wonder Full

1:46 this is the LAB

2:01 let's find out

2:09 REA takes apart any app

3:08 Grok Bot sold a $500 ad + LennyClaw

5:10 Tibo's day 9: Codex on disk

5:29 sponsor: Threadify

6:09 Claude broke git at Anthropic

7:04 DigUp searches your Mac by meaning

7:50 a bug planted in OpenAI's office

9:17 BuilderBench: who caught Opus?

10:52 the lock and the reveal

11:55 every score

13:37 outro

thanks to today's sponsor, my software Threadify - your lead generation agent:

https://www.threadify.app/home?utm_source=youtube&utm_medium=video&utm_campaign=lab-0021&utm_content=sponsor

want to sponsor a future episode? [email protected]

Lenny's AI Builders - the LAB. all the AI news you need, plus something you can actually use, in under 15 minutes.

less scrolling. more "oh shit, i could do that".

watch the full episode: https://www.youtube.com/watch?v=Cb1oaurcx2g

X just merged Grok, Cursor and X Premium into one plan. i switched, and my plan check says it saves about A$74.60 a month. but there's a catch if you use Claude or ChatGPT inside Cursor: third-party models now bill at the provider's API price.

plus Meta's Muse Wonder Full lands tied 2nd on the Artificial Analysis Intelligence Index, Hermes and OpenClaw are back on Claude Max plans, OpenAI's day 7 lifts Sol's default context from 272k to 500k, Google's AI Edge Foresight keeps your voice notes on your Mac, and Satya Nadella says to treat models as insider risks.

the thesis: this week the walls came down.

the practical: your Max plan runs Hermes. my Slack bot Fwed runs Opus 5.5 on the Claude plan i already pay for, through Nous's official Claude subscription plugin (the real Claude app, no API key). set up 7 Oct, checked 13 Oct. Anthropic's help page says the Agent SDK can run on your plan limits, and also says an account is for one person.

chapters

0:00 X just merged Grok and Cursor into one plan

0:13 the X Subscription: tiers, the catch, my bill

2:46 this is the LAB

2:57 the walls came down

3:08 Meta's Muse Wonder: tied 2nd

4:47 Hermes and OpenClaw are back on Claude

5:52 sponsor: Threadify

6:23 OpenAI day 7: 500k context

7:14 Google's notetaker never leaves your Mac

8:07 Nadella: models as insider risks

10:18 your Max plan runs Hermes

12:10 outro

thanks to today's sponsor, my software Threadify - your lead generation agent:

https://www.threadify.app/home?utm_source=youtube&utm_medium=video&utm_campaign=lab-0020&utm_content=sponsor

links

X: the X Subscription: https://x.com/SpaceXSI/status/2109783335296008414

Mark Zuckerberg: Muse Wonder: https://x.com/finkd/status/2109689305073614918

Claude Devs: https://x.com/ClaudeDevs/status/2108228391404966005

Tibo: OpenAI day 7: https://x.com/thsottiaux/status/2109430343346581527

TestingCatalog: Google AI Edge Foresight: https://x.com/testingcatalog/status/2107853628656496657

Satya Nadella: models as insider risks: https://x.com/satyanadella/status/2108931348857827686

want to sponsor a future episode? [email protected]

Lenny's AI Builders - the LAB. all the AI news you need, plus something you can actually use, in under 15 minutes.

less scrolling. more "oh shit, i could do that".

watch the full episode: https://www.youtube.com/watch?v=nlYMQ4NIgKQ

Codex now writes your next prompt. i let it run a whole session and kept 9 of 10. plus Grok Bot gets its own email, Claude Code Projects opens up, ChatGPT dots on your phone, one Claude agent that runs a thousand, Anthropic's report on Claude doing things it shouldn't, and a one-line prompt that makes your AI animations move like Pixar.

the thesis: this week, agents got their own stuff.

the practical: give Claude any animation and this one line:

"Review this animation against John Lasseter's principles of animation and tell me the top three fixes."

attach the video, or point Claude at the code that makes it, apply only the three fixes, then render again. Claude ran it in Claude Code on 10 Oct on one of our own animations: timing, staging and a climax that gets a beat. one run, and Claude's review is advice, not a score.

chapters

0:00 Codex now writes your next prompt

0:06 composer predictions: on, cost, my 9 of 10

2:06 this is the LAB

2:29 Grok Bot gets its own email

3:52 Claude Code Projects opens up

5:13 sponsor: Threadify

5:52 ChatGPT dots, now on your phone

6:58 one Claude agent runs a thousand

9:16 Claude did things it shouldn't

10:46 make it move like Pixar

12:39 outro

thanks to today's sponsor, my software Threadify - your lead generation agent:

https://www.threadify.app/home?utm_source=youtube&utm_medium=video&utm_campaign=lab-0019&utm_content=sponsor

links

OpenAI Devs: composer predictions: https://x.com/OpenAIDevs/status/2108624138369929725

composer predictions details: https://community.openai.com/t/codex-can-now-suggest-your-next-message-composer-predictions-beta/1404842

Thomas Ricouard: https://x.com/Dimillian/status/2108625316855836727

Tibo: https://x.com/thsottiaux/status/2108645667451318747

Grok Bot email: https://x.com/bot/status/2108609764766908772

Elon Musk: https://x.com/elonmusk/status/2108612939842511217

9to5Mac: https://9to5mac.com/2026/10/09/grok-bot-just-got-its-own-email-address-heres-how-to-claim-yours/

Claude Devs: Claude Code Projects: https://x.com/ClaudeDevs/status/2108621476538781878

Projects redesigned: https://claude.com/resources/articles/projects-redesigned

Projects waitlist: https://claude.com/form/projects

ChatGPT: dots updates: https://x.com/ChatGPT/status/2108636745915052037

dots updates: https://community.openai.com/t/fresh-dots-updates-create-in-the-mobile-app-delegate-to-codex/1404834

Poteto: https://x.com/poteto/status/2107566574735651107

Claude Devs: dynamic workflows: https://x.com/ClaudeDevs/status/2108591328732856655

workflow runs docs: https://platform.claude.com/docs/en/managed-agents/workflow-runs

Anthropic: unintended model actions: https://www.anthropic.com/research/investigating-unintended-model-actions

Ashwin: the Lasseter prompt: https://x.com/Shwinnabego/status/2108573799302205935

want to sponsor a future episode of the LAB? [email protected]

Lenny's AI Builders - the LAB. all the AI news you need, plus something you can actually use, in under 15 minutes.

less scrolling. more "oh shit, i could do that".

watch the full episode: https://www.youtube.com/watch?v=JgY1JLNnfRs

OpenAI's coding model just got up to 8x faster - if you pay $500 a month. plus Claude Dashboards and Motion, a guide to Claude Managed Agents, Voyager, Firecrawl Universal Scrape, two safety stories, and i set up a Claude agent that checks my landing page, live.

the thesis: they sell the shovels. we build.

the practical: one command set up a Claude Managed Agent that opens my landing page, follows every button and tells me in Slack what's broken. first real run: about 90 seconds, 21 cents against a $5 cap, three problems found. this version finds and proposes only - the "say yes and it fixes it" step is next. i also tried GPT-6.1 Sol Ultrafast on a real task while filming: it sat for about 10 seconds, then worked for 9m 57s. that is one run, not a benchmark.

try it: run "claude update", then give Claude Code this:

/claude-api managed-agents-onboard https://claude.dev/blog/building-effective-agent-automations/

thanks to today's sponsor, my software Threadify - your lead generation agent for Threads:

https://www.threadify.app/home?utm_source=youtube&utm_medium=video&utm_campaign=lab-0018&utm_content=sponsor

links

OpenAI Devs: GPT-6.1 Sol Ultrafast: https://x.com/OpenAIDevs/status/2108262812489531498

Tibo: day 4, instant steering: https://x.com/thsottiaux/status/2108275041276420573

Claude: Dashboards and Motion: https://x.com/claudeai/status/2108271552991252810

Claude Devs: automations with Claude Managed Agents: https://x.com/ClaudeDevs/status/2108320296323477683

Claude guide: building effective agent automations: https://claude.dev/blog/building-effective-agent-automations/

Anvisha: Voyager: https://x.com/anvisha/status/2108252061209088159

Firecrawl: Universal Scrape: https://x.com/firecrawl/status/2108225206141518107

Jane Manchun Wong on Anthropic's usage policy: https://x.com/wongmjane/status/2108266310492917916

Mikita Balesni: the letter: https://x.com/balesni/status/2108262814003687745

Jasmine Wang: https://x.com/j_asminewang/status/2108263312291180680

OpenAI Newsroom: the reply: https://x.com/OpenAINewsroom/status/2108441580806025712

want to sponsor a future episode of the LAB? [email protected]

Lenny's AI Builders - the LAB. all the AI news you need, plus something you can actually use, in under 15 minutes.

less scrolling. more "oh shit, i could do that".

watch the full episode: https://www.youtube.com/watch?v=a42FHynrhZs

Elon says his own bot will use Claude. plus Claude's new Haiku 5.5, monthly API credits for Max plans, GPT-6 in ChatGPT, and i test Haiku 5.5 against GPT-6 Luna on BuilderBench, live.

the thesis: competition is great - builders are cashing in.

the practical: i ran BuilderBench v3 on Haiku 5.5 and GPT-6 Luna (8 Oct 2026, 4 hours, extra high) and rated every job blind before i knew which model made it. GPT-6 Luna 44.58, Haiku 5.5 39.18 (out of 100, final personal scores, partial evidence). run cost estimates: Luna $1.47 (short-context scenario), Haiku $37.39 (provider-reported). these are API-equivalent estimates, not bills, and the methods differ. one run, my jobs, my taste.

try it: pick three jobs you really do, give both models the same brief, and rate the work blind.

thanks to today's sponsor, my software Threadify - your lead generation agent for Threads:

https://www.threadify.app/home?utm_source=youtube&utm_medium=video&utm_campaign=lab-0017&utm_content=sponsor

links

Elon on Grok Bot's back-end models: https://x.com/elonmusk/status/2107724314451878104

xAI: Grok Bot now works with X: https://x.ai/news/grok-bot-and-x

X API pricing: https://docs.x.com/x-api/getting-started/pricing

Claude: introducing Claude Haiku 5.5: https://x.com/claudeai/status/2107894039626277339

Anthropic: Claude Haiku 5.5: https://www.anthropic.com/claude-haiku-5-5

Claude Devs: monthly API credits for Max and Team: https://x.com/ClaudeDevs/status/2107895957933408429

Claude help: monthly API credits: https://support.claude.com/en/articles/17154008-monthly-api-credits-for-max-and-team-plans

OpenAI: GPT-6 and Intelligent UI in ChatGPT: https://x.com/OpenAI/status/2107894997538525580

Karpathy: ask for visuals, not text: https://x.com/karpathy/status/2105819303471976479

Tibo: day 3: https://x.com/thsottiaux/status/2107913674593644711

OpenAI help: ChatGPT Pro tiers: https://help.openai.com/en/articles/9793128-about-chatgpt-pro-tiers

Theo: tsc-rs: https://x.com/theo/status/2107789940482621795

tsc-rs repo: https://github.com/pingdotgg/ts-rust

want to sponsor a future episode of the LAB? [email protected]

Lenny's AI Builders - the LAB. all the AI news you need, plus something you can actually use, in under 15 minutes.

less scrolling. more "oh shit, i could do that".

watch the full episode: https://www.youtube.com/watch?v=WZkrVGp-jxE

OpenAI promised 28 days of gifts. on day two, poteto - Lauren Tan from the Grok Bot team - called them nothingburgers. there is beef.

76% of almost 75,000 votes on Tibo's poll said day two needed a reset. we got one. my take: build the model, not the calendar.

then Claude moves into Google Docs, Sheets and Slides (beta, paid plans), Hark Pro (the Figure founder's personal agent), OpenAI's AI did new maths (with a model we can't use yet), treg vs Clay for lead gen, and Nano Banana 2.1.

the practical: the one prompt that makes Claude use every drop of my plan. i had 30% of my weekly usage left. Claude checked it every hour and went from 2 sub-agents to 5. copy the prompt from the video.

thanks to today's sponsor, my software Threadify - your lead generation agent for Threads:

https://www.threadify.app/home?utm_source=youtube&utm_medium=video&utm_campaign=lab-0016&utm_content=sponsor

links

poteto: 'is this why your 28 days are full of nothingburgers': https://x.com/poteto/status/2107566574735651107

Tibo: roundup of day 2: https://x.com/thsottiaux/status/2107575657014468879

Claude: now inside Google Docs, Sheets and Slides: https://x.com/claudeai/status/2107522596845822135

Claude help: use Claude in Google Docs, Sheets and Slides: https://support.claude.com/en/articles/16951679-use-claude-in-google-docs-sheets-and-slides

Brett Adcock: Hark Pro: https://x.com/adcock_brett/status/2107504425451536822

Hark: https://hark.com/

OpenAI: new mathematical results: https://x.com/OpenAI/status/2107596713791767021

OpenAI: sharing AI progress in mathematics: https://openai.com/index/sharing-ai-progress-in-mathematics/

github.com/openai/math: https://github.com/openai/math

Jason Zhou: treg: https://x.com/jasonzhou1993/status/2107423956395589784

treg people search: https://treg.to/people-search

Nano Banana 2.1: https://x.com/NanoBanana/status/2107521202466046453

my usage prompt on Threads: https://www.threads.com/share/_s-Z4_Bvq/

Lenny's AI Builders - the LAB. all the AI news you need, plus something you can actually use, in under 15 minutes.

less scrolling. more "oh shit, i could do that".

watch the full episode: https://www.youtube.com/watch?v=E7RqRgxmP4g

Claude gives you about 5x more for your money than OpenAI. that's not my maths - SemiAnalysis measured it.

they ran the same jobs on every big AI plan and watched the usage bars. on the $200 plans, Leo's screenshot of their data shows 28.6 billion tokens of Opus 5.5 vs 10.2 billion of GPT-6.1 Sol. that's 2.8x the raw tokens, and about 5x the value at API prices. one lab even gave 1 of 3 identical subscriptions 20% lower limits. so know what you're paying for. we deserve to know.

then OpenAI's 28 days of gifts (my take: an advent calendar for people who just lost half their limits), HyperFrames Studio (a video editor for agents), Beam (a giant open model you can't run), ChatGPT's new plugin store, and why only 4.5% of people pay for AI.

the practical: i ran Matt Pocock's new /retro skill on the session where my agent edited the last episode 25+ times. it found 6 problems, live, on camera.

try this (make your agent review itself):

1. give your agent this link and say "install this repository of skills": github.com/mattpocock/skills

2. pick one session that felt slow or messy.

3. in that session, type /retro and hit enter.

4. read the list, worst first. approve the fixes you agree with.

sponsor: Threadify, my lead generation agent for Threads. get started: https://www.threadify.app/home?utm_source=youtube&utm_medium=video&utm_campaign=lab-0015&utm_content=sponsor

sources

SemiAnalysis: Anthropic subscriptions offer 5x+ more value than OpenAI: https://newsletter.semianalysis.com/p/anthropic-subscriptions-offer-5x

SemiAnalysis on X: https://x.com/SemiAnalysis_/status/2107204965710053510

Leo (@synthwavedd): tokens by plan: https://x.com/synthwavedd/status/2107206085656580277

OpenAI: ChatGPT Pro tiers (20x to 10x): https://help.openai.com/en/articles/9793128-about-chatgpt-pro-tiers

Tibo: 28 days: https://x.com/thsottiaux/status/2106845241357824205

Tibo: day 1, about 50% faster: https://x.com/thsottiaux/status/2107158998495748264

HyperFrames Studio: https://www.hyperframes.dev/studio

Reflection AI: introducing Beam: https://reflection.ai/blog/introducing-beam

OpenAI: submitting a ChatGPT plugin: https://developers.openai.com/plugins/deploy/submission

Pietro Schirano: MagicPath growth: https://x.com/skirano/status/2107236581442720203

a16z: top 100 gen AI consumer apps: https://a16z.com/100-gen-ai-apps-7/

Matt Pocock's skills (v1.3.1, /retro): https://github.com/mattpocock/skills/releases/tag/v1.3.1

5x is SemiAnalysis's measured API-equivalent value (Opus 5.5 vs GPT-6.1 Sol); 2.8x is raw tokens on Leo's screenshot, medium reasoning. Pro goes from 20x to 10x Plus for new subscribers now and existing ones from 30 Oct. the 28 days and "about 50% faster" are Tibo's own posts - no changelog yet. i haven't tried HyperFrames Studio. Beam's weights aren't out and its scores are self-reported; 250 GB is our estimate. 2,000% is MagicPath's own claim. a16z: US paid personal subscriptions, August 2026.

Lenny's AI Builders - LAB 0015

watch the full episode: https://youtu.be/ExiN_qOGCbs