A/B testing used to mean a paid tool, a developer, and weeks of setup — and the best tools kept leaving the market. Google killed Optimize; VWO and AB Tasty moved their free tiers upmarket toward enterprise. Startups got left behind, guessing and redesigning on a hunch.

Now it's one Claude Code session through PostHog's MCP. You say one sentence, and the agent builds the whole experiment — feature flag, 50/50 split on a single URL, server-side with no flicker, consent-safe, internal traffic filtered — and you keep the judgment calls. Things I used to do with a lot of trickery on the site, I didn't have to touch.

This is move 6 of 6 in the Growth Codex series — conversion optimization, the finale. You'll see the real Bridesmaid test: the conversation-vs-checklist variant, a directional lift of roughly +14% completions / +11% CTA clicks that never crossed statistical significance at our traffic, and the honest pirate posture — throw spaghetti early when traffic is thin, get scientific as the stakes scale. Plus the copy-paste prompts to set up and conclude your own test this week.

The honest part: at our traffic the result was directional, not a proven winner — it would take the better part of a year to reach full significance, so we shipped on the totality of the signal and logged the call. None of that is hidden here.


IN THIS EPISODE

00:00:00 – Intro: A/B testing got complicated, and the best tools left
00:00:37 – The tools we kept losing: Google Optimize → VWO → AB Tasty
00:01:32 – PostHog + Claude Code to the rescue
00:02:24 – What an A/B test is: one URL, two variants on Bridesmaid
00:03:16 – Behind the scenes: built via the PostHog MCP, nothing by hand
00:04:13 – The setup prompt: goals, consent, no flicker, internal filtered
00:06:15 – Don't guess: watch the session replays
00:08:17 – Throw spaghetti, then get scientific
00:10:40 – Reading the dashboard: the signup-completion drop
00:14:12 – Six moves, one self-optimizing engine
00:16:49 – Your spaghetti test: run one this week


DEEP DIVE LINKS

📖 Full Article: https://pirateskills.com/insights/vibe-ab-test

🧪 Pirate Lab — Tue June 23: free weekly live session where we put this week's insight into practice — watch me set up a full A/B testing and conversion stack live with Claude Code and PostHog, no code, no dashboards: https://pirateskills.com/pirate/lab/2026-06-23

Pirate Night - Celebrate 10 years of Pirate Skills in the coop I have been waiting for, with no other than Manuel Koelman, the founder of Pirate X and host of the legendary Pirate Summit, join us here: https://piratesummit.com/pirate-night/

🎧 Spotify: https://open.spotify.com/show/7kFl0BJiOYih8yqQTRR9SN

🍎 Apple Podcasts: https://podcasts.apple.com/us/podcast/build-grow-repeat/id1870561487

🗓️ All upcoming events: https://pirateskills.com/events

💬 Connect with me on LinkedIn: https://www.linkedin.com/in/bensufiani/