A/B testing your AI prompts: a guide that skips the hype
## The vibes problem
I have watched smart engineers spend weeks tuning a SOUL.md based on gut feel. They run a prompt, read the output, think "yeah that's better," and commit the change. This is not engineering. This is astrology with a terminal open.
The problem with vibes-based evaluation is that you're doing two things at once: generating the hypothesis and confirming it. You tweaked the prompt because you expected it to be better. Then you looked at outputs and -- surprise -- it seemed better. Confirmation bias is not a character flaw. It's a cognitive default. You need a process that routes around it.
## What a real A/B test looks like
An A/B test has four components. Miss any one of them and you're back to vibes.
### 1. A control and a variant
Your control is the current production prompt. Your variant is the one change you want to test. One change. Not three changes bundled together because "they're all improvements." If you change the system prompt, the temperature, and the model in the same experiment, and the variant wins, you have no idea which change helped. You might have made two things worse and one thing so much better it covered for the others.
Keep your variant to a single hypothesis. "Adding a chain-of-thought instruction to the system prompt will improve accuracy on multi-step tasks." That's testable. "Making the prompt better" is not.
### 2. A metric you define before the test
Pick your primary metric before you see any results. Task completion rate is the obvious one, but it's often too coarse. Think about what "better" means for your specific use case.
For a customer support agent, you might care about resolution rate, escalation rate, and average response length. For a code generation agent, maybe it's test pass rate and token cost per task. For a summarization agent, maybe it's factual accuracy scored by a grader model.
The point is: write down "I will declare the variant a winner if metric X improves by at least Y%" before you start. If you wait until after, you'll find the metric that looks best and retroactively declare that's what you were testing. Researchers call this p-hacking. I call it lying to yourself with numbers.
### 3. Random assignment
Every task needs to be randomly assigned to either the control or variant. Not "first 50 tasks go to control, next 50 to variant" -- that introduces time-of-day bias, user-mix bias, and whatever else changed between batch one and batch two.
True random assignment means each task has an independent coin flip deciding which variant handles it. ClawSplit handles this automatically, but if you're rolling your own, use a proper random number generator, not hash-based bucketing on user ID (which creates systematic bias if your user IDs have structure).
### 4. Enough data to mean something
This is where most prompt tests fall apart. You run 20 tasks, see a 15% improvement, and ship it. But with 20 tasks, a 15% difference is well within random noise. You haven't learned anything.
How much data you need depends on two things: the size of the effect you're trying to detect and your baseline success rate. If your current prompt succeeds 80% of the time and you're hoping the variant pushes it to 85%, you need roughly 900 tasks per variant to detect that difference with 95% confidence. For a 90% to 95% improvement, you need even more -- around 1,400 per variant -- because you're trying to detect a smaller relative change at a higher baseline.
I know. That's a lot. But running 50 tasks and declaring victory is not saving you time. It's giving you fake confidence.
## Common mistakes
**Stopping early when results look good.** If you peek at results after 100 tasks and the variant is winning, it's tempting to stop and ship. Don't. Early results are noisy. The variant might be ahead by luck. Decide your sample size before you start and stick to it.
**Testing on curated examples.** Your test set should be real production traffic, not a hand-picked set of "representative" examples. Curated sets miss the weird edge cases that actually break your agent. If you must use a synthetic test set, make it at least 5x larger than you think you need.
**Ignoring secondary metrics.** Your variant improved task completion by 8%! But it also increased token cost by 40% and added 2 seconds of latency. Is that a win? Depends on your priorities. Track everything, decide what matters, and don't let one metric blind you to regressions elsewhere.
**Over-interpreting small differences.** A 2% improvement with p=0.04 is technically statistically significant. But a 2% improvement is probably not worth the operational complexity of maintaining a different prompt. Statistical significance tells you the effect is real. It doesn't tell you the effect is worth caring about. Think about practical significance too.
## When to skip the formality
Not every prompt change needs a 2,000-task experiment. If you're fixing an obvious bug -- the agent is returning JSON when it should return plain text -- just fix it. If you're adding a guardrail for a known failure mode, and the failure mode has a clear binary test, a smaller experiment is fine.
The full A/B testing setup matters most when changes are subtle, when you're optimizing rather than fixing, and when the stakes are high enough that a regression would cost real money or user trust.
## Getting started
If you've never run a prompt A/B test before, start small. Pick your worst-performing prompt, write one specific hypothesis about what would improve it, define a metric, and run 200 tasks through each variant. That's enough to detect a 10-15% improvement. You'll learn more from that single experiment than from a month of vibes-based tweaking.