You run a test for three weeks. The result: no significant difference. You shrug and move on. Do that enough times and you conclude testing does not work for you. The real culprit is almost always power.
What statistical power is
Power is the probability your test detects a real effect that actually exists. Low power means even genuine winners come back "inconclusive." Plenty of teams run tests well below the 80% power that is treated as a floor, which means they are flipping a coin on whether they would even notice a win.
The three things that set power
- Sample size. More visitors, more power. This is the big one.
- Effect size. Bigger true differences are easier to detect. A 20% lift is easy. A 2% lift is brutal.
- Baseline rate. Very low conversion rates need a lot more traffic to resolve.
The uncomfortable math
To detect a small relative lift on a typical conversion rate, you often need thousands of conversions per variant, not visitors, conversions. Many pages simply do not get that volume in a sane window. So the test never had a chance from the moment you launched it.
Signs you are running underpowered tests
- Most of your tests end "no significant difference."
- You test on low-traffic pages.
- You add a third or fourth variant, splitting already-thin traffic further.
- You stop early when it "looks" significant, which trades power for false positives.
What to do
- Concentrate tests on your highest-traffic pages.
- Test bolder changes, not button tints. Bigger swings are detectable swings.
- Run fewer variants at once, or use a method that does not split traffic evenly.
- Calculate the required sample size before you start. If you cannot reach it, do not run that idea as a classic A/B test.
The structural fix
Underpowered A/B testing is really a traffic-allocation problem. Methods that move traffic toward likely winners instead of holding a rigid even split pull more signal out of the same visitors. That is why lower and mid-traffic sites often do better with a multi-armed bandit than with a textbook A/B test.
