Strategy
Mastering Experimentation Methods: From Design to Analysis
Learn more
Resources
FEATURED CASE STUDY
Coinbase Saves Millions, Reduces Experiment Analysis Time by 40%, and Restores Trust in Experimentation with Eppo
Learn more
When it comes to experimentation, there's a common misconception that your “win rate” (the percentage of experiments that produce positive results) must be the holy grail of metrics - at least for measuring how efficient your program is. After all, the goal is to win, right? Let’s only run experiments with sound hypotheses and good research behind them, stop any “random acts of testing”, so that our win rate will increase and we will have less wasted effort!
This certainly sounds nice and intuitive, but it’s a trap. The truth is that managing for win rate isn't just counterproductive; it's harmful to what makes experimentation actually work when it comes to driving innovation and competitive advantage. Turning win rate into a goal to be hit can stunt your organization's ability to learn, grow, and take the risks necessary to achieve game-changing breakthroughs.
Below, I'll explain three compelling reasons why managing for success rate is problematic. Then, we'll explore an alternative metric that's far better suited to unlocking the full potential of experimentation.
I. Success Rate Disincentivizes Innovation
Imagine you're part of an experimentation team, and you've been told your performance will be measured by the number of tests that result in measurable success. What's your next move? Chances are, you'd prioritize only the "safe" ideas, the ones you're pretty sure will lead to positive outcomes. After all, why risk failure when your job or reputation could depend on a solid win rate?
But there's a glaring problem with this mindset. Safe tests, ones rooted in consensus or logic that "just makes sense", are rarely the ones that generate standout innovation. They're often incremental improvements, not the bold, boundary-pushing ideas that create market differentiation. Those bold, risky experiments are exactly where the magic happens. They're the tests that others might be too cautious (or too narrowly-focused) to try - and when we find the few unlikely ideas that do work, suddenly our organization can leap ahead.
Focusing on win rate crushes this kind of ambition. It fosters a fear of failure in your teams, steering them away from the unconventional ideas that could yield massive returns. It essentially means playing not to lose rather than playing to win.
II. Success Rate Misses Significant Value Creation
Not all experiments succeed—and that's okay. With the right infrastructure in place, every single experiment run can create real value for the business, even if it’s a “loss” or even an inconclusive result. Let’s start with negative results, since the explanation is a bit more immediately intuitive.
Negative results can be just as instructive as positive ones. Discovering that an idea has a measurable negative impact can help you avoid rolling out harmful changes to users - we have just created real value via risk mitigation. I have seen countless feature launches, website redesigns, and even bug fixes that would have caused precipitous drops in company revenue caught and remedied at the experiment stage, before the issue became much larger in scale.
Similarly, inconclusive experiments may suggest that a new feature adds no measurable value. If you decide against deploying it, you've just saved your team the long-term maintenance and resource costs of supporting a low-impact product. Now we’ve created value by directing more optimal resource allocation.
The problem is that win rate as a metric doesn't capture either of these kinds of value. It's laser-focused on the idea of "winning," completely overlooking the very real benefits that come from identifying poor ideas before they're implemented. Worse still, it devalues the kind of experimentation culture where the goal is learning and growth, not just success.
III. Success Rate Assumes Prediction Is Possible
Jeff Bezos says “if you know in advance that [an idea] is going to work, it is not an experiment.”
He’s right, of course, but in my experience working with dozens of companies on experimentation in a hands-on capacity (and observing dozens more from afar), most professionals are massively overconfident in their ideas - especially when you consider that the average success rate is something like 33%. So I like to consider this idea in advance, i.e., we can’t know for certain that any given idea is going to work, so we need to experiment.
Either way you look at it, experiments are about quantifying uncertainty. Trying to predict their outcomes is a bit like attempting to guess the future with a crystal ball.
Here's the hard truth: no one can accurately predict which ideas will work. The most talented teams can and should make educated guesses, but there will always be surprises, both positive and negative. If we want to optimize for a goal like “win rate”, we are inherently assuming that it is possible to pick winners in advance. This is just fostering unrealistic expectations for your teams and executives.
IV. A Better North Star: Cost Per Experiment
If not the success rate, then what? Enter cost per experiment, an alternative metric that shifts the focus from managing results to scaling learning. Again pulling from the Jeff Bezos quote library, he told Harvard Business Review in 2007 that "the key, really, is reducing the cost of the experiments." Why? Lower costs mean you can run more experiments, and more experiments mean a higher likelihood of discovering game-changing innovations.
The math here is pretty simple. Experiments have a convex payoff function. When you lose, the costs are limited: you suffered a degradation for some percentage of users over a small period of time, but mostly the costs are just the time, effort, and resources expended in building and running the test itself. But when you win, the potential gains can be massive, sometimes producing impacts worth millions (or hundreds of millions) of dollars. The more experiments you run, the more opportunities you create to uncover these high-value wins.
By aiming to reduce the cost of experimentation—whether through better tools, streamlined processes, or self-service platforms—you enable your teams to run more tests and learn faster. Put simply, volume trumps precision in the world of experimentation.
V. How to Pivot Away from Win Rate Focus
Shifting organizational mindsets away from win rates starts with reframing how you talk about experimentation. It's not about success or failure; it's about learning. Drive home the idea that every experiment, regardless of outcome, provides valuable information to make better decisions in the future.
Consider adopting alternative metrics to track success. Cost per experiment is one, but you can also measure the speed of experiments, the proportion of experiments delivering actionable insights, or the percentage of effort spent on bold tests. These metrics encourage behaviors that align with long-term growth and innovation.
Finally, create a culture where learning is celebrated as much as winning. Share examples of failed experiments that led to valuable insights. Help stakeholders and executives understand that even unsuccessful tests can contribute to avoiding risks or reallocating resources more efficiently. By championing learning over outcomes, you empower teams to take risks and think bigger.
Conclusion
Managing experimentation by success rate might seem logical on the surface, but digging deeper, its flaws become evident. It disincentivizes innovation, ignores the value of negative and inconclusive results, and assumes an impossible ability to predict outcomes. Instead, focusing on metrics like cost per experiment offers a scalable, sustainable path to experimentation success.
At the heart of this shift is a fundamental truth worth repeating: in experimentation, the goal isn't always precision; it's volume. The more experiments you run, the faster you learn, and the greater your odds of uncovering the breakthrough ideas that truly move the needle.
Table of contents
Ready to go from knowledge to action?
Talk to our team of experts and see why companies like Twitch, DraftKings, and Perplexity use Eppo to power experimentation for every team.
Get a demo
