Culture
How We Ran Successful Experimentation Office Hours at Groupon
Ideas for training and nurturing experimenters across the company
Learn more
Stop Micromanaging Product Strategy
Leaders should set the vision, not dictate roadmaps
This is the first in a series of blog posts about aligning company leadership and experimentation teams. The technology world is filled with situations where executives aren't fully bought into experimentation, and product experiment teams throw up their hands at the "culture issues" that hold back their practices. These posts are aimed at executives who are hesitant to ramp up more experimentation, and product leaders who want to get more buy-in to run experiments.
This first post is about setting metric policies, which is part of a broader shift to product-led-growth. This was brainstormed during a conversation with Erik Bernhardsson, leader of the team that built Discover Weekly at Spotify, CTO of Better, and the author of Building a data team at a mid-stage startup.
Q: I feel like the biggest blocker to our experimentation practice is cultural. Our CEO doesn't want to run experiments, as [he/she] thinks they get in the way of big changes and grand reveals. How can I get the executive team to support running more experiments?
Experimentation-hesitancy is incredibly common among executives. Most founders have years of building the company without experiments, and it can feel like experimentation is a conservative force that gets in the way of company speed, product marketing, and moonshot innovation.
None of this is true. Saying that experimentation slows down company speed is like saying that testing code locally slows down company speed. After all, not doing experimentation is basically testing in production. Experimentation might not increase speed to shipping, but it increases speed to impact, as you'll quickly know what changes are valuable and innovative, and which ones are not.
Instead, the major disconnect is rooted in executives pursuing a shipping strategy vs. a metric strategy. This post will describe:
- What is the difference between a shipping strategy and a metric strategy
- Why metric strategies still lead to paradigm shifts
- What executives should help determine for metric strategies
- Induced incentives
- Metric prioritization
- How metric strategy naturally leads to experimentation
Ask teams to improve metrics, not to ship features
Technology companies spend their early life putting together shipping strategies. To explain what I mean, I'm going to borrow from Lenny Rachitsky's framework for how to prioritize product work:
- Mission: What are we trying to achieve?
- Vision: What does the world (or product) look like when we’ve achieved it?
- Strategy: How will we achieve our vision?
- Goals: How will we measure our progress towards it?
- Roadmap: What do we need to build to get there?
- Task: What should we do today?
The problem is that for most companies, metrics might not come into play at any stage of this hierarchy. Suppose you have a company with a mission of "Enable everyone to have their own business" and a vision of "Create a platform that anyone can build anything on". Here are two examples of strategy and goals, one which is a product shipping strategy, and one that is a metric strategy
The product shipping strategy is not a good fit for experimentation, and is generally bad. Shipping a bunch of features doesn't mean that they've been used, that customer's experiences have improved, or that it's furthered your progress towards your mission. In contrast, experimentation (and product led growth in general) is a great tool for pursuing a metric strategy.
In essence, metric strategy says leaders should dictate outcomes, and push product strategy down to teams.
Erik Bernhardsson: "As a leader at the top, you need to deliberately keep pushing ideation down to the bottom. People will come to you for product ideas, and the natural tendency is for leaders to say we should probably do this or that, but this unintentionally creates a culture where the leader dictates product strategy for the whole team. If you think about the information flow, knowledge is usually generated at the lowest possible layer, from people shipping things. If information has to go up and down the chain, you lose a lot. You need to push product strategy to the bottom."
Metric strategies lead to paradigm shifts
If executives start declaring metric strategies, companies can still achieve large paradigm shifts. My favorite example of this was Airbnb's instant book feature. For most of my Airbnb career, guests had to first ask permission for hosts to stay in their Airbnb, and hosts would get to decide if they like the booking request. This created a ton of friction, was a terrible experiences for guests, and opened the door for improper bias to enter the process.
A product shipping strategy would have dictated a bunch of features to ship. But Airbnb set up a strong team (led by Lenny himself!) with a metric strategy: make Airbnb reach 100% instant book. The result was an experiment heavy strategy that cumulatively shifted the marketplace. The product changes spanned search ranking, host onboarding, and core host functionality (e.g. guest controls, house rules, lead time settings, etc.) that allowed hosts to be successful in a world where guests book their home instantly.
In effect, leadership's metric strategy pushed the product strategy to the team, who could build and learn quickly.
What goes into a good metric strategy?
So how should a leader make the change from a shipping strategy to a metric strategy? There are a couple factors to weigh.
1. Get your incentives right
Metric strategies seem straightforward at first, with the simplest one being "increase revenue". But there is nuance to how metrics create incentives. Consider the following metrics for Airbnb:
Among these, Airbnb has always prioritized nights booked. This enabled the company to reward growth in lower cost areas like India and China, which was crucial for international network effects. But the company didn't solely prioritize user growth and use unique users making bookings for a metric, which would have ignored effects of increased engagement of existing users.
To illustrate how strong the incentives of this metric choice were, we didn't pursue changes in Airbnb's fee rate until 4 years into my tenure there. The amount of Airbnb host and guest fees were flat for years on end, because those changes never would have reflected in nights booked.
2. Prioritize your metrics
The additional complexity arises when multiple teams have metric strategies that collide. This is when it is crucial for leadership to define how decisions will be made.
As an example, a company like DoorDash has two metrics that reflect different priorities: # orders and revenue. A team working on the ordering experience is likely focused on # orders which indicates customers getting value from the product, whereas a merchant product team would be focused on revenue , which means merchants getting more money.
In this context, what should a company do if in an experiment, orders goes up and revenue goes down, or vice versa? This is not a question that the typical product team is equipped to answer, and it's ultimately a tradeoff that leadership should weigh in on.
Airbnb saw a similar issue between another set of metrics, % Instant Book and # Host Cancellations. As you might expect, hosts not having an option to reject guests before a booking often leads to hosts rejecting guests after the booking, and a number of experiments saw one metric improve and the other deteriorate. Each time, product teams needed a framework for making a call.
At Eppo, we plan on making this process easier. These same teams at Airbnb quantified the long term effects of increased instant book and increased host cancellations, creating a framework for making tradeoffs. We plan on adding this type of capabilities to help navigate these questions of metric prioritization.
Metric strategies lead to experimentation
There are strong reasons for shifting to metric strategies. It decentralizes product strategy and unleashes the entrepreneurial capabilities of your team, while still leading to leadership's vision.
But this type of strategy is intrinsically connected to experimentation. It is critically important to know how product changes affect your metrics, both positively and negatively. And it is always surprising to see the outsized effects on metrics that can happen from small changes.
At Eppo, our mission is to foster entrepreneurial culture within companies, and we think the best way to do this is to enable every company to pursue metric strategies via experimentation. In the next post, we'll dig into how to make metrics tractable for experimentation, creating a flywheel of product development for pursuing metric strategies.
Table of contents
Ready to go from knowledge to action?
Talk to our team of experts and see why companies like Twitch, DraftKings, and Perplexity use Eppo to power experimentation for every team.
Get a demo
