Monitoring your SaaS product’s key business metrics and running experiments to improve them can often feel like a balancing act. Making a big change to one part of your app can positively impact some metrics, while negatively impacting others.
For example, if your new search algorithm provides more relevant results to users but takes much longer to load, it may still have a net negative impact on important business metrics like revenue.
What’s worse, sometimes you might not remember to monitor those “tangential” metrics closely, meaning negative results might not get caught early enough.
This is why it’s important to pay close attention to your guardrail metrics.
In this article, we’ll explore what guardrail metrics are and why you should be making them an integral part of every single one of your experimentation efforts from now on.
We’ll go over:
Let’s jump right into it.
Guardrail metrics are critical business indicators you closely monitor during experiments like A/B tests. Think of them as safety nets designed to catch potential negative side effects while you try to improve one area of your SaaS product.
Unlike your primary success metrics, which focus on the specific target of your experiment, guardrail metrics are designed to safeguard other important areas of your business.
They help make sure your changes don't unintentionally cause problems like reduced website speed, declining customer satisfaction, or a drop in overall revenue.
By tracking guardrail metrics, you get an early warning system. If an experiment starts to negatively impact a guardrail, it signals that adjustments or even a complete halt to your experiment might be needed.
This approach helps you make sure that your efforts to improve one metric don't end up hurting other vital aspects of your business.
Guardrail metrics play a vital role in keeping your business healthy and on the right track. Here's why they matter so much:
The RealReal operates a two-sided marketplace, bringing together both buyers and sellers of luxury goods. This dynamic presents a challenge: How do you encourage one side of the marketplace to grow without hindering the other?
Guardrail metrics offer a solution. Let's take a closer look:
Imagine the consignor (seller) team wants to boost the number of people selling their clothes on The RealReal.
They decide to experiment with a pop-up ad that encourages visitors to become sellers. This sounds great in theory, but what if the pop-up is so distracting that it drives potential buyers away, leading to fewer sales?
This is where guardrail metrics come in. By setting the "orders placed" metric (a key indicator for the buyer team) as a guardrail, the consignor team can track the impact of their experiment in real-time.
If the pop-up causes a significant drop in orders placed, it signals a potential problem. The consignor team can then investigate and adjust the pop-up to ensure it doesn't negatively impact the buyer experience.
So, what’s the takeaway?
This hypothetical approach shows how guardrail metrics can promote collaboration and guarantee balance within a two-sided marketplace.
By prioritizing the key metrics of both buyers and sellers, they can experiment confidently, knowing that efforts to drive growth on one side won't unintentionally damage the other. This creates a win-win situation where the platform thrives as a whole.
Netflix uses guardrail metrics to make sure its content is resonating with its subscriber base. Here's how some of their “do no harm” guardrails work:
Netflix uses specialized statistical checks like equivalence testing and non-inferiority testing to analyze these guardrails. Their focus is ensuring that even if an experiment isn't a win, it also doesn't cause unintended harm.
For more on these statistical checks, you can read the full report here.
Choosing the right guardrail metrics is like setting up a good security system. It's about knowing your weak points and putting alarms in the right places. Here are three key guidelines to help you pick the metrics that will truly protect your business:
Before anything else, take a step back and ask yourself:
What are the biggest risks to my business or product?
A new feature might boost one area, but what could it potentially damage? Maybe faster checkout speeds could lead to more accidental orders, or a focus on new user acquisition might make existing customers feel neglected. Pinpointing these potential pitfalls is the first step toward choosing effective guardrail metrics.
This list should extend to risks posed to all business goals, not just the primary focus of an experiment.
For example, many businesses know that decreases in website load time negatively impact revenue. To ensure that no experiment introduces additional load time, they measure this as a guardrail metric on all experiments.
Once you have a list of risks, you need metrics that directly track them. Let's say you're worried about customer satisfaction dropping. Don't just assume this will show up in revenue — have metrics like customer feedback scores or support ticket volume to give you specific and fast feedback if customer happiness starts slipping.
Remember: The key is to choose metrics that give you a clear signal of trouble in the areas you're most concerned about.
It's not just about the metrics themselves, but also about when they trigger an alert. A 1% drop in revenue might be negligible one week, but a serious warning sign the next.
Your thresholds should be strict enough to catch real problems, but not so strict that they constantly slow you down. Think of them as boundaries that give you time to react before things reach crisis level.
Finding the right balance between protection and innovation is key. Here are some tips on how to integrate guardrail metrics effectively:
A/B tests allow you to focus on specific changes. But it's easy to miss the bigger picture — how does this tweak impact other aspects of your product? Here's how to make guardrail metrics an integrated part of your A/B testing:
You now understand the importance of guardrail metrics for mitigating risk and promoting experimentation. However, the real challenge lies in setting them up with precision and a focus on statistical rigor.
This is where Eppo excels.
Eppo is a powerful experimentation and feature management platform that simplifies your experimentation process, from setting up robust feature flags to providing in-depth analysis tools that protect against unintended outcomes.
Designed for data-driven teams who value accuracy, Eppo allows you to conduct and analyze experiments confidently and define powerful guardrail metrics alongside your key success goals.
Eppo also allows teams to create “collections” of guardrail metrics to ensure they’re added to every experiment, even when anyone in the organization can self-serve experiment creation.
Here’s how Eppo makes the difference:
Ready to see Eppo in action? Book a Demo and Explore Eppo.
Protect your SaaS business during experiments. Learn what guardrail metrics are, why they're vital, real-world examples, and how to set them up in A/B tests.
