WHY EPPO

By Role

Data Scientists Engineers Product Managers Product Managers

Resources

Customers Outperform Updates White Papers White Papers

FEATURED CASE STUDY

Coinbase Saves Millions, Reduces Experiment Analysis Time by 40%, and Restores Trust in Experimentation with Eppo

Learn more

About

Strategy

March 20, 2025

The Long Game: Effective Strategies for Optimizing Delayed-Impact Metrics

Greg Dale

Before Eppo, Greg was the CEO of Tech for Campaigns where he led large consumer advertising campaigns and experimentation programs

When optimizing products, immediate feedback metrics like clicks, signups, or purchases offer quick insights for testing and iteration. Yet, the most impactful business metrics often take much longer to surface, such as:

Customer lifetime value (LTV)
Churn rates
Refund patterns
Customer service ticket volumes

These long-term metrics pose a distinct challenge. Relying on them delays optimization efforts while focusing solely on short-term signals risks prioritizing quick wins at the expense of sustained business success. This creates a core tension between speed and thoughtful, long-term outcomes.

The Challenge of Long-Lead Metrics

Optimization techniques like A/B testing and multi-armed bandit algorithms both excel with instantaneous feedback loops, but serve complementary purposes. A/B testing provides rigorous statistical validation of specific hypotheses, while bandits enable dynamic optimization through rapid exploration and exploitation. Both approaches are particularly valuable for products with short decision cycles, as their results can drive rapid iteration and can be effectively combined to balance statistical confidence with adaptive learning.

However, problems arise when immediate metrics fall short in representing long-term success, such as in these scenarios:

High-investment products where customers need extended time to evaluate and commit, like mattresses, auto, personal finance, and. A customer’s first interaction with the product could be months before purchase: this delay in conversion signals complicates optimization.
Long-term revenue models like SaaS platforms or subscription services, where value accumulates gradually. Relying solely on short-term signals can paint an inaccurate picture of success.
Delayed quality indicators, including customer satisfaction surveys, refund/returns, or support requests, often take weeks or months to surface.

These factors make optimizing for the long run both challenging but no less essential, demanding a thoughtful balance between patience and agility.

Three Effective Approaches for Long-Lead Metric Optimization

1. Proxy/Surrogate Metrics

Proxy or surrogate metrics act as early indicators that align closely with later, more valuable outcomes. Essentially, they serve as substitutes for outcomes that are either not yet observable or inaccessible due to data limitations.

The process involves pinpointing metrics that appear relatively early in the customer journey and consistently predict downstream outcomes. If a software subscription offers a three day free trial, the point at which users add a credit card – showing value and an “aha” moment – might be the metric to focus on. For instance, if data shows that 80% of users who put in payment info during the first week ultimately become long-term customers, then driving seven-day “add payment” becomes a practical proxy for optimizing marketing efforts.

While surrogate metrics can be powerful, they come with key challenges:

Stability is crucial: The correlation between surrogate metrics and long-term outcomes must remain consistent over time and across diverse user groups. Focusing too heavily on the surrogate might disrupt the relationship with long-term metrics.
Tailored metrics could be required: Different initiatives may benefit from distinct surrogate indices depending on how they influence user behavior and outcomes.

By using surrogate metrics thoughtfully, businesses can bridge the gap between short-term visibility and long-term value, but the approach demands ongoing validation and adaptation.

2. Nested Optimization

Eppo offers Contextual Bandits, a powerful implementation of “multi-armed bandit” algorithms that help test choices and then scales the winners automatically. Bandits ingest short-term rapid metrics so they can rapidly adapt to changing circumstances, which creates the need to monitor longer-term metrics for business impact alongside it. Nested optimization merges the adaptability of bandit algorithms with the methodological rigor of A/B testing. This approach layers short-term optimization (via bandits) within long-term experimentation (via A/B tests), creating a framework that addresses both immediate feedback and long-term outcomes.

Implementation Approaches

An effective implementation might look like this:

Launch a bandit aimed at serving the right content or offer to a customer to minimize churn, using short-term surrogate metrics that have mapped well to churn;
Place that bandit within an A/B test that measures the actual churn metric over the long-term; for example a 10-20% holdout of available users.

3. Predictive Modeling

Predictive modeling upgrades surrogate metrics to incorporate multiple features or attributes into making a more accurate prediction of user behavior up front. These models, trained on historical data, allow organizations to forecast long-term outcomes for current experiments, bridging the gap between immediate insights and extended timelines without relying on delayed metrics.

You may choose to train a model that predicts outcomes like lifetime value (LTV) or churn probability within the first few days or weeks of a user’s engagement with your service.

As opposed to simply using a surrogate metric, modeling can capture some important nuances, like:

Situations where a small group of top spenders generates outsized value above simply a metric like “likelihood to pay”
Using multiple user characteristics to make more accurate predictions – a user who is showing multiple weak signs of engagement may need to score higher than someone who has a more mixed profile.

These types of models are an important tool for many different kinds of digital measurement, not just for metric optimization on-site or in-app but also in marketing campaign optimization. Meta recently open sourced a new tool, LTVision, that aims to make it easier for advertisers to incorporate their first-party data into advertising optimization.

Open-source tools like Meta’s LTVision exemplify the practical application of predictive modeling for long-term value.

Measurement Considerations

When analyzing experiments involving long-lead metrics, teams face an important methodological question. Should they:

Only analyze users who have reached a certain time threshold (e.g., 28 days post-exposure), ensuring complete data but potentially excluding recent participants
Include all enrolled users regardless of exposure duration, maximizing sample size but potentially diluting results with incomplete data

The answer depends on specific circumstances, including:

Experiment duration and user acquisition patterns
The stability of metrics over time
Business priorities regarding speed versus certainty
Statistical power considerations
How damaging dropping censored outcomes can potentially be.

Both approaches have merit in different situations, and experimentation platforms like Eppo support flexibility in this regard.

Implementation in Eppo

Eppo’s experimentation platform is designed to support all three strategies for tackling long-lead metric optimization, offering adaptability to suit varying business needs and data availability.

Metric Creation and Management

Eppo enables teams to define metrics at the entity level (e.g., a user), allowing flexibility in how different aspects of the business are measured. This capability supports implementation of the following approaches:

Define surrogate metrics that align closely with long-term outcomes.
Develop nested optimization frameworks that balance near-term and long-term objectives.
Leverage predictive models to project long-term performance metrics.

Cohort Analysis Options

When conducting analysis, Eppo offers flexibility in cohort inclusion methods, empowering teams to choose between two approaches:

Focus only on users who have passed specific time thresholds to ensure complete data accuracy.
Include all enrolled users to maximize sample size and improve statistical power.

Choosing the Best Approach

Organizations addressing challenges related to long-lead metrics should weigh several factors to determine the most appropriate optimization method:

Data Maturity: Predictive modeling depends on a solid foundation of historical data, while surrogate metrics can be implemented even with limited data availability.
Metric Stability: Surrogate metrics are most effective when the relationship between short-term and long-term indicators remains steady over time.
Technical Complexity: Advanced approaches such as nested optimization or predictive modeling typically require more technical expertise compared to simpler proxy metrics.
Business Urgency: Finding the right tradeoff between speed and reliability of results will depend on the specific urgency and competitive dynamics of the business.

Often, the best solution combines multiple strategies, tailoring them to the organization’s unique challenges. By thoughtfully addressing the complexities of long-lead metrics, teams can preserve the fast pace of experimentation while keeping long-term business goals firmly in sight.

Closing Thoughts

Effectively optimizing for long-lead metrics involves striking a balance between the need for quick experimentation and the focus on long-term business impact. Organizations can address this challenge by leveraging proxy metrics, implementing nested optimization strategies, using predictive models, or combining these methods to suit their needs.

The most suitable approach will depend on your organization’s unique business circumstances, data maturity, and technical expertise. Whichever path you take, maintaining a dual focus on both early indicators and long-term outcomes will ensure your optimization efforts contribute to meaningful and sustainable business growth.