Coinbase Saves Millions, Reduces Experiment Analysis Time by 40%, and Restores Trust in Experimentation with Eppo
- 40%
Faster Experiment Analysis
- Millions
In fixed cost savings
- Trust
Rebuilt in Experimentation across the org
- Industry
Financial Services
- Headquarters
Distributed
- Company
Customers around the world discover and begin their journeys with crypto through Coinbase. 245,000 ecosystem partners in over 100 countries trust Coinbase to easily and securely invest, spend, save, earn, and use crypto.
- Data Stack
Snowflake, Databricks, dbt, Looker, Amplitude, Eppo
About
As a leading player in the cryptocurrency market, Coinbase’s rapid growth exposed cracks in its in-house experimentation system, CIFER. Lacking the sophistication required for a large enterprise, CIFER eroded trust, slowed decision-making, and led to inconsistent experimentation practices.
Tuhin Ghosh, Head of Platform Data Science, and Data Scientist Jeff Bliss spearheaded the search for a solution that could restore confidence in experimentation while delivering cost savings and operational efficiencies. They turned to Eppo.
Challenges
- 1. Critical Feature Gaps
- Technical Perspective
CIFER, Coinbase’s in-house system, lacked essential statistical tools like sequential confidence intervals and integrated sample size calculations, leading to inefficiencies. Attempts to build advanced features, such as CUPED, internally resulted in significant cost increases. The absence of an intuitive UI also made it difficult for business teams to engage with experimentation.
“We were missing basic features, and attempts to build them internally skyrocketed costs. For instance, our warehouse costs doubled overnight when we tried to implement CUPED. That’s when we realized these ‘simple’ features are more complex than they seem,” Tuhin explained. - Business Implication
Product launches were delayed, and teams worked with inconsistent data interpretations. Business leaders couldn’t self-serve experiment results, and product managers lacked the guardrails needed to make sound decisions. This hindered a culture of rapid experimentation and slowed innovation.
- 2. Fragmented Methodologies
- Technical Perspective
The absence of standard features forced each Data Scientist at Coinbase to create custom notebooks and artifacts to analyze and interpret experiments, causing inefficiencies and confusion.
“There was no consistency—every team was speaking a different language. New data scientists often found flaws in previous work, creating an ongoing cycle of frustration,” Tuhin noted. - Business Implication
This lack of standardization led to duplicated efforts and lost insights, making it nearly impossible to scale experiments effectively, preventing Coinbase from fully leveraging its data.
- 3. Low trust in results
- Technical Perspective
CIFER lacked robust error-checking mechanisms, and stale data often went unnoticed, leading to widespread skepticism among product managers and business teams.
“We were stuck in a cycle of patchwork fixes—nothing worked the way we needed, and nobody fully trusted the results,” Jeff admitted. - Business Implication
Business stakeholders hesitated to act on experimental data, fearing costly missteps due to incorrect analysis.
Tuhin remarked, “We were constantly firefighting—every experiment was met with skepticism and second-guessing.”
This mistrust slowed decision-making across the organization and compromised the speed at which new features could be rolled out.
The Tipping Point: A Costly Product Misstep
The final straw came when a new recommendation system, initially hailed as a success based on an experiment conducted through CIFER, was later found to have no impact on business metrics when re-tested. The fallout was costly: millions of dollars wasted on development, damaged credibility, and a clear signal that Coinbase’s experimentation process needed a complete overhaul.
The Solution: Eppo
With resource constraints making an in-house rebuild prohibitively expensive—requiring 10 engineers and 18 months—Coinbase sought an external solution. After a thorough four-month evaluation of both Eppo and Statsig, Coinbase ultimately chose Eppo for its superior ability to meet both technical and business needs, while fully complying with Coinbase’s stringent security and performance standards.
Why Coinbase picked Eppo
- Mature Warehouse-Native Offering
Eppo’s fully developed warehouse-native platform integrated seamlessly with Coinbase’s existing infrastructure, addressing security concerns and eliminating the need for a costly data migration. This enabled faster deployment with minimal disruption. “Eppo was built to work directly with our data setup, making the transition painless. We didn’t have to tear down or rebuild anything,” Jeff explained.
- Proven Expertise at Scale
Eppo’s extensive experience with large-scale B2C companies like Twitch reassured Coinbase that the platform could handle its complex experimentation needs. “I spoke with several customers from both Eppo and Statsig, and it quickly became clear that Eppo’s warehouse offering was built for the scale we operate at,” Tuhin emphasized.
- Comprehensive Feature Set
Eppo offered a more robust suite of advanced statistical features, including CUPED++ and sequential testing, paired with error-checking capabilities that ensured teams could trust the data. “Eppo provided everything we needed, out of the box,” Jeff remarked, enabling Coinbase to standardize experimentation across teams.
- Seamless Support and Collaboration
Eppo’s support team worked closely with Coinbase to ensure a smooth and fast transition, allowing both technical and non-technical teams to quickly become proficient with the platform. “We anticipated the migration would take two or three quarters, but Eppo’s support team had us up and running in just over one,” Tuhin shared.
Results
- 1$ Millions in fixed cost savings
By choosing Eppo over building an in-house solution, Coinbase saved the equivalent of 8.5 full-time engineers, translating into $XM in hard cost savings. The decision also avoided an 18-month development timeline.
- 240% Faster Experiment Analysis
Eppo’s streamlined workflows and intuitive UI helped Coinbase’s data team reduce experiment analysis time by 40%, enabling faster decisions and allowing more focus on strategic projects. “The UI is much more intuitive. Our data teams are now happy to engage with the system, and the time savings have been huge,” Jeff noted.
- 3Rebuilt Trust in Experimentation Across the Organization
Eppo restored trust in experimentation, significantly reducing negative chatter about unreliable results. “There’s been a major shift in how business teams view experimentation. They trust the results now, and it’s speeding up our ability to make key decisions,” Tuhin explained.
Conclusion
Eppo has delivered immediate value to Coinbase by addressing technical shortcomings and rebuilding trust in experimentation. While the long-term impact will unfold over the next 2-3 years, Coinbase has already seen major improvements in efficiency, cost savings, and renewed confidence in data-driven decision-making. Eppo has laid the foundation for experimentation to become a key driver of innovation and growth at Coinbase.
