Turning AI hype into tangible business value through experimentation: Inventa's Journey with Snowflake and Eppo
Table of contents
About
Inventa is an online wholesale marketplace in Latin America, offering a range of services, including credit and selling tools, to empower suppliers and small and medium-sized retailers. With over 250 employees and more than $80 million in funding from top venture capital firms like a16z, Greylock, and Founders Fund, Inventa has emerged as a trailblazer in the industry. Today, it serves more than 1,000 brands, 50,000 products, and 25,000 retailers.
Data is at the core of Inventa's competitive advantage. Its digital platform creates visibility and insights for both suppliers and retailers – while also extending credit terms to retailers. Retailers can discover and purchase wholesale inventory, and suppliers can easily list their products for sale and track sales performance. None of this is possible with traditional competitors that use traditional approaches such as phone calls, paper catalogs, and in-person trade shows.
Daniel McAuley serves as the Head of Data at Inventa, overseeing Data Engineering, Data Science, and Data Analytics teams, totaling 25 members. Daniel’s team has been instrumental in using data to develop a variety of data and AI products for retailers and suppliers. Powering those products is a modern data stack using Snowflake, Fivetran, Rudderstack, dbt, OpenAI, Hex and Eppo.
The Problem
"The results could not be trusted, not even by the data team, let alone the broader business. Starting a new experiment was like taking on a full-blown engineering project, which seriously slowed us down from running more experiments. We couldn’t even implement baseline features like setting up default guardrail metrics across the organization. As a result, different teams with their own unique goals were blind to each other’s metrics, and that led to all sorts of clashes."
In order to measure the business impact of data and AI products they build, Daniel’s team runs experiments for every change they ship. These experiments span a diverse range of AI use cases - from predicting shipping cost and shipping time to ranking search results and automating sales quote creation powered by large language models.
Before Eppo, Daniel’s team used another experimentation tool for randomization, relied on ad-hoc Python notebooks for running statistics and analyzing experiment results, and used Google Docs to share results with the broader team. This fragmented tooling came with major issues around trust, efficiency, and collaboration:
“The results could not be trusted, not even by the data team, let alone the broader business. Starting a new experiment was like taking on a full-blown engineering project, which seriously slowed us down from running more experiments. We couldn’t even implement baseline features like setting up default guardrail metrics across the organization. As a result, different teams with their own unique goals were blind to each other’s metrics, and that led to all sorts of clashes.”
Because experimentation is at the core of Inventa’s product development, ramping up their experimentation velocity and building a successful experimentation program was critical to sustaining the growth of the business. Their existing setup just wasn’t cutting it.
Trust and Warehouse Readiness at the Forefront
After a few weeks of vendor evaluation, Daniel selected Eppo as the go-to experimentation platform, noting, “I asked a few data leaders in my network for their recommendation and Eppo kept popping up as the go-to choice when your data warehouse is the source of truth for business metrics.”
Trust and Warehouse Readiness stood out as the key dimensions in Daniel’s evaluation: “Eppo’s rigorous statistics engine and data diagnostic features were exactly what we needed to instill trust in the underlying statistical assumptions and data quality.”
- Statistical Foundation Matters: Daniel observed that while several vendors boasted advanced statistical features like CUPED and sequential testing, their underlying mathematical assumptions and implementations were not the same: “Our aim was to raise the bar in running accurate experiments. The last thing we wanted was to make it easier for folks to launch experiments with shaky stats. When it comes to experimentation, precision is the name of the game, so opting for the tool with the strongest statistical foundation was a no-brainer.”
- Additionally, unlike other vendors, Eppo’s optimized data pipelines allowed Daniel’s team to run incremental data refreshes to only calculate what’s changed: “Getting our experiment velocity up was my main objective but as the data leader in the company, I also wanted to ensure that my Snowflake bill didn’t spiral out of control. Eppo's incremental data refresh provided me with the peace of mind I needed.”
A single platform for all AI use cases
Today, Eppo enables Daniel’s team to run experiments across various AI use cases:
Search and Ranking: This feature allows retailers to easily search and find the right product from the catalog. Running experiments here measure impact changes in search algorithm against top-of-funnel metrics like # searches, conversion rates etc.
Shipping Cost and Shipping Time: When it comes to shipping times, accuracy matters. The dilemma is whether to overpredict or underpredict. Overpredicting ensures timely deliveries but reduces conversions. Similarly, shipping costs also have an elasticity curve: underestimating incurs business costs, while overestimating lowers order numbers. Running experiments in Eppo has helped Daniel’s team unveil subtleties in price elasticity against their business's long-term metrics/goals.
Generative AI: Daniel's team has developed an OpenAI-powered data product that assists salespeople in handling more customer requests more efficiently. Instead of forwarding individual customer product lists to someone in the back office who sifts through catalogs and compiles quotations, the product automatically extracts all items from the list and generates quotes. Currently, Daniel's team is running experiments to optimize task completion times and boost metrics tied to the volume of goods sold by fine-tuning various parameters in the LLM model.
60% time saved per experiment
Eppo natively integrates with Inventa's data models and business metrics in Snowflake, eliminating the need for manual analysis or data exports. This has resulted in around 60% time savings, with the data team saving over 10 hours per experiment.
The benefits of time saved extend beyond data team efficiency; Daniel's team is no longer bogged down with follow-up questions, allowing them to invest more time in developing and fine-tuning machine learning models. This, in turn, leads to more experiments and successes.
Quantifying impact of AI projects
Adopting Eppo has given Inventa the ability to easily quantify the impact of all the work Daniel’s team is putting into AI and ML projects, and ultimately solidifying Inventa's position as a pioneer in Latin America’s wholesale marketplace.
Eppo has allowed us to turn the AI hype into real data products with tangible business value. Our team can now easily run high-quality experiments that help us understand the impact of each model we ship. We’re learning faster about which model changes are working, and more importantly, what’s not working. This has made it easy for me to justify to the CEO and Head of Finance the amount of time and resources we are investing in ML and AI initiatives. Eppo is the AB testing platform I wish I had at my last job.
.webp)