AI Enabled Creators Need AB Experiments

As the cost of implementing ideas goes to zero, evaluating ideas becomes the bottleneck

Che Sharma

Eppo's Founder and CEO, former early data scientist who built experimentation tools and cultures at Airbnb and Webflow

This is the 2nd part of our AI manifesto. The first highlighted a secret lying in plain sight: the best AI companies treat AB experiment infra as part of MLOps, evaluating every model with an AB experiment.

This post is about what’s different in today’s AI models, how they will enable a wider class of creators that need a vehicle to evaluate their ideas. It’s exciting to see prompt-writing democratize idea development by replacing code and visual graphics skills. But with so many ideas, companies need to separate good and bad.

Generative AI will speed up development

The biggest surprise of generative AI models is how effectively they turbocharge creative development. Where legacy AI systems might predict a price or a category, generative AI can discover and implement an entire idea.

To illustrate, even early phases of generative AI models have proven three capabilities:

Large Language Models can power idea generation. In a few seconds, ChatGPT and LLaMa can match the output of hourlong, 10-person brainstorms.
Tools developed on top of Stable Diffusion can rapidly develop creative assets to test.
CoPilot can speed up programming, making it easier and faster to develop and deploy new features.

Together, humanity has a new, powerful flywheel for putting new ideas into the world. If you’re a marketer at an e-commerce platform, you can prompt ChatGPT for fifty different versions of an autumn-themed newsletter. If you’re a designer of athletic shoes, you can prompt Runway to create hundreds of new sneaker mockups.

But what happens when 100x more ideas can be generated and deployed?

More ideas deployed, more ideas to evaluate

As the labor costs of innovation decrease, it becomes easier and faster to test more ideas and hypotheses. A pipeline of products that once required large teams to execute now just needs one person who knows customers and AI prompting.

But Generative AI won’t speed up innovation without faster idea evaluation. Companies who invest in AI-powered productivity tools without AB experimentation will struggle with the firehose of options, whether it’s choosing among a hundred sneaker designs or deciding to ship the 10x volume of feature ideas that were prompt-engineered. They will have to decide between an anarchic clutter that accepts all ideas (including the middling and actively bad ones), or a traffic jam that stymies innovation under the backlog of analysis.

Experimentation is the only way to turn speed into quality.

AI tools and AB experiments will unlock entrepreneurial spirit

At Eppo, we have believed from Day One that experimentation has the power to drive an entrepreneurial culture. What’s most exciting about AI-powered applications is their promise to fundamentally reshape our working environment.

AI tools and AB experimentation will enable all employees to prove the value of their ideas. Today, a customer support agent with a great idea would have to wage a large political campaign to get design and engineering resources for the idea to see the light of day. We are rapidly entering a world where this support agent can prompt engineer a mockup and even the software implementation with a solution all on their own.

But imagine a world with AI tools and no AB experimentation capabilities. Even with zero implementation cost, this idea would never be acknowledged. If the agent’s idea went out, its effect on metrics would be invisible amidst all of the other launches, marketing campaigns, and world events going on. More likely, the idea would never go out, since the lack of guardrails and an established evaluation process creates too much risk.

AB experimentation unlocks entrepreneurial spirit, offering every employee the opportunity to take destiny into their own hands. When offered self-serve, built for trustworthiness, and comprehensively used across business units, AB experimentation creates an arena for great ideas with customers as the judge and jury. The coming AI wave will create more idea contenders, pushing on the arena to let them all compete.

Don’t burn cash, drive metrics

The world of generative AI holds incredible promise, releasing previously unimaginable products and unlocking a new wave of AI-wielding creators.

But even though AI capabilities are exciting to technology leaders, they ultimately have to excite customers. Companies that don’t invest in idea evaluation will see all of AI’s sound and fury amount to flat-lined metrics, with bad ideas cannibalizing good ideas in a frothy soup of activity.

Eppo is here to link AI investments to business outcomes, and to do it at the increased pace of AI-enabled product development. Our AB experimentation platform is warehouse-native and built with the bleeding edge of statistical techniques to speed up evaluation of ideas. But more important than features is the culture we aim to drive. We believe that AI will empower a new class of creators, and Eppo is their chance to prove their mettle.