Culture
How to Hackathon: Where Experimentation Mindset Meets Rapid Action
6 Hackathon tactics we’ll be using again - and you can steal
Learn more
How We Ran Successful Experimentation Office Hours at Groupon
Ideas for training and nurturing experimenters across the company
During my time on Groupon’s experimentation platform team, one of our biggest challenges was building our tooling to allow product managers to run experiments self-serve. Like many teams, we had limited data science and analytic resources. We needed to remove ourselves from the process as much as possible to prevent bottlenecks in launching experiments and free ourselves to expand the platform's capabilities.
Building a platform for product managers presents interesting constraints. First, PMs typically don’t have statistical and analytical skill sets by default. Designing tooling required a thoughtful approach to help set up and interpret results as intuitively as possible and create guardrails to ensure users were implementing sound experimental design.
Even with guardrails in place, users still required quite a lot of support: new employees needed training and onboarding, and our platform lacked coverage for all use cases or custom metrics, so guidance was required for off-platform experiments. Novel experiments requiring new approaches to analysis popped up more and more as our product portfolio became more complex.
That’s why we launched Experimentation Office Hours. Every time I share the story of our office hours, teams are interested in standing up something similar for their end users. If that’s you, here are some ideas and insights you can take from what we did at Groupon:
Start Small and Scale With Demand
When I first joined the team, we had just one weekly office hour. Keeping it to a single session mitigated one-off meetings and protected our data scientists from meeting creep.
Nevertheless, over time, I started to notice more and more one-on-one meetings sneak onto my calendar. In fact, I distinctly remember one week when I had returned from a conference that was booked solid with meeting requests for one-off experiment consultations. We had a larger team by then, so we decided to increase our office hours to four times a week, giving more support at different times and days of the week.
Keep An Open Agenda
Our office hours didn’t have a fixed structure—people could join in at any time, whether they had questions or not. In fact, some PMs who were newer to experimentation would join just to listen to others’ questions and bring their laptops to get work done.
We answered questions in the order of arrival, although sometimes someone suspected they had a more complex question and let someone with a simpler question proceed.
Open office hours were uncomfortable for some people at first. Statistics can be intimidating, and no one wants to ask a dumb question in front of their peers. At the same time, as experimenters, we want to encourage curiosity and discourage the fear of failure. Fortunately, Groupon as a company really celebrated continuous learning as a core value, so the onus was not entirely on our team to create this culture. We simply reinforced it.
It is critical to meet your users where they are and explain things as simply as possible, i.e., don’t explain jargon with more jargon. Your PMs aren’t statisticians—and no one wants them to be. It is more important to give them some intuition for how the stats work than to give them formulas and theory (although some PMs are into it!). We aimed to empower, armed with an abundance of empathy.
Don’t Defer the “FAQs”
Of course, one of the most common questions people most often wanted to know was how long their experiment would run. We helped out with many power questions and how to reason about the right MDE for their experiment.
Some questions were nuanced, while others, like, “What is power?” are pretty straightforward to put into a FAQ. Could we have deployed a bot or some other low-touch solution? In 2016, it wouldn’t have been very good, but sure, let’s say we could have. Our PMs would not have looped us into escalations or perceived us as partners, which is one of the things I see data scientists craving in their collaborations with their product colleagues.
By investing in a regular cadence of touchpoints, you are also investing in the development of trusting relationships and building a community of experimenters along the way. You build trust with your stakeholders, create visibility into your work, and impact on enablement, which isn’t always easy. None of this is small stuff!
Winning Attendance: It’s Not Just the Office Hours
Regarding how we got people to show up, the most important insight I can share is that our office hours were just one part of a more comprehensive support approach.
We had a really good beginner-friendly Stats 101 video of a talk my colleague had delivered at the company many times. We included it in our experimentation onboarding packet when new PMs and analysts joined the company. This ensured everyone started from the same place and began developing an intuition and comfort with statistical concepts and experimentation at Groupon.
We also had a public support channel in Slack monitored by a dedicated developer and data scientist. More specifically, we had a support rotation, and as a data scientist dedicated to product support, your primary job was to monitor the channel and help with support requests. Think, “Why isn’t my experiment getting sessions?” or, “These numbers look wonky.” Sometimes it was quiet, and you could work on sprint work. Other days it would be a full-time job. Supporting an in-house experimentation platform can be quite labor-intensive.
Triaging the support channel was the most effective way for us to redirect to our office hours. “Can you look into xyz?” is a deeper dive more appropriate to the support queue whereas, “Can you help me think through designing my experiment?” is best accomplished one-on-one in an office hour.
Redirecting consistently is really important. In experimentation, rarely is any question “real quick”. Almost whenever I caved and engaged with a DM or took a meeting, it led to a much deeper issue that took me away from my work for hours or days, and I regretted it. So much of our work is tied to human behavior and while I loathe dogma, setting bad precedent can negate your efforts towards standing up a successful experimentation program.
Redirecting should not sound punitive. It can sound like, “Is this something that can wait until office hours on Wednesday?” Most of the time, that’s all it takes. Of course, some things are urgent or cannot be addressed within office hours, and a separate meeting might be required. Again, the goal is to reduce one-off meetings, not to be dogmatic.
We would also frequently have to redirect DMs, which can feel a little annoying: asking your colleagues to repost to the public channel. I found that reminding them that it is a good question and others might have the same question, or that sharing context between my team members was helpful in case we had to pull in a developer, for example, would typically put people at ease. In protecting our time, we introduced friction into someone else’s job. I found people were more understanding when they understood the why.
Know Your “Customer”
There’s no doubt that the relationship building was a huge benefit from our Office Hours program. But they were also invaluable for direct feedback about your tooling and processes.
We could see and hear firsthand what was and wasn’t working, where people’s pain points were, and be tapped into new experiment requirements. When you build trusting relationships with your partners, you’re more likely to get fast, actionable feedback and maintain positive relationships as opposed to deteriorating trust and harboring resentment.
As my first manager and mentor, Chris Powers, said on my first day, “Cardinal Sin #1 is staying stuck.” Office hours can help keep people confident they’re on the right track.
Experimentation is fundamentally a people problem. Of course, we tackle technical challenges as experimenters, but that’s typically the easy stuff - it’s easier to define, scope, and solve technical problems given the right resources.
People problems are “squishy”. By contrast, they can be hard to define and even harder to solve. They require solid communication, change management, negotiation skills, and, most importantly, trust to effectively define processes and influence collaborators and stakeholders across many different functions.
As experimenters, we often solve problems at the intersection of technical and human factors. This isn’t always easy and might require an experimental mindset and trying things out to see what works best for your culture. It definitely takes time. But sometimes, we need to slow down to speed up. It’s a bit cliche, but somehow, we all need reminding from time to time.
Table of contents
Ready to go from knowledge to action?
Talk to our team of experts and see why companies like Twitch, DraftKings, and Perplexity use Eppo to power experimentation for every team.
Get a demo
