Engineering
Mobile Experimentation with Eppo: From Feature Flags to A/B Testing
Learn more
Early on at Opendoor, an engineer we’ll call Joe (because that was his name) added an in-house feature flag system to the codebase. “They’re nice to have,” explained Joe. “We had them at my old job.”
Years later, Joe’s “one-afternoon side project” system was relied on by multiple teams for mission critical work.
The changes in traffic volume and usage patterns had the feature flag system coming apart at the seams. We eventually migrated to a proper feature flag solution, but not until we were forced by a services architecture transition.
Why didn’t we migrate sooner? First, we had work to do, but second, there was a feeling of FUD, that migrating a feature flag tool would both be a time-sink and a potential for outages - in other words, a nightmare.
Seven years and four feature flag migrations later, as an engineering leader at Opendoor and MasterClass, I can tell you: migrating feature flags is fine, actually.
The truth is, feature flagging is one of the easiest systems you can migrate. The migration may not be quick, but it is straightforward - and relatively painless.
If the mere idea fills you with dread, here’s a playbook to help you get it done.
Why Migration Seems Scary, and Why It Isn’t
The process always starts with a lot of FUD (fear, uncertainty, and doubt). Engineers dread potential risk to product stability, while managers worry about the time and resources needed.
Here’s the unlock: it’s not a big deal to have two feature flagging system in use at once. It’s fine. It’s mildly annoying at worst. In fact, it’s already the case at plenty of companies who use different system for platform vs product/growth use cases (See: “What’s Wrong With Feature Flags?” on this blog), or a different system for web vs mobile flags.
The imagined nightmare scenario would be some rogue release that’s breaking the site in a way that’s costing millions of dollars a day, and somebody needs to wake up in the middle of the night to turn it off, only to find that it doesn’t live in feature flagging system one, it lives in system two. And now it’s going to take longer to remedy.
And even that is not so bad. If you have a decent on-call setup, the person being woken up is the person on the team who knows about the experiment. Their team should have context on how to turn it off. Worst case, it’s an extra hop of waking up the right person, adding up to maybe a miniscule chance for an extra hour of downtime.
Once we accept that running two feature flag systems is okay, mapping out a migration becomes far less daunting.
Migrating Feature Flags Without Tears
Start by categorizing your feature flags. Most feature flags fall into two categories:
1. Short-Lived Flags: These are used for experiments or temporary feature rollouts, typically lasting a few weeks or months.
if FeatureFlagTool.get(“redesign_onboarding_v3”) == “new”:
...
else:
...
2. Long-Lived Flags: These remain in the codebase indefinitely, often for use cases like permanent config or kill switches, here for emergencies.
if FeatureFlagTool.get(“failover_db_enabled”):
...
else:
...Here’s how to approach each:
Step 0: Pick a New System
Before kicking off a migration, make sure your engineers both know about and actually like the new system. Do a proof of concept: ask a single team to start using the new system for their next few flags, and see how they like it. If everything goes well, have them give a talk to the rest of engineering sharing their experience.
Step 1: Implement the New System
If and when your team is bought in and you are set on migrating, it’s time to make a new rule: all new flags must be implemented in the new system. If need-be, you can move even more gradually: start implementing the new system within a single organization for a quarter or so, and only migrate the rest of engineering once a clear approach has been established.
Step 2: Phase Out the Old Flags
As the short-lived flags start to be cleaned up in the normal course of work, the number of flags in the old system will start to decrease. However, some teams are often behind on their feature flag cleanup, overwhelmed with other priorities.
To help them along, establish a strict policy for deprecating old flags in somewhere between 6-8 weeks* after their first appearance in the codebase. This way, natural attrition will transition the majority of your feature flags to the new system without going out of your way.
*In general, feature flags should be deleted within a sprint of the rollout they control. To illustrate, imagine a new flag is added to your codebase, and the feature it gates is rolled out to 100% of users thirty days later. You should plan to remove this flag by Day 45.
In particularly large orgs, assessing the state of the rollout may be less obvious, in which case the 6-8 week rule is an easier-to-enforce option.
For many companies, these first two steps will be all you need. The combination of new flags in the new system and old flags being removed will have you magically migrated over in about a quarter or so.
Step 3: Address Long-Lived Flags
Audit the remaining flags - at this point, this is usually a pretty manageable list. For each, ask: “Do we still need this one?” When nobody quite knows what a flag does, it’s important to take the time to understand it before removing or migrating; luckily, this is much easier to do for the small handful of remaining flags.
Upon investigation, it’ll turn out a significant portion of the remaining long-lived flags can be retired. Congratulate yourself on the proactive spring cleaning.
For those that must remain, replicate them in the new system and verify their functionality. This may involve manual testing or implementing a wrapper to compare outputs from both systems.
There are a few edge cases you may encounter that would require more effort to migrate. One example would be flags that use audience attributes - I.E., this flag is on for mobile users in Germany only. You’ll need to recreate the ability to support those attributes in the new system first. Luckily, long-lived infra-ey flags rarely rely on these.
Wrappers and Error Alerting
For particularly risk-averse companies, this can be made even easier by using a wrapper (i.e. feature flagging your feature flags). A wrapper can call both the old and new systems and flag any discrepancies between the decisions returned for a certain user or context, helping to automate away some of the manual testing. This is getting pretty fancy though - plenty of smaller Series B or Series C companies shouldn’t need to worry about this at all.
def evaluate(self, flag_name, *args):
# while migrating, evaluating both systems before we swap
old_answer = OldSystem.evaluate(flag_name, *args)
new_answer = NewSystem.evaluate(flag_name, *args)
if old_answer is not new_answer:
Error.log(“MigrationInconsistencyWarning”, f”evaluated {flag_name} but got Old: {old_answer}, New: {new_answer}”))
return old_answerIf you fall into that top 5% or so of super meticulous engineering orgs and are interested in digging into this approach further, Stripe has written a great guide on “Online Migrations at Scale” that may be helpful.
Keep it Simple, Silly
The key is not to over-engineer the process. Migration should be simple and focused. Resist the urge to “boil the ocean” with exhaustive training sessions or excessive documentation.
Key Signs You’re Overcomplicating It
- Overcomplicating Training: If your new system requires extensive training, it’s the wrong system. Pick Feature Flag systems that are intuitive and user-friendly. Everything end users need to know about how to use the system should fit on a single page. You don’t need to write a doc for “How to Use Feature Flag system X,” you need “How Feature Flag system X works at Our Company” which should include a couple of clarifications and then a link out to the new system’s docs for anything else.
- Worry Over Accelerated Timelines: If your instinct is to do this as quickly as possible, it will seem especially tempting to avoid a migration altogether and just stay with your less-than-ideal current system. If you go slowly, and adopt good flag clean-up practices, the migration will happen by itself over time - typically a quarter. This approach minimizes disruption while allowing teams to adapt naturally.
Final Thoughts
When migrating feature flags, don’t optimize for speed; optimize for simplicity and clarity. The FUD that surrounds these migrations is mostly a smokescreen - clear it away, and you’ll find that you’re okay. And, of course, if you need help - hit me up.
Table of contents
Ready to go from knowledge to action?
Talk to our team of experts and see why companies like Twitch, DraftKings, and Perplexity use Eppo to power experimentation for every team.
Get a demo
