We unveil the story behind Fun With Flags (FwF), our Experimentation Platform at Delivery Hero. Learn how FwF moved from a feature-flagging tool to an experimentation platform, influencing strategic decisions across the organization. Read the article to get a glimpse of where we started and where we are going.
The Study of Flags
The name of our global Experimentation Platform is Fun With Flags. And that is very telling about the origin of experimenting at Delivery Hero.
In the realm of software, a flag signifies specific code conditions, serving as a crucial signalling tool. When it comes to experiments, flagging involves the random assignment of individuals to various code conditions – such as A, B, or even N – determining the path their subsequent code execution follows.
Interestingly, during the platform’s birth around 2018, the popular TV show “The Big Bang Theory” was at its peak, bringing joy to our engineers in Montevideo, Uruguay. Inspired by the characters Sheldon Cooper and Amy Farrah Fowler, who created a YouTube show called Fun with Flags to explore the intriguing realm of vexillology (the study of flags), our platform proudly bears the same name.
At its core, experimentation delves into the study of flags, viewed through a statistical lens, offering profound insights on a higher level of abstraction. This fundamental concept drove us to embrace the name of the show, Fun with Flags, as a testament to our statistical approach and the excitement it brings to the experimentation journey.
Scaling to 100 Million Flag Requests
In the beginning, FunWithFlags started off as a focused solution for feature flagging, catering directly to the needs of software engineers. We recognized the vital importance of providing a reliable tool to toggle features on and off, particularly during quick hotfix deployments.
What followed was nothing short of a transformation. Within just a single year, FunWithFlags wasn’t just another tool in the developer’s toolkit; it became a lifeline for our engineering processes.
However, what transpired over the course of a single year was truly astonishing. FunWithFlags evolved from being just another tool in a developer’s arsenal to becoming the very backbone of our engineering processes. The surge in demand was extraordinary, with an astounding 100 million requests per week pouring in. The remarkable adoption rate spoke volumes—90% of all Delivery Hero platforms seamlessly integrated FunWithFlags into their workflows. What began as a utility has now blossomed into an indispensable pillar of functionality for Delivery Hero’s global operations.
Of course, such rapid growth didn’t come without its fair share of challenges. The sheer volume and diversity of requests put our architecture to the test, forcing us to push the boundaries of performance, reliability, and user-friendliness. These obstacles fueled our evolution. It wasn’t just a matter of scaling up; it was about scaling intelligently and optimizing our services to cater to the intricate demands of a global operation encompassing multiple brands. We were determined to rise above the hurdles and deliver an exceptional user experience at every turn.
Evolving into Experiments
Initially, our laser-focused approach to feature flagging was instrumental in meeting our development velocity needs. However, it came at the expense of experimentation validity. Merely setting up a feature flag, targeting a specific audience, and observing conversion rates after an undefined period of time fall short of proper experimentation practices. The challenge then became: How do we educate thousands of users on the right way to conduct experiments?
To address this, we established constant communication and fostered a community dedicated to Experimentation. Sharing best practices became paramount. But that wasn’t enough. We had to expand our horizons and embrace rigorous, scientifically-based processes for experimentation. This involved implementing hypothesis-driven flag setups, accurately estimating optimal runtime, conducting runtime health checks (such as SRM), and employing appropriate statistical tests with precision and sensitivity.
Fast forward to the present, and automation has become a game-changer. We’ve simplified the experimentation process for users, automating decision-making and ensuring robust, valid choices. This has allowed us to democratize experiments, opening up the opportunity for all Delivery Hero teams to engage in impactful decision-making through experimentation.
Where We Are Going
At Delivery Hero, our primary focus lies in conducting delivery industry-specific, precise, and swift experiments. Leveraging the wealth of data at our disposal, we have developed simulation-based approaches that ensure optimal randomization and statistical testing.
To expedite experiment durations, we employ Group Sequential Testing techniques that maintain the highest levels of accuracy and sensitivity. Additionally, we offer holdout features to facilitate concurrent experiments, providing a meta-view of their impact. Our metrics are fully customizable to cater to the unique needs of the company.
And, of course, we continue to support feature flags as well. This allows switching between experiments and feature flags, enabling a delightful experience of exploring the endless possibilities with flags at Delivery Hero. We prioritize both versatility and precision to empower our teams in their quest for data-driven decision-making.
Conclusion
FunWithFlags has undergone a remarkable transformation, evolving from a handy tool for feature flagging into a grown-up experimentation platform that serves as the beating heart of Delivery Hero’s operations.
We’re not done yet! At FunWithFlags, we constantly explore new frontiers and push the boundaries of what our platform can accomplish. We’re determined to bring even more advanced features to the table, empowering users with the ability to conduct precise and rapid experiments like never before.
If you like what you’ve read and you’re someone who wants to work on open, interesting projects in a caring environment, check out our full list of open roles here – from Backend to Frontend and everything in between. We’d love to have you on board for an amazing journey ahead.
