A Guide to Delivery Hero’s Recommendation System

By Aleksandra Kovachev

From delicious meals to groceries, flowers and medicine, our goal at Delivery Hero is to always deliver an amazing experience to our customers. Today, our customers have access to the best dishes and products they need, all at the convenience of our platform. However, the possibility of having so many choices in the palm of your hands can sometimes make finding and deciding what you want and need challenging.

This is exactly what the Recommendations and Ranking team at Delivery Hero solve for our customers. We show them what they want, and even go a step further – finding the “when” and “where”. 

If you are interested in doing this type of work on a daily basis, have a look at some of our latest Consumer Discovery openings.

The When & Where

The “When” and “Where” are unique challenges tied to the food delivery marketplace, which are not present for other recommendation platforms. For example: stocked products, news, video and audio. The two main differences emerging in the recommendations for the food marketplace compared to other marketplaces are:

  • Our customers tend to have their favourite restaurants that they are happy to order from again, and favourite dishes they wish to order again. They are not always interested in exploring new restaurants.
  • Even if we find a restaurant that best matches their needs, there are limitations we can hit like delivery areas and opening times. This is not the case when you recommend content or stocked products.

To tackle these challenges and show the best and most delicious recommendations to our customers, we strive to reach these objectives: 

  1. Use statistics, machine learning, and state-of-the art Machine Learning methods to develop and deliver relevant recommendation strategies.
  2. Design scalable and efficient data pipelines to transform data, run models and generate predictions.
  3. Develop reliable and fast API solutions to serve online and offline recommendations to our customers. 

How are recommendations organized? 

“In the beginning, the Universe was created. This has made a lot of people very angry and has been widely regarded as a bad move.” – Douglas Adams, The Hitchhiker’s Guide to the Galaxy

As the services offered by Delivery Hero expand, we separate recommendations into different levels and categories. At the top level we have vendor and product recommendations, for which we also have separate APIs. 

Additionally, each level has a sub category, which defines the type of marketplace the service belongs to, for example: restaurants, dmarts and shops.

For each of these combinations we develop and offer different types of recommendations, covering the most basic ones, such as past orders, popular but also personalized recommendations, complementary and similar recommendations. For each strategy we can build different models, upgrade and serve. All of our global brands can request a given recommendation strategy from us, along with a specific version of its implementation they would like to use. 

Organization of the recommendations structure at DH

Why we stand-out

“It is a mistake to think you can solve any major problems just with potatoes.” – Douglas Adams, The Hitchhiker’s Guide to the Galaxy

In order to deliver these different types of recommendations, our Data Scientists implement models that range in their complexities: some use simple heuristics and similarities, but others implement and combine the latest state-of-art Machine Learning methods like XGBoost, Doc2Vec, CNNs, TransferLearning, etc.

For example, some of our restaurant based recommendations are:

  • The “Popular near you” algorithm, which recommends the top most popular restaurants with the highest order and reorder score in a given time range. 
  • The “Trending near you” on the other hand, looks into the relative increase of orders on vendor level to find the most trending restaurants. 
  • The “Recommended for you” predictions can be generated using different algorithms including memory-based or model-based collaborative filtering, and different implementations of Deep Neural Networks.

One of the core differences between these recommendations is whether they are personalized or not. For instance the “Popular near you” strategy is not personalised and is built by learning from the tastes of all our customers during a given timeframe and for a specific location. Whereas the “Recommended for you” strategy is personalized and tries to learn the unique taste of a customer based on their past orders. Depending on the algorithm used in the background, we can either recommend restaurants looking into the past orders from other users with similar taste and ordering behaviour, or we can learn from features from the customer’s previous orders, such as time, date, location, cuisines offered, and try to find other restaurants that satisfy that specific customer’s taste.

In comparison to the restaurant marketplace where we mostly recommend restaurants, in Dmart and Shop marketplaces, we recommend products. Some of the recommendation strategies we provide there are:

  • Your past purchases”, which can be based on the frequency or recency of your past orders.
  • Similar products” strategy, which tries to find the most similar product to the one a customer is looking at – based on textual and image features of the products.
  • “Often bought together products” strategy, which recommends products that are frequently seen together in baskets with the chosen product.

How we measure and evaluate

“Reality is frequently inaccurate.” – Douglas Adams, The Hitchhiker’s Guide to the Galaxy

We continuously test and update our models, and in order to achieve this, we employ an offline evaluation system where we are able to evaluate our work in progress on historical data. The offline evaluation helps us to tune our model parameters, and decide which models are valuable enough to be tested in production, and which are not.

However, we are aware that the recommender systems are not like the classical Machine Learning algorithms, and it is very difficult to evaluate them offline as the data your users see and interact with is usually already biased due to the existing recommendations. This drawback can be overcome by performing A/B tests, which on the other hand, are costly. For that reason, we always try to find a tradeoff and, for example, experiment with the parameters of one model using offline evaluation, but always compare two completely different models using A/B testing.

To bypass the experimentational depth we face with the A/B tests, we are working on an online reinforcement learning system that, based on rewards, allocates the traffic to a winning experiment and minimizes the overall regret. This type of multi-armed bandit-like experimentation helps us to automate the selection of best and winning solutions for different challenges, such as ranking of swimlanes, ranking of restaurants within a swimlane, selecting the best algorithm parameters, etc. 

How is all this possible?

“I may not have gone where I intended to go, but I think I have ended up where I needed to be.” – Douglas Adams, The Hitchhiker’s Guide to the Galaxy

The true magic does not reside only in our Python notebooks. Yes, we develop our models and solutions in Python and its modules like scikit-learn, Tensorflow, Keras. Each recommendation strategy is wrapped in a standardized dockerized application that is continuously integrated and delivered via a combined solution of Jenkins, Github, Google Container Registry and Kubernetes. 

In order to run and execute all of these different models, we use Airflow as our job scheduler. Adding a new strategy to Airflow is easy as pie, enabled by an automatic DAG generator using configuration files. For example: For each new strategy, we build our standardized application with predefined steps, i.e.: preprocess-data, train-model, generate-predictions, validate-predictions, export-predictions, for which automatic DAG is created and scheduled by just providing a .yaml config file. 

Each of the steps or tasks is wrapped in a KubernetesPodOperator and executed on our Kubernetes clusters (Thanks Logistics Team!). Slack notifications are enabled to raise alerts if Airflow tasks are failing. 

The recommendations from the models are stored in BigQuery and/or CloudStorage and further ingested in a Postgres Database. 

On the API side we use FastAPI Framework to serve our recommendations. Our API Gateway serves as a single point of entrance for all the calls we are getting from our consumers. Currently, we have several APIs: Vendors, Products, SwimlanesRanking and OnlineRecos APIs. API monitoring is provided by DataDog and pushes alerts to Slack.

Our recommendations architecture together with our technology stack

Thank you for taking the time to read about Delivery Hero’s Recommendation Systems! I hope you learned something from my experience working with Delivery Hero so far!

Thank you, Aleksandra for your insight! Our Consumer Discovery tribe is always looking for more heroes, so check out some of our latest Consumer Discovery openings, or join our Talent Community to stay up to date with what’s going on at Delivery Hero and receive customized job alerts!