Understanding the customer base goes a long way in powering a number of recommendation models throughout our apps.
Personalisation models make use of various signals and behaviors to build an understanding of user preferences while making sure business criteria are also met. Let us explore what entails such an exploration.
In the previous blog, we discussed key components that make up a personalisation setup with the sole purpose of improving customer experience and helping our business bottomline. Customers are at the very center of any business. At Delivery Hero they are our heroes and no personalisation journey can work without understanding the customers. Let’s dive deeper into this topic and how it shaped our personalisation journey.
It is very important to understand and answer questions related to users’ behavior, motivations, preferences and requirements. Exploratory analysis is a key step which powers not just the initial set of tasks but is an on-going activity which helps throughout the journey. This step generates a number of key insights, ideas and helps identify potential challenges which we try to handle in subsequent phases as the system matures.
To understand how this helps, let us first talk about how the exploratory analysis works out. The most widely used and obvious method is to leverage all the data we have about our customers. This can be in the form of transactions, the kind of items they order, their restaurant and price point preferences and more. This form of analysis makes use of implicit signals which we leverage as feedback to understand different customer behaviors. This also helps in developing personas and categorization of usage patterns.
“Customers in Singapore and Thailand Love ???? Indian vendors “ for late-night dinners
The second way of analyzing customer behavior is through explicit feedback. When customers use our apps, they are presented with the opportunity to rate the packaging, delivery, rider, restaurant/vendor and more such characteristics. They can also provide textual feedback in the form of reviews. These explicit signals are even more impactful towards building an understanding of the customer base. Most modern recommendation systems make use of both implicit and explicit feedback to improve overall experience.
“Malaysian customers just Love???? Burger vendors for their evenings”
The final piece in the list for methods for exploratory analysis is User Research (UXR). This is by far the most meticulous and tedious task but it provides a direct connection with our customers. This is also very costly to conduct, in terms of time, effort and monetary aspects. At Delivery Hero, we typically run multiple UXR studies to understand how our customers use the app, what they use it for, their major pain points and what they love :).
A combination of these three analysis methods helped us develop an understanding of our customers. For instance, UXR studies helped us understand that our customers in certain parts of the world just love bubble teas at certain times of the day over other beverages. This helped us prioritize the visibility of bubble tea serving locations to help our customers find the best ones quickly. Similarly, analysis of implicit signals pointed to hyper-local preference of restaurants. Helping us have a more focused approach to improve our recommendations.
On an average, most customers just love???? to order again from vendors they have already ordered from #nostalgia #cravings
EDA’s are not just helpful in understanding customer behavior. We analyze data to improve upon existing models and system behaviors as well. This acts as a feedback loop to ensure our models are indeed working as intended and to identify gaps to improve further. The following is a high-level overview of a recent analysis to understand the impact of position and what motivates a customer to order from a specific vendor.
This analysis helped us understand what type of vendors were being rated high by our models versus which appealed to our customers better. It helped us narrow down to a list of features which helped in improving our model performance.
Modeling User Behaviour
The insights from the exploratory phase power the modeling phase to a great extent. Right from helping Data Scientists decide on what features to use to even choice of models, evaluation criteria and more. There are a number of intricate details which we are skipping here for the sake of brevity, but in a nutshell, the modeling phase is split into two main components described next.
A New Beginning: Cold Start
If you know someone, it’s comparatively easy to guess what they would like to eat or from where they would like to eat, as opposed to someone whom you just met. The same applies to the recommendation models as well.
The coldstart models are geared towards analyzing generic and macro patterns in a population to provide better recommendations than random listing. The aim is to try to present what a new user might be interested in rather than just a plain random set of choices. The evaluation criteria and business metrics are also specifically defined separately for new customers. Coldstart models mitigate the challenge of not knowing much about new customers by trying to use aggregate information and signals/behaviors from long term customers. Some of our models even make use of platform (our app) usage behavior to tailor recommendations for new customers.
Don’t Worry, We Got You: Personalized Model
Once we know a customer’s behavior based on their interactions with our system, we try to provide more fine-tuned recommendations, tailored to individual tastes and preferences. The more we know about you, the better our recommendations get. The “know you” part is where the whole data science magic happens. Data Scientists leverage both explicitly shared feedback and implicitly deduced behaviors to power such models. Just like every other component in this setup, personalized models also go through an iterative process of improvement. Right from a basic rule based setup where we leverage insights from EDA and domain understanding of experts to our current set of machine learning based models, we have been in constant state of improvement. A resilient and robust infrastructure setup powers this highly iterative data driven ecosystem.
Understanding the data/signals and developing models to improve concerned metrics is only one part of the picture. Another important aspect is “how does one know which model to use?” or “is the latest model any better than what we currently have in production?”. Evaluation of recommendation models which are highly temporal in nature is quite different from evaluation of usual machine learning models. We make use of offline evaluations as well as online A/B tests to evaluate our models using a number of well defined metrics. We will cover more on this in upcoming posts.
Conclusion
Personalisation is an interesting and important aspect of our platform. Understanding user preferences is one of the key steps in developing solutions powered by recommendation engines. It is a continuous and iterative process which keeps evolving as the user preferences evolve. We discussed how implicit and explicit feedback helps in developing an understanding of our customers along with UX Research which involves far more effort and depth. We touched upon a few interesting insights from exploratory analysis of different populations as well. The blog post presented how we personalize experience for new and existing customers by leveraging different approaches. We concluded by hinting about standard methods that help in evaluation of recommendation systems. Stay tuned as we introduce more components and go more in depth in up-coming blog posts.
Would you like to become a Hero? We have good news for you, join our Talent Community to stay up to date with what’s going on at Delivery Hero and receive customized job alerts!
