Navigating the Quantum Leap: Delivery Hero’s Journey to Salesforce Hyperforce

16.08.24 by Saon Kumas Das, Balbhadrasinh Govindsinh Karki, Gaurav Gathania

Navigating the Quantum Leap: Delivery Hero’s Journey to Salesforce Hyperforce

7 min read

Salesforce is a core application at Delivery Hero, centrally managing data and business processes for global vendors (restaurants and shops) and internal teams.

It streamlines the distribution of this data to various customer-facing web and mobile applications for food and grocery ordering.

Salesforce has announced that starting Oct 18, 2023, Salesforce instances will need to migrate to a new infrastructure architecture called Hyperforce, which will be implemented worldwide in phases.

Migrating our Salesforce instances to Hyperforce was a crucial step in enhancing our data security, scalability, and global performance. This move allows us to leverage cutting-edge cloud infrastructure, ensuring that our systems are not only resilient and future-proof but also capable of delivering an even better experience for our customers and teams worldwide
James Carter, SVP Engineering

What is Hyperforce and Why Does it Matter?

Hyperforce is designed to deliver Enhanced Customer Experience, Global Compliance & Data residency, Cost-Effective Operation, Innovation & Agility.

Salesforce is transitioning all instances from its privately managed data centers to public cloud infrastructure for improved reliability and availability. This shift, known as Hyperforce, expands deployment options for customers to align with local data regulations and security requirements. Hyperforce leverages partnerships with cloud providers like AWS and Azure, offering wider geographical availability than Salesforce’s previous infrastructure. While customers can temporarily delay this migration, it is ultimately mandatory.

Objective

“Ensure minimum downtime and impact on the business due to Hyperforce migration”.

“Delivery Hero achieved a milestone of 10 million orders a single day” – you can imagine the volume and criticality we are managing in our business.

Delivery Hero operates in 70 countries and is available 24×7, maintaining high service level agreements (SLAs) over emergencies and downtimes. As a result, scheduling downtime for the Salesforce hyperforce migration has been challenging. Salesforce instances for Delivery Hero support more than 11K active users, ranging from Sales to Customer Service. Hence working with multiple stakeholders and collaborating with various tech teams to ensure availability for the D Day.

We were finally able to secure a lean window of 4 hours on May 18, 2024. This was considered the leanest window for our contact centers, and potential customer cases as it fell during early hours in many regions, reducing the chances of losing leads, conversions, potential revenue and customer satisfaction.

We also wanted to ensure the Hyperforce A teams (Dev/Admins/Product/Business) had sufficient windows for emergency mitigation in case of critical issues. Hence a large team was aligned to support various key business processes like Sales and Service.

Setting the Stage: A Contextual Overview

Our Salesforce journey commenced in 2011, and our customer base has experienced exponential growth since then. Recognizing the necessity of digital transformation to remain competitive, Delivery Hero adopted a multi-instance Salesforce cloud strategy (totalling 9 instances) to address a diverse range of business use cases across various business units.

In early 2023, we embarked on our Hyperforce evaluation and decided on a phased migration approach. We began with the less complex Salesforce instances and gradually moved to the more intricate ones. With a comprehensive migration plan developed in collaboration with Salesforce, incorporating their prerequisites), we involved all necessary teams in pre-checks and post-migration testing. This meticulous planning resulted in a seamless transition for 8 out of 9 Salesforce instances.

However, we encountered challenges during the final migration, which involved a highly complex global instance managing Vendor Management processes. This system, deployed globally, served both sales and service users and featured over 100 integrations, primarily through a custom middleware layer with several point-to-point integrations.

We targeted the new infrastructure for various business units throughout 2023 and the first quarter of 2024. These units include Supplier Relationship Management, Dmarts – Portfolio Management, Corporate Catering, B2B Advertising Partnerships, Global Risk & Compliance (GRC), and Woowaall of which have lower customer volumes and business impact. The final and more complex Vendor Relationship Management underwent a thorough evaluation process in Q3 2023 and Q1 2024, to migrate to the new infrastructure by Q2 2024.

High Level Steps

The high-level steps involved in the Hyperforce migration are as follows:

Confirm migration with Salesforce: Notify Salesforce of the intent to migrate at least 30 days in advance. This is necessary for Salesforce to initiate data replication within the Hyperforce infrastructure. During the migration, only incremental data changes are copied over, which helps maintain a migration window of under 3 hours.
Develop a comprehensive migration plan: This involves detailed planning across various teams and establishing a communication strategy for all stakeholders. At a high level, our deployment plan can be summarized as follows:

Migration Actions	Ownership
Pre Hyperforce Migration
Update stakeholders before downtime start	Project Lead
Showing maintenance page on Web portal	Dev Team
Freeze non-admin users	Admin team
Stop scheduled jobs	Admin team
Switch off Inbound & outbound Integrations	Platform Team / Middleware Team
Hyperforce Migration
Migration of salesforce’s infrastructure	Salesforce team
Post Hyperforce Migration
Sanity testing with admin users for navigation & performance	QA / DEV Team
Revert back to the steps performed in pre-migration	Platform Team / Middleware Team
Update stakeholders before downtime starts	Admin / Dev Team
Smoke testing (end-to-end)	QA Team
Update stakeholders after migration	Project Manager

Migration Schedule

Execute the migration plan: Implement the migration plan, ensuring minimal disruption to ongoing business operations.

Main Identified Issues During Hyperforce Migration Testing (full copy sandbox migrated to Hyperforce)

To ensure a smooth transition for your business, Salesforce provides a comprehensive Hyperforce Overview. Here we will cover the major identified areas for Vendor Relationship Management Salesforce instance.

Instance references: After the Hyperforce migration, the Salesforce Instance name will change (e.g. from EU45 to DEU55). We needed to ensure that there are no references to the old instance names (e.g. EU45) in our code base, including Apex classes, email templates, static resources, site URLs and integration endpoints.
Platform event: Two weeks before the migration, it was identified in collaboration with the Salesforce Tech team that the Platform Event Queue could potentially lose its state. As a result, the Delivery Hero Salesforce team needed to find a solution to preserve integration in such scenarios. Salesforce’s events are categorized into two types: Standard events and High-volume events.
This was probably the biggest roadblock we encountered during the Hyperforce migration and we had to come up with a custom solution to tackle high-volume events. Yes, Salesforce does not offer out-of-the-box support for high-volume event replay ID migration In our integration landscape we have used platform events to communicate with other downstream systems, and we also discovered that our downstream systems rely heavily on Salesforce’s high volume replayid sequences. In a nutshell, downstream systems always expect incremental value in replayId which would not be possible post Hyperforce migration because these IDs are shared among all the tenants in the first-party/Hyperforce architecture and are expected to be reset. To address this issue, the Delivery Hero Salesforce development team implemented the following solution: we created custom logic in our middleware layer to increment the replay ID, ensuring it remains higher than the current value from the first-party infrastructure.
- Value of ReplayId Before HF (X) = 10 M
- Value of ReplayId After HF (Y) = 2 M
- Buffer value of ReplayId (Z) = 1 M
- Offset Value at Platform Layer(Offset) = ( X – Y ) + Z
- Value of ReplayId to downstream = Salesforce replayId after HF + Offset
This makes sure that irrespective of the reset value of salesforce’s replayid, the subscribers always get higher numbers after migration.
Domain redirection: From the Hyperforce Assistant report we observed many URLs having instance name references (e.g. eu43). To ensure these URLs stay valid post migration,we replaced instance names with domain names.
Replacing IP whitelisting: Delivery Hero’s integration landscape is extensive, involving multiple internal systems for end-to-end interaction. An IP whitelisting strategy was crucial for security reasons, and many existing systems depended on it. Consequently, the infrastructure change impacting IP ranges required us to implement domain whitelisting to accommodate these changes.
Login: While executing the final stages of integration testing in the full copy sandbox, we encountered an unexpected issue. Salesforce informed us that the generic login URL (login.salesforce.com) might not be operational for extended periods (over 5 hours) on the newly migrated Hyperforce environments. This would have severely impacted our business operations and required an extended migration outage. To address this, we began engaging with the impacted integrated system teams to replace login.salesforce.com with the DeliveryHero domain URL. However, in the midst of these conversations, after 2-3 weeks, Salesforce informed us that the issue had been fixed and we decided to go ahead with the Hyperforce migration.
Other notable implications: Service use case, Contact center Einstein Chatbot URLs needed updates due to URL changes. Revisiting the synchronization of Marketing Cloud integration.
Assessing dependencies of AppExchange products on the Salesforce instance.

Go-Live Preparation

Effective communication with all relevant stakeholders beforehand, providing necessary details, is a crucial key to any Go-Live. Transparency, collaboration and open for suggestions based on the testing results significantly enhanced the final production migration outcome. Once we received a tentative final date from Salesforce it was shared with all relevant parties.

A detailed migration plan was published, outlining action items and plans across the team.

Migration, Final Take: D day.

After fixing all the above issues and maintaining effective communication with all stakeholders, we successfully transitioned to the Hyperforce infrastructure on May 19th 2024 at midnight. We had to extend our migration window by a few hours due to a permission mismatch in sanity testers’ profiles. However, we were able to overcome all these challenges and had a successful Hyperforce migration. And yes, post-migration no major issues were discovered 😀

Advantages

Moving to Hyperforce offers advantages as mentioned by Salesforce. At Delivery Hero, from the initial observations, we noted some performance improvements such as reduced Apex CPU time, Row lock time and Asynchronous call out failures. We also observed significant improvements in the refresh time of the full copy sandbox environments which decreased from 22 days to 14 days).

Conclusion

Hyperforce migration is mandatory for every customer and represents more than just a regular update. It marks the next transformative step for your cloud solution. Given our enterprise setup, migrating to Hyperforce was not a straightforward process. All aspects should be meticulously cross-checked before proceeding with the migration.