Abstract
How we successfully transitioned the search system for the world’s largest recipe sharing platform to a modern stack – including the successes, failures, team structures, and processes along the way.
A talk I gave at Berlin Buzzwords 2023. The talk description can also be found on the Berlin Buzzwords website, and the recording can be found on YouTube.
Cookpad is the largest recipe sharing platform in the world. Our mission is to make everyday cooking fun, and central to that is our search product. Our search engine helps cooks everywhere find tasty dishes to cook from our ever-growing catalogue of six million recipes created by everyday cooks. As a global recipe search – available in 70+ countries, 30+ languages, and to over 50 million monthly users – delivering this is no mean feat.
In order to prepare for a substantial new iteration of our search product, we realised that our existing search system was not suited to our goals. Over the course of two years, we embarked on a technological and cultural transition, with the aim of giving product teams and engineers (including search engineers, data scientists, and ML engineers) greater ownership over the search experience. This included a shift to a Python-based stack and a data-driven approach to search relevance improvement.
We embarked on a transition to a new system, along with new team structures and team composition, and new strategies for improving the search experience. Over two years we delivered a new system, without halting product development along the way, and without disruption to the user experience.
Our starting point was a team and system with capacity limited to maintenance and bug fixes, where relevance enhancement was delivered through incremental knowledge base tuning by SMEs (non-engineer subject matter experts). Our end point was multiple search teams who have greater ownership over the search experience and relevance improvement, assisted by SMEs, and following a process for rigorously tested hypothesis-driven experimentation.
This change involved transitioning to a new event-driven architecture, along with technologies that were new to Cookpad search, such as Kubernetes, Kafka, Python, and machine learning. In addition – and just as importantly – it also involved a transformation in team structures and team composition, for which we borrowed many concepts and practices learned from the search community, and also ideas from the Team Topologies movement.
This talk will cover our journey, why we did it, as well as the trials, tribulations, and successes along the way. Hopefully, it will give others who are in a similar position new ideas on how to reinvent their own search system and search function, while minimising disruption to product delivery, in order to deliver proven improvements at pace.
