Adopting Flyte at Gojek

Gojek is an on-demand multi-service platform and digital payment technology group operating in Southeast Asia. It has 20+ services ranging from food, transportation to payments and entertainment. In 2021, the Gojek ecosystem contributed 1.7% of the Indonesian GDP, indicating that the user base is quite large. Hence, scalability and reliability are of critical importance.

Machine Learning forms a big part of the services offered by Gojek. It’s used for several use cases, such as generating pickup points, voucher allocation, optimizing prices, etc. Back in 2017-18, ML pipelines were built using an abstraction on top of Clockwork and Airflow, which today has 1000+ DAGs. The abstraction was necessary to run pipelines in separate containers in order to handle the dependency management. However, the abstraction wasn’t a smooth ride due to the less than ideal development experience, unscalability of scheduler, missing DAGs, and infrastructure overhead.

After some reflection, the Gojek team understood that they wanted a platform that is scalable, Kubernetes-native, a data-aware orchestrator, and provides a first-class development experience, and Flyte fit their criteria.

‍Pradithya Aria Pura from the Data Science Platform team at Gojek presents a detailed walkthrough of the adoption of Flyte at Gojek and what they love about Flyte:

Some noteworthy excerpts from the video:

“… we started evaluations of several tools in the market that’ll satisfy our requirements, and eventually, we decided to use Flyte. We’re happy with our experience so far with Flyte.”

“Workflow versioning is quite important. When productionizing a pipeline, there are only a few platforms that provide this kind of versioning, and to us, it’s critical where we want to roll back to a specific workflow version in case there’s a bug introduced in the pipeline.”

“Flytekit (Python SDK) provides a powerful and expressive way of building pipelines. So far, we love the dynamic workflow and flow control.”

“Flyte is fast and scalable. During our evaluation stage, we did stress testing to understand whether Flyte can satisfy our requirements, and it provided us with good results.”