Spotify is a digital music, podcast, and video service that gives access to millions of songs and other content from creators all over the world. As a product, Spotify comprises many departments working on different aspects of the service, be it streaming, automation and tooling, finance, or data analysis.
Every quarter, as part of the compliance objectives, the finance team requires to project 2 years into the future what the profits and losses will be. It’s a projection across all of Spotify’s different business processes, precisely, the revenue that could be raised, the expenditure to increase the revenue, etc.
Initially, as many as 8 different teams configured and ran 15+ models to forecast various aspects of Spotify’s business. The end-to-end forecasting typically took 3-4 weeks every quarter. All the involved teams worked in silos—if team 1 handles subscribers and team 2, revenue, then team 1 needs to manually hand off the number of predicted subscribers to team 2 to compute revenue since revenue relies on subscribers. This hand-off was error-prone and slow; henceforth, Spotify was limited to running the forecast every quarter due to the time and effort involved.
As the finance team became increasingly eager to automate the business casing and scenario analysis, there was a need for a platform that could automate the workflows, aka pipelines.
The team’s final vision was to have a cohesive platform that brings together different teams onto a single platform. Owing to the complexity in building such a model (many models owned by many teams built using many technologies), they were in need of a platform that could handle all of this.
That was when the Spotify team encountered Flyte, and they eventually decided upon Flyte as the main runtime engine of their model.
Dylan Wilder, an engineering manager on a Finance team called “Vivaldi” at Spotify talks about the problems they wrested with while building the financial forecasting model, and how Flyte provided a cohesive solution.
Some noteworthy excerpts from the video:
“As engineers, a lot of this might be table stakes for us, but for data scientists, being able to get up and running on Flyte, and getting all of this stuff for free, is a really big win for them.”
“The ability to share models with each other, compose things easily, out-of-the-box parallelism, …, caching, connectivity with GCS; they (data scientists) have been really excited about all of this stuff.”
“The fact that Flyte supports immutable models, versioning, and graph workflows allows us to think about the state of the graph. We also rest on the fact that Flyte will keep all these things coherent and cache things correctly.”
We enable software users with the tools to take their ideas from conception to high quality production.
.png){kind=small}
.png){kind=small}
