Why Flybuys migrated to dbt Cloud

Like many companies, Australia’s Flybuys found itself in an unenviable position with its data. It had multiple, redundant methods for slicing and dicing data—and none of them were easy to use.

Flybuys knew they needed a single data control plane. The question was which tools and approach would yield the greatest return on investment.

Recently, I sat down with Milos Zikic, Flybuys’ Lead Enterprise Data Architect, to discuss the issues that his company faced in taming data anarchy. We discussed why they ultimately chose dbt Cloud as their data control plane, how they did it, and how it paid off for them.

Four different data architectures

Established in 1994, Flybuys is one of the biggest rewards programs in Australia, with 9.5 million members. On top of offering a traditional loyalty program, they also run campaigns and perform partner offer management through both traditional and digital media channels.

This means Flybuys processes data. A lot of data. According to Zikic, the platform churns through 1.4 billion transactions every month.

Up until 2022, Flybuys had four different data teams with four different approaches to managing data:

• Their data platform team was using dbt Core to perform data transformations, supported by a home-grown infrastructure platform that ran on AWS
• An analytics team was with Snowflake stored procedures, with compute spun up in AWS
• Another analytics team used a combination of Amazon Sagemaker and R scripts to do its data transformation
• Finally, a fourth team used the Sagemaker/R platform along with a separate machine learning platform

While these systems worked for each team, maintaining four separate environments brought a number of problems:

Waste

Every team had to stand up—and support—its own infrastructure for running data transformation pipelines.

Inconsistency

There was no uniformity or standardization in how teams modeled data. Not everyone was testing data (or testing it in the same way). There was also little opportunity for code reuse.

Complexity

All of these systems were hard to understand and use. Data analysts who needed a new type of data product had to rely on data engineers to make it happen, rather than self-servicing their own solutions.

192% ROI: Why Flybuys made the leap

Flybuys knew it wanted to move to a single, standardized system for creating, testing, and managing data transformations. Zikic and his team thought that using a single system would eliminate duplicate effort, boost productivity, and enable greater self-service for analysts and end users.

The team saw two possibilities:

• Self-host dbt Core as a centralized platform; or
• Move to dbt Cloud

Self-hosting dbt Core seemed attractive because it meant not paying for dbt licenses. However, it also meant standing up and maintaining a centralized infrastructure—which would consume an enormous amount of time and resources.

By contrast, dbt Cloud would give them this centralized platform out of the box, with no need to maintain separate infra. It'd also get a lot of features—such as auditing and role-based access control—that would be expensive for them to get right.

The team proposed both future states. It also ran experiments to gauge how each one would work and which one provided the best improvement at the lowest cost. It then built a business case for each tool.

According to Zikic, Flybuys’ analysis revealed that the company could earn a 192% Return on Investment (ROI) by going with dbt Cloud. The move would save 143.6 hours—a full 17 days—per analyst every year. It would save even more time—152.6 hours—per engineer per year.

Most of this cost savings would come from the added features that dbt Cloud provided. With dbt Cloud, testing, version control, and documentation all come out of the box, without the overhead of manual coordination and infrastructure management.

Zikic says that dbt Cloud’s support for Continuous Integration/Continuous Deployment (CI/CD) was also a driving factor. “Our CI/CD was very convoluted,” he said. “Only data engineers were doing it.” dbt Cloud would enable Flybuys’ analysts to set up data pipelines that they could use to test, verify, deploy, and monitor changes to data models.

How dbt Cloud aligned with Flybuys’ values

Flybuys also had to make the case that using dbt Cloud aligned with the company’s core values.

Be financially responsible

Zikic’s team made an effective case that using dbt Cloud would reduce the costs of running data transformations by reducing both complexity as well as duplication of effort. The company also aimed to save money by increasing work velocity for engineering and analytics teams.

In terms of concrete numbers, Flybuys aimed to reduce spend on AWS services for data transformation by 15%. It also predicted a whopping 70% reduction in data transformation infrastructure costs.

Build secure and trustworthy systems

Flybuys needed to verify that their dbt Cloud implementation met the company’s standards for security and trust. More than that, however, Zikic argued that dbt Cloud would improve both the company’s data quality as well as its overall data security posture.

By acting as a data control plane for all data, dbt Cloud would bring improved quality and heightened consistency to its data, as well as help the company standardize on metrics. In addition, dbt Cloud’s embedded documentation support would lead to better clarity and transparency of data analytics code.

“Using a Software as a Service (SaaS) system like dbt Cloud provides better out-of-the-box security than we could provide in our bespoke environment,” Zikic said.

Strengthen company culture

Before dbt Cloud, data analysts were heavily reliant on data engineers to produce new data products. Flybuys wanted to empower its analysts to create more data products on their own. That meant providing a reliable CI/CD deployment process with the proper safeguards in place to ensure no unauthorized changes shipped to production.

Zikic’s team set a goal of increasing the number of quarterly data product deployments by 100%. It also aimed for zero unauthorized production pushes. Hitting these goals would increase the value of data and reduce the time it takes to provide insights to end users.

How Flybuys managed the transition to dbt Cloud

To make this transition smooth, Flybuys took it slowly. It started with a proof of concept to suss out what worked and what didn’t about its planned migration.

Flybuys found immediate value in several dbt Cloud features:

• dbt Cloud’s infrastructure management and focus on application development—particularly, easy-to-use Integrated Development Environment (IDE)—made it easier for anyone to set up a data pipeline. Environment management took away the pain of managing infrastructure and environmental conflicts.
• Easy security set-up using Role-Based Access Controls (RBAC).
• Access controls and audit logs made it easier to verify protection for workloads with sensitive data, such as personally identifiable information (PII). That was a lot more complicated to achieve with dbt Core, as workloads ran on people’s local machines.
• It’s easy to manage dbt Core versions and release in dbt Cloud and keep everyone on the same version.

Flybuys also identified some points where they'd still need to spend some additional time and work. Integration with GitLab required some additional footwork in terms of security configuration and repo management. Additionally, Zikic’s team kept Airflow in the mix to enable integration with non-AWS resources.

After identifying and addressing the gaps, Flybuys started transitioning to dbt Cloud. It started small—just two use cases supported by ten licenses across the data engineering and analytics teams. The team focused its efforts primarily on member segmentation and member engagement scores.

Flybuys ran the pilot across Q3 and Q4 of 2022. It then evaluated the project based on the success metrics defined above. That would determine whether it moved forward with migrating other use cases onto the new system.

The results

So how’s it going two years later?

Zikic says that dbt Cloud is now the official platform for productionizing enterprise-grade data transformation pipelines at Flybuys. The company has 100+ projects on dbt Cloud, with dbt Core only managing a handful of legacy jobs. It’s also integrated dbt Cloud with a number of its other systems, such as FiveTran and Hightouch.

Analysts and data engineers report a high level of satisfaction with the new toolset. The main benefit is that dbt Cloud gives them a single, simple way to build new data projects. Using dbt Cloud, both data engineers and analysts can focus on delivering pipelines instead of managing environments.

Zikic said work remains to be done. In particular, Flybuys is still running some of the legacy tooling it had hoped to replace. The team plans to continue reducing this, introducing additional vendors that easily integrate with dbt Cloud as part of its modern data stack.

Conclusion

Every team’s data journey will be different. No matter what data challenges your team faces, dbt Cloud can help by serving as your data control plane—a single, comprehensive platform that all data stakeholders can use across your organization.

Find out more about how dbt Cloud can address your company’s specific needs—schedule a demo today.