Data ingestion across clouds causes a bottleneck
When companies want to optimize their marketing strategy, they turn to RV. The company’s Red Digital division provides end-to-end performance marketing services that help business-to-consumer (B2C) services providers attract new customers. RV uses a modern, and scalable, data solution to optimize search campaigns, social campaigns, display and landing pages in ways that will turn top-of-funnel marketing leads into repeat customers for their clients.
“For some of our clients, we manage the entire customer journey,” said Brandon Beidel, Director of Product Management at Red Ventures. “Many clients don’t have the in-house marketing expertise to run sophisticated campaigns. It’s our job to help them drive customer growth over time, and we use data and AI to give them the best results possible.”
For RV, delivering greater value to clients — and consumers — is about using timely insights from data to get in front of the right consumers. But the company doesn’t just think of the bottom line. Red Ventures wants customers and prospects to have an outstanding user experience as they explore products and decide whether to buy.
“Today’s consumer expects to be able to complete the entire purchase process for most products online,” said Beidel. “But when they’re buying complex services, it’s challenging for businesses to get their message across effectively. Our goal is to create for our clients an optimal ad experience or many iterations of a website and then connect each of their customers with the user experience that will best help them understand the product. For us to tailor marketing to that level, we need to effectively use data and AI.”
To ensure data integrity, RV maintains each client’s data in a separate cloud environment. From there, RV must integrate each client’s data and perform intensive transformations to generate machine learning predictions. Until recently, data engineers wrote custom scripts to ingest data for each client — a tedious and time-consuming task.
A modern data stack helps simplify and automate ETL workloads
Seeking to process data more efficiently, RV implemented Databricks to scale data engineering pipelines and speed up insights. The company also uses Fivetran to perform data ingestion and dbt to apply data transformations. Databricks is now the engine that performs RV’s heaviest computing tasks. For each client environment, RV has set up a dedicated Databricks workspace, Fivetran connectors and dbt projects that only designated employees can access. All three solutions feed data into a machine learning pipeline that drives functions such as budgeting for clients’ advertising spend.
“Due to the nature of our business, we must maintain a separate data stack for each client,” Beidel explained. “But Databricks, Fivetran and dbt help us avoid reinventing the wheel as we build out the processing, computation and data management pieces for each client. We’ve been able to take work our engineers have done and reuse it across multiple clients. We’ve also put Fivetran in the hands of our marketers to let them set up integrations for ingesting our clients’ data, and we’ve given them access to dbt to define data transformations. Because these tools are so simple, we often don’t need to involve our engineers.”
Most of the onsite events from clients’ websites stream into RV. The company receives click data, event views, page views, scrolling depth and more into its warehouse. From there, RV has simplified and automated its ETL workloads.
“Databricks has allowed us not to have to think about managing infrastructure,” said Beidel. “For many workloads, we simply select the compute size, define our script and let it run. Our data engineers don’t have to be experts in managing Spark clusters. With Databricks, they can focus on the shape of the data and the business logic.”
Red Ventures has decreased its data processing and troubleshooting time — and continues to review data to improve the results they deliver for clients.
Delivering better results for clients with less data work
RV now processes client data more efficiently than ever. The company used to spend 100 to 150 hours per data integration. Today, that figure is 10 hours — which frees up data engineers to work on higher-value tasks. Processing jobs that ran overnight and took up to 20 hours now finish in four or five. And because Fivetran provides the same data set for every client and dbt gives everyone a clear view of data transformations, RV’s engineering team has reduced troubleshooting from 50% of its time to less than 20%. Automated cluster management capabilities in the Databricks Lakehouse Platform have yielded further time savings.
“Not having to manage clusters is a huge time-saver for us,” Beidel remarked. “Our biggest grievance with other warehousing technologies is the lack of effective autoscaling. It’s a must-have capability when you’re working with a large volume of analytical workloads and have fluctuations in demand throughout the day. Having that variability in usage taken care of for us by Databricks probably saves one engineer out of 10.”
This greater efficiency has enabled RV to support 30% more clients. “One of the things that drew me to dbt is that everything is in SQL,” Beidel explained. “It serves as a common language for our team. Not everyone can write SQL, but everyone can read it and have a better understanding of what’s going on underneath.”
Because dbt and Fivetran are so easy to use, more RV employees are involved in data transformations. Engineers will often use dbt to show marketing staff how specific values were calculated — leading to deeper discussions about the data and to knowledge transfer from engineering to business users.
This data democratization is contributing to better results for RV’s clients. “Using the highly intuitive machine learning pipeline we’ve built, we’ve helped our clients increase cost efficiency in some channels by 20 to 30%,” Beidel concluded.
