Andela speeds up product development of its talent matching service with dbt

Andela helps technology leaders build remote engineering teams quickly and cost effectively with global talent. A successful talent match depends on data. Eventually, the MyAndela product will make this matching service automatic. But for today, the Technology team is focused on empowering Andela’s staffing team with accurate, up-to-date information at their fingertips.

Gathering and structuring data

“In late 2019, we started talking about how we could get our talent matching product to market faster,” said Evan Johnson, Director of Data at Andela. The end goal is a user interface in MyAndela that allows the staffing team to search for talent based on skill, experience, availability, etc. “We figured we could dramatically speed up our product development by building the data models in BigQuery using dbt and then prototyping the search functionality in Looker.”

1. Creating the taxonomy: “We needed to understand how different skills interacted with each other,” said Maya Neria, Senior Product Manager at Andela. “Our taxonomy helps us answer questions like: What makes for a well-rounded developer? What are the various experiences that they have across industry specialization, skills and tools that could make them a quality fit for a partner?”

The team used this taxonomy to capture structured data throughout the sourcing and evaluation process. This data was captured and stored in a variety of systems including Greenhouse, Bamboo HR, HubSpot, Salesforce, as well as Andela’s custom-built platform–MyAndela.

1. Centralizing data: Next, all of this data needed to be organized in one place. The team used Stitch (where a connector was available) and custom Python scripts (when an existing connector was not available) to extract data from source systems and load it into their data warehouse, BigQuery.
2. Building dbt models: Business logic stored in a business intelligence tool is only accessible to users of that tool. Business logic built in dbt is accessible to anyone who has access to the data warehouse, regardless of their preferred tool. In Andela’s case, this meant the data team could build models using dbt and that code was available to both the engineering team building the MyAndela product as well as the data team building the prototype in Looker.

The new setup enabled easy collaboration between the data and engineering teams. Not only did dbt make it easy for the data team to apply software engineering best practices like version control, testing, and CI/CD to analytics code; the same data sets were now available in BigQuery for both analytics and product use cases. The result was faster product development.

Releasing the prototype

In February 2020, less than three months after the team landed on the idea of prototyping with dbt and Looker, they released a usable Looker prototype of the search functionality to the staffing team, nearly five months ahead of the scheduled release date for the user interface. This early prototype meant they could also start gathering user feedback.

One of the big things they uncovered from early feedback was the need for enhanced Boolean search, “The difference between an engineer who knows React AND Python vs. React OR Python is big, and we didn’t appreciate this important nuance,” Evan said. With this early feedback, the UI team added search functionality for Boolean search as well as space to specify “nice-to-have” vs. “must-have” skills. “If we hadn’t caught this during the early prototyping stage, it would just have created tech debt that needed to be fixed later on in the actual product,” said Maya.

Moving quicker (and not breaking things)

Modeling data with dbt and prototyping the search functionality in Looker was so successful that the data and product teams are now making it standard practice. “We were able to get the search functionality of our talent matching service shipped ahead of schedule. Now we’re moving onto automated matching and using the same approach” Maya said. “Model the data in dbt, build the prototype in Looker, gather feedback, then build in MyAndela.”