How Andela sped up product development of its talent matching service with dbt

Andela is out to build a talent matching service powered by machine learning, but the product roadmap to get there includes several key milestones. This is the story of how Andela cut nearly five months off the product development timeline. In the process, discovering a new approach to developing data products that involved modeling data in dbt, building the prototype in Looker, and productizing in the Andela platform.

Results

3

Months to build a usable prototype

5

Months shaved off the product development timeline

1

New approach to product development

Andela helps technology leaders build remote engineering teams quickly and cost effectively with global talent. A successful talent match depends on data. Eventually, the MyAndela product will make this matching service automatic. But for today, the Technology team is focused on empowering Andela’s staffing team with accurate, up-to-date information at their fingertips.

Gathering and structuring data

“In late 2019, we started talking about how we could get our talent matching product to market faster,” said Evan Johnson, Director of Data at Andela. The end goal is a user interface in MyAndela that allows the staffing team to search for talent based on skill, experience, availability, etc. “We figured we could dramatically speed up our product development by building the data models in BigQuery using dbt and then prototyping the search functionality in Looker.”

  1. Creating the taxonomy: “We needed to understand how different skills interacted with each other,” said Maya Neria, Senior Product Manager at Andela. “Our taxonomy helps us answer questions like: What makes for a well-rounded developer? What are the various experiences that they have across industry specialization, skills and tools that could make them a quality fit for a partner?”

The team used this taxonomy to capture structured data throughout the sourcing and evaluation process. This data was captured and stored in a variety of systems including Greenhouse, Bamboo HR, HubSpot, Salesforce, as well as Andela’s custom-built platform–MyAndela.

  1. Centralizing data: Next, all of this data needed to be organized in one place. The team used Stitch (where a connector was available) and custom Python scripts (when an existing connector was not available) to extract data from source systems and load it into their data warehouse, BigQuery.

  2. Building dbt models: Business logic stored in a business intelligence tool is only accessible to users of that tool. Business logic built in dbt is accessible to anyone who has access to the data warehouse, regardless of their preferred tool. In Andela’s case, this meant the data team could build models using dbt and that code was available to both the engineering team building the MyAndela product as well as the data team building the prototype in Looker.

The new setup enabled easy collaboration between the data and engineering teams. Not only did dbt make it easy for the data team to apply software engineering best practices like version control, testing, and CI/CD to analytics code; the same data sets were now available in BigQuery for both analytics and product use cases. The result was faster product development.

Andela analytics engineering Workflow

Releasing the prototype

In February 2020, less than three months after the team landed on the idea of prototyping with dbt and Looker, they released a usable Looker prototype of the search functionality to the staffing team, nearly five months ahead of the scheduled release date for the user interface. This early prototype meant they could also start gathering user feedback.

One of the big things they uncovered from early feedback was the need for enhanced Boolean search, “The difference between an engineer who knows React AND Python vs. React OR Python is big, and we didn’t appreciate this important nuance,” Evan said. With this early feedback, the UI team added search functionality for Boolean search as well as space to specify “nice-to-have” vs. “must-have” skills. “If we hadn’t caught this during the early prototyping stage, it would just have created tech debt that needed to be fixed later on in the actual product,” said Maya.

Andela UI

Moving quicker (and not breaking things)

Modeling data with dbt and prototyping the search functionality in Looker was so successful that the data and product teams are now making it standard practice. “We were able to get the search functionality of our talent matching service shipped ahead of schedule. Now we’re moving onto automated matching and using the same approach” Maya said. “Model the data in dbt, build the prototype in Looker, gather feedback, then build in MyAndela.”

Andela

Location

New York, New York

Industry

IT

Founded

2014

Results

3

Months to build a usable prototype

The team modeled underlying data in dbt, then built a prototype of the MyAndela search functionality in Looker in just three months. When the MyAndea interface is ready for release it will query the same dbt data sets currently used by Looker–all that data transformation and modeling work remains intact.

5

Months shaved off the product development timeline

Adopting an iterative approach to development cut five months off the product development timeline and helped the product team get early user feedback faster.

1

New approach to product development

The Andela team is now applying this same approach, modeling in dbt then prototyping in Looker, to its next major feature released–automated matching.

Start using dbt today!

We believe that data analysts should work more like software engineers. In doing so, they unlock their own creative potential and dramatically expand their impact on their organizations.

Start your Free Trial