Founded in 2011 in California as “Riskalyze,” Nitrogen started in the wealth management industry with a portfolio risk analytics tool. Today, under the Nitrogen brand, the company delivers firm-wide analytics, compliance solutions, and dozens of data integrations with custodians, banks, and other fintech vendors.
Nitrogen enables wealth managers to get a full overview of their customers’ portfolios on different custodians, such as Fidelity or Charles Schwab. They can use the product to design new investment strategies or prepare pitches for prospective clients.
“Without current accounts and investment positions data landing early in the morning every day like clockwork, our customers can’t rely on the data to get results for their clients,” said Andrew Waters, Director of Platform Engineering at Nitrogen.
Timely, accurate data is essential for the success of the product and, therefore, the business:
“For us, data is not just for internal teams or business intelligence. It’s a key value of the product we offer,” explained Andrew.
The role of data integrations at Nitrogen
Data integrations, in particular, are the backbone of Nitrogen’s product. The company has data integrations across dozens of firms, including banks and other tech providers.
“Our financial advisors (customers) need to see their data flowing in both directions,” explained Andrew. “We have to work seamlessly with their other tools.”
“Establishing those data feeds is core to anything they do across our product. A key value that we're offering is powered by the data that we have available,” emphasized Andrew.
A unique opportunity to move upmarket comes with a data caveat
In recent years, Nitrogen’s target audience has expanded from SMBs—the “mom-and-pop shops” of financial advisors—to also include mid-sized and large firms.
Earlier this year, a unique opportunity arose to land a mid-sized customer. The prospect wanted to purchase the product for their entire firm of hundreds of advisors. However, it came with a caveat: a new data integration had to be delivered within six weeks before their contract was due to start.
“Our sales team did an incredible job landing this customer,” said Andrew. “We wanted to put every effort to meet their requirements.”
The data team investigated whether they could deliver this new data integration with their existing workflow: “The deadline was impossibly tight, so we evaluated all options. We quickly ruled our existing process of building a new interface in our legacy data pipeline, which typically took over a quarter to complete,” said Andrew. “We knew we needed to come up with an innovative solution as more of these larger wealth management firms would have similar requirements.”
Legacy infrastructure hinders upmarket expansion
The limits of the existing data infrastructure
The question of delivering on the data integration request promptly reopened discussions on Nitrogen's legacy infrastructure—a mix of in-house software data transformation applications and traditional relational databases such as MySQL and Postgres.
“Various components of our stack weren’t scalable, and took patience to develop in,” explained Andrew. “We knew the way we were currently doing things wouldn’t scale to where we want to go in the future...we were pushing the limits of what we could do with our old systems.”
The search for a new data infrastructure
Faced with these challenges, the Nitrogen data team started the search for alternative solutions:
“We saw numerous opportunities if we moved to a central data lake. We could leverage machine learning to improve retention, and also decrease costs by moving out of expensive relational databases,” said Andrew.
Evaluating dbt Cloud and modern data tools
Scoping requirements: including the business & investors
The Nitrogen data team began by interviewing departments across the company to identify data use case opportunities and priorities, and involve other teams in the evaluation process. HG, a majority investor at Nitrogen, also participated in the evaluation process.
Recommending dbt Cloud: past success and ease-of-use
Based on past successes with other portfolio SaaS companies, HG put forward dbt Cloud as their recommendation, kicking off Nitrogen’s evaluation.
“We saw dbt Cloud as an opportunity because of its simple learning curve. It’s an approachable SQL-based skillset, which made it easy to adopt,” said Andrew. “It also looked like an efficient and cost-effective solution that fit our business needs.”
“After a quick evaluation, we determined dbt Cloud suited our requirements and was our only option that could deliver the integration on time; so we greenlit the project.”
Getting started with a hackathon
With dbt Cloud and Snowflake, the team hit the ground running:
“We got a group together from different teams that were interested in the new data stack and did an early hackathon,” shared Andrew. “Half of the team explored the transformation side of things on dbt Cloud, and the other focused on Snowflake.”
Although this was a small-scale project, it was a successful first effort. Armed with product knowledge and confidence in their new solution, the team could move on to their production use case: building the requested data integration for the prospective customer.
Successfully tackling the integration use case, and landing the customer
Unfortunately, most of the data engineering team at Nitrogen was tied up with other commitments and could not assist with the data integration work:
“Normally, I don’t get into development but we only had two people with enough capacity,” explained Andrew. “I paired up with our principal data engineer and together, the two of us dove into learning dbt and building out the first integration use case.”
Despite the compact team size, Nitrogen successfully delivered the integration in time:
“Even though we had a smaller team than all of our past integration projects, we were able to deliver the integration in just six weeks. That’s exactly half of the time that projects with similar scope took previously,” noted Andrew.
To speed up the onboarding process, Nitrogen had a dbt Labs trainer take them through dbt capabilities in a 3-week, 6-course series:
“dbt Learn was a fantastic introduction,” said Andrew. “Having access to those session recordings helped us rapidly onboard the rest of our team. We still use them to onboard newcomers and other teams that want to get started with dbt Cloud.”
Reaping the benefits of migrating to a modern data stack
A substantial decrease in the cost of data maintenance
The new data structure driven by dbt Cloud simplified the set of systems needed to maintain the Nitrogen’s data infrastructure:
“We don't have as many steps in the process, such as going through different queues and then staging the database,” explained Andrew. “The whole architecture has been streamlined. The new workflow on dbt Cloud and Snowflake runs a lot more efficiently.”
One of the sources of increased efficiency was the move away from large and cumbersome relational databases.
“We don’t have a final measurement on saved costs yet, but it’s already very clear that it’s substantially less. We’re probably spending 2 to 4 times less,” said Andrew.
Faster data delivery and improved customer experience
The streamlined infrastructure also had a positive impact on data delivery:
“Our daily imports now run under 4 minutes, which is 30% faster than similar-sized data sets in our old process,” said Andrew.
The speedier data delivery has a direct impact on customer experience, with wealth managers receiving their customers’ and market data earlier:
“Data lands in customers’ hands earlier in the day, and enables us to hit our objective of delivering all account data before financial markets open,” explained Andrew.
Increased visibility and collaboration
Incentivizing collaboration wasn’t an objective for Nitrogen when they formed their new stack, but it’s proven to be a positive consequence of the migration:
“Collaboration is just now becoming a visible benefit of having a central repository and clear data lineage,” said Andrew. “Sometimes knowledge is locked into one person’s head, but now it’s obvious—anybody can explore where the data came from and understand how it was transformed. The dbt lineage feature and how data is linked explicitly in the product is perfect for that.”
A simpler workflow with SQL at the forefront
With SQL-first dbt Cloud in place, the data transformation process is magnitudes simpler:
“Before, the logic and custom applications, such as using JSON objects, for data translation were far too complex,” shared Andrew. “SQL-based development is inherently faster and a better fit for a proper data architecture.”
“Today, to make a simple transformation layer, we only have to worry about three things: getting the data into Snowflake, transforming the data in dbt Cloud, and then exporting to a destination.”
Looking ahead for Nitrogen
Phase out legacy technologies
The first project replacing the legacy data infrastructure was a success for Nitrogen. Moving forward, the data team will continue working on the transition from the legacy stack and the depreciation of their existing, expensive relational databases.
“We’ve proven with the first project that this new stack is better in performance on all factors we care about. It beats the baseline, which is very positive from an early prototype,” said Andrew. “Now over the next year, we’ll continue expanding that logic to handle all our current and future data use cases.”
Spread dbt adoption to other teams, including compliance
Business users and executives have already started using the new dbt data views, but other teams are still being onboarded to the platform. Compliance officers, in specific, see a big opportunity in using the new stack to prove they’re meeting fiduciary responsibilities:
“That’s the second big use case we’re delivering, which is crucial in our industry,” explained Andrew. “We’re building a new data warehouse to serve customer-facing BI in our application. This will assist wealth managers in assessing and visualizing risk for their different customers.”
Improve governance with dbt Cloud’s automated testing
Now past the immediate goal of taking their first use case live, fast, the Nitrogen team will continue to focus on data governance and data quality over the next year.
“We want to explore dbt Cloud’s data quality features, such as alerts and data validation,” said Andrew. “We’re still early in our journey; we see a big opportunity in improved data governance, and we’re excited to layer on standards, oversight, and everything else dbt Cloud has to offer.”
