Blog 4 cloud migration challenges and realistic solutions

4 cloud migration challenges and realistic solutions

Kira Furuichi

May 16, 2023

Cloud migration: it’s the trend we’re all doing (and with good reason), but what are some of the potholes and challenges your team should look out for during your cloud migration?

We’re unpacking the four primary cloud migration challenges (planning and preparation, optimizing for security and cost, matching tools with skill sets, and avoiding business downtime) with pragmatic solutions. It’s vital to understand the realities and complexities of cloud migrations, but also offer up realistic solutions and modern tooling you can adopt to have the smoothest migration possible.

What is cloud migration?

Cloud migration is the process of moving technical infrastructure from an existing on-premise solution to a cloud provider. Things that are typically migrated in a move like this are databases, IT applications, and other technical infrastructure. In the context of data work (and this post), we tend to focus the conversation of cloud migrations around the transition from on-premise storage to cloud storage (cloud data warehouses or data lakes), and the transformation tools and new workflows empowering that migration.

Cloud migration challenges and practical solutions

Let’s just get it out there: cloud migrations are challenging. It’s often difficult to:

• Create feasible plans and timelines,
• Ensure security and compliance,
• Empower your users to use your new platforms and tools, and
• Avoid considerable business downtime.

So why are so many data practitioners putting themselves through the challenges of cloud migrations? Because the long-term benefits outweigh the short-term pains. When you invest in making a move to the cloud, you’re betting on:

• Creating a more governable and accessible data stack,
• Leveraging cloud data tools that scale with your business,
• Securing your data to new standards, and
• Realizing long-term cost gains.

The focus of this article is on the primary challenges we listed above—and more importantly, how addressing these challenges with practical solutions is the best defense to creating a smoother, more organized cloud migration.

(Everyone’s favorite) planning and preparation

Definitely the least fun, but arguably the most important step in conducting a cloud migration: planning for success. Your cloud migration will only be as timely, organized, and manageable as your plan for it is. To create a feasible and organized plan, establish the scope, ownership, and timelines early on for your migration.

Scope

Establish expectations to both your team, end users, and leadership on what will actually happen during this migration.

Questions you should answer at this stage:

• What is migrated to the new stack? What is left behind?
• Which parts of your tech stack are impacted by the migration?
• Which teams need to be involved? When do they need to be brought in?
• What type of migration strategy (e.g. lift and shift or full-modernization approach) is your team taking?

Ownership

Determine responsible parties for your migration.

Questions you should answer at this stage:

• Who is responsible for conducting X part of the migration?
• Which team is responsible for creating documentation around processes?
• Which team is responsible for drafting up and implementing contingency plans?

Timelines

Identify key milestones and goal timing for your migration. Some example milestones might look like:

• Security review completed
• Vendor agreements signed
• Users access and groups created in cloud storage platform and transformation tool
• Raw source data transferred over to new cloud storage platform
• Existing data warehouse is shut down

All of the elements of your migration plan should live in a document accessible to your key stakeholders that is regularly updated with milestone completions and blockers. We also recommend creating a temporary channel (ex. #tmp-cloud-migration) in your internal communication network (ex. Slack or MS Teams) to communicate important updates about the process to the right people.

Addressing security, compliance, and cost concerns

One of the biggest concerns and hesitations with moving to the cloud is a perceived lack of security and compliance. With on-prem solutions, you’re on your own server where you control and maintain updates and security protocols. Being on the cloud comes with connecting to servers outside of your own network, but that doesn’t mean you have to forfeit security and compliance. Before your migration even begins, your data, legal, and security teams should perform reviews on the security policies of your cloud systems and bake these reviews into your overall cloud migration planning document.

Lastly, a cloud migration can come with increased short-term costs to your data infrastructure. In order for a cloud migration to safely happen, it’s inevitable that for a point in time you will have both a cloud storage system and your existing storage both running. These duplicative costs are temporary as you migrate your data and transformation processes to the cloud, but important to budget for in your planning period. (Trust me—it’ll all be worth it when you can finally shut down that final server and turn off the lights.)

Meeting your team where they’re at and offering leveling-up support

One of the greatest concerns during a cloud migration is how you’re going to enable your existing team members to confidently use the new tools and adjust to the new workflows in a reasonable timeframe. When a cloud migration happens, there’s usually a handful of new platforms and tools data team members must get used to:

• ETL tools to load your data into your cloud data storage platform
• The data storage platform (ie data warehouse or data lake)
• The transformation tool
• Analytics/BI reporting softwares

Each of these layers introduces new skills someone may need to use them correctly. For example, the adoption of a new data warehouse may mean that there will need to be slight syntax adjustments to your existing queries to comply with the platform’s SQL flavor.

To reduce the friction in adopting new tools and level-up your team members, we recommend the following:

• Invest in tools that meet your users at their skill-level: While it’s inevitable that data team members will need to learn their way around new UIs and syntax, you can lower the barrier to contribution by leveraging tools that meet users where they’re at. For example, dbt is a SQL and Python-first transformation tool, allowing anyone who knows either one of these languages to be involved in the data transformation process. In addition, dbt Cloud removes the complexities of the command line and git through the use of an intuitive IDE (integrated development environment), enabling both analysts and developers to use dbt with confidence.
• Encourage or incentive trainings and certifications: Modern data tools and cloud storage platforms often have free or affordable training to get your teams up to speed. dbt, for example, offers several courses ranging from the fundamentals of dbt to refactoring existing SQL to modular entities.
• Establish best practices early on: There are few moments more rewarding than feeling like you’ve gotten off on the right foot—and just like in life, you want to start your cloud migration on the right foot. While you may not have solidified ideas on what best practices will look like in your new data warehouse or transformation tool, it’s helpful to create baseline boundaries and practices you want your team to follow. These best practices may change over time with your data and team, but establishing some form of structure and process from the get-go helps make doing the right things easy and doing the wrong things hard. This is especially vital when folks are using new tools or processes and may be unsure on where to start. For teams who are newer to data modeling, we recommend reading through dbt’s best practices guides, so you and your team can start the modern data transformation process on the right foot 🦶
• Lean on community support: The best way to avoid pitfalls in a cloud migration is to learn from folks who have already done it! Lean on peers and the data community to get invaluable insights into the wins and struggles of big migrations. dbt Labs shepherds a community of over 60,000 data practitioners where folks share knowledge, ask questions, and spark conversations around modern data problems.

These efforts will cost you time and effort in the beginning stages of a cloud migration, but have considerable dividends in the long-term that will ultimately save your team development time, energy, and money.

Avoiding considerable business downtime

It’s inevitable that during a migration, there will be some downtimes in data accessibility. To avoid unexpected downtimes, implement the following during your cloud migration:

• Communication is 🔑: Ensure that your team communicates clearly with end users about outages and migration plans. Notify specific teams when they’re specific data will be impacted and provide regular updates on when the data is back up.
• Work in sandbox environments 🏠 when possible: Work in protected development environments before you deploy to production, so your team can confirm things look and feel like they should before changes are fully exposed to end users.
• Start small before moving onto larger projects 📈: When your team first starts using new software and tools, it’s typically easier to start a migration off with smaller, simpler projects before tackling larger, more complex projects. Starting off small allows your team to develop a strong workflow in a relatively lower-risk space as well as helps them develop confidence and speed in their new systems. In addition, getting some small wins early in the process boosts the morale of your team and end users.

Challenges beyond the migration: You’ve migrated, what’s next?

Now it’s time for the fun stuff! Once the majority of your cloud migration is finished, you’ll likely shift focus to optimizing your data transformations for performance, implementing best practices throughout your data stack, and bringing other teams into the fold of your data work:

• Ensuring all of the tables in your data warehouse are documented, tested, and align with best practices.
• Performing cost-analyses on your cloud data storage usage to monitor spend.
• Ramping up analysts and technical business teams to become comfortable in your cloud data storage platform and with your data transformation tool.

Your journey’s just begun!

Migrating to the cloud and making dbt a part of that story

The challenges we outlined in this article are objectively real problems many data teams will face in their data modernization efforts. But it’s important to remember that teams are en masse migrating to the cloud in part because of the modern data tooling that exists to make these obstacles less challenging.

When you migrate your data to a cloud storage platform and pair it with a transformation tool like dbt Cloud, you approach cloud migration from an accessible, secure, and version controlled environment. dbt Cloud:

• Integrates with many different cloud storage providers
• Moves your transformation development, deployment, testing, and documentation to the cloud
• Bases itself in SQL—the lingua franca of both data analysts and data engineers—to create transformations, allowing anyone who knows it to be involved in the process
• Enforces next-level security and compliance
• Offers foundational training and opinionated best practices

A cloud migration is a challenging, but exciting thing: when you modernize your data stack, you’re investing in not only the long-term sustainability, security, and accessibility of your data pipelines, you’re also investing in your team’s ability to continue to drive data and meaningful insights at your organization.

Last modified on: Feb 22, 2024