Everything You Need to Know about Data Migration

Data migration is one of those inevitabilities that comes with success! When your software is gaining traction, this is one of the growing pains. In this episode, we cover everything you need to know about data migration.

Full Transcript Below:

Cris :

And what happens during data migration?

Cody :

Okay. They have a different database technology or something of that sort, or it’s just two different locations physically within the internet. And you’ve got to move data around.

Cris :

You might have grown a lot faster than you anticipated. And so suddenly the option that you have as far as where it’s housing your data just isn’t working anymore.

Cody :

Typically a part of that kind of group of things you’ve got to do is going to involve data migration a lot of the time.

Cris :

So Cody, today we get to talk a little bit about a “how-to” kind of a situation. We’re going to talk about data migration. So I wanted to start off by saying what happens during data migration? What are some of the situations that kind of cause a need for it? How does data migration even come into play as a thing?

Cody :

Well, it definitely is like asking the question, when does moving come into play? And the answer is, well, there’s a lot of situations. A couple of common ones within the case of data migration would be our old hosting provider is charging far too much. We need to move to a new one. They have a different database technology or something of that sort, or it’s just two different locations physically within the internet, and you’ve got to move data around. And that’s one of the most common use cases.

Cody :

Another is simply you found out that early choices in your architecture, like your actual database technology choice or any type of storage mechanism is not either, A, compliant, or B, performant to the current business environment as you have matured as a business.

Cris :

And you might have grown a lot faster than you anticipated. And so suddenly the option that you have as far as where it’s housing your data just isn’t working anymore.

Cody :

Yeah, yeah, exactly. And this is a real common scenario because I mean, as a startup, the startup mentality, and really how to survive is to be as scrappy as possible, use as light a weight solutions as possible, use as quick of solutions as possible. And usually, kind of create a nice cache of tech debt along the way. And that’s really the right way to do it in a lot of cases. But ultimately as you mature, you’re going to have to handle that tech debt, and one of the main things that is typically a part of that kind of group of things you’ve got to do is going to involve a data migration a lot of the time.

Cris :

Got it. So when we’re migrating all this data, obviously things can go wrong because very similar, I’ve actually used the moving experience. I’ve actually had like a box left on the top of the car and you move and it just falls off in the process. So like taking this back to data migration, what can go wrong when you’re migrating data?

Cody :

Well, a lot. I mean the first and most obvious thing is downtime. If you have a business that is entirely reliant on a platform to be online and operating income. If you need to move the database providers or something like that, and you have to take the systems down, well that’s just dollars out of the company coffers, like minute by minute. So that’s a big issue. Now that said, of course, data migrations are possible to be done with no downtime. It’s usually so minimal, like milliseconds that it would be considered no, but there are ways to do it. And then, of course, downtime aside, there’s always the possibility of the migration itself going poorly or taking much longer than anticipated, which can be the case, depending on how things were decided in, I guess, the infancy of a project.

Cris :

I did not Google Map my house and I do not know how to get there. Like …

Cody :

Yeah, well it’s kind of like, well if you bought a house that was smaller than you needed, or I should say larger than you needed when you needed a house initially, and then you had a family.

Cris :

Got you.

Cody :

That house might be now too small.

Cris :

Interesting.

Cody :

But on the flip side, if you, for some reason, bought that house with cash and had no inspection, you may never be able to sell it because you found out that it’s a super fun site, years down the road. That’s definitely a very far and wide analogy, but that’s kind of the scenario we’re talking about is that if there was not enough thought put into the initial technology you might have to migrate down the road.

Cody :

It could end up being extremely complex, especially if the application itself operates on top of this data, whatever it may be, whether it’s a database, or a key-value store, or something of that sort. If there wasn’t a lot of thought put into, “hey, one day we might need to change this.” Chances are, it can be quite the ordeal. Quite the ordeal.

Cris :

So how do we put thought into these data migrations? How do you as the company, the user, when you’re initially setting things up, how do you prepare for these data migrations? How do we prepare for these data migrations? And what can clients expect when they’re going to do a data migration with Bixly?

Cody :

Well, I think the most important thing to realize is that every data migration is ultimately entirely unique and it can mean many different things. As I mentioned, there is, of course, okay, we need to go from this hosting provider to this hosting provider. Basically same technology to the same technology, but something is different in terms of either the size of the hardware, where it’s located, the hosting provider, to the cost because of X, Y, Z reasons. But ultimately you’re taking, let’s say MySQL and you’re moving it to another MySQL server just somewhere else, or it’s configured differently. That is a simpler type of data migration because ultimately you just have to copy it.

Cody :

Now, the other side would be a more complex data migration. Let’s say early on, and this is a really common scenario. A client decided to use a NoSQL technology. Early on. Because they like it, it’s fast. It’s a hot buzzword. There are a lot of reasons to use NoSQL, but there’s also a lot of reasons to not. And unfortunately, when it was kind of buzzwordy, a lot of people ended up using it and found out that, oh, it really does have some limitations.

Cris :

I do need SQL.

Cody :

Yeah. I need to SQL. So taking data from that, which does not have true schema validation and doesn’t have all this kind of structure and relationship reinforcement that is typically present in a database that is relational, and then trying to move it to a relational database. You’re going to have to build a lot of things along the way. There’s really not that much of an automatic process. And there are tools that will make it easier, but that’s before you even get to the point that, oh, we need to also update our whole software and like change everything. And like that kind of migration would probably take a long time. So kind of without getting too much in the weeds here, which I already kind of did.

Cris :

That’s okay. And I think it leads into, well, like, I mean, that’s a lot that obviously can kind of go wrong. How do you ensure that the data is kind of ready to go when it needs to go, as much as you can?

Cody :

Yeah. And I’m going to answer that question with a very vague statement. And that is how do you prepare for anything where there are unknown side effects that cannot be known until you approach it. You ultimately have to do your research. You have to go through and evaluate the code. You have to know the two technologies specifically. And you need to formulate an execution strategy. Sometimes there’s just no good execution strategy, but those variables need to be known before you go in blind, and all of a sudden, just decide like we’re going to migrate it today. And then oops, the platform’s down for a week because X, Y, Z reasons, and we’re too deep to go back now and it’s too important to cancel the transition.

Cody :

So I think that not planning is planning to fail when it comes to data migrations. And I think that’s a big part of this is a lot of the, I guess the core way to make a data migration go off smoothly is to simply spend the time and money it takes to adequately have a strategy. Not just go at it blindly. And that includes not only during, okay, now we know we need to make a data migration, but also think forward when you’re making those important decisions right in the beginning. Understand, okay, what would it actually take to make us have to change our choices? Is this tech just going to be old within a few years? Which is super common, especially in JavaScript. Or do we anticipate that if we have, I don’t know, 10,000 users active at any one time, we’re going to need to change technologies entirely? Do we anticipate that our hosting provider is going to get more expensive? Those are all things to think about.

Cody :

And I wish I could offer a more concrete solution, but ultimately data migrations, both in their core … What’s the word? The core reason to even do them is incredibly unique to every project. And on top of that, the process to do them is even more unique to every project. Unfortunately, development, product development, and computer science, in general, is a very fuzzy science. So because of that, there’s no standard way any project is laid out. Even ones that have kind of a standard structure, usually have vast differences.

Cris :

What I’m hearing is when you’re dealing with data migration, you need to be forward-thinking, and not only forward-thinking in the sense of you, okay, so you start off and you’re like, okay, well, I need to make sure, because I’m going to need to migrate my data. But even when you go through that data migration, you move to the next step. It’s like, you always need to be thinking about the next step much as you possibly can because you want to solve the problem at hand and you don’t want to do that poorly. So you plan. But you need to keep in mind that just because you migrated the data today, it doesn’t mean you’re not going to probably need to migrate it again, and again, and again. You will more than likely need to migrate that data if you’re continuing to grow as a company and something’s going to shift. So planning is extremely important. If you don’t plan, you kind of set yourself up for failure.

Alexandra:

Thank you so much for joining us for this episode of Bixly Tech Tuesday. I hope you enjoyed that conversation with Cris and Cody, as they talked all about data migration, what to expect, mistakes to avoid, all of that great information. If you have any questions at all, go ahead and leave them in this comment section and we’ll get right back to you. And don’t forget to check out the description box down below. We have a bunch of really helpful links for you. And in particular, I want to highlight our free DevOps guide. It has a bunch of great information if you’re interested in a DevOps implementation. And of course, check us out online, bixly.com. It’s a great place to learn about all of the different services that we have, including a free 60-minute call with Cris. All you have to do is hit that button right at the top that says start my project roadmap. Until next time, this has been an episode of Bixly Tech Tuesday.