Organizations often need to develop custom data migration strategies and use proprietary software to successfully complete the data migration process. They also need to decide which data migration approach best suits their needs.
SEE: Data migration test checklist: Via before and after migration (TechRepublic Premium)
In this guide, we’ll cover the basics that data migration entails, but we’ll also cover the different types of data migrations and when you might want to use each in more detail.
What is data migration?
Data migration is the process of transferring data from one location to another. This can be a transfer between databases, storage systems, applications, or various other formats and systems. The data migration process often includes multiple steps to get data ready for migration, including data preparation, extraction, and transformation.
SEE: Introduction to data migration (TechRepublic)
The goals of data migration are to ensure that data is migrated accurately and completely, minimizing data downtime and minimizing migration costs. Common data migration scenarios include website consolidation, legacy system upgrades or replacements, adoption of a cloud-based system, infrastructure maintenance or information systems consolidation.
Data migration types by system format
While many data migration best practices and strategies remain the same no matter what data format or system type you are working with, it is important to understand that certain steps will need to be added or reworked depending on the type of data you are moving. as well as the associated source and target systems.
Database or schema migration
Database or schema migration happens when a database schema is set to a previous or new version of the database to make the migration smoother. Data conversion steps are often an important part of this type of migration, as many companies operate on legacy database and file system formats.
This type of project involves moving datasets from one storage system or format to another. Today, this usually involves moving data from tape or a traditional hard disk drive to a higher capacity hard disk drive or to the cloud.
Data center migration
Data center migration involves moving your entire data center to a new physical location or a new non-physical system such as the cloud. Due to the scale of this project, extensive data mapping and preparation is required to successfully migrate.
Migrating to the cloud happens when organizations migrate from legacy on-premises systems to the cloud or migrate from one cloud provider to another. Applications, databases, and various other business assets will need to be migrated in such a transition. Because of its complexity, most people rely on a third-party vendor or service provider to assist with the cloud migration.
Such migration may involve moving applications from one environment to another, but may also involve moving datasets from one application to another application. This type of migration often happens in parallel with cloud or data center migrations, but it can also happen when moving from one vendor to another, for example, for a project management application.
Business process migration
Business process migration is used to ensure that all information is shared with the target system and acquiring company, especially during mergers and acquisitions as well as other major business transformations. Depending on the industry and region, such a transition may include a special emphasis on data governance and security measures.
Master data migration strategy types
Choosing the right data migration strategy can have a significant impact on the success of the migration and ensure that the migration is smooth and without significant delays. The two primary data migration strategies are a big bang data migration and a trickle data migration.
Big bang data migration approach
The big bang approach involves transferring all data from source to destination in a single transaction. This makes big bang data migrations less complex, less costly, and less time consuming than drop-by-drop data migrations. Some organizations may complete a big data migration on a vacation or weekend when they’re not using related apps.
SEE: Data migration and data integration: What’s the difference? (TechRepublic)
It’s worth noting that there is significant downtime during a large data migration, as the systems using the data will be down and unavailable until the migration is complete. The downtime may be greater for organizations carrying huge amounts of data.
In addition, the limited throughput of networks and APIs can further delay the data migration process. As the complexity and volume of data continues to increase, the big bang data migration approach may become more difficult to implement.
- takes less time
- less complex
- less costly
- Requires data interruption
- Higher risk of costly failure
The big bang data migration approach is best suited for small businesses or data migration projects with small amounts of data. This approach is not ideal for moving mission-critical data that needs to be available 24/7.
Drop data migration approach
The trickle data migration approach is a type of iterative or incremental migration. It uses agile techniques to complete the data transfer.
The entire process is broken down into smaller sub-portions, each with its own timeline, objectives, scope, and quality controls. One of the primary goals of drip data migration is to ensure zero downtime; which makes this strategy ideal for organizations that need 24/7 access to data. Source and target systems run in parallel while data is moved in small increments.
The disadvantages of the drip data migration approach are that it takes longer to complete the migration process and significant resources need to be allocated to the project to keep two parallel systems running simultaneously. Additionally, data engineers must ensure that data is synchronized across both systems in real time.
SEE: Hiring kit: Data engineer (TechRepublic Premium)
A common approach is to run the source system until the end of the migration, and users will only migrate to the target system after the entire migration has been successful. However, data engineers need to be mindful that any updates or changes to the source system should be reflected in the target system.
- Zero downtime
- Less prone to unexpected failures
- More expensive
- takes more time
- Needs extra resources to keep two systems up and running
Midsize and larger organizations may prefer this data migration approach as there is no data disruption. Larger organizations may also have the resources and technical expertise to run two systems simultaneously.
Data migration best practices
The purpose of data backup is to make a copy of the data that can be recovered in case of data failure. It is best to profile all source data before writing mapping scripts.
Create a dedicated team for data migration
Appointing or hiring data migration specialists will ensure smooth completion of the project. If there are problems, a well-trained and qualified team should have the capacity, skills and experience to address them.
Full continuous testing
Data engineers should test data migration at all stages, including planning, design, and maintenance.
Don’t be quick to shut down the old platform
Sometimes, the first attempt to complete the data migration fails and requires a rollback and another attempt. It’s best to wait until the target migration is complete and tested before completely moving away from legacy systems and applications.
Read next: Best cloud and app migration tools (TechRepublic)