Data migration is typically the most forgotten or underestimated component of an IT project which is the process of making a copy of data and moving it from one system to another, preferably without disrupting or disabling active business processes.
On some occasions, it is not easy to understand that a data migration is needed in the project and most of the times data migration is not seen as an item that its requirements needs to be captured during analysis phase. That’s why migration related problems begin during development or testing phase when data need is identified or when the data from the old system refuses to fit properly into your new user interfaces or business rules despite the transportation of the old data.
For a successful project, the need of a data migration and its requirements must to be identified early beginning of analysis phase and further actions should be reflected to project plan accordingly. But how?
Data migration strategies are a whole different detailed topic and they will not be mentioned here but capturing the essentials of data nature is a life-saver activity which is the targeted idea of this work.
1. Will you need a data migration?
Identify it during elicitation phase if possible. If not, analysis phase is also ok but be careful and just identify the need as earlier as possible.
Ask data related simple questions.
- Do your requirements mention from the systems need to be shut down? If yes, does this system keeps any data?
- Is your project a replacement project for an old system? If yes, does this system keeps any data?
- Do your requirements mention from entities that are already used in your company’s systems? In order to use that data do your new system’s data model needs to keep them as well. Watch out, it may mean also a nasty synchronization!
If one of answers is yes, you will need a potential data migration. Just continue to get more details.
2. How to capture requirements and define business rules?
If you determine that you need a data migration, be careful and specify at least following details in your analysis.
2.1 Identify if your company have multiple data sources for that entities. If yes, determine the master and the fate of other systems’ data. You will see how many systems you are dealing with. It is important to know how and where data is stored, backed up, and if it is archived.
2.2 Data profiling: Data profiling is a “must have” activity to understand what are specs of the material you will be working on. Conduct profiling activities and classify the data, such as data with missing unique ID’s or missing first name and last name or false data such as numeric values as first name and other useful information like length, type, candidate for primary key, etc. Results of the profiling will guide you to design user interfaces and capture business rules.
2.3 Do you need data cleansing? According the data profiling, you will see if you need cleansing activities and what will be the final structure of the data you will be working on. Also you will be able to clarify cleansing related activities on the project timeline which will provide a cleaner vision for the project timeline.
If your answer is no, definitely check the section 2.6.
2.4 Data structures must be well understood if your project requires design of user interfaces or data forms. Consider data structures while determining dynamic user interfaces or form context.
For example, old system may store the phone number and its extension in one field by separating data by “–“. This means two numbers are stored in one field and you should consider such constraints while you are designing a new interface with multiple fields on section 2.8 or determine ways to separate phone number and its extension effectively while moving the data as mentioned on section 2.7.
2.5 Fate of the historical data: Take into account if historical data will be migrated or not. It may affect your user interfaces or business processes.
2.6 Fate of the missing or dirty data: After profiling, most probably you will see that some of your data is not clean or adequate to use in further actions. For example, you are working on sensible customer data and national identity number is mandatory but some records do not have identity numbers. It will cause you problems to pinpoint the customer or you will face further problems if this information is mandatory to display customer on the screen. Even worse, if you have validations based on the identity number such as debt control or billing, the system will not be able to conduct such validations.
Always check whether data ownership belongs to a specific business unit. If yes, let them decide to the fate of data.
- Decide whether the data is just enough to use or will it cause problems to conduct business processes. Will you migrate such data? If yes, sections 2.8 and 2.10 will be highly important for you.
- Is it possible to clean or enhance problematic data somehow? If yes, how? Determine the way and related requirements.
- If you decide not to move such data, always consult with your business unit for possible further actions. They may need to find the customer and inform him/her legally according his/her account status or they may need another manual/automation processes.
2.7 Data mapping is basically the activity of creating a map of the existing data model by matching each entity&field with the future data model. Each entity should be mapped correctly and in details to be able to move data successfully. The map is an essential item of data migration strategy which is a whole another topic and not be mentioned here in details.
Based on the mapping, you can see the gap between your legacies and your future and you can use the information on section 2.8
2.8 Screen validations and rules: Results of sections 2.2-2.7 will give you clear information about user interface validations and potential need of new business processes.
2.8.1 Screen validations?
- Define entity specs such as type and length based on the profiling results, the information will guide you to design potential data forms.
- Define the rules for the gap. If entities are not matching neatly, define UI standards and validation rules accordingly. Such as, will be these entities optional or if two different data are kept in one field, will you continue to collect and display them together?
2.8.2 New processes is needed? If you decided to transport problematic data;
- You may need to create new processes to correct such data. For example if the identity number is mandatory and your customer update process originally does not allow to update the number, in order to enable the correction of the customer information, you may need the define rules such as displaying identity number field editable.
- You may create new processes to alert the system or trigger different actions.
2.9 Define security & security measures such as encryption. If data needs to be migrated encrypted, general rules shall be set during analysis phase.
2.10 Define migration acceptance criteria such as data quality, migration duration etc. if it will cause termination on your services
2.11 Define the fate of the legacy data: Will you continue to keep the data on the legacy systems? If yes, determine whether a synchronization is needed with the new system or not? How long the data should be kept on legacy system? What are maintenance rules?
3. What is next?
Of course, plan the future! All answers will guide you for further activities which need to be added to the project plan in details. Identifying these steps at the beginning will prevent you from future unexpected surprises and definitely will help to close the project on time.
3.1 Rehearsals
A clean migration needs rehearsals where you can have a look at your situation.
3.2 Testing scenarios
Requirements need to be tested and since migration activity creates its own requirements, testing scenarios should cover these requirements as well.
Author: Aylin Şen, Senior Business Analyst