Big Data Transformation Testing Based on Data Reverse Engineering

During the transformation of huge volume of data, there might exist data mismatch, miscalculation and/or loss of useful data that leads to an unsuccessful data transformation. To check out the occurrence of such possible errors, testing is a crucial requirement. The existing quality testing methods are either unreliable, return biased results, fail to provide answers for data differences or have several limitations which does not treat each and every part of the data into the process. We propose an approach of big data transformation testing based on the concept of data reverse engineering. It is a comprehensive approach that reverse the whole transformation process and does a comparison testing on each and every entry of the data if the original source data can be constructed back from the target data, once successful ETL process is done.