I created am ETL that parse various files, transform file's data and then push the lines into a DB, until now I did a manual tests and check that all the values parsed correctly and all the lines (that needed) are insert into DB, of course it's become frustrating and error prone and I want to automate those tests.
I thought to create a "tests files" with "foretold" data and then, parse them and try to valid(verify) all the values on the DB, but I'm not sure if I should to test the DB's values or the "transformers" functions in my code.
For example:
I have 3 types of CSV files that contain:
- File 1
- Birth date - DD/MM/YYYY (23/04/1992)
- Height - cm (173)
- Weight - Kg (70)
- File 2
- Birth date - MM/DD/YYYY (04/23/1992)
- Height - inch (68.11)
- Weight - pounds (154.324)
- File 3
- Birth date - YYYY-MM-DD (1992-04-23)
- Height - cm (173)
- Weight - Kg (70)
So the code extracts the lines from each file and then (according to file's mapping) from column value and creates an instance of the appropriate transformer, the transform values insert into DB, in my example, after parsing those files in DB will have 3 rows with same values (23/04/1992, 173, 70)
Maybe there is a more correct way to do the testing?