I chanced to see some job requirements of Oracle SQL programmer or C# developer jobs, who want a person who can handle terabytes of data in ETL or data migration. I have worked on data migration to move data from one system to the other. As a C# developer or Oracle SQL programmer, I am not able to understand how things will be different if large data is involved. (Oracle documentation says that it can easily handle even 8000 terabytes, why do I have to worry about volume then?) Can someone please advice what is special about handling data migration of large volumes?
Simply put inefficient algorithms on small amounts of data is trivial. It take maybe a second longer. No one will miss that amount of time. As the amount of data you work with increases, efficiency really comes into to play. That O(N)^2 loop that took say 5 seconds on a gig of data, now can take hours on 100 gigabytes. You have to know what to look for before you are done writing the code, because in a large data scenario, there is rarely a "Hey let's try it out and see how it works," possibility. In scenarios like this it is not uncommon to take an entire day to load the data into memory on the server to handle processing. You don't get many shots to get it right.
Sadly even writing an algorithm that is O(10n) and writing an algorithm that is O(2,5N) matters in such conditions. Such jobs require VERY careful planning, excellent understanding of pipelining, parallelization (how the ETL engine will parallize it), Task-Based paradigm and other things. Sometimes even You have to be aware of hardware limitation (how the data is split among disk, how are the partitions and logs organized, do YOu need transactions etc.) You need to know that an UPDATE is more expensive than insert for example... Lot's of things. You need to think DATA, memory. This is really NOT that easy as it looks like. luke