Millions of position data is coming in a system which needs to be stored in a database. The data comes in pipe delimited formats in flat files,on a regular basis say twice a day. And most importantly out of a million records only 5% would have any change. Other records remains same as in earlier feed. Suggest an optimal strategy to put this data into database fast. Consider only 5% of incoming data would have any change in it compared to its previous version in database.
I checked a similar question as discussed here. But its about Bulk insert operation, which can be done by first chunking and then bulk insert to DB.
But the idea here is to somehow find out if some record has really changed, if yes, then only insert or update, otherwise just leave that record. This way it might save lot of time.