0

I'm facing a new enterprise project in Java about data migration. I did one of them in the past, now I want to do a better work and I think I need your considerations.

So, let me explain the problem:
I have a lot of DB tables without any logical wire. So in the most of the cases I need to perform a huge logic in the Java application for build the correct results.

In past I have tested some advice, but in the end I used a lot of HashMaps for store in memory the entries retrieved from db and perform the logic I wanted. I was really lucky because the entries were not much for the memory machine.

The question you maybe you think now is: why you need to store all the entries into HashMap?

Example: I have TABLE1 and I need to perform a logic to all the entries. So I have a HashMap with original data taken from db (key: the id of the table, value: the entry POJO) and they can be retrieved by a sequence of queries without load all on memory.

The second HashMap contains the modified data sets of the TABLE 1.

Then I have TABLE 2 to perform another logic but I need the first modified data sets to perform other task.

In the end I must necessarily have in memory all the HashMap that contains the modified data sets for do all the logic.

I searched a lot on the internet to see if you can put a caching mechanism, but I am very confused.

What I would need is an intelligent algorithm that saves me the entries of HashMap on disk in case of need, but it should be fast enough to recover data when I need. Another alternative would be to save the changed data on db and use a massive amount of queries, but I would avoid it because the results that interest me will not be saved to the database.

Improve this question

1 Answer 1

Reset to default
1

If you want a drop-in replacement of HashMap/TreeMap, then MapDB is what you're looking for. It has features such as transparent serialization to disk and multiple caching strategies (intelligent algorithm you've mentioned).

Also you may consider doing all the work inside the DB, using stored procedures and temporary tables. It might be faster, because of the lack of sending data over the network.

Improve this answer
1
  • Thank you for advice Scriptin, i will try to find out if MapDB is the good choise.
    – Mattia Peterle
    Commented Jan 25, 2015 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.