I'm facing a new enterprise project in Java about data migration. I did one of them in the past, now I want to do a better work and I think I need your considerations.

So, let's me explain the problem:
I have a lot of DB tables without any logical wire. So in the most of the cases I need to perform a huge logic in the Java application for build the correct results.

In past I have tested some advices, but in the end I used a lot of HashMap for store in memory the entries retrieved from db and perform the logic I wanted. I was really lucky because the entries were not much for the memory machine.

The question you maybe you think now is: why you need to store all the entries into HashMap?

Example: I have TABLE1 and i need to perform a logic to all the entries. So I have an HashMap with original data taken from db (key: the id of the table, value: the entry POJO) and they can be retrieved by a sequence of queries without load all on memory.

The second HashMap contains the modified data sets of the TABLE 1.

Then I have TABLE 2 to perform another logic but I need the first modified data sets to perform other task.

In the end I must necessarily have in memory all the HashMap that contains the modified data sets for do all the logic.

I searched a lot on the internet to see if you can put a caching mechanism, but I am very confused.

What I would need is an intelligent algorithm that saves me the entries of HashMap on disk in case of need, but it should be fast enough to recover data when I need. Another alternative would be to save the changed data on db and use a massive amount of queries, but I would avoid it because the results that interest me will not be saved to the database.

I'm facing a new enterprise project in Java about data migration. I did one of them in the past, now I want to do a better work and I think I need your considerations.

So, let me explain the problem:
I have a lot of DB tables without any logical wire. So in the most of the cases I need to perform a huge logic in the Java application for build the correct results.

In past I have tested some advice, but in the end I used a lot of HashMaps for store in memory the entries retrieved from db and perform the logic I wanted. I was really lucky because the entries were not much for the memory machine.

The question you maybe you think now is: why you need to store all the entries into HashMap?

Example: I have TABLE1 and I need to perform a logic to all the entries. So I have a HashMap with original data taken from db (key: the id of the table, value: the entry POJO) and they can be retrieved by a sequence of queries without load all on memory.

The second HashMap contains the modified data sets of the TABLE 1.

Then I have TABLE 2 to perform another logic but I need the first modified data sets to perform other task.

In the end I must necessarily have in memory all the HashMap that contains the modified data sets for do all the logic.

I searched a lot on the internet to see if you can put a caching mechanism, but I am very confused.

What I would need is an intelligent algorithm that saves me the entries of HashMap on disk in case of need, but it should be fast enough to recover data when I need. Another alternative would be to save the changed data on db and use a massive amount of queries, but I would avoid it because the results that interest me will not be saved to the database.

I'm facing a new Enterprise Project in Java about data migration. I have did one of them in the past, now I want to do a better work and I think I need your considerations.

So, let's me explain the problem: I have a lot of DB tables without any logical wire. So in the most of the cases i need to perform a huge logic in the java application for build the correct results. In past I have tested some advices, but in the end i used a lot of HashMap for store in memory the entries retrieved from db and perform the logic I wanted. I was really lucky because the entries were not much for the memory machine.

The question you maybe you think now is: why you need to store all the entries into hashMap?

Example: I have TABLE1 and i need to perform a logic to all the entries. So i have an HashMap with original data taken from db (key: the id of the table, value: the entry POJO) and they can be retrieved by a sequence of queries without load all on memory. The second HashMap contains the modified data sets of the TABLE 1. Then I have TABLE 2 to perform another logic but I need the first modified data sets to perform other task.

In the end I must necessarily have in memory all the HashMap that contains the modified data sets for do all the logic.

I searched a lot on the internet to see if you can put a caching mechanism, but I am very confused.

What I would need is an intelligent algorithm that saves me the entries of HashMap on disk in case of need, but it should be fast enough to recover data when I need. Another alternative would be to save the changed data on db and use a massive amount of queries, but I would avoid it because the results that interest me will not be saved to the database

Any ideas would be appreciated, Thank you all!

I'm facing a new enterprise project in Java about data migration. I did one of them in the past, now I want to do a better work and I think I need your considerations.

So, let's me explain the problem: 
I have a lot of DB tables without any logical wire. So in the most of the cases I need to perform a huge logic in the Java application for build the correct results.

In past I have tested some advices, but in the end I used a lot of HashMap for store in memory the entries retrieved from db and perform the logic I wanted. I was really lucky because the entries were not much for the memory machine.

The question you maybe you think now is: why you need to store all the entries into HashMap?

Example: I have TABLE1 and i need to perform a logic to all the entries. So I have an HashMap with original data taken from db (key: the id of the table, value: the entry POJO) and they can be retrieved by a sequence of queries without load all on memory.

The second HashMap contains the modified data sets of the TABLE 1.

Then I have TABLE 2 to perform another logic but I need the first modified data sets to perform other task.

In the end I must necessarily have in memory all the HashMap that contains the modified data sets for do all the logic.

I searched a lot on the internet to see if you can put a caching mechanism, but I am very confused.

What I would need is an intelligent algorithm that saves me the entries of HashMap on disk in case of need, but it should be fast enough to recover data when I need. Another alternative would be to save the changed data on db and use a massive amount of queries, but I would avoid it because the results that interest me will not be saved to the database.