I am currently developing a continuous backup software. The key feature of the software is instant backup meaning that each time a file is created, modified, renamed or deleted the change is immediately reflected on the backup repository.
In order to achieve this we designed the system so that once the client application starts, the entire backed up files database is loaded in memory. The issue arise when backing up large amounts of files say 10 - 15 million files. This first causes the load time to be slow and high memory usage. In some cases reaching 2GB.
Querying the database on every change is also frowned upon as it will cause heavy loads on the disk where the database resides not to mention how slow the query execution time becomes as the database grows.
So I was wondering what options does one have to design such a system to be memory efficient and reasonably fast.