I am currently developing a continuous backup software. The key feature of the software is instant backup meaning that each time a file is created, modified, renamed or deleted the change is immediately reflected on the backup repository.

In order to achieve this we designed the system so that once the client application starts, the entire backed up files database is loaded in memory. The issue arise when backing up large amounts of files say 10 - 15 million files. This first causes the load time to be slow and high memory usage. In some cases reaching 2GB.

Querying the database on every change is also frowned upon as it will cause heavy loads on the disk where the database resides not to mention how slow the query execution time becomes as the database grows.

So I was wondering what options does one have to design such a system to be memory efficient and reasonably fast.

  • 1
    Would much depend on many details. Do you really need the whole database in memory or could you for example just keep the most recent entry for every file and load older versions only on demand? Also with something that seriously wants to handle that amount of files a dedicated server for this database could be an option. Basically 10-15 million records shouldn't be that much of a query problem if the dbms system is highly optimized, columns are indexed properly etc. Maybe researching different database solution (eg NoSQL) would be another option. Aug 28, 2014 at 10:24
  • Are you using a decent dbms, or something home grown? You shouldn't need to keep the full db in memory. At most, the indexes. I think you are prematurely optimizing in an attempt to solve a problem you don't know for sure you'll have. Aug 28, 2014 at 20:25

2 Answers 2


The time taken to query a local database on disk should be orders of magnitude less than the time required to backup a file or communicate over the internet. (Your database isn't on the online server, is it? In which case keep a cached copy locally - assuming this isn't shared between users. Use a version number to check that the local host is using the same version of the database as the remote server).

I would expect some recency effects: after a file is created, it is quite likely to be updated or renamed. So if database access is still an issue, cache some of the recent entries...


Loading the data to memory in a sane manner is the job of the database engine...

You DO NOT need to manually perform that task, your database engine will perform that task for you, its purpose is to process queries and cache the required data. Unless you are firing 1000's of queries a second, dont be afraid to fire multiple queries to a database, it is designed for it

  • 1
    and if you do need a specialist in-memory database (you probably don't) then use one - eg timesten (now owned by Oracle), MemSQL or Altibase for example.. or sqlite
    – gbjbaanb
    Aug 29, 2014 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.