I am responsible for the design and development of a cloud backup application. Currently we are using xDelta to do differential backup so that only the changed parts of the files gets uploaded. This cuts cost for both us the company and the user. One issue we are having with this approach is that in order to do a differential backup the modified file needs to be compared to the original file. To solve this we opted for a quick solution and that is to cache the original files on the users' computer for a quick reference.

Now this solution works brilliantly for desktop/home users where the file sizes are relatively small. The problem arise when say a server administrator tries to backup an sql database for example that is 100GB in size. Though currently we cache the file on disk, this is definitely not appreciated by server administrators since we are filling their drives with cached files.

I am wondering if there is any other approach to do differential backups without the need of having the original files on disk permanently.

  • 4
    Do you know how rsync works? It basically calculates hashes of file blocks on either side, without transferring the files. AFAICT the same does bacula. Hopefully storing small hashes (e.g. 16-byte hash per 16-KiB block) would be more economical. – 9000 Apr 1 '14 at 11:30
  • @9000 thanks for the insight. I am really hoping to exclude the storing process altogether. Though this solution will probably be on the top of my list. – Zaid Amir Apr 2 '14 at 8:22
  • Are you looking at working implementations or design suggestions? If the latter, have you considered putting a limit to the size of cached files on the nodes? – lorenzog Apr 3 '14 at 14:37
  • @lorenzog currently I am in for both, though a reliable working implementation is faster if it has good support as it reduces the development and QA time, if not then I will have to implement my own methods. Currently 9000's suggestion is the one I am most considering but I am open to other suggestions – Zaid Amir Apr 4 '14 at 9:28
  • Seems like you have two basic choices....1. Store enough local information to calculate differences (whole file or rsync approach) 2. Collect the differences as the changes are made. (slowing performance in most cases). You presumably already have a full copy of the file(s) at some base point in time in your cloud environment. – joshp Apr 5 '14 at 18:00

It is really important when implementing a backup solution that the focus is not on the backup process, but on the restore process. It is the ability to get the data back into a working environment that matters, not the filling up of cloud hard drives and LTO tapes.

In that respect, when backing up databases, OS directories, MS Exchange etc... taking the 'grab the files' approach is just storing trouble for later.

With Databases if you do a byte by byte copy of the working files on an in use database then the result is likely to be a corrupt database when you attempt the restoration. If you make a shadow copy of the working files (and manage to get a consistent set with the shadow creation occurring simultaneously across all data files for that database) then you are providing a restoration option that is equivalent to recovery after a power off event. In that most of the time the database software will recover, but you will still have a failure rate of 1-10%.

The only safe way to back up database is to trigger a database backup in the database software. There are specific API's to trigger this, that can be done as part of the backup process which vary from database to database, but this will generate a backup file that WILL enable the database to be recreated again. This is the data that needs backing up. It will be new data each time so there is little point in doing differential analysis.

That said, companies using your services for this, would probably be better with a SAN supplying disks to virtualisation servers as iSCSI and implementing their infrastructure that way. When it comes to restoration, the success rate of restoring a virtual OS environment is much higher than to bare metal, considering that it is highly likely to be restored to different hardware with different device driver needs.

  • 1
    Sorry, but the problem is not solely related to databases. I gave a database file as an example. My main issue is to find a method to perform delta cloud backups without having to keep the original file. – Zaid Amir Apr 2 '14 at 8:18
  • 1
    I Appreciate that, but my key point still stands. You are focused on the backup side and trying to solve the problem of backing up large files, when I'm trying to alert you to the issue of restoring your systems to working order after a significant data loss event. When you have really understood that process, you will probably choose a different strategy for the backup aspect. – Michael Shaw Apr 3 '14 at 15:50
  • Again thanks for the information. That being said your answer does not address my question. Differential backup is a requirement that I am stuck with. Whether it be for sql databases, virtual image backup files or any other type of large files that change frequently. – Zaid Amir Apr 4 '14 at 9:17
  • Using the rsync approach where you treat large files as a series of 16k blocks, store hash values for each of these blocks. If the current block hash value is different, then you need to send that 16k block of that file may save you some network bandwidth.. but I would only consider this kind of service for a home user. – Michael Shaw Apr 4 '14 at 10:51
  • @RedSerpent the definition of "differential backup" is that you store the difference from a well known snapshot - which means that you already have a snapshot (ie backup) of the file. Databases provide this, as do backup programs or services. They also support backing up opened files (which you haven't considered) and metadata, avoid backing up empty pages etc – Panagiotis Kanavos Apr 9 '14 at 7:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.