0

I'm working as an intern at a fund. I spent the last month building a website for internal use, and now I think it's a good time to set up a backup scheme for the MySQL database at its backend. Funny enough, my mentor is reluctant to get me another server because we are not allowed to use external IaaS like AWS and DigitalOcean for security reasons, and it takes weeks to get a usable server from our IT department. Thus, I'm planning to make the backup on the same server running my website. Yeah, I understand the data would be gone if the disk fails or the server brows up, but at least it would be a lifesaver in case of an accidental DROP DATABASE production;. By the way, all "servers" assigned by IT appear to be VPS running on the same physical machine, so I guess backing up on another virtual server can't protect the data against a disk/server/power failure after all?

Anyway, here is my local backup plan: I will run mysqldump and commit it to a local git repository every minute. More concretely, I have set up a cronjob to run the following script with */1 * * * *. Essentially I'm using git as an incremental compression tool, rather than a VCS. I have also included some tags for easy navigation.

#!/usr/bin/env bash
export BACKUP_DIR=/home/foo/Backups/bar
mysqldump \
    --defaults-extra-file=/home/foo/Developer/MYSQL_ROOT.cnf \
    --single-transaction \
    --extended-insert=FALSE \
    production | sed '$d' > $BACKUP_DIR/production.sql
git -C $BACKUP_DIR commit --all -m 'Auto backup via cron' > /dev/null
git -C $BACKUP_DIR tag -f `date +%F`
git -C $BACKUP_DIR tag -f `date +%F@%H`
git -C $BACKUP_DIR tag -f `date +%F@%H-%M`

Currently, the backup script takes about a second to complete, and the resulting production.sql is around 3MiB in size. I estimate it would stay under 10MiB for years. The website in question has ~20 users, and I won't expect more than 1000 requests per day. I'm using MySQL Community Server 8.0.21 on RHEL 7, without any Enterprise subscription.

Can I make a reliable local backup this way? Is one lightweight tag per minute too much? Is there a better alternative?

6
  • 3 MB every minute is ~30 GB every single week. I mean, implementing such strategy may be not the completely worst idea, however you should then perhaps consider also hard removal of old commits to keep the repository as small as possible (with backups every minute - which I personally think is way too often), you should/could keep e.g. only the data for the last 24 hours.
    – Andy
    Jul 15, 2020 at 7:21
  • @Andy That's why I'm using git, which only stores differences between versions of files! :P
    – nalzok
    Jul 15, 2020 at 7:32
  • I didn't realise you are updating a single file, rather than storing each backup as a full file. Sorry.
    – Andy
    Jul 15, 2020 at 7:38
  • 2
    Your IT department should already have a backup system. Ask them to set it up on your server.
    – user253751
    Jul 15, 2020 at 8:54
  • @nalzok: Git does not store deltas. It stores full versions of files. You misunderstood the question you linked to. Jul 15, 2020 at 21:16

3 Answers 3

4

In short

No, this is not a real backup, and it might bring lots of trouble. There are better alternatives to protect you against accidental errors that worry you most.

More details

A real backup must be somewhere else, because the real question is not „if the server will break“ but „when it will“. Having the backup on the same server will not protect you against a hardware crash, nor even ransomeware. As Kain said: go talk to the sysadmin team about backup solutions. They surely can help!

Going for GIT is risky. For example:

  • Kain already talked about volumes issue, since the data is dumped in text form and accumulates over time. Done every minute you‘ll soon run out of time.
  • Privacy issues: not only do you make data more easily accessible to those who do not need to know (something that you can easily address in the db), but you keep forever old data which could have been corrected. This might be illegal for example under GDPR if you handle anything about a person.
  • loss of data: now suppose you make that fatal sql command. If you restore the db, you‘ll have lost transactions that were finished by the end of the db dump, but that was not db-committed at the start. And maybe nobody will remember what this data was. Furthermore, you may not notice the issue immediately (more data lost due to recovery) and it may take time to restore the db (unavailable service = unhappy users). It is never a good idea to experiment manually sql on a productive system. dot.

Alternative: For protecting against your own data manipulation errors, For the kind of operations you intend to do, another alternative could be to replicate the database, write your command in a script. test. if it works, run the script in production.

If you do this often, let your app write in two databases constantly, one allowing test access. Or better, start to automate the techy cases ;-)

3

Yes/No.

What is the growth on your Git repository over a day? over a week? a month? Use that to get a clear picture of how much history you can keep.

Similarly this backup, and all of the prior backups blow up if the repository experiences data corruption. What archival process are you instigating?

Can you liaise with the rest of your IT department, see what backup/archival mechanisms are in place. Many run weekly full backups, nightly diffs, and daily transaction logs. Are these sufficient for you? If not how does your mechanism segway into theirs so that you have sufficient redundancy, while also not over-engineering your own solution.

As for protecting against a drop xyz; might I recommend curating the login permissions being given? That statement cannot hurt you, if it cannot be run.

6
  • Thanks for the input! I just want to point out that setting up permission with some MySQL roles is probably not sufficient, because from time to time I might need to run some UPDATE/DELETE directly to the database to cover the edge cases which the web app cannot handle, in which case a missing or ill-specified WHERE clause can be catastrophic.
    – nalzok
    Jul 15, 2020 at 6:00
  • Fair enough, not everything can be covered, but the service account should be limited. And any administrative account should be only using sql tested on a development orientated data copy. At least obvious issues can be spotted upfront.
    – Kain0_0
    Jul 15, 2020 at 6:03
  • Similarly this backup, and all of the prior backups blow up if the repository experiences data corruption. I think this is inevitable for any local backup. Are you suggesting me to make backups of my git backup? (I can copy them over to my development machine with scp, but that sounds weird...) I'm not sure what you mean by "data corruption", becuase git commit usually doesn't corrupt data!
    – nalzok
    Jul 15, 2020 at 6:31
  • Generally no git won't corrupt data, but that doesn't hold up under historical review. git creates a diff chain. If anyone of those diffs becomes corrupted, and their entire future is corrupted. Those diffs are also stored within a file format that is frequently recompressed, rearranged, and is bit-packed. All of these operations, and layout qualities increase the likely hood or corruption above the background random bit flip. It may no longer be the case, but my recollection is that none of the data has a check code, let alone a self-healing code, so their is little/no corruption resistance.
    – Kain0_0
    Jul 15, 2020 at 8:02
  • @Kain0_0 In git, each object ID is a checksum of the object; Git should notice if the checksum of the data it actually fetches doesn't match the object ID.
    – user253751
    Jul 15, 2020 at 8:55
3

Git is not a backup solution.

There are countless programs available which are specialized for creating regular incremental backups of important files, both paid and free software. There are even some which are specifically designed for MySQL databases. You will likely get far better results by using a tool actually built for that purpose.

That way you won't end up with a Git repository with 1440 commits every day and a half million commits every year.

Also, remember that local backups should only be a temporary solution. Yes, processes in enterprise environments can be slow and bureaucratic, but it is common best practice in every professionally managed datacenter that every physical server and every productive application has at least one backup which is physically separated, managed by the datacenter administrators and can be restored if needed. If you forgot to order the backup for your server when you started your project, then you should better do that now.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.