Test data consistency between multiple servers

Question

I'm currently working on an architecture that have the following structure:

                            (central server)
                           /        |       \
            (local server)    (local server)  (local server)
             /       |          |        |        |       \
          (PC)  ... (PC)      (PC)  ... (PC)     (PC) ... (PC)

There is one central server that talk to multiple local servers (more than 100) and every local server talk to multiple PCs.

The central server receive a big amount of data every day multiple time a day and after some manipulation synchronize the data to the local server, after that the local servers spread the data on their PCs.

All this machines talk to each other via HTTP requests, every exchange of data is done with an HTTP POST of a json file.

Is critical that every information that the central server receive is correctly received and stored on the local server before and on the PCs after.

I have now to test if the synchronization work correctly and I want to automate the testing so that a script will run continuously on the central server and check if the newly arrived data are synchronized with the machines below.

So my first question is: it makes sense to test every time ALL the data the central server receive? (We are talking about tens of thousands of database entry for every single local server, so hundreds of thousand in total.)

Also for the PCs the performance are a big issue, I can't steal too many CPU or RAM resources from them.

If the answer is no, it makes sense to test just a subset of the data taken randomly?

If no, what is the best way to act?

UPDATE THe data are passed to the local server in this way: A service on the central server receive the data and store them on a database, another service is called, this one take the data from the database in COPY format and place the COPY in a file. The file is sended to the local server via an http POST request.

Answer 1

You mention testing quite a lot, but don't mention the mechanism used to trigger the updates or transmit data to the local servers. I fear it may be home-grown, which would make reliability more suspect and testing more essential.

For this kind of high fan-out, master-slave replication, you might be better off investing in a reliable transfer mechanism that gives you high confidence ("guarantees") what's on the local servers (or PCs) is what's on the central server. Rsync, Git, or Mercurial can be very efficiently used to effect the transfer; they all provide strong data integrity techniques. When you see commit 4740488633dd83c175788aa61824205513b825cf on a local server or PC, you can be very sure that's identical to the commit with the same id (hash) on the server.

Or, if you really just want to check that a file (or packet, or other unit of information) is the same on a local server or PC as it is on the central server, take a strong hash (or "checksum") of it, such as SHA-1, and compare to the hash/checksum computed "upstream." This is a hash technique that's often been used by version control systems. You can have extremely high confidence that if hash(a) == hash(b), that a and b are entirely identical. If you're particularly paranoid about the possibility of difference, there are even stronger, longer content hashes you can employ (e.g. the SHA-2 family; IIRC Mercurial is evolving towards its SHA-256 bit instance).

Answer 2

Depends

The purpose of testing is to reduce risk.

How the test is designed depends on the risk you are attempting to mitigate.

Some options

1. If you believe there is some risk that the updates may fail completely, it should be adequate to perform a spot check of the data, since that would be an efficient way of detecting a failure.

2. If you believe there is some risk that the update may succeed but may contain corrupt data, then an exhaustive check of the full data set is in order.

3. If you just need a general end-to-end check, the best sort of test would be to simulate a user (perhaps using a test user designated for this purpose) and perform the most common activities that would fail if the data were not up to date.

Answer 3

My initial thought is to use hash checking extensively, I particularly like MD5. It has an extremely low collision rate and provides a simple mechanism for checking data transfer integrity.

I can't comment re: PC resource use as the question doesn't go into enough detail about the types and frequency of operations performed by PC CPU's

Answer 4

First I would consider Local Server and PC as separate.
It is not the job of the main server to distribute to the PCs.

How often you test is separate from test procedure.

You would need and ID for the push.
Sounds like you should have the following tests:

• From central validate records received (as in PK) at a local server for a push ID
• From central validate records received (as in PK) at a local server and all data is consistent for a push ID
• From central validate a total sync with local
• Not required but from local validate a total sync with central

To the PC is a different story as I doubt you push. Not clear. You could maybe have a privileged accounts that can query both central and local and compare during off hours.