Architecture of data logging and storage system

Question

I've gotten myself a bit more roped into coming up with the design and architecture for a new system due to some departures and looking for some insights.

The system has a bunch of different high speed services on a network that a variety of data will be gathered from at regular intervals (50 to 1000ms, depends on the particular service and possibly 10-50 data points per service).

The state the plan was left before handing it to me was that there were going to be two individual data polling servers that would each run a service to poll the data and maintain their own longer term storage of that data. The issue being there can't be gaps in data (or at least have to be very minimal) for future reporting purposes. So these servers would need to sync up their data to fill gaps if they went down.

It doesn't feel like a very unique problem though so I'm assuming there are some good designs for this. I'm thinking it would make more sense to have a 3rd server (or maybe you can do this with a NAS or something more simple?) to store the long term data and just have a raid there. Then the two data polling services would write to it depending on which is determined to currently be the master that is up.

Answer 1

2 central polling servers will not scale well.

Rather than having 2 central polling servers, have a polling server per service on the same box if possible, have this post data to a load balanced central server.

To deal with these polling services crashing, have a second monitor process on the same box which watches the polling service and starts a second one if it crashes.

Have a central service monitor the monitor services and raise an alert if any of them go down.

Answer 2

Is there a reason that each server has to maintain it's own data? Seems to defeat the purpose of databases.

As far as gaps go, a NoSQL type db (on your 3rd server) would be the way to go.