0

I've gotten myself a bit more roped into coming up with the design and architecture for a new system due to some departures and looking for some insights.

The system has a bunch of different high speed services on a network that a variety of data will be gathered from at regular intervals (50 to 1000ms, depends on the particular service and possibly 10-50 data points per service).

The state the plan was left before handing it to me was that there were going to be two individual data polling servers that would each run a service to poll the data and maintain their own longer term storage of that data. The issue being there can't be gaps in data (or at least have to be very minimal) for future reporting purposes. So these servers would need to sync up their data to fill gaps if they went down.

It doesn't feel like a very unique problem though so I'm assuming there are some good designs for this. I'm thinking it would make more sense to have a 3rd server (or maybe you can do this with a NAS or something more simple?) to store the long term data and just have a raid there. Then the two data polling services would write to it depending on which is determined to currently be the master that is up.

Improve this question
1
  • What is the bandwidth/connect from the polling servers to the ultimate source of data? I'm not sure why you would want two servers polling the ultimate source of data - as that would just slow down the effort (you might want two around in case one went down, but only one active).
    – Lewis Pringle
    Jul 26, 2018 at 0:36

2 Answers 2

Reset to default
1

2 central polling servers will not scale well.

Rather than having 2 central polling servers, have a polling server per service on the same box if possible, have this post data to a load balanced central server.

To deal with these polling services crashing, have a second monitor process on the same box which watches the polling service and starts a second one if it crashes.

Have a central service monitor the monitor services and raise an alert if any of them go down.

Improve this answer
0

Is there a reason that each server has to maintain it's own data? Seems to defeat the purpose of databases.

As far as gaps go, a NoSQL type db (on your 3rd server) would be the way to go.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.