I've gotten myself a bit more roped into coming up with the design and architecture for a new system due to some departures and looking for some insights.
The system has a bunch of different high speed services on a network that a variety of data will be gathered from at regular intervals (50 to 1000ms, depends on the particular service and possibly 10-50 data points per service).
The state the plan was left before handing it to me was that there were going to be two individual data polling servers that would each run a service to poll the data and maintain their own longer term storage of that data. The issue being there can't be gaps in data (or at least have to be very minimal) for future reporting purposes. So these servers would need to sync up their data to fill gaps if they went down.
It doesn't feel like a very unique problem though so I'm assuming there are some good designs for this. I'm thinking it would make more sense to have a 3rd server (or maybe you can do this with a NAS or something more simple?) to store the long term data and just have a raid there. Then the two data polling services would write to it depending on which is determined to currently be the master that is up.