3

If I am collecting data that needs to be posted to a web service on a regular (~5 minutes) basis, and that service has temporarily gone offline, what is the best practice to ensure we don't lose any data?

Of course, we could simply buffer in memory until the service is back online, but what if the service is offline for a great period of time? Should we fall back to simply storing the serialized data to the local file system until the service is back online?

Are there other considerations/practices?

Any 'out of the box' frameworks that handle this for C#?

EDIT:

It's business critical that we don't lose any, or very minimal, data as it's for auditing for sensitive temperature controlled environments. We'll be using Json, and they will simply be sensor readings (not a large amount of data). We'll be using Web API, probably hosted on Azure, so the availability of the service should be good. Of course there are other factors such as localised network concerns. Everything will be under our control, apart from the infrastructure where the sensor software will be installed, this will be at client sites.

Improve this question
5
  • 2
    Would depend on amount of data, data format (CSV, binary, JSON, XML...) and maybe a few other factors (also considering using another service if this one isn't reliable enough, though of course may not be an option for whatever reasons). There are various simple solutions for log data that could handle this maybe, one thing to look into could be the Elasticsearch ELK stack.
    – thorsten müller
    Jul 20, 2015 at 11:30
  • 1
    What resources are available to you? What (e.g. monetary) loss would your company endure if the data is fully lost? What is the volume of data and bandwidth? How serious/reliable is that web service to which you post? Is it under your control?
    – Basile Starynkevitch
    Jul 20, 2015 at 11:34
  • Did you consider doing the opposite. Have some kind of server on your machine, and have the web service pull that data periodically?
    – Basile Starynkevitch
    Jul 20, 2015 at 11:40
  • It's business critical that we don't lose any, or very minimal, data as it's for auditing for sensitive temperature controlled environments. We'll be using Json, and they will simply be sensor readings (not a large amount of data). We'll be using Web API, probably hosted on Azure, so the availability of the service should be good. Of course there are other factors such as localised network concerns. Everything will be under our control, apart from the infrastructure where the sensor software will be installed, this will be at client sites.
    – Darren Young
    Jul 20, 2015 at 12:49
  • 1
    @DarrenYoung: you should edit your question instead of commenting it
    – Basile Starynkevitch
    Jul 21, 2015 at 5:15

1 Answer 1

Reset to default
3

Since you mention that "It's business critical that we don't lose any, or very minimal, data" IMHO buffering in memory is not really a good idea - it's volatile and if your machine/process dies for whatever reason that data is lost. Plus in-memory buffering needs extra care (unknown downtime duration, chewing memory resources, etc).

On upload failure I'd always fallback to storing the data in a local persistent location which can survive a server reboot. Whenever the "current" upload attempt succeeds (i.e. the server is back online) I'd retry uploading all "outstanding" data from the persistent location, maybe in multiple chunks if needed.

Shouldn't be that complex to warrant an 'out of the box' framework solely for this purpose.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.