Azure Service Fabric - Designing a Calculation Engine

Question

Suppose I have the following devices: Device A Device B Device C (...)

Also, for each device, I have the following measurements: Measurement 1 Measurement 2 (...)

Each measurement event includes a measurement id (int), a date (DateTime) and a value (double).

What I'm trying to achieve is, in an event driven scenario, how would you design a calculation engine where, for each device, you want to calculate new measurements based on triggered events which we receive from the device's measurements?

Key problems I'm trying to address here

• Which components of Service Fabric should be used? Assume we have around 10.000 devices times 200 measurements. We can use external services like Rdis Cache or some memory based store (trying to avoid disk writes..)

• Some calculations requires that a previous calculation is completed before its finished. Example:

  DeviceA.CalculatedMeasurement1 = DeviceA.Measurement1 + DeviceA.Measurement2 DeviceA.CalculatedMeasurement2 = DeviceA.CalculatedMeasurement1 x DeviceA.Measurement3

• Some calculations just aggregate data. Like hourly aggregations or daily aggregations. How would you trigger this kind of calculations since they can only be triggered if all the events are received for that particular hour or day?

I'm just looking for something to start with. Right now I'm kinda lost with all the options we have. If someone can point me to the right direction to approach this issues will be much appreciated. Thank you for your time!

Answer 1

How would you design a calculation engine where, for each device, you want to calculate new measurements based on triggered events which we receive from the device's measurements?

See how compute grids are designed I think this would be best technology to address the compute requirements. The grid middleware systems are good at large number of calculations and task dependency handling. For specific design solutions and ideas about solving this kind of problems go to online docs on IBM Spectrum Symphony, MS HPC Server, Techlia Grid, Apache Ignite, Tibco GridServer.

How would you trigger this kind of calculations since they can only be triggered if all the events are received for that particular hour or day?

Cron and batch job at the end of hour or day. Of course, if you need to ensure that all events were received before the job starts then you will have to come up with some kind of validation checks specific to your application.