1

I am looking to enhance my skills in back-end technologies and would need your help in setting up a scalable microservices architecture.

Here's my project.

I have N sensors that send me data (Pressure, temperature, pollution, etc.) every second.

I would like to create a web interface that displays the latest known values from each sensor, as well as a summary with calculations on each type of data based on the sensors. I also need a history for each sensor. I am using Angular with HTTP and websockets.

To meet this requirement, I need several microservices. At the heart of this architecture, I thought of a Redis database. For historization, a MongoDB database. For microservices:

  • I have a acquisition microservice, which connects to the sensors and retrieves the data and writes it into the Redis database.
  • I have a historization service, which is subscribed to all the keys in the Redis database. When a value changes in the Redis database, it gets notified and archives it in its MongoDB database.
  • I have a microservice that performs real-time data calculations. It is subscribed to values. When a value changes, it performs calculations and writes the results of its calculations to other Redis keys.
  • I have a notification microservice. It is subscribed to keys in the Redis database. Upon a change in the Redis database, it sends a notification to web clients connected via websocket.

I deploy all of this with Docker Swarm. I have multiple instances of each microservices. In my case, it's not necessary since I don't need high availability, but the goal is to design something clean.

I have noticed a problem : If I have 5 replicas of my historization service, all 5 will be notified by Redis of a value change, and therefore, I will archive the same data 5 times. How can I avoid that ?

Thanks

2
  • 1
    Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
    – Community Bot
    Aug 28, 2023 at 16:03
  • "In my case, it's not necessary [...] but the goal is to design something clean" - If what you want is to sharpen your skills, I would then advise you to start with a realistically complex scenario where scaling and HA are really needed. The right way(s) to scale depend a lot on your business domain, type of load, nature of your data, etc. You will also need to read up a lot on scaling and redundancy strategies, I don't think a single stackexchange answer will give you a recipe for something that can take tech giants years of refinement to figure out in their own context. Aug 30, 2023 at 7:08

2 Answers 2

7

To meet this requirement, I need several microservices.

Not really. You don't 'need' microservices at all. There may be good reason to decompose your architecture into smaller loosely coupled services but it's important to realize that there are also costs and downsides.

The first thing I question here is: why do you need two separate databases for current and historical data? Any database that can store a full history of the data can store the current data. From a pure data design perspective, the current data is simply the history at T-0 i.e.: the last known history. Again, there may be valid reasons for splitting history from the current data, but I'm not seeing an obvious reason for doing that.

The second question I have is: why do you need to do calculations in real time? If you are storing all the history, why not do calculations on demand? You can always store the results once they have been calculated.

As for your question about duplicate event processing, the simplest solution is for the data to be keyed in a deterministic way e.g.: the timestamp of the change. Then you can either reject duplicates at the target or allow them to overwrite each other.

Again, there could be good reasons for the design you are proposing but I get the sense that it is significantly over-engineered. Experience tells me that it's generally better to with a very simple architecture and then refine it based on real problems that you encounter. Otherwise, you may end up creating problems instead of solving them.

1

(found the actual question; I agree with JimmyJames that this is way harder to do as microservices than necessary)

I have noticed a problem : If I have 5 replicas of my historization service, all 5 will be notified by Redis of a value change, and therefore, I will archive the same data 5 times. How can I avoid that ?

This is a common problem in event-driven systems. You want to achieve "exactly once" semantics, which is very difficult to get right when you consider the possible combinations of services crashing at various stages of event processing.

Possible solutions:

  • design around an existing streaming data platform (e.g. Kafka) rather than building all the pieces yourself

  • a single "event converter" which just turns the event into something that can be dispatched on a message bus or through a load balancer into the actual processor instances

  • "state" flags on the input data, so it is possible for any instance of the service to check if another is already processing the same input

  • "state" flags or unique identifiers on the output data, so if you do accidentally process something twice you can detect that and only save one copy

  • de-replication (you are not Twitter, if your data is merely once a second and your N is not in the thousands, you can probably just put it on one machine)

1
  • You could add "idempotent processing" to the list. If nothing changes when the event is processed more than once, then you can safely reprocess or double process any events. Sep 1, 2023 at 0:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.