-2

I am building a Spring boot REST API app that is part of a microservice architecture project. What I am planning:

  • My app listens for events from two other services and after some business logic, logging, calculating, and persistence I have to publish my results to another rabbit MQ exchange.
  • The first two services listen to pings every 10 seconds from 15,000 devices each and publish their data to queues that I listen to.

My current plan:

  • I need some sort of cache to keep track of events sent from both services because I need data from both at a time interval to do my logic. I am using an in-memory hashtable and I am sorting the values in this hashtable everytime I update it.

  • My hashmap is a Map<String, Map<String, Instant>> where the first key is a regionID and the second map is keyed based on deviceID. This sorting is done on the internal map and I suspect might become the bottleneck.

Is a single shared key value datastructure storage a good way to go with this or is there something else I should try?

5
  • 2
    Whenever performance is involved, you need to implement, measure, and work on your approach if results are not good enough. Nobody here knows what your app will actually do and what the real requirements regarding event correlation would be. Commented Jul 9 at 10:40
  • I understand that I should implement and then see if that works, I just wanted advise on whether a single shared storage where millions of reads and writes are happening daily is something that is done in the industry.
    – dk tammy
    Commented Jul 9 at 13:52
  • You'd be surprised you many different things (including very stupid ones) are done daily in the industry... Commented Jul 10 at 6:14
  • 1
    @dktammy That's sounding dangerously close to "cargo cult programming" - i.e. trying to choose solutions based on whether it happens to work for other people. Keep in mind that other people will be working in a completely different context to yours, and with different requirements. You have nothing to lose by throwing a quick prototype together and trying it out; either it works or it doesn't; random strangers can't know any better than you whether it'll work or not for your particular situation. Commented Jul 10 at 18:53

2 Answers 2

0

An important thing to consider is what kind of hardware you can use, and whether additional hardware or additional development effort is a cheaper way to make it faster (or fast enough).

The reason why you implement and measure first is that you have little idea what affects performance how much. You might make something ten times faster and figure out that the amount of time went from 0.1 to 0.01 out of ten seconds. While reducing 5 seconds by 20% to 4 seconds is much more effective.

0

The most important factor I suppose based on your question will be how many real records are relevant.

In a day you get around: 129.600.000 pings. (15.000*6*60*24)

But are the relevant pings really your number or is the 15.000 your number? For example if the pings are from servers to measure uptime, you would have a list of 15.000 servers.

Region ID - Device ID - Last seen dateTime

Every ping you check update the lastSeen for example. Is that kind of the process you need?

Managing an active queue of 129M is a totally different game compared to 15K.

If you need to manage the 129M a big data solution might help you as shaping timed records into data series is a quite well known problem. The question is where you need to have custom solutions and where you can use proven ones.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.