I am working on Java program which will listen multiple kind of events from multiple sources and store information to multiple sinks.

Each interface will have its own source to listen and own sink to store data. Event frequency can be hundreds of thousands events per second. As event frequency is of great magnitude, there should be no loss of data. Application must be scalable and must give good performance in peek hours.

Should I write different main class to run each interface separately or a multi-threaded application which will handle all interfaces.

Please suggest the right architecture for my requirement.

  • it reads like a homework question but actually I think this is a very interesting question. If you do the 'handle all types' program you will actually make better use of your resources – Ewan Sep 8 '18 at 6:39
  • Kafka Connect might help in this ? – Sikorski Jan 7 at 14:17
  • kafka Connect is too commercial for us... – Ashish Nijai Jan 7 at 14:23

Ok, so first lets get the multi-threaded aspect out of the way.

Both programs should be multithreaded and not just by having multiple single threaded message processors.

Now the question of scaleablity.

The by the book answer is to have many programs each dealing with a single message type.

You put one instance of a program per virtual machine and scale by adding virtual machines as required.

However. spinning up and down virtual machines or even containers isn't instant. amd right sizing those machines to your program is also a difficult problem.

Often its easier to have a VM with more processing power than you need and to put one of each type of message processor on it.

That way, when you spin up a new box you you are only worrying about total load, not the individual load of each message type.

Also its likely that many of your message types are very low volume. Its not economical to keep a whole box running just to handle a couple of messages a day. (really at least two boxes and load balancing infrastructure)

Now running multiple instances on a single box solves these problems, but because they all compete for the same cpu/network/disk resources its not super effecient.

However, if I have a single program that can deal with multiple message types I can have better control over how many I choose to process at the same time.

Because the program knows it has all the resources on the computer and can choose exactly how many messages to process at a time it can allocate those resources more efficently.

There is of course still the argument about keeping your business logic seperate. But in an system which has many low volume per type, low cpu cost to process messages. A multitype message processor could be the way to go.

  • Thank you for ur idea. It will be nice if you suggest high level kind of architecture or any efficient enterprise level patterns. – Ashish Nijai Sep 9 '18 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.