3

I have been exploring the microservice architecture for the batch-based system.

Here is our current setup:

Code: We have 5 systems that are internally connected and they pass data from one system to another. Currently entire logic is sitting in Oracle as PL/SQL, Hadoop(Hive, Impala, Spark, etc) and Shell scripts.

Communications: These systems share the data either through cross DB Table grants or they export the data in files and send it to each other.

Triggers: These systems send trigger through custom workflow engines or processes look for some files in a repetitive mode.

Now coming to the main question: Is it a good idea to convert these processes into microservices(Code) and use Kafka( Communication and Trigger) so that they can share data and we can have a more distributed well-choreographed process flow. Just to give an example, When one system finish process it can send data in Kafka is available(this act as trigger and producer) and all consumer system can start using that data in parallel instead of sending data in files or hitting databases individually.

Edit based on comments: Looking for some insight on microservice based architecture for the Batch based system, Irrespective of the current setup or think we are building a brand new system.

Any suggestion through link/Blog, tools, and technologies would be greatly appreciated.

Improve this question
7
  • 3
    What makes this question hard to answer is that there is no problem defined. If the current approach is meeting requirements and reasonably maintainable, why 'fix' it? If you say what you want to achieve by changing these systems, or what problems you have then it would be easier to give answers. Software isn't really ever any better or worse in the abstract, unless you see it as an art form. It's better or worse in terms of its meeting of requirements, its cost, reliability, etc.
    – joshp
    Commented Feb 12, 2020 at 6:13
  • @joshp : I hear you.To name few problem : 1. Current code is tightly coupled for batch process and those are not reusable. 2. Of course operation and maintenance costs are high as it’s running on its own physical machines. 3. End users are unable to pull large dataset for analysis purpose ( R/python/spark based model) 4. Environment is used for day end reporting only even though data is available. Etc... In nutshell, Current system is few year old and we are planning to rewrite/rearchitect, how the batch based system would look like if we design in today’s world.
    – SMaZ
    Commented Feb 12, 2020 at 14:43
  • 1
    Hi, since you would like to have architecture diagrams, could you create one for the current system? This way, I hope, the question will be clearer and you get more answers.
    – User
    Commented Feb 12, 2020 at 21:16
  • @User: Thanks for the response. I modified the question. I can surely give the current architecture but it will lead ideas/answers in refactoring. I am wondering if anyone comes across a situation or implemented batch-based system using Microservice.
    – SMaZ
    Commented Feb 13, 2020 at 3:01
  • 1
    I believe you Communications part would benefit from a refactoring, if you switched to use something like Kafka, RabbitMQ or similar. About the microservices part, you need to check the pros and cons of it; while it can promote more decoupling, it has deployment complexity drawbacks; and at the same time you can achieve a better decoupling between your systems without this kind of architecture too (by "better" I mean a better decoupling compared to what you have today on your 5 systems). Finally, a question: is this on-prem or cloud-hosted?
    – Emerson Cardoso
    Commented Feb 14, 2020 at 10:33

2 Answers 2

Reset to default
2

It is a decent idea. But what are your concerns(pain points) ? A micro service will mainly be:

a) more complex (it is a distributed system after all)

b) more maintainable

c) more flexible (evolution perspective, as I saw you are concerned with redesigning parts of the system in the future)

I have used the same design as you are asking for batch based system. It is not bad. And what I really like is how you can have multiple components listen to a single event(I suppose that is where the magic happens). Have a look at this book : Designing event driven systems Concepts and Patterns for Streaming Services with Apache Kafka . It gets a bit tiring in some chapters but most of it is really good. I think it misses some aspects too but it will give you plenty of insight and is rather simple.

Improve this answer
0

There are a lot of design choices available and a lot depends on the requirements (size, speed, capacity of the system, it is realtime or near-time or 24h behind, extensibility, reliability etc.)

I would definitely read Enterprise Integration Patterns and maybe Event Streams in Action to have the right tools in your quiver to make good decisions

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.