I have been exploring the microservice architecture for the batch-based system.
Here is our current setup:
Code: We have 5 systems that are internally connected and they pass data from one system to another. Currently entire logic is sitting in Oracle as PL/SQL, Hadoop(Hive, Impala, Spark, etc) and Shell scripts.
Communications: These systems share the data either through cross DB Table grants or they export the data in files and send it to each other.
Triggers: These systems send trigger through custom workflow engines or processes look for some files in a repetitive mode.
Now coming to the main question: Is it a good idea to convert these processes into microservices(Code) and use Kafka( Communication and Trigger) so that they can share data and we can have a more distributed well-choreographed process flow. Just to give an example, When one system finish process it can send data in Kafka is available(this act as trigger and producer) and all consumer system can start using that data in parallel instead of sending data in files or hitting databases individually.
Edit based on comments: Looking for some insight on microservice based architecture for the Batch based system, Irrespective of the current setup or think we are building a brand new system.
Any suggestion through link/Blog, tools, and technologies would be greatly appreciated.