I am considering the options for designing a system in java which accepts data files from different systems applies some transformations to the data, concats data from multiple files to produce some output data files for the downstream systems. The data transformations are not per record transformation, e.g. I need to read a record , then search the file for another record matching the pattern of a given field from the first record

I was thinking of spring batch,but have been asked to consider other options also. Kindly help with any pointers, things to consider while designing and evaluating options.


Well, depending on the complexity vs. load of your application, Spring Batch may or may not be a good solution. This is strictly a personal opinion, but I've found that, while Spring Batch let's you quickly prototype some batching mechanism (you only have to implement the business logic, all the batching mechanism comes out of the box) it can prove slow if you have to deal with very large quantities of data (very complex treatments) and performance is critical.

On the other hand, I've noticed from your question that you say "accepts data files from different systems applies some transformations to the data". If the files come via different channels (such as ftp, web service, etc) then Mule ESB might be a good framework for you. It comes out of the box with connectors for all the various ways of having inter-machines communications (various types of web services, ftp, even email) and gives a simple way to create discrete components for various treatments of those files, which you can chain together any way you like. Besides, the thing is open source so you can easily download it and try it real quick to see if it's a good solution for you.

Other than that, if the overall demand of the project is not very complex in terms of non-business logic needs (like: the sending/receiving of files is only done in one way, and/or the treatment of the files is simple and stable enough to not require any fancy modular system for transform modules, etc), you can go a bit more low level, and just add in a library that handles your communication (like maybe Jersey for web service communications or Apache commons-net for FTP, etc.) and then code everything else by hand. The JDK already comes with quite a few libraries that could be useful, like XML parsers (StAX, JaxB), easy way of handling file I/O (nio2), etc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.