0

I'm developing a Java application that will handle huge XML files (up to 300mb each and hundreds of thousands of lines). They are grouped usually with 20 to 40 similar XML files that also need processing. Work will be mainly CPU-bound, with some IO-bound related to opening the files and saving back the results.

I was wondering what would be better, as talking for performance-wise. Would it be better to create an application that I could invoke through a script like this:

java -jar myApp.jar huge_1.xml
java -jar myApp.jar huge_2.xml
java -jar myApp.jar huge_3.xml

or maybe handle it all in one application that will create up to N threads that could be invoked like this?

java -jar myApp.jar huge_1.xml huge_2.xml huge_3.xml

The xml files can be worked on independently, the final result will be grouped up later for analysis.

In a long-term vision, which version (if any at all) would be of better use?

  • 1
    Note that there's nothing stopping you from running multiple instances of a multi-threaded program. – TMN Jun 23 '16 at 15:57
2

I'd suggest Threads.

Threads are "lighter" than full-blown Processes and Java has far better tools for managing the interactions with/ between Threads than it has for Processes.

BTW, if you were to use the script as you described it, your processes would run sequentially, one after the other. You'd need some way of telling your operating system to start the program but not wait for it to finish, repeat this for each of the processes that you need and then figure out how to marshal all of those processes back together, all in your scripting language of choice.

1

I believe a multithreaded approach would be the best one.

Also have you thought of using a mix of the two? Launch an application (manager) responsible for managing which files to process and their progress, to process each file you'd then launch the generic processing application in a separate process, a little bit like how browsers work today. This way each processing is individual and even if some error occurs that crashes that process all the others are safe.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.