Which language is more suitable heavy file tasks?

Question

I need to write a script (based on basic functions) to process /image/audio/video files. The process is mainly filesystem tasks and converts. The database of files has been stored by mysql. The script is simple but cause heavy tasks on the system; for example renaming/converting/copying thousands of file in a run. The script does not read the content of files into memory, it just manage the commands for sub-processes. The main weight is on the communication with filesystem. The script will be used regularly for new files. My concern is about performance. I am thinking of

1. Shell script
2. a complied language like C

Please advise which programming language is more suitable for this purpose and why?

UPDATE: An example is to scan a folder for images, convert them with ImageMagick, move files to destination folder, get file info, then update the database. As you can see, the process has no room for optimization, and most of languages have similar APIs for popular programs like ImageMagick, MySQL, etc. Thus, it can be written in any language. I just wish to reduce resource usage by speeding up the long loop.

NOTE: I know that questions about comparing languages are not favorable, but I really had problem to choose, because the problems can appear in action.

Answer 1

It sounds to me like you will simply be handing these files to another piece of software to actually read them in. If that's the case, use python or ruby or whatever easy to use high level language you have on hand because this program isn't actually IO intensive.

Now, if you are actually reading the contents of files in yourself to process on your own then I would say if the language you normally use doesn't have any interoperability for making OS level calls, don't use that language.

Ideally in this case you'd want a high level language with low level facilities.

C# for instance does this well in allowing high level handling of the simple stuff like processing the user input from command line and organizing the steps of your task or whatever else, but then allowing OS calls and direct memory management for highest possible performance (if necessary!). Java may do similarly? Not certain. Haskell is very high level as well as having facilities for direct memory manipulation, though Haskell has an extremely high barrier for learning if you don't already know it. C++ is probably the most commonly used language for this type of task for this exact purpose, it is a high level language and with C in it's roots it has complete low level facilities available.

That said, beware of premature optimization. Writing a program in a language that isn't your strongest language will likely under perform the one you could write in your native language as you wouldn't be aware of available optimizations or be using the language appropriately. Further the only way to know if the quickest most robust route of using your main language won't work is by giving it a go which would be far quicker than going all in on a language you don't know. So prototype something and see how it performs, if you don't think it does well enough then whip up a quick prototype in other languages and compare.

Prototyping as a part of your technical solution analysis is an ever important skill for all programmers, so take advantage of this task to practice it. In the scientific method we don't try to prove our hypothesis as that leads to bias, rather we try to disprove them. Your hypothesis is your normal language won't perform well enough for this task, so start by trying to disprove that.

Edit: Based on your edit, your program is absolutely not IO intensive, so write it in the language you are most comfortable with

Answer 2

If you are going to be using this script/program 100x more than the time it takes to write then I would write some simple test in both ways and run some timing test.

Way back when I used VB 6, I could do file I/O using VB, but I could also call the system routines directly in the DLL from VB 6 and sometimes get better than 100x performance. It's not really the language, but how well you can wield it effectively.

If the shell is eaiser to use and can make direct use of the low-level system functions then use that, otherwise go with the C program and use the low-level system functions. The point is the low-level system functions should out run any language trying to do the same thing by a noticeable to signifigant amout. You give up the safety of the script/language for the speed.

EDIT

Based on your update, I would focus on getting ImageMagick to use multiple threads.

See: Multi-threaded operations? which suggest

If you add --enable-openmp to your configure script command line and build ImageMagick, convolution operations run in multiple threads (e.g. blur, sharpen, unsharp mask, edge, etc).

I would do this in whatever language is easiest as it appears the best performance gain cannot be had in the language you choose but configuring the apps.

Answer 3

I'd consider python or any other high-level, dynamic, interpreted language for these reasons:

• Relatively easy to pick up
• Mangling filenames, string slicing, etc. is easy
• DB access is similarly pretty straightforward using a built-in module
• Using the threading module gives quick, simple, easy to understand concurrency, and is effective so long as the python program is I/O bound (waiting on FS copy, for example)