Assuming the following scenario:

A python application that receives file and process that file trying to understand what the file format( any type of data/compressions/archives/packages/mounts/etc ), then it open(cuts) the file and sees which files are inside that file and tries to execute the same logic as before. All relevant information that the script can get from files must be stored in the database in order to present it in the UI( in the form of a tree ).

  • Parent file should be available to share info with his child files
  • After all files are extracted and data collected there are some post process scripts( example: to connect js function names in html file with js file )

how it woks now: one worker come up with x threads that listening to python queue, and when first file_path come into queue it start work and fill that queue with child files. All related information are saved into python list and then it send batches into db(arangodb).

what problems with that solution:

  • Hard to debug with multithreading, specially if thread is stuck. renewing the thread
  • No easy way to catch errors, only in logs


  • do a task queue/job queue like Celery/rq/dramatiq will fit here? If yes, which one on your opinion?
  • publisher-subscriber pattern will fit here or it's overcomplicating?

Maybe you know some articles/companies/products that seems familiar?

  • 3
    Why are you concerned with multithreading and concurrency for this problem? The problem seems quite simple to me, but it does seem that you are severely overcomplicating it by all the technologies you want to use. – Vincent Savard Jan 28 at 13:46
  • You're not actually speeding up any processing by going for multithreading, you're just making your code run concurrently, not in parallel. If you have a connection w/ the frontend, perhaps updating the percentage of files processed in soft-realtime or if you have multiple clients and you care about responsiveness, then you might want concurrency. No reason for bringing in pub/sub whatsoever here. – Milan Velebit Jan 28 at 14:24
  • @VincentSavard for example you received a file.zip that contains 10 other zip files, it will speed up a process if 10 threads will read those files. Work with files is I/O bound and threads are good at such work, tested :) sure if you can advise some other options to speed up process, I'm glad to hear – Edvard Krol Jan 28 at 14:25
  • Just to clarify, at what stage of the development are you? Do you have a complete, working solution and you measured that the bottleneck is caused by I/O? Is the bottleneck actually an issue in practice? My understanding of the question (which may be entirely wrong, I concede) is that you're currently not at the point where you have a working solution, and are already trying to optimize it. Beware of premature optimization! – Vincent Savard Jan 28 at 14:35
  • 2
    @EdvardKrol: That makes no difference. If your random disk read speed is 500MB/s with one thread, with ten threads reading ten files it'll still be 500MB/s just divided ten ways. Unless your disk read speed well exceeds your decompression speed, using multiple threads won't make things any faster in this case. Unless you used the highest compression levels with compression algorithms like lzma or bz2 which has slow decompression speed and you are running on really speedy SSD, usually I/O would be slower than decompression. – Lie Ryan Jan 28 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.