Assuming the following scenario:
A python application that receives file and process that file trying to understand what the file format( any type of data/compressions/archives/packages/mounts/etc ), then it open(cuts) the file and sees which files are inside that file and tries to execute the same logic as before. All relevant information that the script can get from files must be stored in the database in order to present it in the UI( in the form of a tree ).
- Parent file should be available to share info with his child files
- After all files are extracted and data collected there are some post process scripts( example: to connect js function names in html file with js file )
how it woks now: one worker come up with x threads that listening to python queue, and when first file_path come into queue it start work and fill that queue with child files. All related information are saved into python list and then it send batches into db(arangodb).
what problems with that solution:
- Hard to debug with multithreading, specially if thread is stuck. renewing the thread
- No easy way to catch errors, only in logs
- do a task queue/job queue like Celery/rq/dramatiq will fit here? If yes, which one on your opinion?
- publisher-subscriber pattern will fit here or it's overcomplicating?
Maybe you know some articles/companies/products that seems familiar?