We have a python scheduling solution which, at the moment dynamically loads python modules to run each job in a separate process.
The jobs are pretty heavy containing some Tensorflow models which process data retrieved from shared drives or databases and inserting the results back into databases.
I have the task of maintaining the scheduling solution and was considering whether containerizing each 'job' script, using Docker, is a good idea. The scheduling solution would then run the containers using Docker's Python interface.
I like Docker and I feel it segregates each job nicely and gives a cleaner production environment than dumping a load of scripts into a directory. It also seems like it would make testing and deploying the jobs easier - push source and Dockerfile to server, build and voila sorted. I am worried by how maintainable and performant it is, in my experience the containers are quite large for just the source, especially if using nvidia-docker...
TLDR: Is containerizing Python 'job' scripts good practice or overkill?