Should I containerize Python 'job' scripts?

Question

We have a python scheduling solution which, at the moment dynamically loads python modules to run each job in a separate process.

The jobs are pretty heavy containing some Tensorflow models which process data retrieved from shared drives or databases and inserting the results back into databases.

I have the task of maintaining the scheduling solution and was considering whether containerizing each 'job' script, using Docker, is a good idea. The scheduling solution would then run the containers using Docker's Python interface.

I like Docker and I feel it segregates each job nicely and gives a cleaner production environment than dumping a load of scripts into a directory. It also seems like it would make testing and deploying the jobs easier - push source and Dockerfile to server, build and voila sorted. I am worried by how maintainable and performant it is, in my experience the containers are quite large for just the source, especially if using nvidia-docker...

TLDR: Is containerizing Python 'job' scripts good practice or overkill?

Answer 1

From the information you present us here, it is hard to judge whether it makes sense to use Docker at all.

From what you write:

We have a python scheduling solution which, at the moment dynamically loads python modules to run each job in a separate process.

What would a dockerized scenario bring to the table?

If you design for docker, you would achieve the goal of one container doing one job (no dynamically loading needed). Docker's strength is just that: virtualizing at the process level. In order to achieve that goal via docker you have to design images for each "process".

The question is, whether it is cheaper to maintain the docker images or to separate the dynamically loading part in simple python scripts which are otherwise started.

The scheduling solution would then run the containers using Docker's Python interface.

This doesn't make sense for me. If you want to decouple things with docker, your scheduler should be oblivious to the fact, that Docker (or whatever) is involved at all.