Combining web app and CLI to python package

Question

I'm writing a program in Python that has two different entry points. There's a CLI that processes some data and needs to be installed to run hourly on a server, this populates a database. There's also a web app that has a front end for viewing the data in that database.

I'm using sqlalchemy as an ORM for the database that I'd like this to share between the CLI and the web app. My question is, should I write a single package with the code for all three elements (web app, database and CLI) all in one. Or separate out the database into its own package and use this as an API for the web app and CLI in their own packages?

My current thoughts (based on helpful comments below from @Hans-MartinMosner) are to have a project structure like this:

my_project
   database_orm/
      db_model1.py
      db_model2.py
      db_helper.py
   my_webapp/
      __init__.py
      views.py
      templates/
         index.html
         page2.html
      static/
         site.js
         site.css
      setup.py
   my_cli/
      __init__.py
      __main__.py
      magic.py
   tests/
      unit_test.py
   setup.py

Any opinionated advice on a different structure and ways that I can simplify deployment (I'm planning on hosting the web app on Azure App Services) will be greatly received.

I should probably point out that I'm using Python 3.7 and I don't need to worry about earlier Python versions.

Answer 1

Caveat: I've got no experience with python packaging, so I'm extrapolating from experience with other software environments, and I'm not sure whether this isn't already stuff you know better than me. YMMV.

If I understand correctly (looking at https://www.learnpython.org/en/Modules_and_Packages), python packages are not primarily a mechanism for independent deployment (as are rpm or deb packages) but groupings of modules that make it easy to (partially) import complex functionality. Since your CLI and web app parts are top-level (not imported into other modules) it is probably not necessary to put them into a python package structure (directory with an __init__.py file) but it is reasonable to keep them in separate directories anyway since they can be deployed separately. If you use a package structure with a main.py entry point you have the advantage that you can run the app by its directory name (python my_cli or python my_webapp) which looks better than naming the entry point explicitly.

The ORM part can be put into a python package, but a simple module (single database_orm.py file) might be enough, depending on the complexity. If you have a number of classes that are not only simple value objects with persistency but which have extensive behavior of their own, they should go into separate modules within a package.

Your CLI and web apps would import the ORM module or package, so for deployment the structure could look like this:

some_path_on_cli_server/
   my_cli/
      database_orm/
         db_model1.py
         db_model2.py
         db_helper.py
      __init__.py
      __main__.py
      magic.py

some_path_on_web_server/
   my_webapp/
      database_orm/
         db_model1.py
         db_model2.py
         db_helper.py
      __init__.py
      __main__.py
      views.py
      templates/
         index.html
         page2.html
      static/
         site.js
         site.css
      setup.py

In your source code management, you would of course not have database_orm as a duplicate subdirectory within the app directories, but side by side with them.

For deployment, since you don't need to support many anonymous users of your software who need a hassle-free installation, it might be enough to have simple makefile rules that create tar or zip archives of the apps with included ORM, copy these archives to the server machines and unpack there. Alternatively, put them into git repositories and do a git pull on the server to upgrade. The solution does not have to be perfect, it just needs to be reasonably easy to use and very easy to set up.