0

I'm writing a program in Python that has two different entry points. There's a CLI that processes some data and needs to be installed to run hourly on a server, this populates a database. There's also a web app that has a front end for viewing the data in that database.

I'm using sqlalchemy as an ORM for the database that I'd like this to share between the CLI and the web app. My question is, should I write a single package with the code for all three elements (web app, database and CLI) all in one. Or separate out the database into its own package and use this as an API for the web app and CLI in their own packages?

My current thoughts (based on helpful comments below from @Hans-MartinMosner) are to have a project structure like this:

my_project
   database_orm/
      db_model1.py
      db_model2.py
      db_helper.py
   my_webapp/
      __init__.py
      views.py
      templates/
         index.html
         page2.html
      static/
         site.js
         site.css
      setup.py
   my_cli/
      __init__.py
      __main__.py
      magic.py
   tests/
      unit_test.py
   setup.py

Any opinionated advice on a different structure and ways that I can simplify deployment (I'm planning on hosting the web app on Azure App Services) will be greatly received.

I should probably point out that I'm using Python 3.7 and I don't need to worry about earlier Python versions.

  • 2
    Ask yourself which parts can reasonably be installed separately. If it may make sense to have database, data processing and front end on separate machines they should be separate packages. If they should always go together, it doesn't make sense to package them separately, as you will not gain anything and have a higher risk of incomplete or broken installations. – Hans-Martin Mosner May 7 at 8:40
  • @Hans-MartinMosner The database is hosted on a remote server, the CLI will run on multiple independent servers and the web app will likely be hosted on Azure App services. Based on this I'd assume your recommendation would be to have separate packages with a common database API in a separate package? – Nick Martin May 7 at 9:18
  • 1
    Yes, as the database API is a dependency that is shared by the CLI and the web app, it makes sense to have it in a separate package. Deployment of monolithic apps may be simpler, but in this case separation looks cleaner and is likely well manageable. – Hans-Martin Mosner May 7 at 9:30
  • @Hans-MartinMosner Please could you elaborate your answer into an example project structure - feel free to be opinionated - so I can accept? – Nick Martin May 7 at 10:03
  • I can do later - using mobile at the moment. – Hans-Martin Mosner May 7 at 10:12
0

Caveat: I've got no experience with python packaging, so I'm extrapolating from experience with other software environments, and I'm not sure whether this isn't already stuff you know better than me. YMMV.

If I understand correctly (looking at https://www.learnpython.org/en/Modules_and_Packages), python packages are not primarily a mechanism for independent deployment (as are rpm or deb packages) but groupings of modules that make it easy to (partially) import complex functionality. Since your CLI and web app parts are top-level (not imported into other modules) it is probably not necessary to put them into a python package structure (directory with an __init__.py file) but it is reasonable to keep them in separate directories anyway since they can be deployed separately. If you use a package structure with a main.py entry point you have the advantage that you can run the app by its directory name (python my_cli or python my_webapp) which looks better than naming the entry point explicitly.

The ORM part can be put into a python package, but a simple module (single database_orm.py file) might be enough, depending on the complexity. If you have a number of classes that are not only simple value objects with persistency but which have extensive behavior of their own, they should go into separate modules within a package.

Your CLI and web apps would import the ORM module or package, so for deployment the structure could look like this:

some_path_on_cli_server/
   my_cli/
      database_orm/
         db_model1.py
         db_model2.py
         db_helper.py
      __init__.py
      __main__.py
      magic.py

some_path_on_web_server/
   my_webapp/
      database_orm/
         db_model1.py
         db_model2.py
         db_helper.py
      __init__.py
      __main__.py
      views.py
      templates/
         index.html
         page2.html
      static/
         site.js
         site.css
      setup.py

In your source code management, you would of course not have database_orm as a duplicate subdirectory within the app directories, but side by side with them.

For deployment, since you don't need to support many anonymous users of your software who need a hassle-free installation, it might be enough to have simple makefile rules that create tar or zip archives of the apps with included ORM, copy these archives to the server machines and unpack there. Alternatively, put them into git repositories and do a git pull on the server to upgrade. The solution does not have to be perfect, it just needs to be reasonably easy to use and very easy to set up.

  • I'm not sure I fully understand. database_orm is copied twice in your project structure, isn't this code duplication? – Nick Martin May 8 at 10:09
  • This is intended to show the deployment structure; if the same package/module is needed on both servers you just need to copy it somehow, how else will you use it on both? Of course you can install the whole project tree on both servers and just run the code needed on each, but then you have unnecessary code on each. If that isn't a problem, then I might have misunderstood the requirements, not entirely improbable... – Hans-Martin Mosner May 8 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.