I've been using python for several years as a non-developer writing scripts to help business processes. I'm in a new position (still non-developer) and would like to adopt more rigorous coding standards (especially since the projects I will be working on will be bigger). Specifically, this means, in part, creating distinct environments and folder structures for each project rather than just having a bunch of modules from distinct projects all floating together in the same folder getting run manually at the terminal.

I was reading this guide about how to structure projects but I have some questions since I've never done this before.

1) An example project I am tasked with includes a. automating data import into system b. checking downstream output from system to ensure everything was processed correctly.

Data takes at least 1 month to flow through, this is not an immediate process. In other words, these are distinct tasks. But higher ups have phrased it as being one project. In the world of structuring python projects, would it make more sense to create two projects (or in other words, two folder structures with their own virtual environments) out of this? Or include them in one? I don't foresee code needing to be shared between them. I do imagine task b. would need a list of case IDs created by task a. but that would be the only functional link.

I assume two distinct projects makes sense but what thought goes into that?

2) Is a project designed to only ever explicitly run a single module (like a main.py at the top level) while the others are all referenced internally? Or can you structure a project such that you may explicitly run any number of modules depending on what task you are performing at runtime? In other words, what is the scope and intent of a project/top-level package/folder?

3) If you intend to run a main.py from the top level package at the command line, does that top level directory need to be a package if it is not being imported into another program? Can it just be a regular folder as long as you never reference it in other code?

4) To what degree are you "supposed" to combine classes/functions with the nitty-gritty usage of those functions (is there a term for this?) within the same module? In other words, are you supposed to leave classes/functions in their own modules and leave the importing/usage/running of them to another separate module? Or can you combine them into one? And when would you do which and why?

With my messy background, I have only ever had singular modules which contain all the functions and execution in a single file. I don't know what the proper protocol is for delimiting scope.

These may be basic questions that I've taken too long to come around to but thank you.

  • 1
    "But higher ups have phrased it as being one project." Without more context, I would presume the word 'project' is being used in the context of 'project management' meaning it's a single body of work for management and accounting purposes. Is there some reason to think it has anything to do with how you structure your code? – JimmyJames Jan 29 at 13:48
  • Yes that is a correct definition. I don't know, I'm just trying to figure out what is generally done. – Josh Flori Jan 29 at 14:44

It's possible to divide this up into multiple projects and it might make sense. The main reason for keeping it together would be if you have to share modules between the two running programs. There are other ways to deal with this but in the case that they are tightly bound, it can greatly simplify things.

There's one big exception that comes to mind, however. Let's say that you have a shared module that is used to write records to a file/wire format in the first part and parse that format in the second. Then at some point, you need to make a change to that format. You might run into a situation where the data being parsed is from a month ago and is in the old format. If you've deployed this change already, and your code expects the new format, you could have a little trouble. Just something to consider.

I'm not sure you would need to have packages here. You can execute any of your module at the command line. When you do, the top level statements in the module will execute. All the imports, definitions, and code. The same thing happens when you import that module. When you want to have a module do something special when it's executed from the command line you can do this:

if __name__ == "__main__":
  # things that should happen when run at the CL.

The other option here to keep it all in the same project is that you can create special modules that do nothing other than import things from other modules and execute them. This might be preferable if you have extensive code in your top level context as well as a significant amount of declarations in your modules.

I wouldn't get to hung up on this. I would probably try to keep it as simple as possible and revisit the decision later if there are issues.

| improve this answer | |

With my messy background, I have only ever had singular modules which contain all the functions and execution in a single file. I don't know what the proper protocol is for delimiting scope.

This is perfectly fine, if this solves your use cases. In that case there is no real reason to structure your directories in any ways other than what helps you best to find the right script when you need it.

It is possible that you at some point find that you are doing a lot of duplicate work or copy & paste code from one script to the other and run into trouble when you have to make changes to code in multiple different places, then you might consider of having module with reusable code. When going for that I would have one folder structure for commonly used modules and another one for scripts.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.