0

TLDR with bold

I want to create a library (I think this is the right term) for my own reinforcement learning environments (envs for short). Most of the envs would be based on self-implemented games created either in pure python or in c++ with python bindings. What would be the best way to structure this project in a manner that is easy to expand, maintain and that would make most sense? I want to be able to reuse code, such as using a general board class for all my board game implementations (eg. chess, go, gomoku). I plan on making it cross platform with the help of CMake and might even venture to package it as a Conda package.

Through my initial search I found that this layout (and its variants) is popular and decided to rely on that.

My initial plan was to structure my library into projects, create a repo for each project, and include a project into another one as a git submodule. For creating the environment for the game 2048 the structure would been the following (CMakeLists omitted):

(the env-vector relies on the env-2048, which relies on the game-2048 which uses the general-board class)

general-board
├── external
│   └── Catch2/
├── include
│   └── general-board
│       └── file.h
├── src
│   └── file.cpp
└── tests
    └── tests.cpp

game-2048
├── app
│   └── manual_game.cpp
├── external
│   ├── Catch2
│   └── general-board
├── include
│   └── game-2048
│       └── file.h
├── src
│   └── file.cpp
└── tests
    └── tests.cpp

env-2048
├── external
│   ├── Catch2/
│   └── game-2048/
├── include
│   └── env-2048
│       └── file.h
├── src
│   └── file.cpp
└── tests
    └── tests.cpp 

env-vector <---- this would be on the top, bundling the envs together
├── external
│   ├── Catch2/
│   ├── env-2048/ 
│   ├── env-chess/ <---- another board game
│   └── env-go/ <---- another board game
├── include
│   └── env-vector
│       └── file.h
├── python
│   └── pybind11_magic_here
├── src
│   └── file.cpp
└── tests
    └── tests.cpp

After some implementation I got concerned whether the number of submodules and the redundancy got too much. With this structure the project on the top would contain the general-board project n times (where n is the number of games relying on a board), and the catch2 would be included even more. Seems fishy and error prone.

My second idea was to create one big project and to include everything into it in a 'flat' way and not in a 'nested' way as before. It would look like this:

(line ending with '/' depicts a folder)
environments_all_in_one
│
├── external
│   └── Catch2/
├── include
│   └── environments_all_in_one
│       └── **not_even_sure_what_to_put_here**      
├── python
│   └── pybind11_magic_here
├── src
│   ├── env_vector
│   ├── envs
│   │   ├── env-2048/
│   │   ├── env-chess/
│   │   └── env-go/
│   ├── games
│   │   ├── game-2048/
│   │   ├── game-chess/
│   │   └── game-go/
│   └── general-board
│       ├── board_abc/
│       ├── board_array/
│       └── board_vector/
└── tests
    └── tests.cpp

This way code would not be present multiple times and it definitely helps transparency. However, as I am not experienced I must ask:

Is there a better way to do it?

  • 1
    Unrelated: "TLDR with bald" you probably meant to write bold there. – πάντα ῥεῖ Apr 5 at 8:15
1

As with structuring any project...

  • How many people will be working across this code base?
  • How good are the merging abilities of the repositories and the integrated build and verification pipeline?
  • Are you sharing code by using checking it it, by lumping everything together in a single repository, by using git-submodules, or a package manager?
  • How are you intending on deploying/distributing the results of these projects?

These will all have an effect on the pro's and con's of structuring the code base.

Generally speaking the more people working across a code base, the more and better defined walls you will need between pieces of code in order to mitigate cascades caused by errors/flaws/changes. That is a preference for multiple repositories, and package managers, and committing external libraries into the source.

We generally want smaller, well defined modules with great solid APIs. This is limited by the difficulty to separate modules and still ensure integration succeeds. When these tools are poor a single repository is preferable, because it is obvious what is meant to integrate with what, and developers can prove integration before checkin. When the tooling supports merging and automatic verification, having a module per repository and supporting forks (by branching or repository duplication) is possible and even preferable as it allows modules to be easily split when too large, and for many developers to contribute even to very hot code paths.

We always want to ensure code is both up to date, and not breaking the current module. These goals are contradictory in that one prefers change, and the other actively attempts to prevent it. Again tooling is key here.

  • If the tooling can automatically verify code, then its reasonable to leverage package managers to auto-update dependencies.
  • You can use things like git-submodules, (though why you want that pain is personally beyond me). The problem though is that this confuses the separation between repositories, which runs counter to the idea of a strong boundary between modules.
  • A Single repository can be effective, particularly for smaller man teams, or teams (apparently like google) where the cost of managing many smaller repos outways the downsides of having a single repository. It is also a better starting point for projects in tooling anemic environments (lack of CI/CD, lack of Binary Repository, lack of good merging tools, etc...)
  • Checking in third-party source. Personally for me this is a no brainer. If the source is an essential component to even run the compiled source code. It is part of your source code and should be versionned as you would write your own code. If the third-party component is optional in the sense that dropping that component in after everything is built is enough, then it not a dependency and is up to preference.

Many projects don't answer the question of distribution first, even though its the whole point of the project presumably. There is someone/something out their requiring the result of building this project. How does it get it? Are you making a single installer, are you expecting them to copy out from several zips? How you choose to distribute will make some project layouts easier, and others more difficult.


What does this mean?

And here we get to opinion...

While monoliths are much maligned they do have their good points:

  • Small teams (the magic number is somewhere around <= 10 devs depending on code base size) can work with them easily.
  • Integration is self-evident. Everything in the one commit.
  • A Single external libraries directory can support the whole code base.
  • Distribution is a snap, everything is distributed in a single installer. Supporting multiply installers for sub-modules is not simple though without conceeding that everything gets a version bump, even if nothing has changed for that module.

If this fits, I would choose a monolithic structure.

Otherwise I would prefer to split it up into separate repositories. Each repository responsible for a single deliverable end product or component.

  • Good for medium through to very large teams. Because of the number and seperation between repositories.
  • Integration becomes a pain. A good CI system, Binary Repository, and Package Manager become essential.
  • Third-party libraries might be better serviced by a package manager. And internal dependencies are definitely better serviced by a package manger, as each internal component cannot strictly control the end versions actually deployed. Dependencies that aren't passed on (like unit-testing) or for an end product I would still argue should be checked in as source. (As the component/product cannot be even built with them).
  • Distribution is more flexible as many end products can be released. However the dependency management is painful, and end-products may ship with different versions of internal and third-party components. This complicates both the deployment life (if those end products interact) and the bug management as bugs will need to be tracked against multiple versions and variants of the software configuration.

At the end of the day, no one project structure is best just better given X.

| improve this answer | |
  • After reading your reply multiple times, I think the monolith structure is the most suitable for me as of now. I will be most likely the only one to work and to use this library of mine. Furthermore, I do not feel up to the challenge to split up the code and deal with cross-platform package management (my presumtion is that it is not simple) and the other stuff that you mentioned on the last segment of your answer. – Dudly01 Apr 5 at 11:19
  • Also, I want to wait for other possible replies before marking yours as a solution. Yours is good as it is general, but I am at the skill level where specific layout answers would be apriciated as well. (But for those, I might have not given enough detail.) – Dudly01 Apr 5 at 11:23
  • Yeah, Better is not a fixed mark, it moves around a lot, which makes what you are asking for an opinion. Though my opinion with the extra information in your answer is that i'd be leaning toward a monolithic approach (with good sub modularisation). It doesn't mean it is definitively better. I can however give an answer that is a lot less opinionated and much more factual answer by describing the actual issues you need to consider and how they play into decision making. – Kain0_0 Apr 5 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.