5

This question is inspired by comments on this recent question. In that question I asked about a version tagging workflow. A few commenters suggested it was a little strange that I'm tagging and releasing every new feature that is merged into main. It sounds like it is more typically to collect up a few new features and then make a version bump/release. I'm trying to understand how this workflow would look and how I would modify my current workflow.

Note the context here is a small open source python package. Here's my current workflow:

  1. Branch a feature branch off of main
  2. Develop code on feature
  3. Start a PR on github feature -> main
  4. Version bump in source code. For me this means updating the version input in pyproject.toml. I'm working on configuring my source so that this is the only place in source code I need to update the version in a hard-coded way.
  5. Update the changelog to reflect the new version and the changes (just one feature/fix usually..)
  6. Once the PR is good (feature looks good and is tested) I merge the PR feature -> main
  7. Then in my local repo I add a tag to the new merge commit with the version number
  8. I build the code into sdist/wheel
  9. I publish the built artifacts to pypi

The last 3 or 4 steps are what I consider my "release" process. I'm starting to learn about github's "release" feature which I guess would show up as step 10 and be something like: create a github release based on the most recent tag. Copy the changes from CHANGELOG.rst into the github release release notes.

You can see in this workflow that every feature that gets merged into main gets a new tag and is released. This results in e.g. one line change logs per version.

To modify this to allow me to collect multiple feature branches I would do the following:

  • I would skip step 4, that is, I wouldn't version bump in source code on the feature branch.
  • I would keep step 5 (manually writing the changelog for each feature in the feature branch) but instead of putting a version number at the top of the new change I would put a version placeholder like ?.?.? since I don't know what all features are going to be collected.
  • Then after enough changes are collected in main I would make a release branch off main where I would do the version bump in source code (step 4 above) and set the version in the change log. Then when this release branch is merged back into main I would tag the merge commit with the new version and build and publish the new version code.

So the idea is that if main started on 1.1.1 it would stay on that version as it collects a few features (and the changelog would read ?.?.?). Then when the release branch happens it would bump to e.g. 1.2.0 and the changelog would read 1.2.0.

Note that I'm baking in a few decisions to this workflow. First, for now, I want a human to manually choose the new version and write the changelog entries. I also want to maintain a practice of never committing directly to main since that seems like sound advice.

The biggest problem with this workflow is that if multiple features are being developed in parallel then if feature\A is merged into main while feature\B is being worked on, feature\B may end up with a merge conflict in the CHANGELOG. One solution would be to write the CHANGELOG in the release branch after all features have been collected. But I prefer the CHANGELOG to get updated within the feature branch because whoever is writing the feature at that time knows best how to describe the details about that change. So otherwise, with the constraints that I want humans writing the changelog and that I want the changes to happen in the feature branches, I don't see a better strategy.

I know that something like conventional commits could help with this problem, but I'm not totally comfortable using that tool yet for a few reasons*, but I am looking into it.

Questions

I know that the above strategy could work in practice. But I'm trying to understand what best practices are. Or, if not best practices, at least get a sampling of what other projects/teams do for their feature/version bump/release workflows (because I haven't really seen that many examples, and can't find many thorough explanations online).

  • Would the above proposed workflow be considered a standard workflow?
  • If the above workflow is non-standard, how could it be modified/improved so that it is more standard.
  • What are examples of other workflows that address (or also don't address) the issues raised above?
  • Would you consider the changelog issues described above to be a major problem that needs to be addressed? If so how can it be addressed? Am I shooting myself in the foot by wanting humans to write the changelog in the feature branches?

*I'm not comfortable using conventional commits because it's one more thing devs need to learn and adhere to. Alternatively one person could use squash merges or rebases to write a conventional commit when it's time for the changelog to be written. By why complicate the git history when you could just manually write the changelog... This is not a question about conventional commits so I don't want to get too deep into this.

edit: Above I propose collecting change in a changelog section titled ?.?.?. I think a much more standard approach would be to have an unreleased section at the top of the changelog where these changes are collected.

3
  • 3
    A merge conflict on the CHANGELOG is about one of the easiest to resolve. The resolution is nearly always "apply both changes, with the feature branch coming first in reading order". Aug 24, 2023 at 9:28
  • 1
    Why is a merge conflict in the changelog such a big problem? Usually it's just a plain text file that is only appended, which should be rather easy to solve any conflicts in. Or is it in a more complicated file format?
    – kapex
    Aug 24, 2023 at 10:16
  • Yes I agree. I’m not personally worried about a merge conflict in the changelog. Just mentioned it because other people commented about it on the other post.
    – Jagerber48
    Aug 24, 2023 at 12:26

4 Answers 4

13

Answering your questions below:

  1. Would the above proposed workflow be considered a standard workflow?

    There isn't a strict "standard" workflow that everyone adheres to, but your workflow is a variation of the "Feature Branch Workflow" and "Gitflow Workflow." The idea of creating feature branches, merging them into a main or develop branch, and then creating a release branch is a common practice in many teams. However, the specifics, like how you handle versioning and changelogs, can vary from team to team.

  2. If the above workflow is non-standard, how could it be modified/improved so that it is more standard?

    Your workflow is already quite standard, but here are a few suggestions to align it more with common practices:

    • Consider having a develop branch where all features are merged first. Once you're ready for a release, you can merge develop into main or create a release branch from develop. This way, main always represents the latest stable release.
    • For the changelog conflict issue, one common practice is to have a CHANGELOG directory instead of a single file. Each feature branch can add a new file to this directory detailing its changes. During the release process, these files can be consolidated into the main CHANGELOG.rst file.
  3. What are examples of other workflows that address (or also don't address) the issues raised above?

    • Trunk Based Development: Developers work in short-lived branches or directly in the trunk, which ensures more frequent integrations and encourages simpler, smaller changes. This doesn't necessarily solve the changelog issue but does promote more frequent releases.
    • GitHub Flow: It's a simplified workflow where there's a main branch, and every new feature or fix is a branch off of main. Once the feature is tested and reviewed, it's deployed to production and then merged into main. This keeps things simple but might not be suitable for projects that need more structured releases.
  4. Would you consider the changelog issues described above to be a major problem that needs to be addressed? If so how can it be addressed? Am I shooting myself in the foot by wanting humans to write the changelog in the feature branches?

    The changelog issue can be a pain point, especially with multiple parallel feature developments. However, it's not insurmountable. Having humans write the changelog is a good practice because it ensures that the changes are described in a way that's meaningful to users. A few solutions:

    • As mentioned earlier, use a CHANGELOG directory. Each feature branch adds a new file to this directory. During the release, these files are consolidated.
    • Use a tool or script to automate the consolidation of changelog entries during the release process.
    • Encourage developers to pull the latest changes from main before finalizing their PRs, which can help in resolving changelog conflicts before they reach the main branch.

Lastly, regarding conventional commits: they're useful for generating changelogs automatically, but they do require discipline. I am a bit biased here too when it comes to conventional commits, I absolutely love them. But if you feel that the overhead of learning and adhering to the conventional commit format is too much for your team, it's okay to stick with manual changelogs. The key is to find a balance that works for your project and team.

2
  • 2
    This is very helpful, thank you! The changelog directory is very interesting to me. I found this more in-depth discussion of the changelog directory/fragment structure.
    – Jagerber48
    Aug 23, 2023 at 18:54
  • 1
    Also, you're suggestion about encouraging developers to pull latest changes from main before finalizing PRs is what makes me not really scared at all about merge conflicts coming from the changelog being modified on multiple branches. The merge conflicts will force devs/authors to take some time to make the changelog look nice and reflect all the upcoming changes.
    – Jagerber48
    Aug 23, 2023 at 18:56
6

There's no such thing as a "standard workflow". There are some models that are commonly used - look at gitflow, GitHub flow, GitLab flow, oneflow, three-flow, and trunk-based development for some examples. I'm sure there are others out there that I'm not aware of or maybe that haven't been written about. I wouldn't consider any of these "standards", though. Without understanding the specific context and problems, it's hard to say which flows are appropriate or best suited for a particular case.

Personally, once you have a system that is operational, whether that's a stable published library or a deployed system or however you make your work available for use, I like having release branches. Whether that's gitflow, trunk-based development with branch for release, or something else, depends on the context. But you can accomplish something very similar with tagging.

When you have a branch for release, you would be able to pick your version identifier and update your changelog and release notes for changes. When you branch from your mainline branch (develop in gitflow, main or trunk in trunk-based development), you effectively create a safe snapshot for your release activities that don't impede development of the next version. You can run additional, execute longer-running tests and fix any issues (but making sure to back merge any fixes into the mainline), and carry out any release-specific work. Another advantage is that if you keep branches for supported versions around, you'll always be able to do a patch if your release branches are only for major or minor versions and patches are continued development along that release branch.

But if you don't have any problems now, I wouldn't change. As your development model changes - you accept contributions from other developers, the number of developers grows, you want to change the distribution model or support for old versions - you'll run into various problems. As long as you have a stable mainline, you can change your flow in the future.

If you keep your mainline stable, you'll be in good shape for future changes to both your product and your delivery processes.

5

No, mainly because of :

"4. Version bump in source code..."

It's been said a million times; Version published apps, not source code.

If you want to get the version into the source, then you need some out of process build step that updates the sourcecode when you push to master

3
  • This is a fair point. As I'm learning more on this subject I'm beginning to see the wisdom in this idea. I think I could get version out of my source code using setuptools-scm or something similar without too much trouble. Then I think the main questions become (1) on what branch in source control do I want features to accumulate (main? develop? should there be a release branch somewhere?) and (2) how is the changelog handled.
    – Jagerber48
    Aug 23, 2023 at 18:59
  • Note here that "published" may mean internally published as well. If you have a nightly build to QA, that's probably worth bumping the version number for as well so bugs can specify what version of code was tested. Of course that should be automated by the build system, not done by hand. Aug 24, 2023 at 6:33
  • @Jagerber48 My first thought when I started reading this question was to suggest using setuptools-scm. It's great and I use it for all of my own projects. Of course that doesn't necessarily mean you will like it too but I think you would do yourself a big favor by at least trying it for one project. One notable thing about it is that it determines versions based on tags, not branches, so it doesn't matter what branch structure you use for feature development.
    – David Z
    Aug 24, 2023 at 23:59
0

Here's my current workflow half a year later.

  • Branch feature off main.
  • Develop code on feature.
  • Start a PR feature -> main
  • Once changes are complete and recorded in an unreleased section of the changelog then merge the PR into main.
  • Repeat the above few steps a few times for some duration of time collecting changes/features (and logging them in the unreleased section of the changelog).
  • When enough changes are collected or enough time has passed (these are arbitrary milestones) then it is time for a release.
  • Manually read the changelog and determine what the version number for the release will be, e.g. 1.2.3.
  • Make a release\1.2.3 branch.
  • Relabel the unreleased section of the changelog to the new version number and recreate an empty unreleased section of the changelog.
  • Make any last minute changes to the release (this step should be avoided if possible and handled in the feature branches).
  • Merge the release branch into main
  • Make a github release. The title of the release matches the version name and the github release will also add a tag to main with the same name as the version name. The body of the release is copy and pasted from the changelog.
  • The creation of the tag on the main branch triggers readthedocs to pull the latest and stable versions of code and build new docs.
  • A github action runs on release which builds the code and pushes it to PyPi. I use in pyproject.toml I use setuptools-scm to resolve the version number from the git tag. In the docs conf.py I use importlib.metadata.version to programmatically determine the version number that was extracted during the build.

This workflow has worked well for my project. The main realizations that got me to this better workflow were (1) using setuptools-scm to get the __version__ attribute out of my python source code and even pyproject.toml file and (2) using the unreleased section of the changelog to collect changes. Both of these mean that I don't need to know what version number the next release is going to be until I create the release branch right before the release happens.

There are still a few manual steps when it comes to actually doing the release, but I'm overall comfortable with this workflow. The main downside for me is that I have to manually copy the version number and changelog from the changelog file (which is updated with each PR) into the github release form. Perhaps I could find/write github actions that detect when a branch called release/... is merged into main and trigger an automated release from there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.