1

Currently my code base has 3 deployments: "development", where devs unit test and develop features; "testing", where testers run end user verification for the features; and "production", the deployment available to clients.

The issue we're facing is that all 3 routinely go out of sync. One solution is to allow only development to merge into testing, only allow testing to merge into production. Issue with that is that if a feature F1 is done development, we would like to merge it into testing but that might inadvertently push some incomplete F2, F3 to testing as well. Similar issue happens if F1 is verified on testing we can't release it to production because of F2 and F3 being unverified.

Wanted to ask if there is any smooth branching workflow that would be ideal for this case, or if the only correct solution is consistent catchups.

7
  • 4
    Standard practice is to maintain separate new features on separate branches. Is there a particular reason why you don't want to do this? Aug 16, 2023 at 6:03
  • Each deployment communicates with a number of external resources such as databases and has credential configurations which allow for testing to be done accurately. Would not want to expose all of those configurations to each developer/ it would be quite arduous to do the same. Therefore even if each feature has a different branch, its not possible to test it until it goes into a deployment Aug 16, 2023 at 6:08
  • 2
    Is it a prerequisite that the 50 developers work on the same codebase (in source control terms: monorepo)? If so, why? Have you delineated big functional areas in the codebase, identified dependency relationships between them and mapped them to dev teams? Aug 16, 2023 at 7:30
  • These is a logical separation between different files in the codebase but the code base has to consistently be deployed and tested at the same public URL, hence all the tests are being done at the same testing and dev URLs Aug 16, 2023 at 9:02
  • 1
    There's insufficient information here. Are your "deployments" equivalent to your branches? What is your branching strategy? What is your release strategy? To what extent do you have automated testing in place?
    – Thomas Owens
    Aug 16, 2023 at 11:44

4 Answers 4

15

Your problems have existed for decades and people have realized that there isn't a good way to solve it without significantly altering their way of working. Thats where Trunk-based Development, Continuous Integration, Deployment Automation and Feature toggles came from.

First, Trunk-based Development: have only one canonical branch and package version. Cherypicking features between branches is never going to work at scale.

Have a solid build and deployment pipeline. Have only one built package version, that is deployed to different environments, changing only configuration. No one will be able to keep up with having different branches and built packages going into different environments.

With features that take long to implement, use feature toggles. This way, you don't need to keep track of what features are in which branch and which package. Instead, they are all in single package and can be toggled by configuration per-environment.

Minimize amount of feature branches. Thanks to the above, it would be uncomfortable to deploy and test a separate feature branch. Instead, promote continuous integration, where features are commited into trunk frequently, at least once a day, preferably multiple times a day. Each commit should have a automated tests run against it, so you can get fast feedback if something breaks.

1
  • Additionally, you should have enough local regression testing capability (without deploying to any of the environments) that you can be happy that a feature, with its feature toggle off, will not have any impact on the rest of the system. If you can't ensure that, then feature toggles wont help protect against breakages in development preventing release to testing. Aug 16, 2023 at 12:51
9

Excellent answer by Euphoric here, but let me add another (additional) option, which also means a significant change to your current way of working: try to break down your product into individual, separate components which can be developed, tested and deployed mostly independent from each other. This will allow you to break the team of 50 people into smaller individual, teams, each one maintaining a different set of component in its own - and that's a good thing, since 50 devs in one team are way-too-many to work together effectively.

In the world of web application development, this approach has become famous as the concept of "micro services" - I am sure you have heard of it. But if you are developing a large desktop or embedded application, the situation is not much different: one team can develop a library or program, which is tested and released from time to time, and other teams use that library or program as part of their application.

The important part here is that such components or libraries are technically and organizationally decoupled from the main application or other libs. Their code lives in a separate repo. They have their own requirements and feature planning, their own backlog, their own development iterations, their own documentation. A component dev team has to care for providing a stable and well documented interface, backwards compatibility, use proper versioning, communicate new features and breaking changes like any other 3rd party library vendor. Such a library will become a product of its own, maybe with its own life cycle. It will have its own test cycle, maybe with dedicated testers.

Of course, "end users" of those components are the other teams, especially the team which integrates the components into the main application system. So those other teams are the "clients" for which component team work, and they are the ones having to communicate new requirements. But having small, individual teams will reduce the communication overhead by an order of magnitude, and reduce your out-of-sync problems by an equal degree.

0

The keyword you want to look at if you really need to keep that type of work organization is monorepo. The answer is not necessarily a "smooth branching workflow" but a mix of team practices and advanced tooling, as explained here. Big companies like Google developed tools to enforce custom code compartmentalization at build time, way beyond what just Git branches and repos can give you. Maybe check out Bazel for instance.

That said, I wouldn't advise going down that path without a thorough analysis of your business domain areas, delineating corresponding parts of your codebase and defining the relationships between these. Domain Driven Design's Context Mapping technique can help you do that.

It's also an organizational problem since ultimately, you want to find the right team structure with small and independent enough responsibility areas to avoid the issues you mention, but still big enough to avoid too many dependencies and too much communication overhead.

-3

A preferred way in my opinion would be to create a new feature branch from main branch for every feature and have 3 environments. Dev, staging and production.

  1. Dev is where you test your features locally/on a machine in cloud.
  2. Staging is where all the feature branches(across teams) would be merged to after testing. This is where you would have your integration testing and all your automated test cases running.
  3. Prod is self explanatory.

If you have different services and databases your service is connected to, the ideal way would be to configure your service's dev setup(includes database and cache as well). For other services that are independent of the feature, connect them to that service's staging setup.(Assuming they have a functioning one)

All new feature branches are to be created from the main and updated on a daily basis(assuming daily deployments).

All feature branches to be merged in staging env for testing. No direct feature to main merge. Only stage can be merged to main. If several teams are using this, there has to be a cutoff policy decided, for example pull requests can be merged till 1pm, post which testing and deployment would start. Exceptions can be made on a case by case basis.

2
  • 1
    In this scenario, the stage to main merge would have to wait for all features on staging to finish their testing and we therefore would not be able to merge as soon as any 1 feature has completed testing correct? Aug 16, 2023 at 7:01
  • If testing a feature takes too long of a time, that team can be asked to delay it for the next deployment Aug 16, 2023 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.