7

I've started in this new company a few weeks ago, this is the CTO CI strategy:

Current GIT process

Current: Developer team has the repo prod/master and they merge everything into master (no branching strategy). Once the code is ready in prod/master they'll ask Infrastructure team to start the deployment process which uses Jenkins.

The Infrastructure team executes a job in Jenkins that performs this actions:

  1. Clone the whole prod/master into build/master (so they don't mess with the developers)
  2. Execute scripts to build the binary(ies)
  3. Generate a .txt file with the version of the build
  4. Commit and push this changes into build/master (reason: prepare the deployment)
  5. Apply environment specific settings and push, configurations, binaries and code to distro/master

We end up with three repos at the end of the day for each application, that means, if we have 10 applications we would have 30 repositories

Reasons of the CTO for this:

  1. prod/master: For developers and their code (no branching, only master)
  2. build/master: For Infra team to generate versions (to prepare the deployment)
  3. distro/master: Binaries + code + specific environment configurations (to perform rollbacks, traceability and havebackup)

Cons:

  • Really complex process
  • Unnecesary large data ammounts in repositories and slower processing when performing deployments
  • Only works for FileSystem deployments (Databases are not considered in this sceneario and that kind of changes are manually performed)
  • No instant feedback for developers
  • Complexity when crossed patches/fixes and deployments
  • Developers are involved in the production deployment (quite often, in order to test and apply changes on hot)
  • Most of the deployments are performed directly into production

Pros:

  • There's backup and posibility to rollback
  • Easy traceability (for rollbacks, not for development)
  • Specific configurations per environment are stored in the repos with the code and binaries

And this is my approach:

approach

  1. Developers create a JIRA ticket, which will be used as tag for the build and to create the branch
  2. Developers will deploy and test in a Q.A/PRE-PROD environment
  3. Once the code works, it will be integrated to master
  4. Once integrated with master, the binary goes to a "binary repo like artifactory or other"

Pros:

  1. Traceability: The code deployed is easy to find through the tag (JIRA-XXX) for an specific build.
  2. Rollback: Taking the binary from the repo (Artifactory)
  3. One Repository per project, it means 10 projects are 10 repos, not 30.
  4. Instant feedback to developers, if the deployment is not sucessful they can change their code
  5. This design contemplates db scripts as hooks
  6. The configurations per environment will be handled with Ansible + GIT, generating templates with placeholders and a backup of each configuration.

Cons:

  • Re-educate developers to work in branches
  • Force developers to integrate code only when it really works
  • Change the CTO mindset only will happen through examples (working on it)
  • We must create new infra (new environments to create deployments and not going to production directly)
  • Lots of hours automating through hooks, rest apis
  • Need to implement new technologies

I'd like to know the opinion of people with expertise on this git strategies and the balance between development and operations.

Regards.

H.

2
  • Is the division between dev and infra set in stone? To me that looks like the real problem here, the git strategy merely being a product thereof.
    – Guran
    Jan 6 at 9:16
  • @Guran kind of, the CTO is a developer so his solutions are dev-oriented, and we are forced to comply with his definitions (no negotiations available by now). Jan 6 at 13:57
7

By having three repo you kind of have a branching strategy already but a complex one.

I think you're not facing a technical issue but a cultural issue. Even if your solution is good (I think it is but I did not go into details) you won't be able to have it accepted as is.

In those kind of situations you need first to understand why it has come to this. What problems where the team faced that they worked this kind of solutions. It may have been:

  • No understanding/fear of git. If it was the first SCM tool implemented by the CTO or ops team, the may have not understood how git works and have think multiple repo safer to ork with.
  • Lack of trust in developer (or ops, or other): Maybe someday, someone broke something and now everyone pay the price.
  • Politics : Someone if protecting himself or his team from any kind of change, because if it works... it woks and any change that would break something could come to bite them.

In fact writing this I always go back to a lack of knowledge of some tools be it Git, JIRA, or any other too in the CI stack.

Steps to proceed:

  • Book a meeting with someone in charge/knowledgeable. If you CTO is accessible, it may be best. Don't discuss solution or issue. Just try to understand why things are how they are. (try the five whys)
  • Find the deepest issue (lack of trust, fear of change, fear of something broken...) and find a single change that address this issue. It may be hard, but it may be harder to push for an overhaul that fixes every issues.
  • Alternatively you can propose practices that don't impact the current workflow but can render it obsolete in the future. For example having developer working on branches to avoid merge on master (if it is an issue) or add JIRA ID in commit title to track work easier. Having things in place and proven to work is a good argument to push them to "higher" level (infra, prod, ...)
3
  • "Alternatively you can propose practices that don't impact the current workflow": Yes I'm doing that, working in a new process which do not interferes with the current workflow. Once I have something working it will be easier to demonstrate why is better (statistics, performance, maintainability) Jan 6 at 17:09
  • "Book a meeting with someone in charge/knowledgeable. If you CTO is accessible": Not accesible, he WANTS IT TO HAPPEN, that's why I'm working in the other workflow in the meantime. Jan 6 at 17:10
  • "Having things in place and proven to work is a good argument to push them to "higher" level (infra, prod, ...)" Exactly what I had in mind! Jan 6 at 17:11
0

At a previous client we switched from no branches to branch-based development. At my current client we are doing mostly branch-based development.

The huge downside I have is that all of our work is not integrated most of the time and it is a lot of pain to work with others who also work on branches.

The branching thing is good in theory, and git is up to the task. But in practice I at least don't see neatly separated tasks, where I don't have to use other people's current work. We don't always have small (couple of hours) tickets, so the window where the software is not integrated is always larger. And there's dependency hell, because I can't just depend on "SNAPSHOT" versions, I always need to build branches, or worse, integrate myself and have that dependency somewhere.

Ideally I would want everybody to work on master. Of course that also at some point becomes untenable when there are multiple teams or products with dependencies and different release cycles. Devops and CD would solve most of these problems, but none of the places I worked with have that, not even close.

6
  • What about the use of 3 repos for 1 product/development? Seems like the problem you point out is only from the developers point of view (their workflow) but the "current" git strategy makes the things complex for the operations team (scalability and maintainability) where's the balance in a situation like this? Jan 5 at 23:23
  • About this: "The huge downside I have is that all of our work is not integrated most of the time and it is a lot of pain to work with others who also work on branches". The first developer in finishing his feature integrates with master, the other one must pull from master and adequate his own work to the new changes (resolve conflicts). Jan 5 at 23:30
  • @MiguelOrtiz About the 3 repos: I don't like that either, but it has some merit if you assume there are different teams (dev & infra) working on the same thing. Normally that would be one team (i.e. Devops-like), that would get rid of the complicated setup. Jan 6 at 8:58
  • @MiguelOrtiz About branch & merge: First, it is an overhead. Second, it only works if the tasks are really small & fully independent. If the tasks are more than a day or so, with a decent sized team, you'll have problems. Yes, you can solve those, but it just adds up. Again, that is just my experience, I would imagine it varies greatly with team size, task size, project size, type of tasks, etc. Jan 6 at 9:06
  • 1
    @RobertBräutigam If you're working on feature too big/that take too long and constantly face merge issues you have either a probleme of sizing of work (you should be able to devide work in smaller units you can integrate earlier/easier) or a code quality issue where everything is a big plate of spaghetto.
    – JayZ
    Jan 6 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.