1

We have an issue of maintainability for our pipelines at the organization where I work, whereby making a change and ensuring the change is tested and bug-free for our customers (other developers in the organization) can take at least a month or more in some cases for seemingly simple enough changes.

We provide standard pipelines for our code templates (Java for ECS and NodeJS for Lambda, Angular for SPA) that developers in the org use. The pipelines are somewhat straightforward (build, unit test, deploy, health check). But do use other techs like terraform, docker, serverless etc within the pipelines. Bitbucket is our only VCS and some developers (mostly contractors) may use bitbucket pipelines alongside our pipelines for faster testing feedback.

We also have other pipelines that handle consumer creations (an API wanting to call another API in our org), which handles the usual delete/update create operations on two different pipelines that handle all of these requests. There can be some other business logic-esque (but not many) parts in the code template pipelines.

To break it down further, I provide the pain points of making and testing a change below. For reference, these pipelines are pretty large and do a bunch of custom logic, all in Groovy using Jenkins.

  1. No static typing; we have shared global objects, but these have become quite monstrous in size and are just a hashmap where the maps are hierarchical, but still some property names seem to change. When creating any change whatsoever, I need to commit my change, run the pipelines (depending on the stage, this can take 10+ minutes) and wait for a failed run because I am calling an undefined property or variable, and the cycle begins of fixing.

  2. There is no automated testing for testing the pipelines themselves and QA will have to manually test the pipelines, if the change involves one of the starter code templates, the QA will have to go through the whole flow of creating a new project, registering it on our in house full-stack app (we have control and can make changes to this as we please), creating a pipeline and then run differing cases which can take around 30 minutes each for the pipeline to complete (the caching of builds and etc makes this slower, a WIP to make this faster is underway). The QA then manually checks AWS and other services to ensure it was created correctly

  3. Because of the large amounts of code in this repo, developers often have to re-learn what is happening under the scenes in the set-up stage when making small changes.

Overall, it is unpleasant to work on these, but we have to maintain and upgrade parts of the pipelines quite regularly. So my questions are below if someone has encountered the same sort of problems.

How do you ensure pipeline code is tested through automated test suites? How do you get fast feedback on code changes when altering CI/CD pipelines? Are there alternative products or techniques with Jenkins which were more maintainable/"fun" to work with?

2
  • The answer to the general question "How to increase maintainability in CI/CD pipelines" is to select tools that facilitate local testing and treat the code the same way as regular production code. We don't do tool recommendations though. With a quick Google search I found this StackOverflow topic which may be of help: stackoverflow.com/questions/36309063/…
    – Rik D
    Apr 2 at 16:10
  • I've had a look at post too, the replay feature I've found to be well...weird, it seems like when I get an error which causes the script to fail, fixing it in the next commit and doing a replay will make the pipeline fail around the same place but with an even more obscure error message. But I will try out the local part (the second top answer) and see if there is much benefit. Apr 2 at 19:54

1 Answer 1

2

The answer is simple, don't make complicated pipelines with shared templates and custom scripts!

It's a common mistake at large companies. You want to standardise and "make things easy" with templates, but they just add more and more complexity, are boring to code, so developers move on to other things, and pipelines generally lack the tools for testing, version control, roll backs etc

some developers (mostly contractors) may use bitbucket pipelines alongside our pipelines for faster testing feedback.

This looks like a red flag to me, your (presumably highly skilled and paid) staff don't want to learn your company bespoke pipeline stuff and use off the shelf generic alternatives instead, AND get better results!

I would advise removing the responsibility for anything but building the code from the build pipeline. Then let people write their own pipelines. The end result of which is publishing the build artifact.

You can then do the remaining tasks with other software, hopefully off the shelf stuff, Octopus, Helm, Terraform, or your own scripts if you want.

1
  • Yeah, unfortunately, quite a few technologies are used due to architectural or compliance reasons, leading to an unmaintainable messy state. Currently, only our team is offering these pipelines and other devs have mostly not created pipelines before. However, I've proposed to the team that we migrate our code to use Jenkins Templating Engine to offer a standard template while being easier to modify (by other devs too), test, and maintain. I'll try my best to simplify and encourage simpler stages and hopefully, I can update this thread with a success story in due time. Apr 9 at 6:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.