3

The company I work at has a (in my opinion) slow build process. We use TFS, and checking in code takes several hours. It goes like this:

(Assuming change is made and code review has been approved)

First, you make your change and submit it for a preapproval step via a button in visual studio. This builds the entire visual studio solution and runs a subset of tests that we consider to be critical to the project. These builds are run in parallel for efficiency. If this succeeds, the next step is automatically kicked off. If this fails, you get a build email saying what tests failed and for what reason, at which point you must start over.

[This first step takes several hours to complete]

Second, the same set of tests is run as before with your change, except instead of these builds being run in parallel, they are serialized. If the build succeeds here, your changes are checked into the "mainline". (We used to use branches, but don't anymore). If this fails, you get a build email saying what tests failed and for what reason, and you must start over.

[This second step takes several more hours to complete]

Third, (remember the code is already checked in at this point), a more comprehensive build is run that runs all of the tests instead of just a subset. Because this build takes so long, changesets are batched up, and as before if the build fails you get an email saying which tests failed. Unfortunately, it can be hard to figure out if your change was the culprit.

[This third step takes several more hours to complete]

We have the concept of an "intermittent test". If a test fails, we run it up to two more times to see if it passes. We know this is bad, but have not made the effort to eliminate these kinds of tests.

Moreover, this process has lately been brought to a halt by issues with tests hogging too many resources on the build machines, or people skirting the process and directly checking in code, which breaks the first step.

For reference, the project size is on the order of a million lines of code. I don't expect builds to take several minutes, but after reading about people doing continuous integration and how it should take 10 minutes at most, you should be able to run all of your tests immediately, etc. I'm becoming very disheartened with the state of our process.

How could we improve our build process so that there is a shorter turnaround time between having a change ready and getting into the mainline?

  • 3
    There is no such thing as "typical." – Robert Harvey Nov 2 '16 at 19:45
  • @RobertHarvey I'm not sure what you mean by that. – Michael Hagar Nov 2 '16 at 19:47
  • 2
    Anecdotally, it sounds like what happened at your organization is that you started having "scaling problems" with your software development process, and instead of analyzing those problems, identifying possible solutions and adopting a better process, you simply piled more software (your CI machine) on top of a process that was already too complicated to begin with. – Robert Harvey Nov 2 '16 at 19:49
  • 2
    Opinion polls like this are discussion questions which are essentially opinion-based questions. Removing the "what do you do?" aspect to the question, adding objective criteria for fixing your build process, and focusing it a bit more would be a huge improvement. As it is now, it is both primarily opinion-based and too broad. – user22815 Nov 2 '16 at 19:51
  • In my experience, it is not at all unusual for an organization's process from code-change to production-deployable to take several hours. This should only be an issue for production patches, though. It should always be possible for developers to test their code with much quicker turnaround. – antlersoft Nov 2 '16 at 19:58
4

Some things I can see right away (although I have no idea how attainable they are):

  • add profiling to your unit tests. figure out which ones take the longest and make those work faster.
  • refactor your project into smaller projects that each can run separately. A single project that's millions of LoC is really hard to maintain, and it's probably better if you divide your project into chunks that are more or less separate. Whether you choose horizontal or vertical separation or a combination depends on your project.
  • Do something about your intermittent tests. Tests failing means that your software is broken, simple as that. At the very least, monitor which tests fail 3 times in a row and make an effort to fix those.
  • if your developers aren't using high-end equipment already, invest in fast equipment for your developers and your build machine (or machines, more on that in a moment). If you compare the price of hardware to that of a good developer, it's almost always more cost-effective to throw extra hardware at the issue than it is to spend more devops time on the issue. at roughly 100K a year and 2K hours per dev, you're looking at 50 USD/h. If spending 1500 USD per dev on high-end hardware (1 TB M.2 SSD, hexacore Intel CPU, 64 GB of RAM as a ballpark*) saves each dev an hour a week in faster build and test times, you're still saving a thousand dollars per dev a year.
  • Split your batch tests into chunks. Get 2, 3 or more build machines (again, make them fast), split your unit tests into roughly equal groups and let each machine run their own group. if running all unit tests took you 5 hours before, splitting them into 5 groups and running them on separate machines means you're looking at a build that should only take 1 hour.

* 1 TB M.2 SSD costs 329 USD. 6 core Intel i7 6850K with X99 chipset costs 617 USD. 32 GB of RAM costs 200 USD. X99 motherboard costs about 2-300 USD. for about 14-1500 USD, you got a machine that can brute-force it's way through a build.

  • 1
    Indeed, if it takes several hours to run your tests, you have a whole ton of tests, or the ones you have are severely inefficient. – mmathis Nov 3 '16 at 0:03
  • Instead of buying test machines you could put the test infrastructure into the cloud. Azure would probably have some good options for TFS... – mcottle Nov 3 '16 at 2:47
  • Intermittent tests don't necessarily mean the sw is broken - sometimes the tests themselves are at fault. But indeed, regardless of the root cause, intermittent failures can ruin the process performance and should be addressed. – Dan Cornilescu Nov 3 '16 at 5:18
  • A faster machine can help speedup a build only if the build is limited by the computing resources. But it won't help much if the build doesn't even use the existing resources at max, for example if it is too serialized. A build analysis would be required to determine the limitations first. – Dan Cornilescu Nov 3 '16 at 5:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.