How can we improve our build process?

Question

The company I work at has a (in my opinion) slow build process. We use TFS, and checking in code takes several hours. It goes like this:

(Assuming change is made and code review has been approved)

First, you make your change and submit it for a preapproval step via a button in visual studio. This builds the entire visual studio solution and runs a subset of tests that we consider to be critical to the project. These builds are run in parallel for efficiency. If this succeeds, the next step is automatically kicked off. If this fails, you get a build email saying what tests failed and for what reason, at which point you must start over.

[This first step takes several hours to complete]

Second, the same set of tests is run as before with your change, except instead of these builds being run in parallel, they are serialized. If the build succeeds here, your changes are checked into the "mainline". (We used to use branches, but don't anymore). If this fails, you get a build email saying what tests failed and for what reason, and you must start over.

[This second step takes several more hours to complete]

Third, (remember the code is already checked in at this point), a more comprehensive build is run that runs all of the tests instead of just a subset. Because this build takes so long, changesets are batched up, and as before if the build fails you get an email saying which tests failed. Unfortunately, it can be hard to figure out if your change was the culprit.

[This third step takes several more hours to complete]

We have the concept of an "intermittent test". If a test fails, we run it up to two more times to see if it passes. We know this is bad, but have not made the effort to eliminate these kinds of tests.

Moreover, this process has lately been brought to a halt by issues with tests hogging too many resources on the build machines, or people skirting the process and directly checking in code, which breaks the first step.

For reference, the project size is on the order of a million lines of code. I don't expect builds to take several minutes, but after reading about people doing continuous integration and how it should take 10 minutes at most, you should be able to run all of your tests immediately, etc. I'm becoming very disheartened with the state of our process.

How could we improve our build process so that there is a shorter turnaround time between having a change ready and getting into the mainline?

Answer 1

Some things I can see right away (although I have no idea how attainable they are):

• add profiling to your unit tests. figure out which ones take the longest and make those work faster.
• refactor your project into smaller projects that each can run separately. A single project that's millions of LoC is really hard to maintain, and it's probably better if you divide your project into chunks that are more or less separate. Whether you choose horizontal or vertical separation or a combination depends on your project.
• Do something about your intermittent tests. Tests failing means that your software is broken, simple as that. At the very least, monitor which tests fail 3 times in a row and make an effort to fix those.
• if your developers aren't using high-end equipment already, invest in fast equipment for your developers and your build machine (or machines, more on that in a moment). If you compare the price of hardware to that of a good developer, it's almost always more cost-effective to throw extra hardware at the issue than it is to spend more devops time on the issue. at roughly 100K a year and 2K hours per dev, you're looking at 50 USD/h. If spending 1500 USD per dev on high-end hardware (1 TB M.2 SSD, hexacore Intel CPU, 64 GB of RAM as a ballpark*) saves each dev an hour a week in faster build and test times, you're still saving a thousand dollars per dev a year.
• Split your batch tests into chunks. Get 2, 3 or more build machines (again, make them fast), split your unit tests into roughly equal groups and let each machine run their own group. if running all unit tests took you 5 hours before, splitting them into 5 groups and running them on separate machines means you're looking at a build that should only take 1 hour.

* 1 TB M.2 SSD costs 329 USD. 6 core Intel i7 6850K with X99 chipset costs 617 USD. 32 GB of RAM costs 200 USD. X99 motherboard costs about 2-300 USD. for about 14-1500 USD, you got a machine that can brute-force it's way through a build.