Alternative to "Passing/Broken build" indicator?

Question

When having a continuous integration executing the tests at each commit, a common best practice is to have all the tests passing at all times (aka "do not break the build").

I find some problems with that:

For example one cannot help an open source project by creating tests corresponding to tickets. I know if I propose a Pull Request to an open source project containing failing test, the build will be marked as failed and the project will not want that merged into its repository because it would "break the build".

And I don't believe it is a bad thing to have failing tests in your repo, it's like having open issues in your tracker. These are just things waiting to be fixed.

The same goes in a company. If you work with TDD, you can't write tests, commit and then write the logic code that fulfills the test. That means if I have written 4-5 tests on my laptop, I can't commit them before going on holidays. Nobody can take back my work. I can't even "share" them with a colleague except by sending them by email for example. It also prevents working with one person writing the tests, the other one writing the model.

All that to say, am I misusing/misunderstanding the build process/continuous integration? It seems to me that "passing"/"not passing" is a too narrow indicator.

Is there a way to make continuous integration and TDD compatible?

Maybe there is a standard solution/practice to distinguish "new tests" (that can fail) and "regression tests" (that should not fail because they used to work)?

Answer 1

I see where you're getting at, but these types of problems are typically solved in other ways. There is a good reason why this is standard protocol. If someone submits code that doesn't compile, everyone updating his/her code will have a program that doesn't compile. That includes programmers who are currently working on something completely different and somehow find themselves in a situation in which they need to wait before they can compile and test what they're working on.

Standard protocol is that you can commit changes even for complete or even incomplete work, so long as it compiles so that programmers can update their code every day if necessary.

However, I still see what you're getting at. Sometimes you want to commit in order to simply save your code. For this, most source repositories support branching. This allows you to create a private branch, work on it without disturbing others, then merge into the trunk when the work is completed. This allows you to commit when you want without any of the backlash associated with causing the build to break.

If that's not suitable, GIT allows you to commit (pushing) to repositories on your local machine, but conceivably the repository could be anywhere. You could create a repository for potentially partial/incomplete work and another repository for finished work, and on that repository you can add a nightly build.

Again, I can't stress the importance enough. Do not commit broken code to trunk ever! Your contributions cannot impact the work of other programmers.

Edit

I see that you intended broken tests, but in my humble opinion, there is little difference. The whole point of a test is to determine if a particular aspect of a program passes or fails. If it always fails and you do nothing, then the test, in the traditional usage of unit testing, serves nothing. If you use it to perform some other metric which doesn't necessarily entail a "failed" commit if one such test fails, then I would strongly recommend you find another way to do the same thing.

Otherwise you risk that the test is never taken into consideration or if it causes your build to fail, that your fellow programmers ignore failed builds. It's more important that programmers realize when they've broken a build than to perform a test which offers no real insight and may only result in bad practices.

Answer 2

Given a master branch with failing tests, how can you be sure - without comparing that list against previous builds - that you haven't introduced bugs?

Simply tracking the number of failing tests is insufficient: you might fix one test, and break another. And if you're on holiday, it won't be clear to others looking at the failing build.

Keep your master branch clean and green at all times. Work in a branch. Keep the branch under CI, in a separate job, and have failing tests to your heart's content. Just don't break master.

Have the branch's reviewer only merge your branch if it passes all tests. (More strongly: have the reviewer only able to merge your branch if the result of merging the branch into master passes all tests!)

Answer 3

There are ways to solve your problems without throwing away well understood and accepted practices about continuous integration.

I will start with the problem of committing a 'broken test' that corresponds to a ticket. One solution is to create one or more breaking tests exposing the problem, then actually fix the problem, so they can be merged back to the main code line together. Second solution is to have the broken tests, but use some type of ignore flag so they do not actually run and break the build. Possibly add a comment or a special annotation that makes it very obvious that this is a broken test for Ticket#N. Also attach a note to the ticket itself that refers to the created tests that are waiting to be unignored and ran. This would help a person fixing the ticket, but also would not be a red-flag for someone who comes across the test.

And onto your next problem with TDD. TDD is about writing a small test and then writing a small chunk of code to make that test pass. Then keep iterating until you have a small functional module. I feel that if you write 4-5 tests, then you go on vacation, you might be doing it wrong. You can pair program with someone in a way that one of you write the test, the other the corresponding code. You should however not use the main code line repository to share this code between you two before a completed module is ready to be committed. As others suggested, a shared branch would solve your problems there.

Trying to break the continuous integration mantra can lead down to unexpected and scary paths. For example, what would code coverage mean in this type of environment? How would developers not feel that the system has many "Broken Windows"? How would one make a change, run the test and know whether they are actually breaking anything new, or its just the old stuff?

Answer 4

I think your fundamental problem is that you are including test RESULTS as part of the build. While obviously some people agree with you, others don't. Breaking the build occurs when it doesn't build. Not when it doesn't build without errors.

Consider a major project like Windows or Linux, or even something like Firefox -- do you think they ship bug free? Of course not. Now these projects aren't doing TDD, but that's really irrelevant -- TDD doesn't change two fundemental facts: bugs exist, and it takes time to fix them. Time which a project (open source or not) just can't afford to waste on low priority bugs. KDE recently had a bug that was over a decade old fixed. When was the last time you heard someone say "I'm glad we waited a decade to ship our project"?

TDD, in a way, probably makes it EASIER to ship with bugs -- because you have a better understanding of what the flaw is. If you can precisely define what causes the bug, you have an excellent basis for weighing the cost of fixing it.

My recommendation is to find a project that doesn't mind some red amoung the green.

Answer 5

 > a common best practice is to have all the tests passing (green) at all times.

I prefer to have all tests not failing (not red).

With this slightly different definition you can also define tests that are

• not implemented yet (grey in nunit if there is a NotImplementedException)
• known to be failing="needs a to do" by marking/annotating the test as ignored (yellow)

I you check these into the repository your continous build is not broken and therefore valid.

Answer 6

You could consider two different CI build "concepts".

1. Regular CI builds. The regular CI build should compile and run only those tests for which code has been written to make them pass, so that CI reports of test pass/fails is a clear, unambiguous indicator of regression against the previously accepted state of the code.

2. A "future" CI build. This build compiles and runs only those tests for which no code has been written specifically to make them pass. There can be several reasons to have such a test:

   • Tests can added for specific failure cases from the issue tracker, for which no fix has been attempted yet. It's clearly useful to already have a codified, running test for an issue even without a fix.

   • Tests added for new required functionality that has not been implemented yet.

   • Often it's easier to know how to test a failure or feature than to know how to implement the feature or fix, and separating the two steps by committing the test to source control can be useful to ensure no information is lost.

As in "standard" CI, in the regular development regime the team would only be looking at the daily build results of the regular build.

The team can also keep an eye on the evolution of test cases from the "future" CI build - specifically to see if any changes made for the regular CI actually fix issues from the "future" build, which can be an indication of an important underlying issue or design improvement.

Finally, if a team member has extra time on their hands they could take a look at fixing one of the issues from "future", and if they succeed migrate it to "regular" (while updating the issue tracker status).

Answer 7

And I don't believe it is a bad thing to have failing tests in your repo, it's like having open issues in your tracker. These are just things waiting to be fixed.

The problem isn't failing tests, it's a simple, context less indicator of regression. Aka: as a developer, can I check a single indicator and know if I introduced a regression or breaking code.

The moment you introduce the concept of 'soft' failures (it's ok, we're working on it/not implemented yet/we're waiting for the new version/it's going to pass again once this other build is fixed), you require everyone that might run or look at the test to know the expected state. Which in a large enough team is going to change by the hour: your indicator becomes meaningless. In a smaller context (team private integration test for instance), then I think it's like technical debt and is ok - just needs to be managed.

The way most tool address that is that 'green/passing' reflects what the expected result is - not that the code is working:

• Fitness has the concept of expected failure.
• JUnit has @Ignored / @Test(expect=)
• Cucumber has the 'not yet implemented' (or whatever it's called) status

Those come with their own issues (how do you distinguish between 'yes, we know it's broken, working on it' and 'the correct behavior is an exception') - but they help.

Answer 8

I use skipped tests.

In the particular unit testing framework I use, I can raise a SkipTest exception. The test isn't actually run, and its failure won't break the build. However, I can see the number of skipped tests and see if there is work to be done in that area.