6

I'm managing a team in .net. They are writing unit tests, use them locally on a regular basis, and love it. However, they keep pushing to have the unit tests kept as a separate project and only want to run them on a communal build server once a week or so.

In contrast, our JS teams are running tests on the build server on every pull request.

I keep trying to explain to them the benefits of having the tests near the code, and running the tests often, but they keep bringing up fears of the builds being too slow, or what happens when the tests prevent an urgent build.

It's boggling my mind and I'm wondering if there is something I'm missing.

What are likely reasons for this kind of pushback? They are happy to write the tests and they tell me they have close to 100 tests already written on branches they refuse to merge into their main branches until "we come to a decision." ( I.e. I tell them it's ok to keep tests in an isolated project far away from the main code.)

The team was quick to adopt writing unit tests and they find them useful and continue to write more. I am now trying to get them to run the tests in the CI and that is where I'm hitting resistance.

  • 2
    How long do the tests actually take to run? – Telastyn Mar 16 '18 at 15:19
  • 16
    "or what happens when the tests prevent an urgent build" But isn't that build somehow useless if there are failing tests? – πάντα ῥεῖ Mar 16 '18 at 15:27
  • 3
    Possible duplicate of What's the point of running unit tests on a CI server? <--- TL;DR stop wasting developers time and let machine automatically run the tests – gnat Mar 16 '18 at 15:27
  • 1
    @telastyn currently they take 3 seconds to run. – Bob Mar 16 '18 at 15:29
  • 3
    sorry but per my reading you simply misunderstood what they want, they don't really care how frequently tests are run, they only want them to run in a non-blocking manner, and that is perfectly doable with CI and you didn't show any reasons for why it shouldn't be that way (as opposed to your devs who seem to tell sensible reasons why it should). Whether it is done by extracting unit tests to separate project or by keeping within and only running these with a separate (non-default) CI config doesn't matter much, it's only a minor technical detail – gnat Mar 16 '18 at 15:44
7

Firstly, thank you to @Gnat for finding insight into the real issue here and giving me what I needed to know to get to the real issue.

We have agreed to isolate the unit tests and integration tests into their own projects to ensure that the added test code doesn't get compiled into the build.

We are also going to be running the tests in a non-blocking workflow until we are able to separate out the code into smaller repositories so that each build doesn't take 15-30 min.

And for now, we have agreed to not build the project on every Pull Request(PR), but instead to run the tests on a nightly build. This is what they really meant by the tests slowing things down... They actually meant doing a build on every PR. They do PRs dozens of times a day. Their main concern about tests blocking an urgent build, was really their concern that urgent builds often end up with dozens of PRs, when everyone is trying to merge their code at once, and they wanted to be able to just run the tests at the end of that process, and not require a separate build at every point along the way.

There wasn't one single answer which gave me all the information I needed, and most of the best answers came in comments, so I've created this answer to summarize up everything that made the most sense.

The important pieces of information I was missing were:

  • If the tests are too close to the code under tests, then its very time consuming to tell .net not to build the test files into the .dlls, and it's much easier to just keep them as separate projects.
  • The real issue was the build required to make the tests run. It wasn't the tests they didn't want to run, it was the build. They didn't want the build to prevent them from being able to merge code as frequently as they do. Locally, on their own computers, they are able to do a debug build and run the tests in under a minute, but on the CI servers, it takes much longer.
  • 1
    This paints a picture of a much healthier team. Sounds like a good plan. – candied_orange Mar 18 '18 at 11:53
6

I'm managing a team in .net.

Doesn't matter.

They are writing unit tests ... and only wants to run them once a week or so.

Unit tests are not run once a week. You run them before and after writing the tiniest amount of code you can to make them pass. Tests are usually good but whatever these are they ain't unit tests.

and love it.

Good.

However they keep pushing to have the unit tests kept as a separate project

Doesn't matter.

Our js teams are running tests on every commit.

That's all? Either you commit very often or you don't test much. I test locally first.

What's been left out of this story is how often you're refactoring the tests. Are these tests some entrenched guardian that takes herculean effort to change? Refactoring tests should be made as easy, and as often, as refactoring code.

I keep trying to explain to them the benefits of having the tests near the code, and running the tests often, but they keep bringing up fears of the builds being too slow, or what happens when the tests prevent an urgent build.

They should be allowed to put the tests anywhere they can work. They also should know how to localize a build to the classes that they actually touched so that this is a non issue.

It would be crazy to take tests designed to run "once a week" and shove them into continuous integration. Instead have them add new tests designed to run in the continuous integration that will be tuned to the performance needed there.

The fancy term for these new tests is: continuous integration tests. They are also not unit tests. The best way to categorize tests is based on the performance you need from them. Not the tools you use to run them. That way tests that behave the same way are together.

Do not take control of the tests away from the developers. Give them tools that let them control new kinds of tests.

  • 1
    "You run them before and after writing the tiniest amount of code you can to make them pass. " I doubt this team or many other follow this strict form of test first development. – JeffO Mar 16 '18 at 21:13
  • @JeffO which is fine. But don't call them unit tests if they aren't unit tests. – candied_orange Mar 16 '18 at 21:15
  • 3
    Based on the link for Unit Test, there's nothing in there about how or when they are written. it's about the scope of what they test and don't test. – JeffO Mar 16 '18 at 21:18
  • 1
    @Laiv If you mean rebuild the entire project rather than only what you just changed then that makes them regression tests. I don't classify tests based on their structure. I classify them based on what I can expect from them, how they're used, how they'll perform. How they're written is an implementation detail. – candied_orange Mar 17 '18 at 12:57
  • 1
    @CandiedOrange: but the way you worded it, it gives the impression the OP has the wrong tests (and not that they use them wrongly) because they are no unit tests, so his team should write a different kind of tests. I think you agree that focussing on how to work differently with those tests is more important than how they are called. Currently, the naming focus might give some readers a wrong impression of what you are talking of. – Doc Brown Mar 17 '18 at 22:45
5

but they keep bringing up fears of the builds being too slow, or what happens when the tests prevent an urgent build.

What happens if you should prevent an urgent build because you're about to commit some buggy code, but don't because you didn't run the tests that would tell you that you're introducing bad code?

It seems like your team is looking at the tests as an obstacle rather than a tool. The tests are their safety net, their canary in the coal mine. If the builds are too slow, then perhaps that's telling them something they need to know about the software. If they continue to sweep the slow build problem under the rug, it will continue to be a problem.

  • according to OP comments builds seem to be fast, not slow: "currently they take 3 seconds to run" – gnat Mar 16 '18 at 16:33
  • 2
    @gnat: right. I think the concern is that the overall build takes a long time, even though the testing part of the build is quick (ie: "build" is not synonymous with "test"). – Bryan Oakley Mar 16 '18 at 16:34
  • 1
    The problem looks to me as a issue of confidence. Devs don't trust in their own tests. Maybe these are not as "unitary" (deterministic) as they think. – Laiv Mar 17 '18 at 15:33
  • @gnat the build takes 15 min. The tests take 3 seconds. – Bob Mar 18 '18 at 0:14
  • I think you hit the nail on the head here. They’re worried they won’t be able to push out an “emergency” patch. The goal here is to convince them they shouldn’t be putting code on prod if the tests are failing. – RubberDuck Mar 18 '18 at 14:14
3

Firstly, separate projects for unit tests are good. Keeping the unit test code away from the main code is separation of concerns and all that kind of cool stuff.

Secondly, there is no getting away from the fact that unit tests, if run as part of the build will slow things down and queuing for builds can happen. You don't mention what you're using to build (TFS, TeamCity, Jenkins etc) but there are things you can do to mitigate this like multiple queues and priorities. Speak to your build manager.

If there is an urgent build that has to jump the queue, the build manager again should be your first port of call. They may be able to can lower priority builds and push the urgent build higher up the queue.

Finally, you mention CI but don't mention whether it is gated. That is: new code is kicked out in its entirety if the build or unit tests fail. This can undoubtedly be contentious as the code can in theory work, but some conditions may cause the tests to fail.

There is alas, no silver bullet here. When I was a build manager, I often encountered huge resistance when code got kicked out - especially if a deadline was looming. How often I heard the refrain: "TFS keeps crashing. Just turn the setting off and let me check the code in!". Without exception the developer was at fault. They'd either forgotten to include a file in the check in or simply failed to update the test when the code changed as it would have taken too long.

You seriously need to get into the habit of pushing back on the developer here otherwise they will run roughshod over you for ever. If another developer pulls broken code that you allowed through just because another developer starting whining, you will not be popular.

2

I believe its because of the typical "reluctance to change" problem we study in software engineering all the time. I can't think of a benefit to keep Unit Tests separate (I have worked in JS, Java, .Net and ABAP) but I can see the usual reason, "we have been doing it for a long time and it always worked, so need to change".

Hope that helped. i would also be interested to know if there is a genuine reason.

2

they keep pushing to have the unit tests kept as a separate project

Tests should go in a separate project. This is standard practice

mySolution
    myProject
    myProject.Tests

and only wants to run them once a week or so.

This is a little odd, although if they are UI tests you might run them overnight. However. Just agree and get the tests checked in.

You can have the 'why don't we run them on checkin?' and 'why don't we not allow merges when tests fail' conversations at a later date.

protected by gnat Mar 18 '18 at 4:11

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.