13

Our code base is growing for 20 years now. We are about 10 devs + sqa working with 500kloc. Some time ago a small team of us (2 devs, one from sqa) started working on an automated test program. Currently one run takes 11h and is somehow an integration test. We are working on it to get this down and reduce false positives and are making good progress in that. But details shouldn't matter.

It is working okay and we continue to improve it. We (the small team) like it very much. If we break something, we notice a day later and not 2 months later when sqa takes a look. Also, our managers (dev + sqa) like the idea. But other people in the team just ignore the testresults. In their mind, if tests are failing after a checkin, it's a problem of the test and not of the code change and it's just our toy project. We had discussions several times if a failing test is a real error. Most times it is.

We can't and don't want to enforce something. How can we show that automated testing is a thing?

closed as primarily opinion-based by Robert Harvey, Greg Burghardt, gnat, amon, 17 of 26 Oct 13 '17 at 15:07

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 11
    This isn't a Software Engineering problem; it is a people problem. – Robert Harvey Oct 12 '17 at 19:07
  • @RobertHarvey I got downvotes on SO because "opinion based" and a comment that this site would suit perfectly (and upvotes on that comment). So: where should I ask? Educate me. – Peter Schneider Oct 12 '17 at 19:12
  • 2
    I am with @RobertHarvey about this being a people problem. But as per Workplace, your question will probably we considered a dupe. For example see this question which is fundamentally what you are asking workplace.stackexchange.com/questions/44964/… – Peter M Oct 12 '17 at 19:23
  • 1
    Do not let those downvoters (or even close-votes) discourage you! Some people may understand that such questions are important, and perhaps can provide help. By the way, also my colleagues fail to see the usefulness of automated tests, despite the previous version (without any automated tests) is a box of bugs. Just change one thing, and break a few other, seemingly unrelated things. Some people just do not want to learn (there is open resistance against learning new things). – Bernhard Hiller Oct 13 '17 at 9:52
  • 1
    It's a shame this question has been closed. If software engineering means anything it means the problems of working with actual people, and the answers to such problems will involve opinion. That said, a couple of quick ideas: (1) if you tests give false negatives, this will definitely increase pushback because the results will feel like a waste of time; (2) bring the runtime down, if at all possible. 11 hours doesn't feel immediate, even if it's far better than two months; (3) will sqa adopt these tests as metrics they watch. they're already recognized by your org in this area. – Dale Hagglund Nov 3 '17 at 15:58
4

Disclaimer

Though I may sound like a manager, I wrote this as a developer who also needed to be persuaded that automated tests are good.


You must understand the basic psychology of developers. It is an ingrained need of developers to commit code. Anything that prevents them from doing so is a very, very bad thing. Failed test is definitely something that prevents them from doing so, ergo it is a bad thing. Hence the resistance.

What you must point out is that, while the automated tests slow them down short-term, in the long run it will save them a lot of grief and will actually speed them up, because they will be able to focus more on the development of new things, and will lose less time doing the other thing that developers hate to do: fixing bugs.

And yes, you must enforce it. You must get the unconditional support from the management and make writing automated tests mandatory and non-negotiable. Over time, the developers will get used to them. What will help is if you can devise some metrics that will show how much more new development was done, and by how much the number of bugs was reduced since you introduced the automatic tests. Words are volatile. Numbers are solid. And numbers are something an average developer understands better than words. If you can prove using solid numbers that automated tests are good, you will get little to no resistance to them.

11

In their mind, if tests are failing after a checkin, it's a problem of the test and not of the code change and it's just our toy project. We had discussions several times if a failing test is a real error. Most times it is.

There's your problem. If your tests are flaky (even if they are reliable 'most of the time'), then people will tend to ignore the results. Your automation team should focus on eliminating those false negatives. Only then will the rest of the team gain enough confidence in the results to actually trust them.

  • 5
    Then again, it could be something else. Like resistance to change. If a test fails, there is always something to fix, either the code or the test, so the attitude that people will ignore tests because the tests are flaky is misplaced. – Robert Harvey Oct 12 '17 at 19:23
  • We talked to them when we were sure that something broke (e.g. Test failed 3 times in a row after his checkin affecting the functionality in question). But yes, increasing reliability is the current priority. – Peter Schneider Oct 12 '17 at 19:29
  • 1
    @PeterSchneider a test can fail 100 times in a row, it will do that especially if the test is wrong. – Pieter B Oct 13 '17 at 7:33
  • On the other hand, a test that never fails is most likely completely useless – Vladimir Stokic Oct 13 '17 at 9:01
  • 1
    +1 Brittle tests are definitely a problem. Developers must be convinced the tests they write are useful, do not introduce needless complications, and are not busywork. – Andres F. Oct 13 '17 at 21:17
6

We can't and don't want to enforce something.

You definitely should enforce it! If someone pushes new code and the tests fail the code should be rejected! It is the only way to reliably maintain a larger software project.

  • I guess that depends on the system, the tests, the scale, and the development process. Not all bugs can be resolved right away, nor do all issues have to be resolved right away. – NickL Oct 15 '17 at 16:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.